From dan at tombstonezero.net Tue May 1 00:50:05 2018 From: dan at tombstonezero.net (Dan Sommers) Date: Tue, 1 May 2018 04:50:05 +0000 (UTC) Subject: [Python-ideas] Objectively Quantifying Readability References: <20180501004252.GG7400@ando.pearwood.info> Message-ID: On Tue, 01 May 2018 10:42:53 +1000, Steven D'Aprano wrote: > - people are not good judges of readability; WTF? By definition, people are the *only* judge of readability.? I happen to be an excellent judge of whether a given block of code is readable to me. OTOH, if you mean is that I'm a bad judge of what makes code readable to you, and that you're a bad judge of what makes code readable to me, then I agree. :-) Dan ? Well, okay, compilers will tell you that your code is unreadable, but they're known to be fairly pedantic. From greg.ewing at canterbury.ac.nz Tue May 1 01:49:38 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 01 May 2018 17:49:38 +1200 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: References: <87po2hrfek.fsf@vostro.rath.org> Message-ID: <5AE7FFF2.4090505@canterbury.ac.nz> Tim Peters wrote: > I expected the following would work, but it doesn't :-) > > iseven = lambda n: ( > lambda n=n, \ > even = (lambda n: n == 0 or odd(n-1)), \ > odd = (lambda n: False if n == 0 else even(n-1)): > even(n))() > > Ugly and obscure, but why not? In the inner lambda, `n`, `even`, and > `odd` are all defined in its namespace, But 'even' and 'odd' not defined in the environment of the lambdas assigned to them, because default values of a function's arguments are evaluated outside of that function. > For `even` to know at compile-time that `odd` will show up later in > its enclosing lambda's arglist requires that Python do `letrec`-style > binding instead. For a start ;-) One could envisage adding a letrec-like construct, but making the argument list of an ordinary lambda behave like a letrec would be warping things rather too much, IMO. Personally I'd rather just add a "where" clause, and backwards compatibility be damned. :-) -- Greg From greg.ewing at canterbury.ac.nz Tue May 1 02:06:06 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 01 May 2018 18:06:06 +1200 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: <20180430162027.GE7400@ando.pearwood.info> References: <20180428093334.GU7400@ando.pearwood.info> <20180429025005.GB7400@ando.pearwood.info> <20180430162027.GE7400@ando.pearwood.info> Message-ID: <5AE803CE.8070104@canterbury.ac.nz> Steven D'Aprano wrote: > Pascal, for example, had lexical scoping back in the 1970s, but > no closures. Well, it kind of had a limited form of closure. You could pass a procedure or function *in* to another procedure or function as a parameter, but there was no way to return one or pass it out in any way. This ensured that the passed-in procedure or function couldn't outlive its lexical environment. -- Greg From j.van.dorp at deonet.nl Tue May 1 03:00:44 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Tue, 1 May 2018 09:00:44 +0200 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: References: <20180501004252.GG7400@ando.pearwood.info> Message-ID: I must say my gut agrees that really_long_identifier_names_with_a_full_description don't look readable to me. Perhaps it's my exposure to (py)Qt, but I really like my classes like ThisName and my methods like thisOne. I also tend to keep them to three words max (real code from yesterday: getActiveOutputs(), or at most setAllDigitalOutputs()). I also really dislike more than 3 or 4 arguments. A question for another type of science would be, do I agree with this study because it agrees with me ? It should be noted that the snippets used were short and small. This might cause a bias towards short identifiers - after all, if you only got 3 to keep track of it, they're more likely to be distinct enough compared to when you have 20. I couldn't give a source, but IIRC people can hold up to around 5 to 7 concepts in their head at one time - which means that if you got less identifiers than that, you don't remember the names, but their concepts.(further reading shows this is supported with their strongest negative correlation - # of identifiers strongly decreases readability.). Compare it to RAM - it's only big enough for 5 to 7 identifiers, and after that you have to switch them out to the harddisk. *nobody* wants to code that does this switching, and our brains don't like running it either. I think this is one of the main reasons list/generator comprehensions increase readability so much. You can get rid of 1 or 2 variable names. From steve at pearwood.info Tue May 1 03:11:46 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 1 May 2018 17:11:46 +1000 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: <5AE803CE.8070104@canterbury.ac.nz> References: <20180428093334.GU7400@ando.pearwood.info> <20180429025005.GB7400@ando.pearwood.info> <20180430162027.GE7400@ando.pearwood.info> <5AE803CE.8070104@canterbury.ac.nz> Message-ID: <20180501071146.GJ7400@ando.pearwood.info> On Tue, May 01, 2018 at 06:06:06PM +1200, Greg Ewing wrote: > Steven D'Aprano wrote: > >Pascal, for example, had lexical scoping back in the 1970s, but > >no closures. > > Well, it kind of had a limited form of closure. You could pass > a procedure or function *in* to another procedure or function > as a parameter, but there was no way to return one or pass it > out in any way. This ensured that the passed-in procedure or > function couldn't outlive its lexical environment. So what was the closure? If the surrounding function was still running, there was no need to capture the running environment in a closure? Not a rhetorical question, I'm genuinely unsure. -- Steve From njs at pobox.com Tue May 1 04:29:16 2018 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 1 May 2018 01:29:16 -0700 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: References: <20180501004252.GG7400@ando.pearwood.info> Message-ID: On Mon, Apr 30, 2018 at 8:46 PM, Matt Arcidy wrote: > On Mon, Apr 30, 2018 at 5:42 PM, Steven D'Aprano wrote: >> (If we know that, let's say, really_long_descriptive_identifier_names >> hurt readability, how does that help us judge whether adding a new kind >> of expression will hurt or help readability?) > > A new feature can remove symbols or add them. It can increase density > on a line, or remove it. It can be a policy of variable naming, or it > can specifically note that variable naming has no bearing on a new > feature. This is not limited in application. It's just scoring. > When anyone complains about readability, break out the scoring > criteria and assess how good the _comparative_ readability claim is: > 2 vs 10? 4 vs 5? The arguments will no longer be singularly about > "readability," nor will the be about the question of single score for > a specific statement. The comparative scores of applying the same > function over two inputs gives a relative difference. This is what > measures do in the mathematical sense. Unfortunately, they kind of study they did here can't support this kind of argument at all; it's the wrong kind of design. (I'm totally in favor of being more evidence-based decisions about language design, but interpreting evidence is tricky!) Technically speaking, the issue is that this is an observational/correlational study, so you can't use it to infer causality. Or put another way: just because they found that unreadable code tended to have a high max variable length, doesn't mean that taking those variables and making them shorter would make the code more readable. This sounds like a finicky technical complaint, but it's actually a *huge* issue in this kind of study. Maybe the reason long variable length was correlated with unreadability was that there was one project in their sample that had terrible style *and* super long variable names, so the two were correlated even though they might not otherwise be related. Maybe if you looked at Perl, then the worst coders would tend to be the ones who never ever used long variables names. Maybe long lines on their own are actually fine, but in this sample, the only people who used long lines were ones who didn't read the style guide, so their code is also less readable in other ways. (In fact they note that their features are highly correlated, so they can't tell which ones are driving the effect.) We just don't know. And yeah, it doesn't help that they're only looking at 3 line blocks of code and asking random students to judge readability ? hard to say how that generalizes to real code being read by working developers. -n -- Nathaniel J. Smith -- https://vorpus.org From greg.ewing at canterbury.ac.nz Tue May 1 05:26:09 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 01 May 2018 21:26:09 +1200 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: <20180501071146.GJ7400@ando.pearwood.info> References: <20180428093334.GU7400@ando.pearwood.info> <20180429025005.GB7400@ando.pearwood.info> <20180430162027.GE7400@ando.pearwood.info> <5AE803CE.8070104@canterbury.ac.nz> <20180501071146.GJ7400@ando.pearwood.info> Message-ID: <5AE832B1.8000601@canterbury.ac.nz> Steven D'Aprano wrote: > So what was the closure? If the surrounding function was still running, > there was no need to capture the running environment in a closure? You seem to be interpreting the word "closure" a bit differently from most people. It doesn't imply anything about whether a surrounding function is still running or not. A closure is just a piece of code together with a runtime environment. In typical Pascal implementations, a closure is represented by a (code_address, frame_pointer) pair, where the frame_pointer points to a chain of lexically enclosing stack frames. The language rules make it possible to manage the frames strictly stack-wise, which simplifies the memory management, but that doesn't make the closure any less of a closure. Contrast this with Modula-2, where only top-level functions can be passed as parameters. When you pass a function in Modula-2, only the code address is passed, with no environment pointer. -- Greg From solipsis at pitrou.net Tue May 1 05:51:23 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 1 May 2018 11:51:23 +0200 Subject: [Python-ideas] Objectively Quantifying Readability References: <20180501004252.GG7400@ando.pearwood.info> Message-ID: <20180501115123.21f09c9b@fsol> Yes, it seems that this study has many limitations which don't make its results very interesting for our community. I think the original point was that readability *can* be studied rationnally and scientifically, though. Regards Antoine. On Tue, 1 May 2018 09:00:44 +0200 Jacco van Dorp wrote: > I must say my gut agrees that > really_long_identifier_names_with_a_full_description don't look > readable to me. Perhaps it's my exposure to (py)Qt, but I really like > my classes like ThisName and my methods like thisOne. I also tend to > keep them to three words max (real code from yesterday: > getActiveOutputs(), or at most setAllDigitalOutputs()). I also really > dislike more than 3 or 4 arguments. > > A question for another type of science would be, do I agree with this > study because it agrees with me ? > > It should be noted that the snippets used were short and small. This > might cause a bias towards short identifiers - after all, if you only > got 3 to keep track of it, they're more likely to be distinct enough > compared to when you have 20. I couldn't give a source, but IIRC > people can hold up to around 5 to 7 concepts in their head at one time > - which means that if you got less identifiers than that, you don't > remember the names, but their concepts.(further reading shows this is > supported with their strongest negative correlation - # of identifiers > strongly decreases readability.). Compare it to RAM - it's only big > enough for 5 to 7 identifiers, and after that you have to switch them > out to the harddisk. *nobody* wants to code that does this switching, > and our brains don't like running it either. I think this is one of > the main reasons list/generator comprehensions increase readability so > much. You can get rid of 1 or 2 variable names. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From marcidy at gmail.com Tue May 1 05:55:05 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Tue, 1 May 2018 02:55:05 -0700 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: References: <20180501004252.GG7400@ando.pearwood.info> Message-ID: On Tue, May 1, 2018 at 1:29 AM, Nathaniel Smith wrote: > On Mon, Apr 30, 2018 at 8:46 PM, Matt Arcidy wrote: >> On Mon, Apr 30, 2018 at 5:42 PM, Steven D'Aprano wrote: >>> (If we know that, let's say, really_long_descriptive_identifier_names >>> hurt readability, how does that help us judge whether adding a new kind >>> of expression will hurt or help readability?) >> >> A new feature can remove symbols or add them. It can increase density >> on a line, or remove it. It can be a policy of variable naming, or it >> can specifically note that variable naming has no bearing on a new >> feature. This is not limited in application. It's just scoring. >> When anyone complains about readability, break out the scoring >> criteria and assess how good the _comparative_ readability claim is: >> 2 vs 10? 4 vs 5? The arguments will no longer be singularly about >> "readability," nor will the be about the question of single score for >> a specific statement. The comparative scores of applying the same >> function over two inputs gives a relative difference. This is what >> measures do in the mathematical sense. > > Unfortunately, they kind of study they did here can't support this > kind of argument at all; it's the wrong kind of design. (I'm totally > in favor of being more evidence-based decisions about language design, > but interpreting evidence is tricky!) Technically speaking, the issue > is that this is an observational/correlational study, so you can't use > it to infer causality. Or put another way: just because they found > that unreadable code tended to have a high max variable length, > doesn't mean that taking those variables and making them shorter would > make the code more readable. > I think you are right about the study, but are tangential to what I am trying to say. I am not inferring causality when creating a measure. In the most tangible example, there is no inference that the euclidean measure _creates_ a distance, or that _anything_ creates a distance at all, it merely generates a number based on coordinates in space. That generation has specific properties which make it a measure, or a metric, what have you. The average/mean is another such object: a measure of central tendency or location. It does not infer causality, it is merely an algorithm by which things can be compared. Even misapplied, it provides a consistent ranking of one mean higher than another in an objective sense. Even if not a single person agrees that line length is a correct measure for an application, it is a measure. I can feed two lines into "len" and get consistent results out. This result will be the same value for all strings of length n, and for a string with length m > n, the measure will always report a higher measured value for the string of length m than the string of length n. This is straight out of measure theory, the results are a distance between the two objects, not a reason why. The same goes for unique symbols. I can count the unique symbols in two lines, and state which is higher. This does not infer a causality, nor do _which_ symbols matter in this example, only that I can count them, and that if count_1 == count_2, the ranks are equal aka no distance between them, and if count_1 > count_2, count 1 is ranked higher. The cause of complexity can be a number of things, but stating a bunch of criteria to measure is not about inference. Measuring the temperature of a steak doesn't infer why people like it medium rare. It just quantifies it. > This sounds like a finicky technical complaint, but it's actually a > *huge* issue in this kind of study. Maybe the reason long variable > length was correlated with unreadability was that there was one > project in their sample that had terrible style *and* super long > variable names, so the two were correlated even though they might not > otherwise be related. Maybe if you looked at Perl, then the worst > coders would tend to be the ones who never ever used long variables > names. Maybe long lines on their own are actually fine, but in this > sample, the only people who used long lines were ones who didn't read > the style guide, so their code is also less readable in other ways. > (In fact they note that their features are highly correlated, so they > can't tell which ones are driving the effect.) We just don't know. > Your points here are dead on. It's not like a single metric will be the deciding factor. Nor will a single rank end all disagreements. It's a tool. Consider line length 79, that's an explicit statement about readability, it's "hard coded" in the language. Disagreement with the value 79 or even the metric line-length doesn't mean it's not a measure. Length is the euclidean measure in one dimension. The measure will be a set of filters and metrics that combine to a value or set of values in a reliable way. It's not about any sense of correctness or even being better, that is, at a minimum, an interpretation. > And yeah, it doesn't help that they're only looking at 3 line blocks > of code and asking random students to judge readability ? hard to say > how that generalizes to real code being read by working developers. Respectfully, this is practical application and not a PhD defense, so it will be generated by practical coding. People can argue about the chosen metrics, but it is a more informative debate than just the label "readability". If 10 people state a change badly violates one criteria, perhaps that can be easily addressed. if many people make multiple claims based on many criteria, there is a real readability problem (assuming the metric survived SOME vetting of course) > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org From kenlhilton at gmail.com Tue May 1 07:22:52 2018 From: kenlhilton at gmail.com (Ken Hilton) Date: Tue, 1 May 2018 19:22:52 +0800 Subject: [Python-ideas] A way to subscript a single integer from bytes Message-ID: Hi all, So I'm pretty sure everyone here is familiar with how the "bytes" object works in Python 3. It acts mostly like a string, with the exception that 0-dimensional subscripting (var[idx]) returns an integer, not a bytes object - the integer being the ordinal number of the corresponding character. However, 1-dimensional subscripting (var[idx1:idx2]) returns a bytes object. Example: >>> a = b'hovercraft' >>> a[0] 104 >>> a[4:8] b'rcra' Though this isn't exactly unexpected behavior (it's not possible to accidentally do 1-dimensional subscripting and expect an integer it's a different syntax), it's still a shame that it isn't possible to quickly and easily subscript an integer out of it. Following up from the previous example, The only way to get 493182234161465432041076 out of b'hovercraft' in a single expression is as follows: list(__import__('itertools').accumulate((i for i in a), lambda x, y: (x << 8) + y))[-1] Now, I'm not proposing changing the 1-dimensional subscripting syntax to return an integer - that would be backwards incompatible, tsk tsk! No, instead, I'm simply suggesting a method of bytes objects, which would do something like this (assume the method is called "subint"): >>> a = b'hovercraft' >>> a.subint(0, -1) # -1 is equivalent to len(a) 493182234161465432041076 Much as I would think that such subscripting would deserve special syntax (perhaps bytes{idx1:idx2}), I don't think this special case is special enough to break the rules. So I'm sticking with the method idea. What are your thoughts? Sincerely, Ken; -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue May 1 07:30:55 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 1 May 2018 13:30:55 +0200 Subject: [Python-ideas] A way to subscript a single integer from bytes References: Message-ID: <20180501133055.2bea1a0f@fsol> Hi Ken, On Tue, 1 May 2018 19:22:52 +0800 Ken Hilton wrote: > > So I'm pretty sure everyone here is familiar with how the "bytes" object > works in Python 3. It acts mostly like a string, with the exception that > 0-dimensional subscripting (var[idx]) returns an integer, not a bytes > object - the integer being the ordinal number of the corresponding > character. > However, 1-dimensional subscripting (var[idx1:idx2]) returns a bytes > object. Example: > > >>> a = b'hovercraft' > >>> a[0] > 104 > >>> a[4:8] > b'rcra' > > Though this isn't exactly unexpected behavior (it's not possible to > accidentally do 1-dimensional subscripting and expect an integer it's a > different syntax), it's still a shame that it isn't possible to quickly and > easily subscript an integer out of it. Following up from the previous > example, The only way to get 493182234161465432041076 out of b'hovercraft' > in a single expression is as follows: > > list(__import__('itertools').accumulate((i for i in a), lambda x, y: (x > << 8) + y))[-1] Let's see: >>> a = b'hovercraft' >>> int.from_bytes(a, 'big') 493182234161465432041076 Regards Antoine. From rhodri at kynesim.co.uk Tue May 1 07:11:56 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Tue, 1 May 2018 12:11:56 +0100 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: <20180501004252.GG7400@ando.pearwood.info> References: <20180501004252.GG7400@ando.pearwood.info> Message-ID: <2e459330-6d2e-d163-859b-d62af0cd9e30@kynesim.co.uk> On 01/05/18 01:42, Steven D'Aprano wrote: > That's a really nice study, and thank you for posting it. Seconded! > There are some > interested observations here, e.g.: > > - line length is negatively correlated with readability; > > (a point against those who insist that 79 character line > limits are irrelevant since we have wide screens now) There are physiological studies of how we read that support this. I don't have any references to hand (I'd have to go hunting, and no promises because the ones I'm faintly aware of happened at least 40 years ago!), but the gist is that we don't sweep our eyes continuously along a line of text as you might assume, we jerk our attention across in steps. > - length of identifiers was strongly negatively correlated > with readability: long, descriptive identifier names hurt > readability while short variable names appeared to make > no difference; I'd be interested to know if there is a readability difference between really_long_descriptive_identifier_name and ReallyLongDescriptiveIdentifierNames. I rather suspect there is :-) and the study may be biased by picking on Java as its example language here. -- Rhodri James *-* Kynesim Ltd From steve at pearwood.info Tue May 1 08:37:11 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 1 May 2018 22:37:11 +1000 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: References: <20180501004252.GG7400@ando.pearwood.info> Message-ID: <20180501123711.GK7400@ando.pearwood.info> On Tue, May 01, 2018 at 04:50:05AM +0000, Dan Sommers wrote: > On Tue, 01 May 2018 10:42:53 +1000, Steven D'Aprano wrote: > > > - people are not good judges of readability; > > WTF? By definition, people are the *only* judge of readability.? We're discussing an actual study that attempted, with some reasonable amount of success, to objectively measure readability without human judgement (aside from the initial judgement on which factors to measure and which to ignore). So given the the fact that there exists at least one non-human objective measurement of readability which corresponds reasonably well with human judgement, how do you reach the conclusion that only humans can be the judge of readability? Besides, even if we agreed that only people can do something, that doesn't mean that they are necessarily *good at it*. > I happen to be an excellent judge of whether a given block of code is > readable to me. In the same way that 93% of people say that they are an above-average driver, I'm sure that most people think that they are an excellent judge of readability. Including myself in *both* of those. (My wife thinks I'm a crappy driver, but her standard of "just barely adequate" is close to professional racecar drivers, so I don't let that worry me.) https://en.wikipedia.org/wiki/Illusory_superiority There are at least three ways that I can justify my statement, one objective, and two anecdotal: 1. The correlation between judgements of readability from different people is not very good: in the study discussed, it was about 0.6 or so. A correlation of 0.5 is equivalent to having everyone agree on half the samples and then rate the other half at random. The paper states: "This analysis seems to confirm the widely-held belief that humans agree significantly on what readable code looks like, but not to an overwhelming extent." 2. Anecdotally, we all know that many programmers are just awful. https://thedailywtf.com/ And presumably most of them think they are writing readable code. (There may be a small minority who are deliberately writing obfuscated code, but I doubt that's a significant demographic.) Most of us have had the experience of making code choices that are inexplicable and unreadable when we come back to it. http://www.threepanelsoul.com/comic/on-perl 3. Anecdotally, I have first hand experience with many people, including programmers, making dramatically sub-optimal choices while declaring that it is the most effective choice. To pick one example that applies to coders, I have known many people who swear black and blue that they work best with their editor configured to show code in a tiny, 8pt font, and I've watched them peering closely at the screen struggling to read the text and making typo after typo which they failed to notice. In other words, people are often not even a great judge of what is readable to *themselves*. -- Steve From greg.ewing at canterbury.ac.nz Tue May 1 08:54:03 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 02 May 2018 00:54:03 +1200 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: <2e459330-6d2e-d163-859b-d62af0cd9e30@kynesim.co.uk> References: <20180501004252.GG7400@ando.pearwood.info> <2e459330-6d2e-d163-859b-d62af0cd9e30@kynesim.co.uk> Message-ID: <5AE8636B.1030301@canterbury.ac.nz> Rhodri James wrote: > I'd be interested to know if there is a readability difference between > really_long_descriptive_identifier_name and > ReallyLongDescriptiveIdentifierNames. As one data point on that, jerking my eyes quickly across that line I found it much easier to pick out the component words in the one with underscores. -- Greg From steve at pearwood.info Tue May 1 09:09:19 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 1 May 2018 23:09:19 +1000 Subject: [Python-ideas] A way to subscript a single integer from bytes In-Reply-To: References: Message-ID: <20180501130918.GL7400@ando.pearwood.info> On Tue, May 01, 2018 at 07:22:52PM +0800, Ken Hilton wrote: > The only way to get 493182234161465432041076 out of b'hovercraft' You seem to be using a bytes object as a base-256 number. Under what circumstances is this desirable? > in a single expression is as follows: What's so special about this operation that it needs to be a single expression? The easy way to do this is: from functools import reduce # not needed in Python 2 reduce(lambda m, n: m*256 + n, b'hovercraft', 0) Do the import once at the top of your module, and then you can call reduce as many times as you like. If you really, really, really want to make it a one-liner, perhaps to win a bet, or because the Enter key on your keyboard is broken, then you can use the __import__('functools') trick. __import__('functools').reduce(lambda m, n: m*256 + n, b'hovercraft', 0) But don't do that. > list(__import__('itertools').accumulate((i for i in a), lambda x, y: (x > << 8) + y))[-1] '(i for i in a)' is best written as 'iter(a)' if you must have an iterator, or just 'a' if you don't care what sort of iterable it is. > Now, I'm not proposing changing the 1-dimensional subscripting syntax to > return an integer - that would be backwards incompatible, tsk tsk! No, > instead, I'm simply suggesting a method of bytes objects, which would do > something like this (assume the method is called "subint"): > > >>> a = b'hovercraft' > >>> a.subint(0, -1) # -1 is equivalent to len(a) > 493182234161465432041076 > > Much as I would think that such subscripting would deserve special syntax > (perhaps bytes{idx1:idx2}), I don't think this special case is special > enough to break the rules. So I'm sticking with the method idea. > > What are your thoughts? I don't even know why you would want to do it in the first place, let alone why you think it is special enough to dedicate syntax to doing it. -- Steve From ncoghlan at gmail.com Tue May 1 09:58:40 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 1 May 2018 23:58:40 +1000 Subject: [Python-ideas] A way to subscript a single integer from bytes In-Reply-To: <20180501133055.2bea1a0f@fsol> References: <20180501133055.2bea1a0f@fsol> Message-ID: On 1 May 2018 at 21:30, Antoine Pitrou wrote: > > Hi Ken, > > On Tue, 1 May 2018 19:22:52 +0800 > Ken Hilton wrote: > > > > So I'm pretty sure everyone here is familiar with how the "bytes" object > > works in Python 3. It acts mostly like a string, with the exception that > > 0-dimensional subscripting (var[idx]) returns an integer, not a bytes > > object - the integer being the ordinal number of the corresponding > > character. > > However, 1-dimensional subscripting (var[idx1:idx2]) returns a bytes > > object. Example: > > > > >>> a = b'hovercraft' > > >>> a[0] > > 104 > > >>> a[4:8] > > b'rcra' > > > > Though this isn't exactly unexpected behavior (it's not possible to > > accidentally do 1-dimensional subscripting and expect an integer it's a > > different syntax), it's still a shame that it isn't possible to quickly > and > > easily subscript an integer out of it. Following up from the previous > > example, The only way to get 493182234161465432041076 out of > b'hovercraft' > > in a single expression is as follows: > > > > list(__import__('itertools').accumulate((i for i in a), lambda x, > y: (x > > << 8) + y))[-1] > > Let's see: > > >>> a = b'hovercraft' > >>> int.from_bytes(a, 'big') > 493182234161465432041076 > It's also worth noting that if there's more than one integer of interest in the string, than using the struct module is often going to be better than using multiple slices and int.from_bytes calls: >>> import struct >>> data = b"hovercraft" >>> struct.unpack(">IIH", data) (1752135269, 1919119969, 26228) (The struct module doesn't handle arbitrary length integers, but it handles 8, 16, 32, and 64 bit ones, which is enough for a lot of common use cases) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenlhilton at gmail.com Tue May 1 10:08:54 2018 From: kenlhilton at gmail.com (Ken Hilton) Date: Tue, 01 May 2018 14:08:54 +0000 Subject: [Python-ideas] A way to subscript a single integer from bytes Message-ID: Whoops! Never seen that before. Nothing I searched up pointed me to it. Sorry for wasting your time! Ken; -- Sincerely, Ken; -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.van.dorp at deonet.nl Tue May 1 11:04:36 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Tue, 1 May 2018 17:04:36 +0200 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: <5AE8636B.1030301@canterbury.ac.nz> References: <20180501004252.GG7400@ando.pearwood.info> <2e459330-6d2e-d163-859b-d62af0cd9e30@kynesim.co.uk> <5AE8636B.1030301@canterbury.ac.nz> Message-ID: 2018-05-01 14:54 GMT+02:00 Greg Ewing : > Rhodri James wrote: >> >> I'd be interested to know if there is a readability difference between >> really_long_descriptive_identifier_name and >> ReallyLongDescriptiveIdentifierNames. > > > As one data point on that, jerking my eyes quickly across > that line I found it much easier to pick out the component > words in the one with underscores. > > -- > Greg Which is funny, because I had the exact opposite. Might it be that we've had different conditioning ? Jacco From dan at tombstonezero.net Tue May 1 11:02:27 2018 From: dan at tombstonezero.net (Dan Sommers) Date: Tue, 1 May 2018 15:02:27 +0000 (UTC) Subject: [Python-ideas] Objectively Quantifying Readability References: <20180501004252.GG7400@ando.pearwood.info> <20180501123711.GK7400@ando.pearwood.info> Message-ID: On Tue, 01 May 2018 22:37:11 +1000, Steven D'Aprano wrote: > On Tue, May 01, 2018 at 04:50:05AM +0000, Dan Sommers wrote: >> On Tue, 01 May 2018 10:42:53 +1000, Steven D'Aprano wrote: >> >> > - people are not good judges of readability; >> I happen to be an excellent judge of whether a given block of code is >> readable to me. > In the same way that 93% of people say that they are an above-average > driver, I'm sure that most people think that they are an excellent > judge of readability. Including myself in *both* of those. Are you claiming that I'm not an excellent judge of whether a given block of code is readable to me? > 2. Anecdotally, we all know that many programmers are just awful. No argument here! :-) > https://thedailywtf.com/ Readability is only one criterion by which to judge code. Most code on thedailywtf is bad due to varieties of bugs, inefficiencies, misunderstandings, or unnecessary complexities, regardless of how readable it is. > And presumably most of them think they are writing readable code ... Why would you presume that? I haved worked with plenty programmers who didn't consider readability. If it passes the tests, then it's good code. If I have difficulty reading it, then that's my problem. Also, when I write code, I put down my text editor to review it myself before I submit it for formal review. My authoring criteria for good code is different from my reviewing criteria for good code; the latter includes more readability than the former. > ... I have known many people who swear black and blue that they work > best with their editor configured to show code in a tiny, 8pt font, > and I've watched them peering closely at the screen struggling to read > the text and making typo after typo which they failed to notice. > In other words, people are often not even a great judge of what is > readable to *themselves*. On that level, aesthetics definitely count (typographers have known this for centuries), but in an entirely different way. Is their code better when their editor shows it at 14.4pt? At 17.3pt? When we first started with color displays and syntax coloring editors, it was popular to make a language's keywords really stand out. But the keywords usually aren't the important part of the code (have you ever programmed in lisp?), and I find it easier to read algol-like code when the keywords are lowlighted rather than highlighted. In Python, for example, the word "import" is far less important than the name of the module being imported, especially when all the imports are grouped together near the top of the source file. Dan From steve at pearwood.info Tue May 1 11:23:16 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 2 May 2018 01:23:16 +1000 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: References: <5AE61F76.60900@stoneleaf.us> <5AE66F61.9000900@stoneleaf.us> <4a5496f7-c180-4a01-0de8-dcd9b07681ad@gmail.com> <78764cbe-ec31-626f-35e8-6d0f5dd6fb06@mrabarnett.plus.com> Message-ID: <20180501152316.GM7400@ando.pearwood.info> On Mon, Apr 30, 2018 at 08:52:13PM -0500, Tim Peters wrote: > > Would/should it be possible to inject a name into a local scope? You can't > > inject into a function scope, and names in a function scope can be > > determined statically (they are allocated slots), so could the same kind of > > thing be done for names in a local scope? > > Sorry, I'm unclear on what "inject a name into a local scope" means. > Do you mean at runtime? I don't know what MRAB means by "inject", but I know what *I* mean, and I have a real use-case for it. There is a long-running micro-optimization, often championed by Raymond, for avoiding slow global lookups (and even slower builtin lookups, since they require a global lookup to fail first) by turning them in local lookups at function-definition time. E.g. some methods from the random.Random class: def randrange(self, start, stop=None, step=1, _int=int): ... def _randbelow(self, n, int=int, maxsize=1< References: <20180501004252.GG7400@ando.pearwood.info> <2e459330-6d2e-d163-859b-d62af0cd9e30@kynesim.co.uk> <5AE8636B.1030301@canterbury.ac.nz> Message-ID: On Tue, May 1, 2018 at 8:04 AM, Jacco van Dorp wrote: > 2018-05-01 14:54 GMT+02:00 Greg Ewing : > > Rhodri James wrote: > >> > >> I'd be interested to know if there is a readability difference between > >> really_long_descriptive_identifier_name and > >> ReallyLongDescriptiveIdentifierNames. > > > > > > As one data point on that, jerking my eyes quickly across > > that line I found it much easier to pick out the component > > words in the one with underscores. > > Which is funny, because I had the exact opposite. > > Might it be that we've had different conditioning ? ?Almost certainly. I started using CamelCase in the mid-'80s and it seems very natural to me, since we still use it for (as you mention) GUI packages derived from C extension modules with that convention. On the other hand, I've also written a lot of snake_form identifiers in non-GUI Python, so that seems fairly natural to me, too.? -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Tue May 1 12:27:16 2018 From: elazarg at gmail.com (Elazar) Date: Tue, 01 May 2018 16:27:16 +0000 Subject: [Python-ideas] A way to subscript a single integer from bytes In-Reply-To: References: Message-ID: I think this method is easy to miss, since people look at the docs for bytes (e.g. using dir(bytes)). It might be worthwhile to either add a `bytes.to_int(...)` method (better, IMHO), or to point to int.from_bytes on the relevant part of the docs. Elazar On Tue, May 1, 2018 at 5:09 PM Ken Hilton wrote: > Whoops! Never seen that before. Nothing I searched up pointed me to it. > Sorry for wasting your time! > > Ken; > -- > Sincerely, > Ken; > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Tue May 1 13:08:43 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 1 May 2018 12:08:43 -0500 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: References: <20180428093334.GU7400@ando.pearwood.info> <5AE61F76.60900@stoneleaf.us> <5AE66F61.9000900@stoneleaf.us> <4a5496f7-c180-4a01-0de8-dcd9b07681ad@gmail.com> <78764cbe-ec31-626f-35e8-6d0f5dd6fb06@mrabarnett.plus.com> Message-ID: [Tim] > ... > Here's an example where I don't know what the consequences of "the > rules" should be: > > def f(): > a = 10 > local a: > def showa(): > print("a is", a) > showa() # 10 > a = 20 > showa() # 20 > a = 30 > showa() # 10 > > The comments show what the output would be under the "nothing about > scope rules change" meaning. They're all obvious (since there is is > no new scope then - it's all function-local). > > But under the other meaning ...? > > The twist here is that `def` is an executable statement in Python, and > is a "binding site" for the name of the function being defined. So > despite that `showa` appears to be defined in a new nested lexical > scope, it's _actually_ bound as a function-local name. That's bound > to be surprising to people from other languages: "I defined it in a > nested lexical scope, but the name is still visible after that scope > ends?". > > I don't know what the first `showa()` is intended to do. Presumably > `a` is unbound at the start of the new nested scope? So raises > NameError? If so, comment that line out so we can make progress ;-) > > It seems clear that the second `showa()` will display 20 under any reading. > > But the third? Now we're out of the `local a:` scope, but call a > function whose textual definition was inside that scope. What does > `showa()` do now to find a's value? f's local `a` had nothing to do > with the `a` in the nested scope, so presumably it shouldn't display > 10 now. What should it do? > Does the final state of the nested scope's locals need to preserved so > that showa() can display 30 instead? Or ...? Just noting that a sane way to answer such questions is to model the intent with nested functions in current Python. It's annoying because you have to explicitly declare the _non_-local names instead, and also make dummy assignments to those names to tell Python in which scope they _should_ be bound. So, e.g., for the above you can write this: def f(): a = 10 # Have to give `showa` _some_ binding in its intended scope, # else the "nonlocal showa" below is a compile-time error showa = None def local_a(): nonlocal showa def showa(): print("a", a) #showa() # raises NameError a = 20 showa() # 20 a = 30 local_a() del local_a showa() # 30 print("but f's a is", a) # 10 The comments show what happens when you call `f()`, matching the guesses in the original message. One thing to note: if this does model the intent, it's essentially proof that it's implementable without intractable effort; e.g., the machinery needed to implement funky closures already exists. But another: since I've never seen code anything like this before, that casts doubt on whether the semantics are actually useful in real life. That's why I always ask for real use cases ;-) From python at mrabarnett.plus.com Tue May 1 13:45:34 2018 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 1 May 2018 18:45:34 +0100 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: References: <20180428093334.GU7400@ando.pearwood.info> <5AE61F76.60900@stoneleaf.us> <5AE66F61.9000900@stoneleaf.us> <4a5496f7-c180-4a01-0de8-dcd9b07681ad@gmail.com> <78764cbe-ec31-626f-35e8-6d0f5dd6fb06@mrabarnett.plus.com> Message-ID: <957e7e1c-5cad-9e1d-d8e7-ecab70d38a58@mrabarnett.plus.com> On 2018-05-01 02:52, Tim Peters wrote: > [MRAB ] > > ... > > The intention is that only the specified names are local. > > > > After all, what's the point of specifying names after the 'local' if _any_ > > binding in the local scope was local? > > Don't look at me ;-) In the absence of use cases, I don't know which > problem(s) you're trying to solve. All the use cases I've looked at > are adequately addressed by having some spelling of "local:" change > nothing at all about Python's current scope rules. If you have uses > in mind that require more than just that, I'd need to see them. > > >> ... > >> If you agree that makes the feature probably unusable, you don't get > >> off the hook by saying "no, unlike current Python scopes, binding > >> sites have nothing to do with what's local to a new lexical scope > >> introduced by 'local:'". The same question raised in the example > >> above doesn't go away: in which scope(s) are 'r1' and 'r2' to be > >> bound? > > > Any binding that's not specified as local is bound in the parent scope: > > Reverse-engineering the example following, is this a fair way of > making that more precise? > > Given a binding-target name N in scope S, N is bound in scope T, where > T is the closest-containing scope (which may be S itself) for which T > is either > > 1. established by a "local:" block that declares name N > > or > > 2. not established by a "local: block > By "parent scope" I meant the function or global scope, a scope that's not a "local:" scope. I couldn't think of a good name for it. [snip] From python at mrabarnett.plus.com Tue May 1 14:00:03 2018 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 1 May 2018 19:00:03 +0100 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: References: <20180428093334.GU7400@ando.pearwood.info> <5AE61F76.60900@stoneleaf.us> <5AE66F61.9000900@stoneleaf.us> <4a5496f7-c180-4a01-0de8-dcd9b07681ad@gmail.com> <78764cbe-ec31-626f-35e8-6d0f5dd6fb06@mrabarnett.plus.com> Message-ID: <41dd3483-0a12-9aca-d181-66b555083480@mrabarnett.plus.com> On 2018-05-01 04:40, Tim Peters wrote: > [MRAB] > >> Any binding that's not specified as local is bound in the parent scope: > > [Tim] > > Reverse-engineering the example following, is this a fair way of > > making that more precise? > > > > Given a binding-target name N in scope S, N is bound in scope T, where > > T is the closest-containing scope (which may be S itself) for which T > > is either > > > > 1. established by a "local:" block that declares name N > > > > or > > > > 2. not established by a "local: block > > Here's an example where I don't know what the consequences of "the > rules" should be: > > def f(): > a = 10 > local a: > def showa(): > print("a is", a) > showa() # 10 > a = 20 > showa() # 20 > a = 30 > showa() # 10 The body of "showa" is lexically nested in 'local', so its name "a" is the local scope's "a", not the function scope's "a". Imagine renaming the specified names that are declared 'local' throughout the nested portion: def f(): ??? a = 10 ??? local local_a: ??????? def showa(): ??????????? print("a is", local_a) ??????? showa() # Raises NameError in showa because nothing bound to local_a yet. ??????? local_a = 20 ??????? showa() # 20 ??????? local_a = 30 ??? showa() # Raises NameError in f because local_a not defined. > The comments show what the output would be under the "nothing about > scope rules change" meaning. They're all obvious (since there is is > no new scope then - it's all function-local). > > But under the other meaning ...? > > The twist here is that `def` is an executable statement in Python, and > is a "binding site" for the name of the function being defined. So > despite that `showa` appears to be defined in a new nested lexical > scope, it's _actually_ bound as a function-local name. That's bound > to be surprising to people from other languages: "I defined it in a > nested lexical scope, but the name is still visible after that scope > ends?". > > I don't know what the first `showa()` is intended to do. Presumably > `a` is unbound at the start of the new nested scope? So raises > NameError? If so, comment that line out so we can make progress ;-) > > It seems clear that the second `showa()` will display 20 under any reading. > > But the third? Now we're out of the `local a:` scope, but call a > function whose textual definition was inside that scope. What does > `showa()` do now to find a's value? f's local `a` had nothing to do > with the `a` in the nested scope, so presumably it shouldn't display > 10 now. What should it do? > Does the final state of the nested scope's locals need to preserved so > that showa() can display 30 instead? Or ...? > > Not necessarily complaining - just fleshing out a bit my earlier > claim that a world of semantics need to be defined if anything akin to > a "real scope" is desired. > From steve at pearwood.info Tue May 1 14:00:23 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 2 May 2018 04:00:23 +1000 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: <5AE832B1.8000601@canterbury.ac.nz> References: <20180428093334.GU7400@ando.pearwood.info> <20180429025005.GB7400@ando.pearwood.info> <20180430162027.GE7400@ando.pearwood.info> <5AE803CE.8070104@canterbury.ac.nz> <20180501071146.GJ7400@ando.pearwood.info> <5AE832B1.8000601@canterbury.ac.nz> Message-ID: <20180501180023.GN7400@ando.pearwood.info> On Tue, May 01, 2018 at 09:26:09PM +1200, Greg Ewing wrote: > Steven D'Aprano wrote: > >So what was the closure? If the surrounding function was still running, > >there was no need to capture the running environment in a closure? > > You seem to be interpreting the word "closure" a bit > differently from most people. It doesn't imply anything > about whether a surrounding function is still running or > not. > > A closure is just a piece of code together with a runtime > environment. That's not *all* it is, because obviously *every* function has both a piece of code and a runtime environment. y = 1 def func(): x = 2 return x+y Here, there's a local environment as well as an implicit global one. Surely we don't want to call this a closure? If we do, then all functions are closures and the term is just a synonym for function. Wikipedia gives a definition: a closure (also lexical closure or function closure) is a technique for implementing lexically scoped name binding in a language with first-class functions. Operationally, a closure is a record storing a function together with an environment. https://en.wikipedia.org/wiki/Closure_%28computer_programming%29 but also warns As different languages do not always have a common definition of the lexical environment, their definitions of closure may vary also and explicitly says that Pascal (at least standard Pascal) didn't do closures: Traditional imperative languages such as Algol, C and Pascal either do not support nested functions (C) or do not support calling nested functions after the enclosing function has exited (GNU C, Pascal), thus avoiding the need to use closures. Neal Gafter gives what I believe is *the* standard definition of closures, at least in academic and functional programming circles, probably taken from Guy Steele and Scheme: A closure is a function that captures the bindings of free variables in its lexical context. http://gafter.blogspot.com.au/2007/01/definition-of-closures.html Since Steele basically wrote the book on closures, I think we ought to take his definition seriously :-) (That's also the definition I was thinking of.) By this definition, if there are no free variables, or no captured bindings, then its not a closure. func() above isn't a closure, since "x" isn't a free variable, and while "y" is, the value of it isn't captured. Unfortunately, the terminology has become a real mess, especially since the technique has moved into languages like Ruby and Javascript. Martin Fowler declares that closures, lambdas and blocks are the same thing: https://martinfowler.com/bliki/Lambda.html although he does admit that technically you can have lambdas that aren't closures. I think Fowler is abusing the terminology since all three of his terms are orthogonal: - closures can be created with def or lambda and are not necessarily anonymous; - lambda comes from the lambda calculus, where it refers to anonymous function expressions, not specifically whether they capture the value of free variables in their environment; - "block" typically refers to the structure of the source code; in Ruby it also refers to a kind of first-class anonymous callable object. Such blocks may, or may not, contain free variables. At least I don't use this definition of closure: "a closure is a callback that Just Works." https://reprog.wordpress.com/2010/02/27/closures-finally-explained/ *wink* > In typical Pascal implementations, a closure > is represented by a (code_address, frame_pointer) pair, > where the frame_pointer points to a chain of lexically > enclosing stack frames. The language rules make it possible > to manage the frames strictly stack-wise, which simplifies > the memory management, but that doesn't make the closure > any less of a closure. Well... it's a pretty feeble closure, since Pascal doesn't support functions as first-class values. More like second-class. The c2 wiki describes Pascal: However, when a function is passed to another function and later called, it will execute in the lexical context it was defined in, so it is, IN SOME SENSE [emphasis added], "closed over" that context. http://wiki.c2.com/?LexicalClosure but I think that sense is the same sense that func() above is "closed over" the free value y. The loosest possible sense. If we accept that "Pascal has closures", they're pretty crap closures, since they don't let you do any of the interesting and useful things you can do in languages with "real" closures like Scheme and Python. And because it is based on an implementation detail (as you say, it is only *typical*, not mandatory, for Pascal inner functions to be implemented this way), we have to consider what happens if we come across a Pascal implementation which *doesn't* use a (code_address, frame_pointer) pair. Is that no longer Pascal? This is the interesting, and frustrating, thing about definitions: deciding where the boundaries lie, and we could argue the boundaries for days without getting anywhere :-) -- Steve From python at mrabarnett.plus.com Tue May 1 14:08:02 2018 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 1 May 2018 19:08:02 +0100 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: <20180501152316.GM7400@ando.pearwood.info> References: <5AE61F76.60900@stoneleaf.us> <5AE66F61.9000900@stoneleaf.us> <4a5496f7-c180-4a01-0de8-dcd9b07681ad@gmail.com> <78764cbe-ec31-626f-35e8-6d0f5dd6fb06@mrabarnett.plus.com> <20180501152316.GM7400@ando.pearwood.info> Message-ID: On 2018-05-01 16:23, Steven D'Aprano wrote: > On Mon, Apr 30, 2018 at 08:52:13PM -0500, Tim Peters wrote: > >> > Would/should it be possible to inject a name into a local scope? You can't >> > inject into a function scope, and names in a function scope can be >> > determined statically (they are allocated slots), so could the same kind of >> > thing be done for names in a local scope? >> >> Sorry, I'm unclear on what "inject a name into a local scope" means. >> Do you mean at runtime? > > I don't know what MRAB means by "inject", but I know what *I* mean, and > I have a real use-case for it. > By "inject" I mean putting a name into a namespace: import my_module my_module.foo = 'FOO' You can't insert a name into a function's body to make a new local variable. [snip] From tim.peters at gmail.com Tue May 1 14:12:13 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 1 May 2018 13:12:13 -0500 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: <41dd3483-0a12-9aca-d181-66b555083480@mrabarnett.plus.com> References: <20180428093334.GU7400@ando.pearwood.info> <5AE61F76.60900@stoneleaf.us> <5AE66F61.9000900@stoneleaf.us> <4a5496f7-c180-4a01-0de8-dcd9b07681ad@gmail.com> <78764cbe-ec31-626f-35e8-6d0f5dd6fb06@mrabarnett.plus.com> <41dd3483-0a12-9aca-d181-66b555083480@mrabarnett.plus.com> Message-ID: [MRAB] > Imagine renaming the specified names that are declared 'local' throughout > the nested portion: > > def f(): > a = 10 > local local_a: > def showa(): > print("a is", local_a) > showa() # Raises NameError in showa because nothing bound to local_a yet. > local_a = 20 > showa() # 20 > local_a = 30 > showa() # Raises NameError in f because local_a not defined. In a later message I showed executable-today Python code that I believe models your intent. That code does indeed display "30" for the last call, and while I myself don't see that it's _useful_ yet, the model is incomprehensible if it doesn't. It doesn't matter that local_a isn't defined at function scope, because showa() refers to the locals _in_ the "local a:" scope. The last value bound to local_a was 30, so that's what showa() needs to show. That the name `showa` is _bound_ in f's locals has nothing to do with whose locals showa() _uses_. Other current messages about "closures" go on more about that. From tim.peters at gmail.com Tue May 1 14:33:05 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 1 May 2018 13:33:05 -0500 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: References: <5AE61F76.60900@stoneleaf.us> <5AE66F61.9000900@stoneleaf.us> <4a5496f7-c180-4a01-0de8-dcd9b07681ad@gmail.com> <78764cbe-ec31-626f-35e8-6d0f5dd6fb06@mrabarnett.plus.com> <20180501152316.GM7400@ando.pearwood.info> Message-ID: [MRAB] > By "inject" I mean putting a name into a namespace: > > import my_module > my_module.foo = 'FOO' > > You can't insert a name into a function's body to make a new local variable. So you do mean at runtime, I think. Then as before, you can do that with module and the builtin namespaces now, but all function locals need to be identifiable at compile time now. People often get confused about that when they read that "it's a binding" that makes a name local to a scope, but it's not "_doing_ a binding" that makes it local, merely that the name _syntactically_ appears as a target in a binding construct. That analysis is entirely done at compile time. In Python's very early days, all namespaces could be dynamically altered at any time in any way. As time went on, more & more restrictions were placed on runtime alteration of local scopes. I don't believe that, in current Python 3, dynamic alteration of local scopes is supported in any case anymore. Perhaps the last to go was runtime alteration of locals via `exec`. This still works in Python 2: >>> def f(): ... exec 'i = 1+2' ... print i >>> f() 3 >>> i # showing that it was function-local `i`, not global Traceback (most recent call last): File "", line 1, in NameError: name 'i' is not defined `exec` was a statement type in Python 2, so the compiler could recognize that and generate radically different name-lookup code in a scope containing an `exec` statement. But in Python 3, `exec` is just another builtin function, and the compiler knows nothing about what it does. Similar code run in Python 3 has no effect on f's locals. From njs at pobox.com Tue May 1 14:30:39 2018 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 01 May 2018 18:30:39 +0000 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: References: <20180501004252.GG7400@ando.pearwood.info> Message-ID: On Tue, May 1, 2018, 02:55 Matt Arcidy wrote: > > I am not inferring causality when creating a measure. No, but when you assume that you can use that measure to *make* code more readable, then you're assuming causality. Measuring the > temperature of a steak doesn't infer why people like it medium rare. > It just quantifies it. > Imagine aliens who have no idea how cooking works decide to do a study of steak rareness. They go to lots of restaurants, order steak, ask people to judge how rare it was, and then look for features that could predict these judgements. They publish a paper with an interesting finding: it turns out that restaurant decor is highly correlated with steak rareness. Places with expensive leather seats and chandeliers tend to serve steak rare, while cheap diners with sticky table tops tend to serve it well done. (I haven't done this study, but I bet if you did then you would find this correlation is actually true in real life!) Now, should we conclude based on this that if we want to get rare steak, the key is to *redecorate the dining room*? Of course not, because we happen to know that the key thing that changes the rareness of steak is how it's exposed to heat. But for code readability, we don't have this background knowledge; we're like the aliens. Maybe the readability metric in this study is like quantifying temperature; maybe it's like quantifying how expensive the decor is. We don't know. (This stuff is extremely non-obvious; that's why we force scientists-in-training to take graduate courses on statistics and experimental design, and it still doesn't always take.) > > And yeah, it doesn't help that they're only looking at 3 line blocks > > of code and asking random students to judge readability ? hard to say > > how that generalizes to real code being read by working developers. > > Respectfully, this is practical application and not a PhD defense, so > it will be generated by practical coding. > Well, that's the problem. In a PhD defense, you can get away with this kind of stuff; but in a practical application it has to actually work :-). And generalizability is a huge issue. People without statistical training tend to look at studies and worry about how big the sample size is, but that's usually not the biggest concern; we have ways to quantify how big your sample needs to be. There bigger problem is whether your sample is *representative*. If you're trying to guess who will become governor of California, then if you had some way to pick voters totally uniformly at random, you'd only need to ask 50 or 100 of them how they're voting to get an actually pretty good idea of what all the millions of real votes will do. But if you only talk to Republicans, it doesn't matter how many you talk to, you'll get a totally useless answer. Same if you only talk to people of the same age, or who all live in the same town, or who all have land-line phones, or... This is what makes political polling difficult, is getting a representative sample. Similarly, if we only look at out-of-context Java read by students, that may or may not "vote the same way" as in-context Python read by the average user. Science is hard :-(. -n > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Tue May 1 15:07:29 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 1 May 2018 14:07:29 -0500 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: References: <20180501004252.GG7400@ando.pearwood.info> Message-ID: Objectively quantifying is easy. For example, def objective_readability_score(text): "Return the readability of `text`, a float in 0.0 .. 1.0" return 2.0 * text.count(":=") / len(text) Then >>> objective_readability_score("if value:") 0.0 >>> objective_readability_score("if value := f():") 0.125 >>> objective_readability_score(":=:=:=") 1.0 QED ;-) From pradyunsg at gmail.com Tue May 1 15:13:01 2018 From: pradyunsg at gmail.com (Pradyun Gedam) Date: Tue, 01 May 2018 19:13:01 +0000 Subject: [Python-ideas] Should __builtins__ have some kind of pass-through print function, for debugging? In-Reply-To: References: <20180427112733.GP7400@ando.pearwood.info> Message-ID: On Fri, Apr 27, 2018 at 6:24 PM Nick Coghlan wrote: > On 27 April 2018 at 21:27, Steven D'Aprano wrote: > >> Obviously dp() would have to be magic. There's no way that I know of for >> a Python function to see the source code of its own arguments. I have no >> idea what sort of deep voodoo would be required to make this work. But >> if it could work, wow, that would really be useful. And not just for >> beginners. >> > > If you relax the enhancement to just noting the line where the debug print > came from, it doesn't need to be deep compiler magic - the same kind of > stack introspection that warnings and tracebacks use would suffice. (Stack > introspection to find the caller's module, filename and line number, > linecache to actually retrieve the line if we want to print that). > I? spent a bit of time and now there's a dprint project on PyPI. It uses stack introspection to print out some details. If someone wants to take it for a spin and provide feedback on how it feels, this thread is as good a place as any, I suppose. :) Cheers, Pradyun > Cheers, > Nick. > > P.S. While super() is a *little* magic, it isn't *that* magic - it gets > converted from "super()" to "super(name_of_first_param, __class__)". And > even that limited bit of magic has proven quirky enough to be a recurring > source of irritation when it comes to interpreter maintenance. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- -- Pradyun -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue May 1 15:08:41 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 2 May 2018 05:08:41 +1000 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: References: <20180501004252.GG7400@ando.pearwood.info> <20180501123711.GK7400@ando.pearwood.info> Message-ID: <20180501190841.GO7400@ando.pearwood.info> On Tue, May 01, 2018 at 03:02:27PM +0000, Dan Sommers wrote: > >> I happen to be an excellent judge of whether a given block of code is > >> readable to me. > > > In the same way that 93% of people say that they are an above-average > > driver, I'm sure that most people think that they are an excellent > > judge of readability. Including myself in *both* of those. > > Are you claiming that I'm not an excellent judge of whether a given > block of code is readable to me? Of course not. I don't know you. I wouldn't dream of making *specific* claims about you. I speak only in broad generalities which apply to people in general. I'm reminded that in the 1990s during the UI wars between Apple and Microsoft, people had *really strong* opinions about the useability of the two OSes' GUIs. Macs required the user to use the mouse to navigate menus, while Windows also allowed the use to navigate them using the Alt key and arrow keys. Not surprisingly, *both* Mac users and Windows users were absolutely convinced that they were much more efficient using the method they were familiar with, and could justify their judgement. For example, Windows users typically said that having to move their hand from the keyboard to grab the mouse was slow and inefficient, and using the Alt key and arrows was much faster. But when researchers observed users in action, and timed how long it took them to perform simple tasks requiring navigating the menus, they found that using the mouse was significantly faster for *both* groups of users, both Windows and Mac users. The difference was that when Windows users used the mouse, even though they were *objectively* faster to complete the task compared to using the arrow keys, subjectively they swore that they were slower, and were *very confident* about their subjective experience. This is a good example of the overconfidence effect: https://en.wikipedia.org/wiki/Overconfidence_effect This shouldn't be read as a tale about Mac users being superior. One of the two methods had to be faster, and it happened to be Macs. My point is not about Macs versus Windows, but that people in general are not good at this sort of self-reflection. Another example of this is the way that the best professional athletes no longer rely on their own self-judgement about the best training methods to use, because the training techniques that athletes think are effective, and those which actually are effective, are not strongly correlated. Athletes are not great judges of what training works for themselves. The psychological processes that lead to these cognitive biases apply to us all, to some degree or another. Aside from you and me, of course. -- Steve From mikhailwas at gmail.com Tue May 1 16:22:29 2018 From: mikhailwas at gmail.com (Mikhail V) Date: Tue, 1 May 2018 23:22:29 +0300 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: <20180501004252.GG7400@ando.pearwood.info> References: <20180501004252.GG7400@ando.pearwood.info> Message-ID: On Tue, May 1, 2018 at 3:42 AM, Steven D'Aprano wrote: > On Mon, Apr 30, 2018 at 11:28:17AM -0700, Matt Arcidy wrote: > > - people are not good judges of readability; People are the only judges of readability. Just need the right people. From python at mrabarnett.plus.com Tue May 1 16:24:28 2018 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 1 May 2018 21:24:28 +0100 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: References: <20180428093334.GU7400@ando.pearwood.info> <5AE61F76.60900@stoneleaf.us> <5AE66F61.9000900@stoneleaf.us> <4a5496f7-c180-4a01-0de8-dcd9b07681ad@gmail.com> <78764cbe-ec31-626f-35e8-6d0f5dd6fb06@mrabarnett.plus.com> <41dd3483-0a12-9aca-d181-66b555083480@mrabarnett.plus.com> Message-ID: On 2018-05-01 19:12, Tim Peters wrote: > [MRAB] > > Imagine renaming the specified names that are declared 'local' throughout > > the nested portion: > > > > def f(): > > a = 10 > > local local_a: > > def showa(): > > print("a is", local_a) > > showa() # Raises NameError in showa because nothing bound to local_a yet. > > local_a = 20 > > showa() # 20 > > local_a = 30 > > showa() # Raises NameError in f because local_a not defined. > > In a later message I showed executable-today Python code that I > believe models your intent. That code does indeed display "30" for > the last call, and while I myself don't see that it's _useful_ yet, > the model is incomprehensible if it doesn't. > > It doesn't matter that local_a isn't defined at function scope, > because showa() refers to the locals _in_ the "local a:" scope. The > last value bound to local_a was 30, so that's what showa() needs to > show. That the name `showa` is _bound_ in f's locals has nothing to > do with whose locals showa() _uses_. Other current messages about > "closures" go on more about that. > Ah, yes. I see what you mean. There's another question that hasn't been asked yet: what should locals() and globals() return? From tim.peters at gmail.com Tue May 1 16:59:42 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 1 May 2018 15:59:42 -0500 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: References: <20180428093334.GU7400@ando.pearwood.info> <5AE61F76.60900@stoneleaf.us> <5AE66F61.9000900@stoneleaf.us> <4a5496f7-c180-4a01-0de8-dcd9b07681ad@gmail.com> <78764cbe-ec31-626f-35e8-6d0f5dd6fb06@mrabarnett.plus.com> <41dd3483-0a12-9aca-d181-66b555083480@mrabarnett.plus.com> Message-ID: [MRAB] >>> Imagine renaming the specified names that are declared 'local' >>> throughout the nested portion: >>> >>> def f(): >>> a = 10 >>> local local_a: >>> def showa(): >>> print("a is", local_a) >>> showa() # Raises NameError in showa because nothing bound to >>> local_a yet. >>> local_a = 20 >>> showa() # 20 >>> local_a = 30 >>> showa() # Raises NameError in f because local_a not defined. [Tim] >> In a later message I showed executable-today Python code that I >> believe models your intent. That code does indeed display "30" for >> the last call, and while I myself don't see that it's _useful_ yet, >> the model is incomprehensible if it doesn't. >> >> It doesn't matter that local_a isn't defined at function scope, >> because showa() refers to the locals _in_ the "local a:" scope. The >> last value bound to local_a was 30, so that's what showa() needs to >> show. That the name `showa` is _bound_ in f's locals has nothing to >> do with whose locals showa() _uses_. Other current messages about >> "closures" go on more about that. [MRAB] > Ah, yes. I see what you mean. > > There's another question that hasn't been asked yet: what should locals() > and globals() return? Under my meaning, the same answer as always: exactly the same as if "local a:' were replaced by "if True:" ;-) Under yours, you're not going to like the answer. It seems obvious that since globals() refers to the module dict, nothing about that would change. You can add prints to my workalike-code-under-current-Python snippet to see what locals() _does_ return in various places. It's not what you'd expect (that locals() inside the `local a:` block is a dict with only key "local_a"), because of what I believe are undocumented implementation details. This is much easier to see in a bare-bones example: def f(): b = 12 def g(): nonlocal b b = 17 a = 42 print("inner", locals()) g() print("outer", locals()) f() The output: inner {'a': 42, 'b': 17} outer {'g': .g at 0x0000024E28C44AE8>, 'b': 17} That is, despite that `b` is explicitly declared as `nonlocal` in the inner function, and does in fact belong to the outer scope, it nevertheless shows up inside the inner function's locals(). The compiler isn't confused, but `locals()` existed long before nested scopes were added, and, e.g., no "nonlocals()` builtin was added later. PEP 227 (which introduced nested scopes) explicitly declined to add one, and said "it will not be possible to gain dictionary-style access to all visible scopes". I was quite surprised to see "b" as a key in the inner locals() above! But so long as it is, any new gimmick building on the current implementation would inherit that behavior. From mikhailwas at gmail.com Tue May 1 20:35:41 2018 From: mikhailwas at gmail.com (Mikhail V) Date: Wed, 2 May 2018 03:35:41 +0300 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: References: <20180501004252.GG7400@ando.pearwood.info> <2e459330-6d2e-d163-859b-d62af0cd9e30@kynesim.co.uk> <5AE8636B.1030301@canterbury.ac.nz> Message-ID: On Tue, May 1, 2018 at 6:04 PM, Jacco van Dorp wrote: > 2018-05-01 14:54 GMT+02:00 Greg Ewing : >> Rhodri James wrote: >>> >>> I'd be interested to know if there is a readability difference between >>> really_long_descriptive_identifier_name and >>> ReallyLongDescriptiveIdentifierNames. >> >> >> As one data point on that, jerking my eyes quickly across >> that line I found it much easier to pick out the component >> words in the one with underscores. >> >> -- >> Greg > > > Which is funny, because I had the exact opposite. > > Might it be that we've had different conditioning ? > > Jacco The one with underscores reads fairly better though. Might it be that it does read better? Ok, that's not scientific enough. But the scores 2:1 so far ;-) to be pedantic - ReallyLongDescriptiveIdentifierNames has also an issue with "I" which might confuse because it looks same as little L. Just to illustrate that choice of comparison samples is very sensitive thing. In such a way an experienced guy can even scam the experimental subjects by making samples which will show what he wants in result. Mikhail From marcidy at gmail.com Tue May 1 21:03:22 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Tue, 1 May 2018 18:03:22 -0700 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: References: <20180501004252.GG7400@ando.pearwood.info> <2e459330-6d2e-d163-859b-d62af0cd9e30@kynesim.co.uk> <5AE8636B.1030301@canterbury.ac.nz> Message-ID: On Tue, May 1, 2018 at 5:35 PM, Mikhail V wrote: > to be pedantic - ReallyLongDescriptiveIdentifierNames > has also an issue with "I" which might confuse because it > looks same as little L. Just to illustrate that choice of > comparison samples is very sensitive thing. > In such a way an experienced guy can even scam > the experimental subjects by making samples which > will show what he wants in result. I love this discussion, but I think anything that isn't included in a .py file would have to be outside the scope, at least of the alpha version :). I am really interested in these factors in general, however. Now I'm surprised no one asks which font each other are using when determining readability. "serif? are you mad? no wonder!" "+1 on PEP conditional on mandatory yellow (#FFEF00) keyword syntax highlighting in vim" -Matt > > > Mikhail > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From cfkaran2 at gmail.com Tue May 1 21:37:26 2018 From: cfkaran2 at gmail.com (CFK) Date: Tue, 1 May 2018 21:37:26 -0400 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: References: <20180501004252.GG7400@ando.pearwood.info> <2e459330-6d2e-d163-859b-d62af0cd9e30@kynesim.co.uk> <5AE8636B.1030301@canterbury.ac.nz> Message-ID: Matt, you took the words right out of my mouth! The fonts that are being used will have a big difference in readability, as will font size, foreground and background coloring, etc. It would be interesting to see if anyone has done a serious study of this type though, especially if they studied it over the course of several hours (I'm getting older, and I've noticed that after about 8-10 hours of coding it doesn't matter what I'm looking at, I can't focus enough to read it, but I don't know when I start to degrade, nor do I know if different fonts would help me degrade more slowly) Thanks, Cem Karan On Tue, May 1, 2018 at 9:03 PM, Matt Arcidy wrote: > On Tue, May 1, 2018 at 5:35 PM, Mikhail V wrote: > > > to be pedantic - ReallyLongDescriptiveIdentifierNames > > has also an issue with "I" which might confuse because it > > looks same as little L. Just to illustrate that choice of > > comparison samples is very sensitive thing. > > In such a way an experienced guy can even scam > > the experimental subjects by making samples which > > will show what he wants in result. > > I love this discussion, but I think anything that isn't included in a > .py file would have to be outside the scope, at least of the alpha > version :). I am really interested in these factors in general, > however. Now I'm surprised no one asks which font each other are > using when determining readability. > > "serif? are you mad? no wonder!" > "+1 on PEP conditional on mandatory yellow (#FFEF00) keyword syntax > highlighting in vim" > > -Matt > > > > > > > Mikhail > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Tue May 1 22:07:41 2018 From: mikhailwas at gmail.com (Mikhail V) Date: Wed, 2 May 2018 05:07:41 +0300 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: References: <20180501004252.GG7400@ando.pearwood.info> <2e459330-6d2e-d163-859b-d62af0cd9e30@kynesim.co.uk> <5AE8636B.1030301@canterbury.ac.nz> Message-ID: On Wed, May 2, 2018 at 4:03 AM, Matt Arcidy wrote: > On Tue, May 1, 2018 at 5:35 PM, Mikhail V wrote: > >> to be pedantic - ReallyLongDescriptiveIdentifierNames >> has also an issue with "I" which might confuse because it >> looks same as little L. Just to illustrate that choice of >> comparison samples is very sensitive thing. >> In such a way an experienced guy can even scam >> the experimental subjects by making samples which >> will show what he wants in result. > > I love this discussion, but I think anything that isn't included in a > .py file would have to be outside the scope, at least of the alpha > version :). I am really interested in these factors in general, > however. Now I'm surprised no one asks which font each other are > using when determining readability. > > "serif? are you mad? no wonder!" > "+1 on PEP conditional on mandatory yellow (#FFEF00) keyword syntax > highlighting in vim" > Well, I am asking. Looking at online PEPs I am under impression everyone should use huge-sized Consolas and no syntax highlighting at all. Just as with "=" and "==". Making samples without highlighting will show a similarity issue. Making it with different highlighting/font style will show that there is no issue. Or, ":=" looks ok with Times New Roman, but with Consolas - it looks like Dr. Zoidberg's face. Mikhail From tim.peters at gmail.com Tue May 1 23:51:05 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 1 May 2018 22:51:05 -0500 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: <20180501152316.GM7400@ando.pearwood.info> References: <5AE61F76.60900@stoneleaf.us> <5AE66F61.9000900@stoneleaf.us> <4a5496f7-c180-4a01-0de8-dcd9b07681ad@gmail.com> <78764cbe-ec31-626f-35e8-6d0f5dd6fb06@mrabarnett.plus.com> <20180501152316.GM7400@ando.pearwood.info> Message-ID: [MRAB] >>> Would/should it be possible to inject a name into a local scope? You can't >>> inject into a function scope, and names in a function scope can be >>> determined statically (they are allocated slots), so could the same kind of >>> thing be done for names in a local scope? ... [Steven D'Aprano ] > I don't know what MRAB means by "inject", but I know what *I* mean, and > I have a real use-case for it. > > There is a long-running micro-optimization, often championed by Raymond, > for avoiding slow global lookups (and even slower builtin lookups, since > they require a global lookup to fail first) by turning them in local > lookups at function-definition time. Before nested scopes, it was also used to help nested functions call each other; e.g., def f(): def g(): ... # h needs to call g, but can't see f's locals def h(g=g): # but default arg expressions can see f's locals g() # works great :-) > E.g. some methods from the random.Random class: > > def randrange(self, start, stop=None, step=1, _int=int): > ... > def _randbelow(self, n, int=int, maxsize=1< Method=_MethodType, BuiltinMethod=_BuiltinMethodType): > ... > > (copied from 3.5, I don't know what the most recent version looks like) Doesn't matter; code like that has been in the std lib since the start, although Raymond is inordinately fond of it ;-) > That's a nice way to make the binding: > > _int = int > > occur once only, instead of putting it inside the function body which > then needs to be executed on ever call. Effectively it's a static > variable for the method, one which persists from one call to the next > without requiring re-initialisation. > > But it's ugly :-( Error-prone too, because absolutely nothing stops a user from calling these things with more positional arguments than are intended. I don't want to bother looking now, but there was _some_ "bug report" complaining that one of these "default arguments" went away somewhere, which a user was specifying intentionally to override some accidental implementation detail. However, while it's dead obviously "error prone" in theory, it's remarkable how few times I've heard of anyone getting burned by it in practice. > The function signature is full of *implementation details* instead of > the parameters needed for the method's interface. Look at _randbelow > which takes one actual parameter, n, plus FIVE fake parameters, int, > maxsize, type, Method and BuiltinMethod, none of which should ever be > passed arguments. > > So when I talk about injecting values into a function, that is the sort > of thing I'm referring to: at function definition time, push or inject a > reference to a known value (say, the builtin int) into something which > behaves as a static variable. > > It would be nice if we could do that without polluting the function > signature. I'm surprised that hasn't already been done. > I'll admit that the name "inject" came to me when I was > thinking of some hypothetical decorator: > > @inject(int=int, maxsize=1< def _randbelow(self, n): > ... > > that somehow pushed, or *injected*, those bindings into the function, > turning them into locals, but in an alternative universe where Guido > loved making new keywords, I'd use a static initialisation block and > stick it inside the def: > > def _randbelow(self, n): > static: > # this gets executed once only, at function definition time > int=int > maxsize=1< type=type > # body of _randbelow Blame computer scientists. I started my paying career working on Cray Research's Fortran compiler, a language in which NO words were reserved. The syntax was such that it was always possible to determine whether a language keyword or a user-supplied name was intended. That worked out great. But it seemed to require ad hoc parsing tricks. Computer scientists invented "general" parser generators, and then every new language started declaring that some words were reserved for the language's use. That was just to make things easier for parser-generator authors, not for users ;-) Maybe we could steal a trick from the way the async statements ("async def", "async for", "async with") were added without making "async" a reserved word? Even if user code already uses the name "static", just as in Fortran the only way to parse static: that makes sense is that it's introducing a block; USER_NAME: as a statement has no meaning. In any other context, the block-opening meaning of "static" makes no sense, so it must mean a user-defined name then. We could also add, e.g., MRAB local a, b, c: and Uncle Timmy not really local at all a, b, c: without introducing any actual ambiguity in code already using any of the leading words on those lines. It would be fun to deprive Guido of the dead easy "but it's a new reserved word!" way to veto new features ;-) From dan at tombstonezero.net Wed May 2 00:03:02 2018 From: dan at tombstonezero.net (Dan Sommers) Date: Wed, 2 May 2018 04:03:02 +0000 (UTC) Subject: [Python-ideas] Objectively Quantifying Readability References: <20180501004252.GG7400@ando.pearwood.info> <20180501123711.GK7400@ando.pearwood.info> <20180501190841.GO7400@ando.pearwood.info> Message-ID: On Wed, 02 May 2018 05:08:41 +1000, Steven D'Aprano wrote: > The difference was that when Windows users used the mouse, even though > they were *objectively* faster to complete the task compared to using > the arrow keys, subjectively they swore that they were slower, and > were *very confident* about their subjective experience. Another driving analogy: when I get stuck at a stoplight, sometimes I take advantage of turn on red or a protected turn, even though I know that it's going to take longer to get where I'm going. But I feel better because I'm not just sitting there at the stoplight. Call it cognitive dissonance, I guess. Some of my coworkers claim that using vi is objectively faster or requires fewer keystrokes than using emacs. I counter that I've been using emacs since before they were born, and that I now do so with the reptilian part of my brain, which means that I can keep thinking about the problem at hand rather than about editing the source code. Who remembers the One True Brace Style holy wars? If we agreed on anything, it was to conform to existing code rather than to write new code in a different style. Reading a mixture of styles was harder, no matter which particular style you thought was better or why you thought it was better. > Athletes are not great judges of what training works for themselves. Wax on, wax off? ;-) Dan From kenlhilton at gmail.com Wed May 2 02:21:19 2018 From: kenlhilton at gmail.com (Ken Hilton) Date: Wed, 2 May 2018 14:21:19 +0800 Subject: [Python-ideas] Yet another idea for assignment expressions Message-ID: Hi all, I've been following the discussion of assignment expressions and what the syntax for them should be for awhile. The ones that seem to crop up most are the original spelling, :=, the "as" keyword (and variants including it), and the recently "local" pseudo-function idea. I have another idea for a spelling of assignment expressions, mostly inspired by real sentence structure. In a review I was writing recently, I referred to CGI, Computer Generated Imagery. As a way of introducing the full term and its abbreviation, I said "Computer Generated Imagery (CGI)". That got me thinking: why not have something similar in Python? Obviously simple parentheses ("expr(name)") definitely wouldn't work, that's a function call. Similarly, brackets ("expr[name]") would be interpreted as subscription. So why not use curly brackets, "expr{name}"? That doesn't conflict with anything (currently it's a syntax error), and could be documented appropriately. Going back to the regex example, this is how it would look in that case: if re.match(exp, string){m}: print(m.group(0)) I am currently unsure how it would affect scope. I think it should be effectively equivalent to a regular assignment statement (and hence follow identical scope rules), i.e. the above example would be equivalent to the following in all ways except syntax: m = re.match(exp, string): if m: print(m.group(0)) Thoughts? Please do let me know if there's some critical flaw with this idea that I missed (or tell me if it's an amazing idea ;)), and just give feedback, I guess. Sincerely, Ken; -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.van.dorp at deonet.nl Wed May 2 02:48:14 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Wed, 2 May 2018 08:48:14 +0200 Subject: [Python-ideas] A way to subscript a single integer from bytes In-Reply-To: References: Message-ID: > I think this method is easy to miss, since people look at the docs for bytes > (e.g. using dir(bytes)). It might be worthwhile to either add a > `bytes.to_int(...)` method (better, IMHO), or to point to int.from_bytes on > the relevant part of the docs. > > Elazar A note in the docs about int.from_bytes seems more than warranted to me to. Jacco From j.van.dorp at deonet.nl Wed May 2 03:16:25 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Wed, 2 May 2018 09:16:25 +0200 Subject: [Python-ideas] Yet another idea for assignment expressions In-Reply-To: References: Message-ID: I kind of strangely like this, but it does something completely different from parens or []. Those do have something in common - func(param) and indexable[index] both result in some value obtained in some way by combining the two names - either it's the result of func when called with param, or it's the some part of indexable identified by index. expr{name} would have a value equal to that of expr, and actually change name (and the change to the contained name is something [] never does (AFAIK), and I daresay most people would agree a function call shouldn't either.) From greg.ewing at canterbury.ac.nz Wed May 2 03:34:18 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 02 May 2018 19:34:18 +1200 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: <20180501180023.GN7400@ando.pearwood.info> References: <20180428093334.GU7400@ando.pearwood.info> <20180429025005.GB7400@ando.pearwood.info> <20180430162027.GE7400@ando.pearwood.info> <5AE803CE.8070104@canterbury.ac.nz> <20180501071146.GJ7400@ando.pearwood.info> <5AE832B1.8000601@canterbury.ac.nz> <20180501180023.GN7400@ando.pearwood.info> Message-ID: <5AE969FA.50100@canterbury.ac.nz> Steven D'Aprano wrote: > y = 1 > def func(): > x = 2 > return x+y > > Here, there's a local environment as well as an implicit global one. > Surely we don't want to call this a closure? Python probably isn't the best choice of language for the point you're making, because even top-level functions in Python *do* carry a reference to their environment (the dict of the module they were defined in). C would be a better choice: int y = 1; int func(void) { int x = 2; return x + y; } int (*fp)() = func; Even here, I don't think there's anything wrong with calling fp a closure. It's just that it's a particularly simple form of closure -- since there's only one global environment, we don't need to bother explicitly carrying it around. However, that doesn't mean all functions are closures. An example of a non-closure might be a C++ member function pointer. It points to a piece of code, but you can't use it on its own, because it's missing a piece of environmental information (a value for "this") that you need to supply by hand when you call it. > we have to consider what happens if we come > across a Pascal implementation which *doesn't* use a (code_address, > frame_pointer) pair. Is that no longer Pascal? In anything that implements Pascal semantics, a function parameter is going to have to be represented by *something* that specifies both code and environment. Code alone is not enough. > If we accept that "Pascal has closures", they're pretty crap closures, > since they don't let you do any of the interesting and useful things you > can do in languages with "real" closures like Scheme and Python. Not all of them, but it can do some of them. They're more useful than in Modula-2, for example, where nested functions can't even be passed as parameters. Anyway, that's why I said "a limited form of closure". You could read that two ways: (1) It's totally a closure, but there are limitations on what you can do with it (i.e. it's not first class). Or (2) it has some of the characteristics of a closure, but not all of them. In my mind I tend towards (1), but it doesn't really matter. -- Greg From vincent.maillol at gmail.com Wed May 2 04:08:55 2018 From: vincent.maillol at gmail.com (Vincent Maillol) Date: Wed, 2 May 2018 10:08:55 +0200 Subject: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter Message-ID: Hi everybody, Our PEP idea would be to purpose to add a global default value for itemgeet and attrgetter method. This was inspired from bug 14384 (https://bugs.python.org/issue14384); opened by Miki TEBEKA. For example, we could do: p1 = {'x': 43; 'y': 55} x, y, z = itemgetter('x', 'y', 'z', default=0)(values) print(x, y, z) 43, 55, 0 instead of: values = {'x': 43; 'y': 55} x = values.get('x', 0) y = values.get('y', 0) z = values.get('z', 0) print(x, y, z) 43, 55, 0 The goal is to have have concise code and improve consistency with getattr, attrgetter and itemgetter What are you thinking about this? MAILLOL Vincent GALODE Alexandre From greg.ewing at canterbury.ac.nz Wed May 2 05:00:46 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 02 May 2018 21:00:46 +1200 Subject: [Python-ideas] Objectively Quantifying Readability In-Reply-To: References: <20180501004252.GG7400@ando.pearwood.info> Message-ID: <5AE97E3E.6000404@canterbury.ac.nz> Tim Peters wrote: > def objective_readability_score(text): > "Return the readability of `text`, a float in 0.0 .. 1.0" > return 2.0 * text.count(":=") / len(text) A useful-looking piece of code, but it could be more readable. It only gives itself a readability score of 0.0136986301369863. -- Greg From rhodri at kynesim.co.uk Wed May 2 08:13:59 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Wed, 2 May 2018 13:13:59 +0100 Subject: [Python-ideas] Yet another idea for assignment expressions In-Reply-To: References: Message-ID: On 02/05/18 07:21, Ken Hilton wrote: > Going back to the regex example, this is how it would look in that case: > > if re.match(exp, string){m}: > print(m.group(0)) The various other options at least suggest that in some manner what is going on is an assignment. This really doesn't. Sorry, it's just too magic. -- Rhodri James *-* Kynesim Ltd From tim.peters at gmail.com Wed May 2 15:09:48 2018 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 2 May 2018 14:09:48 -0500 Subject: [Python-ideas] A "local" pseudo-function In-Reply-To: References: <20180428093334.GU7400@ando.pearwood.info> <5AE61F76.60900@stoneleaf.us> <5AE66F61.9000900@stoneleaf.us> <4a5496f7-c180-4a01-0de8-dcd9b07681ad@gmail.com> <78764cbe-ec31-626f-35e8-6d0f5dd6fb06@mrabarnett.plus.com> <41dd3483-0a12-9aca-d181-66b555083480@mrabarnett.plus.com> Message-ID: [MRAB] >> There's another question that hasn't been asked yet: what should locals() >> and globals() return? [Tim, "globals()" is obvious, "locals()" can be surprising now] > ... And here recording the results of some code spelunking Dicts don't really have anything to do with how locals are implemented anymore; calling "locals()" now inside a function _constructs_ a dict on the fly out of lower-level implementation gimmicks, to be kinda-compatible with what locals() returned before nested scopes were introduced. The real distinctions of interest are recorded in the code object now, under these attributes, where I'm giving a fuller explanation than can be found in the docs, and where "referenced" doesn't distinguish between "binding merely retrieved" and "binding is changed": 1. co_varnames: tuple of names of locals not referenced in an enclosed local scope 2. co_cellvars: tuple of names of locals that are referenced in an enclosed local scope 3 .co_freevars: tuple of names local to an enclosing local scope and referenced in this enclosed code "locals()" now generally builds a dict out of all three of those name sources. #1 is the only one that made sense before nested scopes were introduced. The union of #1 and #2 is the set of names local to the code's scope; CPython implements their bindings in different ways, so needs to distinguish them. The names in #3 aren't local to the code block at all (they are local to some enclosing local scope), but for whatever reason are included in the code's "locals()" dict anyway. I would not have included them. For concreteness: def disp(func): print("for", func.__name__) code = func.__code__ for attr in "co_varnames", "co_cellvars", "co_freevars": print(" ", attr, getattr(code, attr)) def outer(): outer_only = 1 outer_used_inner = 2 outer_bound_by_inner = 3 def inner(): nonlocal outer_bound_by_inner inner_only = 4 outer_bound_by_inner = 5 inner_only = outer_used_inner inner() disp(outer) disp(inner) outer() And its output: for outer co_varnames ('outer_only', 'inner') co_cellvars ('outer_bound_by_inner', 'outer_used_inner') co_freevars () for inner co_varnames ('inner_only',) co_cellvars () co_freevars ('outer_bound_by_inner', 'outer_used_inner') From mistersheik at gmail.com Wed May 2 17:46:09 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 2 May 2018 14:46:09 -0700 (PDT) Subject: [Python-ideas] Please consider adding partialclass to functools Message-ID: Essentially, functools.partial is almost good enough for specifying some of the parameters of an object's initializer, but the partial object doesn't respond properly to issubclass. Please consider adding something like partialclass described here: https://stackoverflow.com/questions/38911146/python-equivalent-of-functools-partial-for-a-class-constructor I ended up doing the same thing here: https://stackoverflow.com/questions/50143864/is-there-a-nice-way-to-partially-bind-class-parameters-in-python/50143893#50143893 Adding functools.partialclass would be similar to the addition of partialmethod in 3.4. Best, Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed May 2 22:41:04 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 3 May 2018 12:41:04 +1000 Subject: [Python-ideas] Please consider adding partialclass to functools In-Reply-To: References: Message-ID: <20180503024104.GC9562@ando.pearwood.info> On Wed, May 02, 2018 at 02:46:09PM -0700, Neil Girdhar wrote: > Essentially, functools.partial is almost good enough for specifying some of > the parameters of an object's initializer, but the partial object doesn't > respond properly to issubclass. [...] I think that ought to be an uncontroversial enough change that it can be taken to the bug tracker. I think you should just raise a feature enhancement request for 3.8 on the tracker, and link to this thread on the mailing list archives in case some discussion follows. -- Steve From steve at pearwood.info Wed May 2 23:40:57 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 3 May 2018 13:40:57 +1000 Subject: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter In-Reply-To: References: Message-ID: <20180503034057.GD9562@ando.pearwood.info> On Wed, May 02, 2018 at 10:08:55AM +0200, Vincent Maillol wrote: > Hi everybody, > > Our PEP idea would be to purpose to add a global default value for > itemgeet and attrgetter method. I'm sorry, I'm not sure I understand what you mean by a global default. My interpretation of that would be the ability to set a global variable which acts as default for *all* calls to itemgetter. itemdefault = "spam" f = itemgetter(2, 99) f('abcd') # => returns ('c', 'spam') If that's what you mean, I hate it :-) [...] > For example, we could do: > > p1 = {'x': 43; 'y': 55} > x, y, z = itemgetter('x', 'y', 'z', default=0)(values) > print(x, y, z) > 43, 55, 0 I wouldn't call that a *global* default. If the default has to be specified on each call, I think that's both acceptible and desirable. Bug 14384 (https://bugs.python.org/issue14384) suggests that it might be more desirable to specify the default when you call the getter function, not when you call the factory: # this f = itemgetter(2, 99) f('abcd', default='spam') # => returns ('c', 'spam') # rather than this f = itemgetter(2, 99, default='spam') f('abcd') # => returns ('c', 'spam') I could go either way. In fact, at the risk of making a more complicated API, I'm almost inclined to allow both forms... -- Steve From raymond.hettinger at gmail.com Thu May 3 01:28:33 2018 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 2 May 2018 22:28:33 -0700 Subject: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter In-Reply-To: References: Message-ID: > On May 2, 2018, at 1:08 AM, Vincent Maillol wrote: > > Our PEP idea would be to purpose to add a global default value for > itemgeet and attrgetter method. My preference is to not grow that API further. It is creep well beyond its intended uses. At some point, we're really better off just using a lambda. Raymond From steve at pearwood.info Thu May 3 02:32:09 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 3 May 2018 16:32:09 +1000 Subject: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter In-Reply-To: References: Message-ID: <20180503063208.GE9562@ando.pearwood.info> On Wed, May 02, 2018 at 10:28:33PM -0700, Raymond Hettinger wrote: > > > > On May 2, 2018, at 1:08 AM, Vincent Maillol wrote: > > > > Our PEP idea would be to purpose to add a global default value for > > itemgeet and attrgetter method. > > My preference is to not grow that API further. It is creep well > beyond its intended uses. Intended by whom? I think you are being too dismissive of actual use-cases requested by actual users. Default values might not have been the primary use considered when the API was first invented, but the fact that people keep asking for this feature should tell us that at least some people have intended uses that are remaining unmet. > At some point, we're really better off just using a lambda. Maybe I'm slow today, but I'm having trouble seeing how to write this as a lambda. The single index case would be simple using the proposed binding- expression from PEP 572: # replace INDEX and DEFAULT with the values you want lambda seq: s[0] if (s := seq[INDEX:INDEX+1]) else DEFAULT but without the binding-expression, I have no idea how to do it cleanly and correctly in a lambda. For what it is worth, on my computer, itemgetter with a single index is 20% faster than a lambda using seq[INDEX] (where INDEX is hard-coded in the lambda), and 60% faster than a lambda using seq[INDEX:INDEX+1][0]. According to the bug report linked to earlier, one of the intended uses for itemgetter is when performance matters. -- Steve From storchaka at gmail.com Thu May 3 05:06:46 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 3 May 2018 12:06:46 +0300 Subject: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter In-Reply-To: References: Message-ID: 03.05.18 08:28, Raymond Hettinger ????: >> On May 2, 2018, at 1:08 AM, Vincent Maillol wrote: >> >> Our PEP idea would be to purpose to add a global default value for >> itemgeet and attrgetter method. > > My preference is to not grow that API further. It is creep well beyond its intended uses. At some point, we're really better off just using a lambda. +1 From raiderrobert at gmail.com Thu May 3 08:41:47 2018 From: raiderrobert at gmail.com (Robert Roskam) Date: Thu, 03 May 2018 12:41:47 +0000 Subject: [Python-ideas] Pattern Matching Syntax Message-ID: Hi Everyone, Never posted in here before, so I hope that I'm not violating any particular procedure for intros or something. Over time, there have been various switch or match statement proposal; some that have gotten as far as PEPs: 2001 Nov - https://www.python.org/dev/peps/pep-0275/ 2006 Jun - https://www.python.org/dev/peps/pep-3103/ 2014 Apr - https://groups.google.com/d/msg/python-ideas/J5O562NKQMY/DrMHwncrmIIJ 2016 May - https://groups.google.com/d/msg/python-ideas/aninkpPpEAw/wCQ1IH5mAQAJ However, I don't see that the conversation ever really resolved, so I'd like restart the conversation on some kind of pattern matching syntax in Python. The main objections I've seen are in the following buckets: - One--and Preferably Only One--Obvious Way. Basically, we have if/elif and that's all we need, so this is syntactical sugar bloat. I'd submit that there are specific cases where this kind of syntax would be the obviously correct way to do something - Specific Syntax Objections. There have been several specific objections that usually come down to "unreadable" or "ugly", which are subjective statements that don't really bring any good way to continue the discussion in a productive manner. I cannot handle all syntax objections ahead of time, but I can handle the "only way" objection. At high level, pattern matching provides similar syntactical sugar to list comprehensions. We could argue that they are unnecessary since we have for loops. But more importantly, pattern matching is powerful for what it restricts you to. More specifically: - Assignment. Many of the implementations offer the ability to immediately assign the value from the matching pattern. However, assignment is prevented in the middle of all of the patterns, which is possible in if/elif. - No Fall Through. Once a pattern is matched, there's no way to break to try another branch. Prevents having to look at multiple cases to figure out how something resolved. If/elif can have this happen, of course, but even more confusing sometimes breaks will be mixed with returns or other control flows, which makes figuring how large if/elifs are resolved. - Automatic Unpacking. Some implementations offer the ability unpack a dictionary equivalent automatically into keys or select ranges of values like slicing. Compared to if/elif, this is tremendously more DRY than doing the "does the key exists?" and then "what is that keys value?" - Guards. Often times you can embed another check to go along with the simple pattern matching. Absolutely possible with if/elif, but crucially are implementations generally happen after the pattern check. Again, keeps code DRY and improves readability. I figured maybe a good way to continue the discussion is to offer a straw-man example syntax: # Simple pattern matching x = 1 number = match x: 1 => "one" 2 => "two" 3 => "three" 10 => "ten" _ => "anything" print(number) # one # Final Pattern that matches anything x = 3 number = match x: 1 => "one" 2 => "two" _ => "anything" print(number) # anything # Pattern matching without any match returns None number = match x: 1 => "one" 2 => "two" print(number) # None # Pattern matching with guards x = 'three' number = match x: 1 => "one" y if y is str => f'The string is {y}' _ => "anything" print(number) # The string is three # Pattern matching with multiple values x = 1 number = match x: 1, 2, 3, 4 => "one to four" _ => "anything" print(number) # one to four # Pattern matching with types x = 1. number = match x: x:int => f'{x} is a int' x:float => f'{x} is a float' x:str => f'{x} is a string' print(number) # x is a float # Supports destructuring dicts x = {'foo': 1} number = match x: {'foo': 1} => "foo is 1" _ => "anything" print(number) # foo is 1 # Supports binding with destructuring dicts x = {'foo': 1, 'bar': 2} number = match x: {'foo': y} => f'got foo {y}' {'bar': z} => f'got bar {z}' {'foo': y, 'bar': z} => f'got foo {y} and bar {z}' _ => "anything" print(number) # got foo 1 and bar 2 # Supports destructuring other types too class Point(): def __init__(self, x, y): self.x = x self.y = y point = Point(1,2) number = match point: Point(x,y) => f'point has an x of {x} and y of {y}' _ => "anything" print(number) # point has an x of 1 and y of 2 As a continued defense for this specific syntax choixe, lets see how two other languages with this feature handle it. I'm going to try to offer as nearly as possible similar examples. Scala https://docs.scala-lang.org/tour/pattern-matching.html val x: Int = 1 def makeMatch(x: Any) = x match { case 1 => "one" case 2 => "two" case _ => "anything" } val number = makeMatch(x) Rust https://doc.rust-lang.org/1.5.0/book/match.html let x = 1; let number = match x { 1 => "one", 2 => "two", _ => "anything", } And for the sake of completeness, here are other languages with similar syntax features and their associated documentation F# https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/pattern-matching Elixir https://elixir-lang.org/getting-started/case-cond-and-if.html Clojure https://github.com/clojure/core.match/wiki/Basic-usage JavaScript (ES2018?) https://github.com/tc39/proposal-pattern-matching Haskell https://en.wikibooks.org/wiki/Haskell/Pattern_matching Swift https://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/Patterns.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Thu May 3 09:15:43 2018 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Thu, 3 May 2018 10:15:43 -0300 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: Message-ID: What about instead of number = match x: 1 => "one" 2 => "two" 3 => "three" 10 => "ten" _ => "anything" number = { 1 => "one" 2 => "two" 3 => "three" 10 => "ten" }.get(x, "anything") No magic syntax with blocks starting inside an assignment, just to start with. On 3 May 2018 at 09:41, Robert Roskam wrote: > Hi Everyone, > > Never posted in here before, so I hope that I'm not violating any particular > procedure for intros or something. > > Over time, there have been various switch or match statement proposal; some > that have gotten as far as PEPs: > > 2001 Nov - https://www.python.org/dev/peps/pep-0275/ > > 2006 Jun - https://www.python.org/dev/peps/pep-3103/ > > 2014 Apr - > https://groups.google.com/d/msg/python-ideas/J5O562NKQMY/DrMHwncrmIIJ > > 2016 May - > https://groups.google.com/d/msg/python-ideas/aninkpPpEAw/wCQ1IH5mAQAJ > > However, I don't see that the conversation ever really resolved, so I'd like > restart the conversation on some kind of pattern matching syntax in Python. > > The main objections I've seen are in the following buckets: > > One--and Preferably Only One--Obvious Way. Basically, we have if/elif and > that's all we need, so this is syntactical sugar bloat. I'd submit that > there are specific cases where this kind of syntax would be the obviously > correct way to do something > Specific Syntax Objections. There have been several specific objections that > usually come down to "unreadable" or "ugly", which are subjective statements > that don't really bring any good way to continue the discussion in a > productive manner. > > I cannot handle all syntax objections ahead of time, but I can handle the > "only way" objection. At high level, pattern matching provides similar > syntactical sugar to list comprehensions. We could argue that they are > unnecessary since we have for loops. But more importantly, pattern matching > is powerful for what it restricts you to. More specifically: > > Assignment. Many of the implementations offer the ability to immediately > assign the value from the matching pattern. However, assignment is prevented > in the middle of all of the patterns, which is possible in if/elif. > No Fall Through. Once a pattern is matched, there's no way to break to try > another branch. Prevents having to look at multiple cases to figure out how > something resolved. If/elif can have this happen, of course, but even more > confusing sometimes breaks will be mixed with returns or other control > flows, which makes figuring how large if/elifs are resolved. > Automatic Unpacking. Some implementations offer the ability unpack a > dictionary equivalent automatically into keys or select ranges of values > like slicing. Compared to if/elif, this is tremendously more DRY than doing > the "does the key exists?" and then "what is that keys value?" > Guards. Often times you can embed another check to go along with the simple > pattern matching. Absolutely possible with if/elif, but crucially are > implementations generally happen after the pattern check. Again, keeps code > DRY and improves readability. > > I figured maybe a good way to continue the discussion is to offer a > straw-man example syntax: > > # Simple pattern matching > x = 1 > > number = match x: > 1 => "one" > 2 => "two" > 3 => "three" > 10 => "ten" > _ => "anything" > > print(number) # one > > > # Final Pattern that matches anything > x = 3 > > number = match x: > 1 => "one" > 2 => "two" > _ => "anything" > > print(number) # anything > > > # Pattern matching without any match returns None > number = match x: > 1 => "one" > 2 => "two" > > print(number) # None > > > # Pattern matching with guards > x = 'three' > > number = match x: > 1 => "one" > y if y is str => f'The string is {y}' > _ => "anything" > > print(number) # The string is three > > > # Pattern matching with multiple values > x = 1 > > number = match x: > 1, 2, 3, 4 => "one to four" > _ => "anything" > > print(number) # one to four > > > # Pattern matching with types > x = 1. > > number = match x: > x:int => f'{x} is a int' > x:float => f'{x} is a float' > x:str => f'{x} is a string' > > print(number) # x is a float > > > # Supports destructuring dicts > > x = {'foo': 1} > > number = match x: > {'foo': 1} => "foo is 1" > _ => "anything" > > print(number) # foo is 1 > > > # Supports binding with destructuring dicts > > x = {'foo': 1, 'bar': 2} > > number = match x: > {'foo': y} => f'got foo {y}' > {'bar': z} => f'got bar {z}' > {'foo': y, 'bar': z} => f'got foo {y} and bar {z}' > _ => "anything" > > print(number) # got foo 1 and bar 2 > > > # Supports destructuring other types too > > class Point(): > def __init__(self, x, y): > self.x = x > self.y = y > > point = Point(1,2) > > number = match point: > Point(x,y) => f'point has an x of {x} and y of {y}' > _ => "anything" > > print(number) # point has an x of 1 and y of 2 > > > As a continued defense for this specific syntax choixe, lets see how two > other languages with this feature handle it. I'm going to try to offer as > nearly as possible similar examples. > > Scala https://docs.scala-lang.org/tour/pattern-matching.html > > val x: Int = 1 > > def makeMatch(x: Any) = x match { > case 1 => "one" > case 2 => "two" > case _ => "anything" > } > > val number = makeMatch(x) > > Rust https://doc.rust-lang.org/1.5.0/book/match.html > > let x = 1; > > let number = match x { > 1 => "one", > 2 => "two", > _ => "anything", > } > > And for the sake of completeness, here are other languages with similar > syntax features and their associated documentation > > F# > https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/pattern-matching > > Elixir https://elixir-lang.org/getting-started/case-cond-and-if.html > > Clojure https://github.com/clojure/core.match/wiki/Basic-usage > > JavaScript (ES2018?) https://github.com/tc39/proposal-pattern-matching > > Haskell https://en.wikibooks.org/wiki/Haskell/Pattern_matching > > Swifthttps://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/Patterns.html > > > > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From jsbueno at python.org.br Thu May 3 09:16:40 2018 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Thu, 3 May 2018 10:16:40 -0300 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: Message-ID: (sorry - my aboe message is about using a dictionary - the "=>" weird tokens should j=be just plain ":" - the point is that Python a?ready has syntax to do what is asked) On 3 May 2018 at 10:15, Joao S. O. Bueno wrote: > What about instead of > > number = match x: > 1 => "one" > 2 => "two" > 3 => "three" > 10 => "ten" > _ => "anything" > > number = { > 1 => "one" > 2 => "two" > 3 => "three" > 10 => "ten" > }.get(x, "anything") > > No magic syntax with blocks starting inside an assignment, just to start with. > > > On 3 May 2018 at 09:41, Robert Roskam wrote: >> Hi Everyone, >> >> Never posted in here before, so I hope that I'm not violating any particular >> procedure for intros or something. >> >> Over time, there have been various switch or match statement proposal; some >> that have gotten as far as PEPs: >> >> 2001 Nov - https://www.python.org/dev/peps/pep-0275/ >> >> 2006 Jun - https://www.python.org/dev/peps/pep-3103/ >> >> 2014 Apr - >> https://groups.google.com/d/msg/python-ideas/J5O562NKQMY/DrMHwncrmIIJ >> >> 2016 May - >> https://groups.google.com/d/msg/python-ideas/aninkpPpEAw/wCQ1IH5mAQAJ >> >> However, I don't see that the conversation ever really resolved, so I'd like >> restart the conversation on some kind of pattern matching syntax in Python. >> >> The main objections I've seen are in the following buckets: >> >> One--and Preferably Only One--Obvious Way. Basically, we have if/elif and >> that's all we need, so this is syntactical sugar bloat. I'd submit that >> there are specific cases where this kind of syntax would be the obviously >> correct way to do something >> Specific Syntax Objections. There have been several specific objections that >> usually come down to "unreadable" or "ugly", which are subjective statements >> that don't really bring any good way to continue the discussion in a >> productive manner. >> >> I cannot handle all syntax objections ahead of time, but I can handle the >> "only way" objection. At high level, pattern matching provides similar >> syntactical sugar to list comprehensions. We could argue that they are >> unnecessary since we have for loops. But more importantly, pattern matching >> is powerful for what it restricts you to. More specifically: >> >> Assignment. Many of the implementations offer the ability to immediately >> assign the value from the matching pattern. However, assignment is prevented >> in the middle of all of the patterns, which is possible in if/elif. >> No Fall Through. Once a pattern is matched, there's no way to break to try >> another branch. Prevents having to look at multiple cases to figure out how >> something resolved. If/elif can have this happen, of course, but even more >> confusing sometimes breaks will be mixed with returns or other control >> flows, which makes figuring how large if/elifs are resolved. >> Automatic Unpacking. Some implementations offer the ability unpack a >> dictionary equivalent automatically into keys or select ranges of values >> like slicing. Compared to if/elif, this is tremendously more DRY than doing >> the "does the key exists?" and then "what is that keys value?" >> Guards. Often times you can embed another check to go along with the simple >> pattern matching. Absolutely possible with if/elif, but crucially are >> implementations generally happen after the pattern check. Again, keeps code >> DRY and improves readability. >> >> I figured maybe a good way to continue the discussion is to offer a >> straw-man example syntax: >> >> # Simple pattern matching >> x = 1 >> >> number = match x: >> 1 => "one" >> 2 => "two" >> 3 => "three" >> 10 => "ten" >> _ => "anything" >> >> print(number) # one >> >> >> # Final Pattern that matches anything >> x = 3 >> >> number = match x: >> 1 => "one" >> 2 => "two" >> _ => "anything" >> >> print(number) # anything >> >> >> # Pattern matching without any match returns None >> number = match x: >> 1 => "one" >> 2 => "two" >> >> print(number) # None >> >> >> # Pattern matching with guards >> x = 'three' >> >> number = match x: >> 1 => "one" >> y if y is str => f'The string is {y}' >> _ => "anything" >> >> print(number) # The string is three >> >> >> # Pattern matching with multiple values >> x = 1 >> >> number = match x: >> 1, 2, 3, 4 => "one to four" >> _ => "anything" >> >> print(number) # one to four >> >> >> # Pattern matching with types >> x = 1. >> >> number = match x: >> x:int => f'{x} is a int' >> x:float => f'{x} is a float' >> x:str => f'{x} is a string' >> >> print(number) # x is a float >> >> >> # Supports destructuring dicts >> >> x = {'foo': 1} >> >> number = match x: >> {'foo': 1} => "foo is 1" >> _ => "anything" >> >> print(number) # foo is 1 >> >> >> # Supports binding with destructuring dicts >> >> x = {'foo': 1, 'bar': 2} >> >> number = match x: >> {'foo': y} => f'got foo {y}' >> {'bar': z} => f'got bar {z}' >> {'foo': y, 'bar': z} => f'got foo {y} and bar {z}' >> _ => "anything" >> >> print(number) # got foo 1 and bar 2 >> >> >> # Supports destructuring other types too >> >> class Point(): >> def __init__(self, x, y): >> self.x = x >> self.y = y >> >> point = Point(1,2) >> >> number = match point: >> Point(x,y) => f'point has an x of {x} and y of {y}' >> _ => "anything" >> >> print(number) # point has an x of 1 and y of 2 >> >> >> As a continued defense for this specific syntax choixe, lets see how two >> other languages with this feature handle it. I'm going to try to offer as >> nearly as possible similar examples. >> >> Scala https://docs.scala-lang.org/tour/pattern-matching.html >> >> val x: Int = 1 >> >> def makeMatch(x: Any) = x match { >> case 1 => "one" >> case 2 => "two" >> case _ => "anything" >> } >> >> val number = makeMatch(x) >> >> Rust https://doc.rust-lang.org/1.5.0/book/match.html >> >> let x = 1; >> >> let number = match x { >> 1 => "one", >> 2 => "two", >> _ => "anything", >> } >> >> And for the sake of completeness, here are other languages with similar >> syntax features and their associated documentation >> >> F# >> https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/pattern-matching >> >> Elixir https://elixir-lang.org/getting-started/case-cond-and-if.html >> >> Clojure https://github.com/clojure/core.match/wiki/Basic-usage >> >> JavaScript (ES2018?) https://github.com/tc39/proposal-pattern-matching >> >> Haskell https://en.wikibooks.org/wiki/Haskell/Pattern_matching >> >> Swifthttps://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/Patterns.html >> >> >> >> >> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> From sf at fermigier.com Thu May 3 09:17:58 2018 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Thu, 3 May 2018 15:17:58 +0200 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: Message-ID: On Thu, May 3, 2018 at 2:41 PM, Robert Roskam wrote: > > And for the sake of completeness, here are other languages with similar > syntax features and their associated documentation [...] > Still for the sake of completeness, and without any judgement from me at this point, a couple more, which are more related to Python: Coconut: http://coconut.readthedocs.io/en/master/DOCS.html#match Mochi: https://github.com/i2y/mochi#pattern-matching S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group @ Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From alberto at metapensiero.it Thu May 3 09:36:48 2018 From: alberto at metapensiero.it (Alberto Berti) Date: Thu, 03 May 2018 15:36:48 +0200 Subject: [Python-ideas] Pattern Matching Syntax References: Message-ID: <878t90oo7j.fsf@ender.lizardnet> >>>>> "St?fane" == St?fane Fermigier writes: St?fane> On Thu, May 3, 2018 at 2:41 PM, Robert Roskam St?fane> wrote: >> >> And for the sake of completeness, here are other languages with similar >> syntax features and their associated documentation [...] >> St?fane> Still for the sake of completeness, and without any judgement from me at St?fane> this point, a couple more, which are more related to Python: St?fane> Coconut: http://coconut.readthedocs.io/en/master/DOCS.html#match St?fane> Mochi: https://github.com/i2y/mochi#pattern-matching There's also macropy http://macropy3.readthedocs.io/en/latest/pattern.html -- Alberto Berti - Information Technology Consultant "gutta cavat lapidem" From j.van.dorp at deonet.nl Thu May 3 10:02:15 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Thu, 3 May 2018 16:02:15 +0200 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: <878t90oo7j.fsf@ender.lizardnet> References: <878t90oo7j.fsf@ender.lizardnet> Message-ID: > # Pattern matching with guards > x = 'three' > > number = match x: > 1 => "one" > y if y is str => f'The string is {y}' > _ => "anything" > > print(number) # The string is three I think you meant to use isinstance(y, str) ? This looks like an incomplete ternary as well, missing the else clause, so it wouldn't be a valid expression either way. And a NameError. y is never defined anywhere. Since there's no name bound to the variable being matched, would that mean a new keyword ? Also, in the rest you seem to be testing "x == {matchidentifier}", but here it suddenly looks like a boolean True would trigger a match ? And if so, would other boolean truthy values also trigger one, making the entire construct rather....limited ? It looks like you're attempting to suggest at least 3 new language syntax elements here. 4 if I count the type matching of "x:int", which you could sell as type annotation, if those hadn't been explicitly optional and ignored when actually binding the names. And almost every other example can be solved with a dict and .get(). The remainder uses a dict as match, and still work on if/elif perfectly fine. Also, I'd say "but other people do it" isn't a valid reason for implementation. There's plenty people doing stupid things, that doesn't mean it's a good idea to do it to. If they idea can't stand on it's own, it's not worth it. >From the syntax corner, it also doesn't really look like Python to me. (my apologies if I sound a bit hostile. I've attempted 3 rewrites to get that out. I only really tried to look at the syntax with what I suppose is it's intended meaning here.) 2018-05-03 15:36 GMT+02:00 Alberto Berti : >>>>>> "St?fane" == St?fane Fermigier writes: > > St?fane> On Thu, May 3, 2018 at 2:41 PM, Robert Roskam > St?fane> wrote: > >> > >> And for the sake of completeness, here are other languages with similar > >> syntax features and their associated documentation [...] > >> > St?fane> Still for the sake of completeness, and without any judgement from me at > St?fane> this point, a couple more, which are more related to Python: > > St?fane> Coconut: http://coconut.readthedocs.io/en/master/DOCS.html#match > > St?fane> Mochi: https://github.com/i2y/mochi#pattern-matching > > There's also macropy http://macropy3.readthedocs.io/en/latest/pattern.html > -- > Alberto Berti - Information Technology Consultant > > "gutta cavat lapidem" > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From raiderrobert at gmail.com Thu May 3 10:29:54 2018 From: raiderrobert at gmail.com (Robert Roskam) Date: Thu, 03 May 2018 14:29:54 +0000 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: Message-ID: Hey Joao, Thanks for providing me feedback on this idea! For the simplistic example at that you select, yes, you absolutely can do this as it stands atm. However, the examples I provided further along aren't as easily accomplished, nor is something like this: x = -1 result = match x: x:int if x > 0 => 'greater than 0' x:int, x:float if x == 0 => 'equal to 0' x:int if x < 0 => 'less than 0' print(result) # 'less than 0' Accomplishing the above with a just a dictionary would be not be the current Pythonic solution, imo, you'd do it with if/elif: x = -1 result = None if type(x) is int and x > 0: result = 'greater than 0' elif (type(x) is int or type(x) is float) and x == 0: result = 'greater than 0' elif type(x) is int and x < 0: result = 'greater than 0' print(result) # 'less than 0' So yes, Python has the syntax to handle these problems. However, my point is there's value in the kind of feature I'm proposing, and the value is stated in the above proposal. If the specific syntax choice `=>` offends the sensibilities, I simply chose mine because a good number of other languages already use `=>`. I've also considered the following: -> to then case And I think it would be good to hear if anyone has a specific preference for those others. On Thu, May 3, 2018 at 9:16 AM Joao S. O. Bueno wrote: > (sorry - my aboe message is about using a dictionary - the "=>" weird > tokens should j=be just plain ":" - > the point is that Python a?ready has syntax to do what is asked) > > On 3 May 2018 at 10:15, Joao S. O. Bueno wrote: > > What about instead of > > > > number = match x: > > 1 => "one" > > 2 => "two" > > 3 => "three" > > 10 => "ten" > > _ => "anything" > > > > number = { > > 1 => "one" > > 2 => "two" > > 3 => "three" > > 10 => "ten" > > }.get(x, "anything") > > > > No magic syntax with blocks starting inside an assignment, just to start > with. > > > > > > On 3 May 2018 at 09:41, Robert Roskam wrote: > >> Hi Everyone, > >> > >> Never posted in here before, so I hope that I'm not violating any > particular > >> procedure for intros or something. > >> > >> Over time, there have been various switch or match statement proposal; > some > >> that have gotten as far as PEPs: > >> > >> 2001 Nov - https://www.python.org/dev/peps/pep-0275/ > >> > >> 2006 Jun - https://www.python.org/dev/peps/pep-3103/ > >> > >> 2014 Apr - > >> https://groups.google.com/d/msg/python-ideas/J5O562NKQMY/DrMHwncrmIIJ > >> > >> 2016 May - > >> https://groups.google.com/d/msg/python-ideas/aninkpPpEAw/wCQ1IH5mAQAJ > >> > >> However, I don't see that the conversation ever really resolved, so I'd > like > >> restart the conversation on some kind of pattern matching syntax in > Python. > >> > >> The main objections I've seen are in the following buckets: > >> > >> One--and Preferably Only One--Obvious Way. Basically, we have if/elif > and > >> that's all we need, so this is syntactical sugar bloat. I'd submit that > >> there are specific cases where this kind of syntax would be the > obviously > >> correct way to do something > >> Specific Syntax Objections. There have been several specific objections > that > >> usually come down to "unreadable" or "ugly", which are subjective > statements > >> that don't really bring any good way to continue the discussion in a > >> productive manner. > >> > >> I cannot handle all syntax objections ahead of time, but I can handle > the > >> "only way" objection. At high level, pattern matching provides similar > >> syntactical sugar to list comprehensions. We could argue that they are > >> unnecessary since we have for loops. But more importantly, pattern > matching > >> is powerful for what it restricts you to. More specifically: > >> > >> Assignment. Many of the implementations offer the ability to immediately > >> assign the value from the matching pattern. However, assignment is > prevented > >> in the middle of all of the patterns, which is possible in if/elif. > >> No Fall Through. Once a pattern is matched, there's no way to break to > try > >> another branch. Prevents having to look at multiple cases to figure out > how > >> something resolved. If/elif can have this happen, of course, but even > more > >> confusing sometimes breaks will be mixed with returns or other control > >> flows, which makes figuring how large if/elifs are resolved. > >> Automatic Unpacking. Some implementations offer the ability unpack a > >> dictionary equivalent automatically into keys or select ranges of values > >> like slicing. Compared to if/elif, this is tremendously more DRY than > doing > >> the "does the key exists?" and then "what is that keys value?" > >> Guards. Often times you can embed another check to go along with the > simple > >> pattern matching. Absolutely possible with if/elif, but crucially are > >> implementations generally happen after the pattern check. Again, keeps > code > >> DRY and improves readability. > >> > >> I figured maybe a good way to continue the discussion is to offer a > >> straw-man example syntax: > >> > >> # Simple pattern matching > >> x = 1 > >> > >> number = match x: > >> 1 => "one" > >> 2 => "two" > >> 3 => "three" > >> 10 => "ten" > >> _ => "anything" > >> > >> print(number) # one > >> > >> > >> # Final Pattern that matches anything > >> x = 3 > >> > >> number = match x: > >> 1 => "one" > >> 2 => "two" > >> _ => "anything" > >> > >> print(number) # anything > >> > >> > >> # Pattern matching without any match returns None > >> number = match x: > >> 1 => "one" > >> 2 => "two" > >> > >> print(number) # None > >> > >> > >> # Pattern matching with guards > >> x = 'three' > >> > >> number = match x: > >> 1 => "one" > >> y if y is str => f'The string is {y}' > >> _ => "anything" > >> > >> print(number) # The string is three > >> > >> > >> # Pattern matching with multiple values > >> x = 1 > >> > >> number = match x: > >> 1, 2, 3, 4 => "one to four" > >> _ => "anything" > >> > >> print(number) # one to four > >> > >> > >> # Pattern matching with types > >> x = 1. > >> > >> number = match x: > >> x:int => f'{x} is a int' > >> x:float => f'{x} is a float' > >> x:str => f'{x} is a string' > >> > >> print(number) # x is a float > >> > >> > >> # Supports destructuring dicts > >> > >> x = {'foo': 1} > >> > >> number = match x: > >> {'foo': 1} => "foo is 1" > >> _ => "anything" > >> > >> print(number) # foo is 1 > >> > >> > >> # Supports binding with destructuring dicts > >> > >> x = {'foo': 1, 'bar': 2} > >> > >> number = match x: > >> {'foo': y} => f'got foo {y}' > >> {'bar': z} => f'got bar {z}' > >> {'foo': y, 'bar': z} => f'got foo {y} and bar {z}' > >> _ => "anything" > >> > >> print(number) # got foo 1 and bar 2 > >> > >> > >> # Supports destructuring other types too > >> > >> class Point(): > >> def __init__(self, x, y): > >> self.x = x > >> self.y = y > >> > >> point = Point(1,2) > >> > >> number = match point: > >> Point(x,y) => f'point has an x of {x} and y of {y}' > >> _ => "anything" > >> > >> print(number) # point has an x of 1 and y of 2 > >> > >> > >> As a continued defense for this specific syntax choixe, lets see how two > >> other languages with this feature handle it. I'm going to try to offer > as > >> nearly as possible similar examples. > >> > >> Scala https://docs.scala-lang.org/tour/pattern-matching.html > >> > >> val x: Int = 1 > >> > >> def makeMatch(x: Any) = x match { > >> case 1 => "one" > >> case 2 => "two" > >> case _ => "anything" > >> } > >> > >> val number = makeMatch(x) > >> > >> Rust https://doc.rust-lang.org/1.5.0/book/match.html > >> > >> let x = 1; > >> > >> let number = match x { > >> 1 => "one", > >> 2 => "two", > >> _ => "anything", > >> } > >> > >> And for the sake of completeness, here are other languages with similar > >> syntax features and their associated documentation > >> > >> F# > >> > https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/pattern-matching > >> > >> Elixir https://elixir-lang.org/getting-started/case-cond-and-if.html > >> > >> Clojure https://github.com/clojure/core.match/wiki/Basic-usage > >> > >> JavaScript (ES2018?) https://github.com/tc39/proposal-pattern-matching > >> > >> Haskell https://en.wikibooks.org/wiki/Haskell/Pattern_matching > >> > >> Swifthttps:// > developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/Patterns.html > >> > >> > >> > >> > >> > >> > >> > >> _______________________________________________ > >> Python-ideas mailing list > >> Python-ideas at python.org > >> https://mail.python.org/mailman/listinfo/python-ideas > >> Code of Conduct: http://python.org/psf/codeofconduct/ > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu May 3 13:13:21 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 May 2018 03:13:21 +1000 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: Message-ID: On Fri, May 4, 2018 at 12:29 AM, Robert Roskam wrote: > Hey Joao, > > Thanks for providing me feedback on this idea! > > For the simplistic example at that you select, yes, you absolutely can do > this as it stands atm. However, the examples I provided further along aren't > as easily accomplished, nor is something like this: > > x = -1 > > result = match x: > x:int if x > 0 => 'greater than 0' > x:int, x:float if x == 0 => 'equal to 0' > x:int if x < 0 => 'less than 0' > > print(result) # 'less than 0' > > > Accomplishing the above with a just a dictionary would be not be the current > Pythonic solution, imo, you'd do it with if/elif: Correct. So the best way to 'sell' this idea is NOT the simple examples, as they're going to be just as simple with a dictionary. I'd like to see a complete definition of the valid comparisons. Some of your examples are based on equality ("5 => ..." matches the number 5, presumably whether it's an int or a float) and have no variable, others use annotation-like syntax and types to presumably do an isinstance check, and then there's some conditions that I'm not sure about. What are all the options and how would each one be written? How do you combine different options? Is the 'if x == 0' part a modifier to a previous comparison, or is it a separate chained comparison? How does this work? ChrisA From e+python-ideas at kellett.im Thu May 3 13:18:31 2018 From: e+python-ideas at kellett.im (Ed Kellett) Date: Thu, 3 May 2018 18:18:31 +0100 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> Message-ID: <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> On 2018-05-03 15:02, Jacco van Dorp wrote: >> # Pattern matching with guards >> x = 'three' >> >> number = match x: >> 1 => "one" >> y if y is str => f'The string is {y}' >> _ => "anything" >> >> print(number) # The string is three > > I think you meant to use isinstance(y, str) ? > This looks like an incomplete ternary as well, missing the else > clause, so it wouldn't be a valid expression either way. > And a NameError. y is never defined anywhere. Since there's no name > bound to the variable being matched, would that mean a new keyword ? > Also, in the rest you seem to be testing "x == {matchidentifier}", but > here it suddenly looks like a boolean True would trigger a match ? And > if so, would other boolean truthy values also trigger one, making the > entire construct rather....limited ? > > It looks like you're attempting to suggest at least 3 new language > syntax elements here. 4 if I count the type matching of "x:int", which > you could sell as type annotation, if those hadn't been explicitly > optional and ignored when actually binding the names. It's a proposal for new syntax. I suspect that you're trying to read the left-hand side of the match cases as Python expressions. They're a distinct thing: unbound names like 'y' are an essential component of any non-trivial destructuring pattern match, as opposed to an error in an expression. I believe the intention in the example you quoted is syntax something like: ::= | "if" where the expression is a guard expression evaluated in the context of the matched pattern. IOW, it could be written like this, too: number = match x: 1 if True => "one" y if isinstance(y, str) => f'The string is {y}' _ if True => "anything" I do see a lot of room for bikeshedding around the specific spelling. I'm going to try to resist the temptation ;) > And almost every other example can be solved with a dict and .get(). > The remainder uses a dict as match, and still work on if/elif > perfectly fine. How about this? def hyperop(n, a, b): return match (n, a, b): (0, _, b) => b + 1 (1, a, 0) => a (2, _, 0) => 0 (_, _, 0) => 1 (n, a, b) => hyperop(n-1, a, hyperop(n, a, b-1)) versus: def hyperop(n, a, b): if n == 0: return b + 1 if n == 1 and b == 0: return a if n == 2 and b == 0: return 0 if b == 0: return 1 return hyperop(n-1, a, hyperop(n, a, b-1)) Of course the latter *works* (sort of: implementing tetration by recursively adding ones never really works) but it's excessively verbose, it's easy to make mistakes when writing it out, and at least in my opinion it's harder to see what it does. It also raises some small but annoying questions, like "do I use if or elif?", "do I use an if for the last case too?", "do I nest the b == 0 cases?", and, if we had used an if for the last case, what we do if we get to the end anyway. > Also, I'd say "but other people do it" isn't a valid reason for > implementation. There's plenty people doing stupid things, that > doesn't mean it's a good idea to do it to. If they idea can't stand on > it's own, it's not worth it. > > From the syntax corner, it also doesn't really look like Python to me. I agree, but I'm sure someone can come up with something prettier. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From rhodri at kynesim.co.uk Thu May 3 13:53:54 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Thu, 3 May 2018 18:53:54 +0100 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> Message-ID: <702c13e0-5e42-b36b-17f8-e0355cbcb698@kynesim.co.uk> On 03/05/18 18:18, Ed Kellett wrote: > It's a proposal for new syntax. I snipped the rest because fundamentally you have failed to explain your new syntax in any clear way. You've given examples of varying levels of complexity but failed to explain what any of them should actually do in words. It wasn't even obvious from your introduction that you were talking about match *expressions* rather than switch statements. Sorry, but this is too unclear to comment on at the moment. -- Rhodri James *-* Kynesim Ltd From e+python-ideas at kellett.im Thu May 3 14:00:12 2018 From: e+python-ideas at kellett.im (Ed Kellett) Date: Thu, 3 May 2018 19:00:12 +0100 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: <702c13e0-5e42-b36b-17f8-e0355cbcb698@kynesim.co.uk> References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <702c13e0-5e42-b36b-17f8-e0355cbcb698@kynesim.co.uk> Message-ID: <899993f1-a65a-cb0f-9d88-073c3ecff1a6@kellett.im> On 2018-05-03 18:53, Rhodri James wrote: > On 03/05/18 18:18, Ed Kellett wrote: >> It's a proposal for new syntax. > > I snipped the rest because fundamentally you have failed to explain your > new syntax in any clear way.? You've given examples of varying levels of > complexity but failed to explain what any of them should actually do in > words.? It wasn't even obvious from your introduction that you were > talking about match *expressions* rather than switch statements. > > Sorry, but this is too unclear to comment on at the moment. It's not my new syntax. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Thu May 3 14:01:55 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 May 2018 04:01:55 +1000 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> Message-ID: On Fri, May 4, 2018 at 3:18 AM, Ed Kellett wrote: > I believe the intention in the example you quoted is syntax something like: > > ::= > | "if" > > where the expression is a guard expression evaluated in the context of > the matched pattern. > > IOW, it could be written like this, too: > > number = match x: > 1 if True => "one" > y if isinstance(y, str) => f'The string is {y}' > _ if True => "anything" > > I do see a lot of room for bikeshedding around the specific spelling. > I'm going to try to resist the temptation ;) Okay, let me try to tease apart your example. 1) A literal matches anything that compares equal to that value. 2) A name matches anything at all, and binds it to that name. 2a) An underscore matches anything at all. It's just a name, and follows a common convention. 3) "if cond" modifies the prior match; if the condition evaluates as falsey, the match does not match. 4) As evidenced below, a comma-separated list of comparisons matches a tuple with as many elements, and each element must match. Ultimately, this has to be a series of conditions, so this is effectively a syntax for an elif tree as an expression. For another example, here's a way to use inequalities to pick a numeric formatting: display = match number: x if x < 1e3: f"{number}" x if x < 1e6: f"{number/1e3} thousand" x if x < 1e9: f"** {number/1e6} million **" x if x < 1e12: f"an incredible {number/1e9} billion" _: "way WAY too many" I guarantee you that people are going to ask for this to be spelled simply "< 1e3" instead of having the "x if x" part. :) > How about this? > > def hyperop(n, a, b): > return match (n, a, b): > (0, _, b) => b + 1 > (1, a, 0) => a > (2, _, 0) => 0 > (_, _, 0) => 1 > (n, a, b) => hyperop(n-1, a, hyperop(n, a, b-1)) > > versus: > > def hyperop(n, a, b): > if n == 0: > return b + 1 > if n == 1 and b == 0: > return a > if n == 2 and b == 0: > return 0 > if b == 0: > return 1 > return hyperop(n-1, a, hyperop(n, a, b-1)) I have no idea what this is actually doing, and it looks like a port of Haskell code. I'd want to rewrite it as a 'while' loop with maybe one level of recursion in it, instead of two. (Zero would be better, but I think that's not possible. Maybe?) Is this something that you do a lot of? Is the tuple (n, a, b) meaningful as a whole, or are the three values independently of interest? Not sure how this is useful without a lot more context. ChrisA From mertz at gnosis.cx Thu May 3 14:25:48 2018 From: mertz at gnosis.cx (David Mertz) Date: Thu, 03 May 2018 18:25:48 +0000 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> Message-ID: Calculating the Ackermann function as Knuth up-arrows really has little practical user. The first few values are well known, the rest won't be calculated before the heat death of the universe. On Thu, May 3, 2018, 2:02 PM Chris Angelico wrote: > On Fri, May 4, 2018 at 3:18 AM, Ed Kellett > wrote: > > I believe the intention in the example you quoted is syntax something > like: > > > > ::= > > | "if" > > > > where the expression is a guard expression evaluated in the context of > > the matched pattern. > > > > IOW, it could be written like this, too: > > > > number = match x: > > 1 if True => "one" > > y if isinstance(y, str) => f'The string is {y}' > > _ if True => "anything" > > > > I do see a lot of room for bikeshedding around the specific spelling. > > I'm going to try to resist the temptation ;) > > Okay, let me try to tease apart your example. > > 1) A literal matches anything that compares equal to that value. > 2) A name matches anything at all, and binds it to that name. > 2a) An underscore matches anything at all. It's just a name, and > follows a common convention. > 3) "if cond" modifies the prior match; if the condition evaluates as > falsey, the match does not match. > 4) As evidenced below, a comma-separated list of comparisons matches a > tuple with as many elements, and each element must match. > > Ultimately, this has to be a series of conditions, so this is > effectively a syntax for an elif tree as an expression. > > For another example, here's a way to use inequalities to pick a > numeric formatting: > > display = match number: > x if x < 1e3: f"{number}" > x if x < 1e6: f"{number/1e3} thousand" > x if x < 1e9: f"** {number/1e6} million **" > x if x < 1e12: f"an incredible {number/1e9} billion" > _: "way WAY too many" > > I guarantee you that people are going to ask for this to be spelled > simply "< 1e3" instead of having the "x if x" part. :) > > > How about this? > > > > def hyperop(n, a, b): > > return match (n, a, b): > > (0, _, b) => b + 1 > > (1, a, 0) => a > > (2, _, 0) => 0 > > (_, _, 0) => 1 > > (n, a, b) => hyperop(n-1, a, hyperop(n, a, b-1)) > > > > versus: > > > > def hyperop(n, a, b): > > if n == 0: > > return b + 1 > > if n == 1 and b == 0: > > return a > > if n == 2 and b == 0: > > return 0 > > if b == 0: > > return 1 > > return hyperop(n-1, a, hyperop(n, a, b-1)) > > I have no idea what this is actually doing, and it looks like a port > of Haskell code. I'd want to rewrite it as a 'while' loop with maybe > one level of recursion in it, instead of two. (Zero would be better, > but I think that's not possible. Maybe?) Is this something that you do > a lot of? Is the tuple (n, a, b) meaningful as a whole, or are the > three values independently of interest? > > Not sure how this is useful without a lot more context. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raiderrobert at gmail.com Thu May 3 14:36:27 2018 From: raiderrobert at gmail.com (Robert Roskam) Date: Thu, 3 May 2018 11:36:27 -0700 (PDT) Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> Message-ID: <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> Hey Chris, So I started extremely generally with my syntax, but it seems like I should provide a lot more examples of real use. Examples are hard. Here's my hastily put together example from an existing piece of production code: # Existing Production Code from datetime import timedelta, date from django.utils import timezone def convert_time_to_timedelta(unit:str, amount:int, now:date): if unit in ['days', 'hours', 'weeks']: return timedelta(**{unit: amount}) elif unit == 'months': return timedelta(days=30 * amount) elif unit == 'years': return timedelta(days=365 * amount) elif unit == 'cal_years': return now - now.replace(year=now.year - amount) # New Syntax for same problem def convert_time_to_timedelta_with_match(unit:str, amount:int, now:date): return match unit: 'days', 'hours', 'weeks' => timedelta(**{unit: amount}) 'months' => timedelta(days=30 * amount) 'years' => timedelta(days=365 * amount) 'cal_years' => now - now.replace(year=now.year - amount) On Thursday, May 3, 2018 at 2:02:54 PM UTC-4, Chris Angelico wrote: > > On Fri, May 4, 2018 at 3:18 AM, Ed Kellett > wrote: > > I believe the intention in the example you quoted is syntax something > like: > > > > ::= > > | "if" > > > > where the expression is a guard expression evaluated in the context of > > the matched pattern. > > > > IOW, it could be written like this, too: > > > > number = match x: > > 1 if True => "one" > > y if isinstance(y, str) => f'The string is {y}' > > _ if True => "anything" > > > > I do see a lot of room for bikeshedding around the specific spelling. > > I'm going to try to resist the temptation ;) > > Okay, let me try to tease apart your example. > > 1) A literal matches anything that compares equal to that value. > 2) A name matches anything at all, and binds it to that name. > 2a) An underscore matches anything at all. It's just a name, and > follows a common convention. > 3) "if cond" modifies the prior match; if the condition evaluates as > falsey, the match does not match. > 4) As evidenced below, a comma-separated list of comparisons matches a > tuple with as many elements, and each element must match. > > Ultimately, this has to be a series of conditions, so this is > effectively a syntax for an elif tree as an expression. > > For another example, here's a way to use inequalities to pick a > numeric formatting: > > display = match number: > x if x < 1e3: f"{number}" > x if x < 1e6: f"{number/1e3} thousand" > x if x < 1e9: f"** {number/1e6} million **" > x if x < 1e12: f"an incredible {number/1e9} billion" > _: "way WAY too many" > > I guarantee you that people are going to ask for this to be spelled > simply "< 1e3" instead of having the "x if x" part. :) > > > How about this? > > > > def hyperop(n, a, b): > > return match (n, a, b): > > (0, _, b) => b + 1 > > (1, a, 0) => a > > (2, _, 0) => 0 > > (_, _, 0) => 1 > > (n, a, b) => hyperop(n-1, a, hyperop(n, a, b-1)) > > > > versus: > > > > def hyperop(n, a, b): > > if n == 0: > > return b + 1 > > if n == 1 and b == 0: > > return a > > if n == 2 and b == 0: > > return 0 > > if b == 0: > > return 1 > > return hyperop(n-1, a, hyperop(n, a, b-1)) > > I have no idea what this is actually doing, and it looks like a port > of Haskell code. I'd want to rewrite it as a 'while' loop with maybe > one level of recursion in it, instead of two. (Zero would be better, > but I think that's not possible. Maybe?) Is this something that you do > a lot of? Is the tuple (n, a, b) meaningful as a whole, or are the > three values independently of interest? > > Not sure how this is useful without a lot more context. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Thu May 3 14:34:22 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 03 May 2018 18:34:22 +0000 Subject: [Python-ideas] Please consider adding partialclass to functools In-Reply-To: <20180503024104.GC9562@ando.pearwood.info> References: <20180503024104.GC9562@ando.pearwood.info> Message-ID: Done: https://bugs.python.org/issue33419 On Wed, May 2, 2018 at 10:42 PM Steven D'Aprano wrote: > On Wed, May 02, 2018 at 02:46:09PM -0700, Neil Girdhar wrote: > > Essentially, functools.partial is almost good enough for specifying some > of > > the parameters of an object's initializer, but the partial object > doesn't > > respond properly to issubclass. > [...] > > I think that ought to be an uncontroversial enough change that it can be > taken to the bug tracker. I think you should just raise a feature > enhancement request for 3.8 on the tracker, and link to this thread on > the mailing list archives in case some discussion follows. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/LWGXM1U20vs/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu May 3 14:40:02 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 4 May 2018 04:40:02 +1000 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> Message-ID: <20180503184001.GG9562@ando.pearwood.info> On Fri, May 04, 2018 at 04:01:55AM +1000, Chris Angelico wrote: > On Fri, May 4, 2018 at 3:18 AM, Ed Kellett wrote: > > def hyperop(n, a, b): > > return match (n, a, b): > > (0, _, b) => b + 1 > > (1, a, 0) => a > > (2, _, 0) => 0 > > (_, _, 0) => 1 > > (n, a, b) => hyperop(n-1, a, hyperop(n, a, b-1)) > > > > versus: > > > > def hyperop(n, a, b): > > if n == 0: > > return b + 1 > > if n == 1 and b == 0: > > return a > > if n == 2 and b == 0: > > return 0 > > if b == 0: > > return 1 > > return hyperop(n-1, a, hyperop(n, a, b-1)) > > I have no idea what this is actually doing It is the Hyperoperation function. https://en.wikipedia.org/wiki/Hyperoperation n is the parameter that specifies the "level" of the operation, and a, b are the arguments to the operation. hyperop(0, a, b) returns the successor of b (a is ignored) -- e.g. the successor of 1 is 2, the successor of 2 is 3, etc. hyperop(1, a, b) returns a+b (addition, or repeated successor); hyperop(2, a, b) returns a*b (multiplication, or repeated addition); hyperop(3, a, b) returns a**b, or a^b in the more usual mathematical notation (exponentiation, or repeated multiplication); hyperop(4, a, b) returns a^^b (tetration: repeated exponentiation; e.g. 3^^4 = 3^3^3^3 = 3^3^27 = 3^7625597484987 = a moderately large number); hyperop(5, a, b) returns a^^^b (pentation: repeated tetration, and if you thought 3^^4 was big, it's nothing compared to 3^^^4); and so forth. While this is really useful to mathematicians, in practice we're going to run out of memory before being able to calculate any of the larger values. If we converted the *entire universe* into memory, we'd still run out. So while it's a fascinating example for maths geeks, in practical terms we might as well re-write it as: def hyperop(n, a, b): raise MemoryError("you've got to be kidding") which aside from a few values close to zero, is nearly always the right thing to do :-) -- Steve From rosuav at gmail.com Thu May 3 14:53:27 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 May 2018 04:53:27 +1000 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> Message-ID: On Fri, May 4, 2018 at 4:36 AM, Robert Roskam wrote: > Hey Chris, > > So I started extremely generally with my syntax, but it seems like I should > provide a lot more examples of real use. Examples are hard. Here's my > hastily put together example from an existing piece of production code: > > > # New Syntax for same problem > > > def convert_time_to_timedelta_with_match(unit:str, amount:int, now:date): > return match unit: > 'days', 'hours', 'weeks' => timedelta(**{unit: amount}) > 'months' => timedelta(days=30 * amount) > 'years' => timedelta(days=365 * amount) > 'cal_years' => now - now.replace(year=now.year - amount) Okay, here we may have a problem. You're expecting a comma separated set of values to indicate "any of these", but elsewhere, pattern matching against a list of values is making an assertion about a tuple. So if you have any pattern matching that isn't based on equality, you're going to need to clearly stipulate how your syntax works. If you are NOT going to support tuple pattern matching (but only dict), you'll need to make this VERY clear, because people are going to expect it. ChrisA From rosuav at gmail.com Thu May 3 14:57:15 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 May 2018 04:57:15 +1000 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: <20180503184001.GG9562@ando.pearwood.info> References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <20180503184001.GG9562@ando.pearwood.info> Message-ID: On Fri, May 4, 2018 at 4:40 AM, Steven D'Aprano wrote: > On Fri, May 04, 2018 at 04:01:55AM +1000, Chris Angelico wrote: >> On Fri, May 4, 2018 at 3:18 AM, Ed Kellett wrote: > >> > def hyperop(n, a, b): >> > return match (n, a, b): >> > (0, _, b) => b + 1 >> > (1, a, 0) => a >> > (2, _, 0) => 0 >> > (_, _, 0) => 1 >> > (n, a, b) => hyperop(n-1, a, hyperop(n, a, b-1)) >> > >> > versus: >> > >> > def hyperop(n, a, b): >> > if n == 0: >> > return b + 1 >> > if n == 1 and b == 0: >> > return a >> > if n == 2 and b == 0: >> > return 0 >> > if b == 0: >> > return 1 >> > return hyperop(n-1, a, hyperop(n, a, b-1)) >> >> I have no idea what this is actually doing > > It is the Hyperoperation function. > > https://en.wikipedia.org/wiki/Hyperoperation > > n is the parameter that specifies the "level" of the operation, and a, b > are the arguments to the operation. > > hyperop(0, a, b) returns the successor of b (a is ignored) -- e.g. the > successor of 1 is 2, the successor of 2 is 3, etc. > > hyperop(1, a, b) returns a+b (addition, or repeated successor); > > hyperop(2, a, b) returns a*b (multiplication, or repeated addition); > > hyperop(3, a, b) returns a**b, or a^b in the more usual mathematical > notation (exponentiation, or repeated multiplication); > > hyperop(4, a, b) returns a^^b (tetration: repeated exponentiation; e.g. > 3^^4 = 3^3^3^3 = 3^3^27 = 3^7625597484987 = a moderately large number); > > hyperop(5, a, b) returns a^^^b (pentation: repeated tetration, and if > you thought 3^^4 was big, it's nothing compared to 3^^^4); > > and so forth. Oh. So... this is a crazy recursive way to demonstrate that a Python integer really CAN use up all your memory. Cool! > While this is really useful to mathematicians, in > practice we're going to run out of memory before being able to calculate > any of the larger values. If we converted the *entire universe* into > memory, we'd still run out. So while it's a fascinating example for > maths geeks, in practical terms we might as well re-write it as: > > def hyperop(n, a, b): > raise MemoryError("you've got to be kidding") > > which aside from a few values close to zero, is nearly always the right > thing to do :-) Got it. Well, I don't see why we can't use Python's existing primitives. def hyperop(n, a, b): if n == 0: return 1 + b if n == 1: return a + b if n == 2: return a * b if n == 3: return a ** b if n == 4: return a *** b if n == 5: return a **** b if n == 6: return a ***** b ... This is a MUCH cleaner way to write it. ChrisA From raiderrobert at gmail.com Thu May 3 15:02:47 2018 From: raiderrobert at gmail.com (Robert Roskam) Date: Thu, 3 May 2018 12:02:47 -0700 (PDT) Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> Message-ID: <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> Hey Chris, Thanks for bringing that up! Before submitting this, I actually had the syntax for multiple matches for one arm being separated by or. And frankly I just didn't like how that looked for more than 3 items: '1' or '2' or '3' or '4' or '5' vs '1', '2', '3', '4', '5' But you're right. The syntax should be for tuples instead. Here's my revised syntax, using a guard instead for the moment: def convert_time_to_timedelta_with_match(unit:str, amount:int, now:date): return match unit: x if x in ('days', 'hours', 'weeks') => timedelta(**{unit: amount}) 'months' => timedelta(days=30 * amount) 'years' => timedelta(days=365 * amount) 'cal_years' => now - now.replace(year=now.year - amount) On Thursday, May 3, 2018 at 2:54:24 PM UTC-4, Chris Angelico wrote: > > On Fri, May 4, 2018 at 4:36 AM, Robert Roskam > wrote: > > Hey Chris, > > > > So I started extremely generally with my syntax, but it seems like I > should > > provide a lot more examples of real use. Examples are hard. Here's my > > hastily put together example from an existing piece of production code: > > > > > > # New Syntax for same problem > > > > > > def convert_time_to_timedelta_with_match(unit:str, amount:int, > now:date): > > return match unit: > > 'days', 'hours', 'weeks' => timedelta(**{unit: amount}) > > 'months' => timedelta(days=30 * amount) > > 'years' => timedelta(days=365 * amount) > > 'cal_years' => now - now.replace(year=now.year - amount) > > Okay, here we may have a problem. You're expecting a comma separated > set of values to indicate "any of these", but elsewhere, pattern > matching against a list of values is making an assertion about a > tuple. So if you have any pattern matching that isn't based on > equality, you're going to need to clearly stipulate how your syntax > works. > > If you are NOT going to support tuple pattern matching (but only > dict), you'll need to make this VERY clear, because people are going > to expect it. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu May 3 15:17:37 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 May 2018 05:17:37 +1000 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> Message-ID: On Fri, May 4, 2018 at 5:02 AM, Robert Roskam wrote: > Hey Chris, > > Thanks for bringing that up! Before submitting this, I actually had the > syntax for multiple matches for one arm being separated by or. And frankly I > just didn't like how that looked for more than 3 items: > > '1' or '2' or '3' or '4' or '5' vs '1', '2', '3', '4', '5' > > But you're right. The syntax should be for tuples instead. Agreed. > Here's my revised syntax, using a guard instead for the moment: > > def convert_time_to_timedelta_with_match(unit:str, amount:int, now:date): > return match unit: > x if x in ('days', 'hours', 'weeks') => timedelta(**{unit: amount}) > 'months' => timedelta(days=30 * amount) > 'years' => timedelta(days=365 * amount) > 'cal_years' => now - now.replace(year=now.year - amount) And then this comes down to the same as all the other comparisons - the "x if x" gets duplicated. So maybe it would be best to describe this thus: match : | ( ) => If it's just an expression, it's equivalent to a comp_op of '=='. The result of evaluating the match expression is then used as the left operand for ALL the comparisons. So you could write your example as: return match unit: in ('days', 'hours', 'weeks') => timedelta(**{unit: amount}) 'months' => timedelta(days=30 * amount) 'years' => timedelta(days=365 * amount) 'cal_years' => now - now.replace(year=now.year - amount) Then there's room to expand that to a comma-separated list of values, which would pattern-match a tuple. ChrisA From e+python-ideas at kellett.im Thu May 3 16:04:40 2018 From: e+python-ideas at kellett.im (Ed Kellett) Date: Thu, 3 May 2018 21:04:40 +0100 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <20180503184001.GG9562@ando.pearwood.info> Message-ID: <38b1c699-7df1-6e3c-db90-9931d1a0ce11@kellett.im> On 2018-05-03 19:57, Chris Angelico wrote: > Got it. Well, I don't see why we can't use Python's existing primitives. > > def hyperop(n, a, b): > if n == 0: return 1 + b > if n == 1: return a + b > if n == 2: return a * b > if n == 3: return a ** b > if n == 4: return a *** b > if n == 5: return a **** b > if n == 6: return a ***** b > ... Well, it'd be infinitely long, but I suppose I'd have to concede that that's in line with the general practicality level of the example. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From mertz at gnosis.cx Thu May 3 16:13:40 2018 From: mertz at gnosis.cx (David Mertz) Date: Thu, 03 May 2018 20:13:40 +0000 Subject: [Python-ideas] Fwd: Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> Message-ID: That's buggy code in either version. A month is not necessarily 30 days, and a year is not necessarily 365 days. An example without such painful bugs might be more compelling... It also likely makes the use case less obvious for the bug-free version. On Thu, May 3, 2018, 2:37 PM Robert Roskam wrote: > Hey Chris, > > So I started extremely generally with my syntax, but it seems like I > should provide a lot more examples of real use. Examples are hard. Here's > my hastily put together example from an existing piece of production code: > > > # Existing Production Code > from datetime import timedelta, date > from django.utils import timezone > > > def convert_time_to_timedelta(unit:str, amount:int, now:date): > if unit in ['days', 'hours', 'weeks']: > return timedelta(**{unit: amount}) > elif unit == 'months': > return timedelta(days=30 * amount) > elif unit == 'years': > return timedelta(days=365 * amount) > elif unit == 'cal_years': > return now - now.replace(year=now.year - amount) > > > > > # New Syntax for same problem > > > def convert_time_to_timedelta_with_match(unit:str, amount:int, now:date): > return match unit: > 'days', 'hours', 'weeks' => timedelta(**{unit: amount}) > 'months' => timedelta(days=30 * amount) > 'years' => timedelta(days=365 * amount) > 'cal_years' => now - now.replace(year=now.year - amount) > > > > > > > > On Thursday, May 3, 2018 at 2:02:54 PM UTC-4, Chris Angelico wrote: >> >> On Fri, May 4, 2018 at 3:18 AM, Ed Kellett >> wrote: >> > I believe the intention in the example you quoted is syntax something >> like: >> > >> > ::= >> > | "if" >> > >> > where the expression is a guard expression evaluated in the context of >> > the matched pattern. >> > >> > IOW, it could be written like this, too: >> > >> > number = match x: >> > 1 if True => "one" >> > y if isinstance(y, str) => f'The string is {y}' >> > _ if True => "anything" >> > >> > I do see a lot of room for bikeshedding around the specific spelling. >> > I'm going to try to resist the temptation ;) >> >> Okay, let me try to tease apart your example. >> >> 1) A literal matches anything that compares equal to that value. >> 2) A name matches anything at all, and binds it to that name. >> 2a) An underscore matches anything at all. It's just a name, and >> follows a common convention. >> 3) "if cond" modifies the prior match; if the condition evaluates as >> falsey, the match does not match. >> 4) As evidenced below, a comma-separated list of comparisons matches a >> tuple with as many elements, and each element must match. >> >> Ultimately, this has to be a series of conditions, so this is >> effectively a syntax for an elif tree as an expression. >> >> For another example, here's a way to use inequalities to pick a >> numeric formatting: >> >> display = match number: >> x if x < 1e3: f"{number}" >> x if x < 1e6: f"{number/1e3} thousand" >> x if x < 1e9: f"** {number/1e6} million **" >> x if x < 1e12: f"an incredible {number/1e9} billion" >> _: "way WAY too many" >> >> I guarantee you that people are going to ask for this to be spelled >> simply "< 1e3" instead of having the "x if x" part. :) >> >> > How about this? >> > >> > def hyperop(n, a, b): >> > return match (n, a, b): >> > (0, _, b) => b + 1 >> > (1, a, 0) => a >> > (2, _, 0) => 0 >> > (_, _, 0) => 1 >> > (n, a, b) => hyperop(n-1, a, hyperop(n, a, b-1)) >> > >> > versus: >> > >> > def hyperop(n, a, b): >> > if n == 0: >> > return b + 1 >> > if n == 1 and b == 0: >> > return a >> > if n == 2 and b == 0: >> > return 0 >> > if b == 0: >> > return 1 >> > return hyperop(n-1, a, hyperop(n, a, b-1)) >> >> I have no idea what this is actually doing, and it looks like a port >> of Haskell code. I'd want to rewrite it as a 'while' loop with maybe >> one level of recursion in it, instead of two. (Zero would be better, >> but I think that's not possible. Maybe?) Is this something that you do >> a lot of? Is the tuple (n, a, b) meaningful as a whole, or are the >> three values independently of interest? >> >> Not sure how this is useful without a lot more context. >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python... at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Thu May 3 16:26:23 2018 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Thu, 3 May 2018 13:26:23 -0700 Subject: [Python-ideas] [offlist] Re: Add "default" kw argument to operator.itemgetter and operator.attrgetter In-Reply-To: <20180503063208.GE9562@ando.pearwood.info> References: <20180503063208.GE9562@ando.pearwood.info> Message-ID: <1A093C40-C22F-47DC-8633-CC21BECE95E5@gmail.com> > On May 2, 2018, at 11:32 PM, Steven D'Aprano wrote: > > Intended by whom? By me. I proposed itemgetter() in the first place. That rationale I gave convinced Guido and python-dev to accept it. I then wrote the code, docs, tests and have maintained it for over a decade. So, I have a pretty good idea of what it was intended for. > I think you are being too dismissive of actual use-cases requested by > actual users. Wow, I don't know what to do with this. Over the years, I've added a lot of things requested by users. I really don't like the tone you've struck and what you've implied about me as developer. That feels somewhat pushy and aggressive. Why not just give a +1 to things that are a good idea and -1 for things we're better off without -- no need for ad hominem comments about the person making the post rather than its content -- that feels somewhat disrespectful. > Default values might not have been the primary use > considered when the API was first invented, but the fact that people > keep asking for this feature should tell us that at least some people > have intended uses that are remaining unmet. When I've seen the request in the past, it always alway "it might be nice if ..." but there were no legitimate use cases presented, just toy examples. Also, I'm concerned that about increasing the complexity of itemgetter() API to serve an occasional exotic use case rather that being easy to learn and remember for the common cases. Raymond From tjreedy at udel.edu Thu May 3 17:43:45 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 3 May 2018 17:43:45 -0400 Subject: [Python-ideas] Please consider adding partialclass to functools In-Reply-To: References: <20180503024104.GC9562@ando.pearwood.info> Message-ID: On 5/3/2018 2:34 PM, Neil Girdhar wrote: > Done: https://bugs.python.org/issue33419 I agree with Steven. Add the functools 'experts' as nosy. And add links to the SO items. > On Wed, May 2, 2018 at 10:42 PM Steven D'Aprano > > wrote: > > On Wed, May 02, 2018 at 02:46:09PM -0700, Neil Girdhar wrote: > > Essentially, functools.partial is almost good enough for > specifying some of > > the parameters of an object's initializer, but the partial object > doesn't > > respond properly to issubclass. > [...] > > I think that ought to be an uncontroversial enough change that it > can be > taken to the bug tracker. I think you should just raise a feature > enhancement request for 3.8 on the tracker, and link to this thread on > the mailing list archives in case some discussion follows. -- Terry Jan Reedy From tjreedy at udel.edu Thu May 3 17:50:54 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 3 May 2018 17:50:54 -0400 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: Message-ID: On 5/3/2018 9:16 AM, Joao S. O. Bueno wrote: > (sorry - my aboe message is about using a dictionary - the "=>" weird > tokens should j=be just plain ":" - > the point is that Python a?ready has syntax to do what is asked) > > On 3 May 2018 at 10:15, Joao S. O. Bueno wrote: >> What about instead of >> >> number = match x: >> 1 => "one" >> 2 => "two" >> 3 => "three" >> 10 => "ten" >> _ => "anything" >> >> number = { >> 1 => "one" >> 2 => "two" >> 3 => "three" >> 10 => "ten" >> }.get(x, "anything") >> >> No magic syntax with blocks starting inside an assignment, just to start with. This was my initial response until I read the further examples that cannot be done with a dict. -- Terry Jan Reedy From mistersheik at gmail.com Thu May 3 18:00:45 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 03 May 2018 22:00:45 +0000 Subject: [Python-ideas] Please consider adding partialclass to functools In-Reply-To: References: <20180503024104.GC9562@ando.pearwood.info> Message-ID: Thank you. Done. On Thu, May 3, 2018 at 5:44 PM Terry Reedy wrote: > On 5/3/2018 2:34 PM, Neil Girdhar wrote: > > Done: https://bugs.python.org/issue33419 > > I agree with Steven. > Add the functools 'experts' as nosy. And add links to the SO items. > > > > > On Wed, May 2, 2018 at 10:42 PM Steven D'Aprano > > > > wrote: > > > > On Wed, May 02, 2018 at 02:46:09PM -0700, Neil Girdhar wrote: > > > Essentially, functools.partial is almost good enough for > > specifying some of > > > the parameters of an object's initializer, but the partial object > > doesn't > > > respond properly to issubclass. > > [...] > > > > I think that ought to be an uncontroversial enough change that it > > can be > > taken to the bug tracker. I think you should just raise a feature > > enhancement request for 3.8 on the tracker, and link to this thread > on > > the mailing list archives in case some discussion follows. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/LWGXM1U20vs/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu May 3 18:09:32 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 3 May 2018 18:09:32 -0400 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: Message-ID: On 5/3/2018 8:41 AM, Robert Roskam wrote: > However, I don't see that the conversation ever really resolved, so I'd > like restart the conversation on some kind of pattern matching syntax in > Python. For the cases not handled by dicts, I believe chained conditional expressions work. """ # Pattern matching with guards x = 'three' number = match x: 1 => "one" y if y is str => f'The string is {y}' _ => "anything" print(number) # The string is three """ Is handled by def f(x): return ('one' if x == 1 else f'The string is {x}' if isinstance(x, str) else 'anything') for x in 1, '2', 3: print(f(x)) I don't like the ordering, but this was Guido's decision. > 1, 2, 3, 4 => "one to four" "one to four' if x in (1,2,3,4) > x:int => f'{x} is a int' > x:float => f'{x} is a float' > x:str => f'{x} is a string' tx = type(x) f'{x} is a {tx}' if tx in (int, float, str) else None -- Terry Jan Reedy From e+python-ideas at kellett.im Thu May 3 18:34:58 2018 From: e+python-ideas at kellett.im (Ed Kellett) Date: Thu, 3 May 2018 23:34:58 +0100 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> Message-ID: On 2018-05-03 20:17, Chris Angelico wrote: >> def convert_time_to_timedelta_with_match(unit:str, amount:int, now:date): >> return match unit: >> x if x in ('days', 'hours', 'weeks') => timedelta(**{unit: amount}) >> 'months' => timedelta(days=30 * amount) >> 'years' => timedelta(days=365 * amount) >> 'cal_years' => now - now.replace(year=now.year - amount) > > And then this comes down to the same as all the other comparisons - > the "x if x" gets duplicated. So maybe it would be best to describe > this thus: > > match : > | ( ) => > > If it's just an expression, it's equivalent to a comp_op of '=='. The > result of evaluating the match expression is then used as the left > operand for ALL the comparisons. So you could write your example as: > > return match unit: > in ('days', 'hours', 'weeks') => timedelta(**{unit: amount}) > 'months' => timedelta(days=30 * amount) > 'years' => timedelta(days=365 * amount) > 'cal_years' => now - now.replace(year=now.year - amount) > > Then there's room to expand that to a comma-separated list of values, > which would pattern-match a tuple. I believe there are some problems with this approach. That case uses no destructuring at all, so the syntax that supports destructuring looks clumsy. In general, if you want to support something like: match spec: (None, const) => const (env, fmt) if env => fmt.format(**env) then I think something like the 'if' syntax is essential for guards. One could also imagine cases where it'd be useful to guard on more involved properties of things: match number_ish: x:str if x.lower().startswith('0x') => int(x[2:], 16) x:str => int(x) x => x #yolo (I know base=0 exists, but let's imagine we're implementing base=0, or something). I'm usually against naming things, and deeply resent having to name the x in [x for x in ... if ...] and similar constructs. But in this specific case, where destructuring is kind of the point, I don't think there's much value in compromising that to avoid a name. I'd suggest something like this instead: return match unit: _ in {'days', 'hours', 'weeks'} => timedelta(**{unit: amount}) ... So a match entry would be one of: - A pattern. See below - A pattern followed by "if" , e.g.: (False, x) if len(x) >= 7 - A comparison where the left-hand side is a pattern, e.g.: _ in {'days', 'hours', 'weeks'} Where a pattern is one of: - A display of patterns, e.g.: {'key': v, 'ignore': _} I think *x and **x should be allowed here. - A comma-separated list of patterns, making a tuple - A pattern enclosed in parentheses - A literal (that is not a formatted string literal, for sanity) - A name - A name with a type annotation To give a not-at-all-motivating but hopefully illustrative example: return match x: (0, _) => None (n, x) if n < 32 => ', '.join([x] * n) x:str if len(x) <= 5 => x x:str => x[:2] + '...' n:Integral < 32 => '!' * n Where: (0, 'blorp') would match the first case, yielding None (3, 'hello') would match the second case, yielding "hello, hello, hello" 'frogs' would match the third case, yielding "frogs" 'frogs!' would match the fourth case, yielding "fr..." 3 would match the fifth case, yielding '!!!' I think the matching process would mostly be intuitive, but one detail that might raise some questions: (x, x) could be allowed, and it'd make a lot of sense for that to match only (1, 1), (2, 2), ('hi', 'hi'), etc. But that'd make the _ convention less useful unless it became more than a convention. All in all, I like this idea, but I think it might be a bit too heavy to get into Python. It has the feel of requiring quite a lot of new things. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From j.van.dorp at deonet.nl Fri May 4 03:26:57 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Fri, 4 May 2018 09:26:57 +0200 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> Message-ID: Would this be valid? # Pattern matching with guards x = 'three' number = match x: 1 => "one" y if y is str => f'The string is {y}' z if z is int => f'the int is {z}' _ => "anything" print(number) # The string is three If so, why are y and z both valid here ? Is the match variable rebound to any other ? Or even to all names ? ofc, you could improve the clarity here with: number = match x as y: or any variant thereof. This way, you'd explicitely bind the variable you use for testing. If you don't, the interpreter would never know which ones to treat as rebindings and which to draw from surrounding scopes, if any. I also haven't really seen a function where this would be better than existing syntax, and the above is the only one to actually try something not possible with dicts. The type checking one could better be: x = 1 d = { int:"integer", float:"float", str:"str" } d.get(type(x), None) The production datetime code could be: def convert_time_to_timedelta_with_match(unit:str, amount:int, now:date): return { "days":timedelta(**{unit: amount}), "hours":timedelta(**{unit: amount}), "weeks":timedelta(**{unit: amount}), # why not something like subtracting two dates here to get an accurate timedelta for your specific interval ? "months":timedelta(days = 30*amount), # days = (365.25 / 12)*amount ? Would be a lot more accurate for average month length. (30.4375) "years":timedelta(days=365*amount), # days = 365.25*amount ? "cal_years":timedelta(now - now.replace(year=now.year - amount)), }.get(unit) I honestly don't see the advantages of new syntax here. Unless you hate the eager evaluation in the dict literal getting indexed, so if it's performance critical an if/else might be better. But I can't see a match statement outperforming if/else. (and if you really need faster than if/else, you should perhaps move that bit of code to C or something.) 2018-05-04 0:34 GMT+02:00 Ed Kellett : > On 2018-05-03 20:17, Chris Angelico wrote: >>> def convert_time_to_timedelta_with_match(unit:str, amount:int, now:date): >>> return match unit: >>> x if x in ('days', 'hours', 'weeks') => timedelta(**{unit: amount}) >>> 'months' => timedelta(days=30 * amount) >>> 'years' => timedelta(days=365 * amount) >>> 'cal_years' => now - now.replace(year=now.year - amount) >> >> And then this comes down to the same as all the other comparisons - >> the "x if x" gets duplicated. So maybe it would be best to describe >> this thus: >> >> match : >> | ( ) => >> >> If it's just an expression, it's equivalent to a comp_op of '=='. The >> result of evaluating the match expression is then used as the left >> operand for ALL the comparisons. So you could write your example as: >> >> return match unit: >> in ('days', 'hours', 'weeks') => timedelta(**{unit: amount}) >> 'months' => timedelta(days=30 * amount) >> 'years' => timedelta(days=365 * amount) >> 'cal_years' => now - now.replace(year=now.year - amount) >> >> Then there's room to expand that to a comma-separated list of values, >> which would pattern-match a tuple. > > I believe there are some problems with this approach. That case uses no > destructuring at all, so the syntax that supports destructuring looks > clumsy. In general, if you want to support something like: > > match spec: > (None, const) => const > (env, fmt) if env => fmt.format(**env) > > then I think something like the 'if' syntax is essential for guards. > > One could also imagine cases where it'd be useful to guard on more > involved properties of things: > > match number_ish: > x:str if x.lower().startswith('0x') => int(x[2:], 16) > x:str => int(x) > x => x #yolo > > (I know base=0 exists, but let's imagine we're implementing base=0, or > something). > > I'm usually against naming things, and deeply resent having to name the > x in [x for x in ... if ...] and similar constructs. But in this > specific case, where destructuring is kind of the point, I don't think > there's much value in compromising that to avoid a name. > > I'd suggest something like this instead: > > return match unit: > _ in {'days', 'hours', 'weeks'} => timedelta(**{unit: amount}) > ... > > So a match entry would be one of: > - A pattern. See below > - A pattern followed by "if" , e.g.: > (False, x) if len(x) >= 7 > - A comparison where the left-hand side is a pattern, e.g.: > _ in {'days', 'hours', 'weeks'} > > Where a pattern is one of: > - A display of patterns, e.g.: > {'key': v, 'ignore': _} > I think *x and **x should be allowed here. > - A comma-separated list of patterns, making a tuple > - A pattern enclosed in parentheses > - A literal (that is not a formatted string literal, for sanity) > - A name > - A name with a type annotation > > To give a not-at-all-motivating but hopefully illustrative example: > > return match x: > (0, _) => None > (n, x) if n < 32 => ', '.join([x] * n) > x:str if len(x) <= 5 => x > x:str => x[:2] + '...' > n:Integral < 32 => '!' * n > > Where: > (0, 'blorp') would match the first case, yielding None > (3, 'hello') would match the second case, yielding > "hello, hello, hello" > 'frogs' would match the third case, yielding "frogs" > 'frogs!' would match the fourth case, yielding "fr..." > 3 would match the fifth case, yielding '!!!' > > I think the matching process would mostly be intuitive, but one detail > that might raise some questions: (x, x) could be allowed, and it'd make > a lot of sense for that to match only (1, 1), (2, 2), ('hi', 'hi'), etc. > But that'd make the _ convention less useful unless it became more than > a convention. > > All in all, I like this idea, but I think it might be a bit too heavy to > get into Python. It has the feel of requiring quite a lot of new things. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From steve at pearwood.info Fri May 4 07:39:14 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 4 May 2018 21:39:14 +1000 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: <38b1c699-7df1-6e3c-db90-9931d1a0ce11@kellett.im> References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <20180503184001.GG9562@ando.pearwood.info> <38b1c699-7df1-6e3c-db90-9931d1a0ce11@kellett.im> Message-ID: <20180504113913.GH9562@ando.pearwood.info> On Thu, May 03, 2018 at 09:04:40PM +0100, Ed Kellett wrote: > On 2018-05-03 19:57, Chris Angelico wrote: > > Got it. Well, I don't see why we can't use Python's existing primitives. > > > > def hyperop(n, a, b): > > if n == 0: return 1 + b > > if n == 1: return a + b > > if n == 2: return a * b > > if n == 3: return a ** b > > if n == 4: return a *** b > > if n == 5: return a **** b > > if n == 6: return a ***** b > > ... > > Well, it'd be infinitely long, but I suppose I'd have to concede that > that's in line with the general practicality level of the example. Yes, but only countably infinite, so at least we can enumerate them all. Eventually :-) And aside from the tiny niggle that *** and higher order operators are syntax errors... Its not a bad example of the syntax, but it would be considerably more compelling a use-case if it were something less obscure and impractical. -- Steve From ncoghlan at gmail.com Fri May 4 08:06:38 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 4 May 2018 22:06:38 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses Message-ID: (Note: Guido's already told me off-list that he doesn't like the way this spelling reads, but I wanted to share it anyway since it addresses one of the recurring requests in the PEP 572 discussions for a more targeted proposal that focused specifically on the use cases that folks had agreed were reasonable potential use cases for inline assignment expressions. I'll also note that another potential concern with this specific proposal is that even though "given" wasn't used as a term in any easily discovered Python APIs back when I first wrote PEP 3150, it's now part of the Hypothesis testing API, so adopting it as a keyword now would be markedly more disruptive than it might have been historically) Recapping the use cases where the inline assignment capability received the most agreement regarding being potentially more readable than the status quo: 1. Making an "exactly one branch is executed" construct clearer than is the case for nested if statements: if m := pattern.search(data): ... elif m := other_pattern.search(data): ... else: ... 2. Replacing a loop-and-a-half construct: while m := pattern.search(remaining_data): ... 3. Sharing values between filtering clauses and result expressions in comprehensions: result = [(x, y, x/y) for x in data if (y := f(x))] The essence of the given clause concept would be to modify *these specific cases* (at least initially) to allow the condition expression to be followed by an inline assignment, of the form "given TARGET = EXPR". (Note: being able to implement such a syntactic constraint is a general consequence of using a ternary notation rather than a binary one, since it allows the construct to start with an arbitrary expression, without requiring that expression to be both the result of the operation *and* the value bound to a name - it isn't unique to the "given" keyword specifically) While the leading keyword would allow TARGET to be an arbitrary assignment target without much chance for confusion, it could also be restricted to simple names instead (as has been done for PEP 572. With that spelling, the three examples above would become: # Exactly one branch is executed here if m given m = pattern.search(data): ... elif m given m = other_pattern.search(data)): ... else: ... # This name is rebound on each trip around the loop while m given m = pattern.search(remaining_data): ... # "f(x)" is only evaluated once on each iteration result = [(x, y, x/y) for x in data if y given y = f(x)] Constraining the syntax that way (at least initially) would avoid poking into any dark corners of Python's current scoping and expression execution ordering semantics, while still leaving the door open to later making "result given NAME = expr" a general purpose ternary operator that returns the LHS, while binding the RHS to the given name as a side effect. Using a new keyword (rather than a symbol) would make the new construct easier to identify and search for, but also comes with all the downsides of introducing a new keyword. (Hence the not-entirely-uncommon suggestion of using "with" for a purpose along these lines, which runs into a different set of problems related to trying to use "with" for two distinct and entirely unrelated purposes). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri May 4 08:11:23 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 4 May 2018 22:11:23 +1000 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> Message-ID: <20180504121122.GI9562@ando.pearwood.info> On Thu, May 03, 2018 at 11:36:27AM -0700, Robert Roskam wrote: > So I started extremely generally with my syntax, but it seems like I should > provide a lot more examples of real use. Yes, real-life examples will be far more compelling and useful than made up examples and pseudo-code. Also, I think that you should delay talking about syntax until you have explained in plain English what pattern matching does, how it differs from a switch/case statement (in languages that have them) and why it is better than the two major existing idioms in Python: - chained if...elif - dict dispatch. I'll make a start, and you can correct me if I get any of it wrong. (1) Pattern matching takes a value, and compares it to a series of *patterns* until the first match, at which point it returns a specified value, skipping the rest of the patterns. (2) Patterns typically are single values, and the match is by equality, although other kinds of patterns are available as well. (3) Unlike a case/switch statement, there's no implication that the compiler could optimise the order of look-ups; it is purely top to bottom. (4) Unlike if...elif, each branch is limited to a single expression, not a block. That's a feature: a match expression takes an input, and returns a value, and typically we don't have to worry about it having side-effects. So it is intentionally less general than a chain of if...elif blocks. (5) We could think of it as somewhat analogous to a case/switch statement, a dict lookup, or if...elif, only better. (Why is it better?) Here is a list of patterns I would hope to support, off the top of my head: * match by equality; * match by arbitrary predicates such as "greater than X" or "between X and Y"; * match by string prefix, suffix, or substring; * match by type (isinstance). I think that before we start talking about syntax, we need to know what features we need syntax for. There's probably more to it, because so far it doesn't look like anything but a restricted switch statement. Over to someone else with a better idea of why pattern matching has become ubiquitous in functional programming. -- Steve From dmoisset at machinalis.com Fri May 4 09:37:05 2018 From: dmoisset at machinalis.com (Daniel Moisset) Date: Fri, 4 May 2018 14:37:05 +0100 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> Message-ID: Note that most languages that you mentioned as references are functional (so they don't have a statement/expression distinction like Python has), and those that are not, have matching statements. The only exception is Javascript, but in Javascript the distinction is not that hard given that it has the idiom (function() {stmt; stmt; stmt})() to have any statement block as an expression. And again, as I mentioned it's an outlier. Other imperative languages like C, Java, have of course switch statements which are similar Making a quick search for real code that could benefit for this, I mostly found situations where a matching *statement* would be required instead of a matching *expression*. To give you the examples I found in the stdlib for Python3.6 (I grepped for "elif" and looked for "similar" branches manually, covering the first ~20%): fnmatch.translate (match c: ... string options) telnetlib.Telnet.process_rawq (match len(self.iacseq): ... integer options) mimetypes[module __main__ body] (match opt: ... multiple str options per match) typing._remove_dups_flatten (match p: ... isinstance checks + custom condition) [this *could* be an expression with some creativity] typing.GenericMeta.__getitem__ (match self: ... single and multiple type options by identity) turtle.Shape.__init__ (match type_:... str options) turtle.TurtleScreen._color (match len(cstr): ... int options) turtle.TurtleScreen.colormode (match cmode: ... mixed type options) turtle.TNavigator.distance (match x: ... isinstance checks) [could be an expression] turtle.TNavigator.towards (match x: ... isinstance checks) [could be an expression] turtle.TPen.color (match l: ... integer options. l is set to len(args) the line before) turtle._TurtleImage._setshape (match self._type: ... str options) [could be an expression] turtle.RawTurtle.__init__ (match canvas: ... isinstance checks) turtle.RawTurtle.clone (match ttype: ... str options) [ could be an expression] turtle.RawTurtle._getshapepoly (match self._resizemode: ... str options, one with a custom condition or'ed) turtle.RawTurtle._drawturtle (match ttype: ... str options) turtle.RawTurtle.stamp (match ttype: ... str options) turtle.RawTurtle._undo (match action: ... str options) ntpath.expandvars (match c: ... str optoins) sre_parse.Subpattern.getwidth (match op: ... nonliteral int constants, actually a NamedIntConstant which subclasses int) sre_parse._class_escape (match c: ... string options with custom conditions, and inclusion+equality mixed) sre_parse._escape (match c: ... string options with custom conditions, and inclusion+equality mixed) sre_parse._parse ( match this: ... string options with in, not in, and equality) sre_parse._parse ( match char: ... string options with in, and equality) sre_parse.parse_template (match c: ... string options with in) netrc.netrc._parse (match tt: ... string options with custom conditions) netrc.netrc._parse (match tt: ... string options with custom conditions) [not a duplicate, there are two possible statements here] argparse.HelpFormatter._format_args (match action.nargs: ... str/None options) [this *could* be an expression with some creativity/transformations] argparse.ArgumentParser._get_nargs_pattern (match nargs: ... str/None options) [could be an expression] argparse.ArgumentParser._get_values (match action.nargs: ... str/None options with extra conditions) _strptime._strptime (match group_key: ... str options) datetime._wrap_strftime (match ch: ... str optoins) pickletools.optimize (match opcode,name: ... str options with reverse inclusion and equiality) json/encoder._make_iterencode(match value: ... mixed options and isinstance checks) json/encoder._make_iterencode._iterencode dict (match key: ... mixed options and isinstance checks) json/encoder._make_iterencode._iterencode dict (match value: ... mixed options and isinstance checks) json/encoder._make_iterencode._iterencode (match o: ... mixed options and isinstance checks) json/scanner.py_make_scanner._scan_once (match nextchar: ... str options) [could be turned into expression with some transformation] unittest.mock._Call.__new__ (match _len: ... int options) unittest.mock._Call.eq__ (match len_other: ... int options) (I'm not saying that all these should be match statements, only that they could be). Cases where an expression would solve the issue are somewhat uncommon (there are many state machines, including many string or argument parsers that set state depending on the option, or object builders that grow data structures). An usual situation is that some of the branches need to raise exceptions (and raise in python is a statement, not an expression). This could be workarounded making the default a raise ValueError that can be caught and reraised as soemthing else, but that would end up making the code deeper, and IMO, more complicated. Also, many of the situations where an expression could be used, are string matches where a dictionary lookup would work well anyway. My conclusions for this are: 1. It makes more sense to talk about a statement, not an expression 2. good/clear support for strings, ints and isinstancechecks is essential (other fancier things may help more circumstancially) 3. the "behaviour when there's no match" should be quite flexible. I saw many "do nothing" and many "do something" (with a large part of the latter being "raise an exception") 4. There's a pattern of re-evaluating something on each branch of an if/elif (like len(foo) or self.attr); and also common to create a dummy variable just before the if/elif. This can also be fodder for PEP-572 discussion That's what I have for now On 4 May 2018 at 08:26, Jacco van Dorp wrote: > Would this be valid? > > # Pattern matching with guards > x = 'three' > > number = match x: > 1 => "one" > y if y is str => f'The string is {y}' > z if z is int => f'the int is {z}' > _ => "anything" > > print(number) # The string is three > > If so, why are y and z both valid here ? Is the match variable rebound > to any other ? Or even to all names ? > > ofc, you could improve the clarity here with: > > number = match x as y: > > or any variant thereof. This way, you'd explicitely bind the variable > you use for testing. If you don't, the interpreter would never know > which ones to treat as rebindings and which to draw from surrounding > scopes, if any. > > I also haven't really seen a function where this would be better than > existing syntax, and the above is the only one to actually try > something not possible with dicts. The type checking one could better > be: > > x = 1 > d = { > int:"integer", > float:"float", > str:"str" > } > d.get(type(x), None) > > The production datetime code could be: > > def convert_time_to_timedelta_with_match(unit:str, amount:int, now:date): > return { > "days":timedelta(**{unit: amount}), > "hours":timedelta(**{unit: amount}), > "weeks":timedelta(**{unit: amount}), > # why not something like subtracting two dates here to get an > accurate timedelta for your specific interval ? > "months":timedelta(days = 30*amount), # days = (365.25 / > 12)*amount ? Would be a lot more accurate for average month length. > (30.4375) > "years":timedelta(days=365*amount), # days = 365.25*amount ? > "cal_years":timedelta(now - now.replace(year=now.year - amount)), > }.get(unit) > > I honestly don't see the advantages of new syntax here. > Unless you hate the eager evaluation in the dict literal getting > indexed, so if it's performance critical an if/else might be better. > But I can't see a match statement outperforming if/else. (and if you > really need faster than if/else, you should perhaps move that bit of > code to C or something.) > > 2018-05-04 0:34 GMT+02:00 Ed Kellett : > > On 2018-05-03 20:17, Chris Angelico wrote: > >>> def convert_time_to_timedelta_with_match(unit:str, amount:int, > now:date): > >>> return match unit: > >>> x if x in ('days', 'hours', 'weeks') => timedelta(**{unit: > amount}) > >>> 'months' => timedelta(days=30 * amount) > >>> 'years' => timedelta(days=365 * amount) > >>> 'cal_years' => now - now.replace(year=now.year - amount) > >> > >> And then this comes down to the same as all the other comparisons - > >> the "x if x" gets duplicated. So maybe it would be best to describe > >> this thus: > >> > >> match : > >> | ( ) => > >> > >> If it's just an expression, it's equivalent to a comp_op of '=='. The > >> result of evaluating the match expression is then used as the left > >> operand for ALL the comparisons. So you could write your example as: > >> > >> return match unit: > >> in ('days', 'hours', 'weeks') => timedelta(**{unit: amount}) > >> 'months' => timedelta(days=30 * amount) > >> 'years' => timedelta(days=365 * amount) > >> 'cal_years' => now - now.replace(year=now.year - amount) > >> > >> Then there's room to expand that to a comma-separated list of values, > >> which would pattern-match a tuple. > > > > I believe there are some problems with this approach. That case uses no > > destructuring at all, so the syntax that supports destructuring looks > > clumsy. In general, if you want to support something like: > > > > match spec: > > (None, const) => const > > (env, fmt) if env => fmt.format(**env) > > > > then I think something like the 'if' syntax is essential for guards. > > > > One could also imagine cases where it'd be useful to guard on more > > involved properties of things: > > > > match number_ish: > > x:str if x.lower().startswith('0x') => int(x[2:], 16) > > x:str => int(x) > > x => x #yolo > > > > (I know base=0 exists, but let's imagine we're implementing base=0, or > > something). > > > > I'm usually against naming things, and deeply resent having to name the > > x in [x for x in ... if ...] and similar constructs. But in this > > specific case, where destructuring is kind of the point, I don't think > > there's much value in compromising that to avoid a name. > > > > I'd suggest something like this instead: > > > > return match unit: > > _ in {'days', 'hours', 'weeks'} => timedelta(**{unit: amount}) > > ... > > > > So a match entry would be one of: > > - A pattern. See below > > - A pattern followed by "if" , e.g.: > > (False, x) if len(x) >= 7 > > - A comparison where the left-hand side is a pattern, e.g.: > > _ in {'days', 'hours', 'weeks'} > > > > Where a pattern is one of: > > - A display of patterns, e.g.: > > {'key': v, 'ignore': _} > > I think *x and **x should be allowed here. > > - A comma-separated list of patterns, making a tuple > > - A pattern enclosed in parentheses > > - A literal (that is not a formatted string literal, for sanity) > > - A name > > - A name with a type annotation > > > > To give a not-at-all-motivating but hopefully illustrative example: > > > > return match x: > > (0, _) => None > > (n, x) if n < 32 => ', '.join([x] * n) > > x:str if len(x) <= 5 => x > > x:str => x[:2] + '...' > > n:Integral < 32 => '!' * n > > > > Where: > > (0, 'blorp') would match the first case, yielding None > > (3, 'hello') would match the second case, yielding > > "hello, hello, hello" > > 'frogs' would match the third case, yielding "frogs" > > 'frogs!' would match the fourth case, yielding "fr..." > > 3 would match the fifth case, yielding '!!!' > > > > I think the matching process would mostly be intuitive, but one detail > > that might raise some questions: (x, x) could be allowed, and it'd make > > a lot of sense for that to match only (1, 1), (2, 2), ('hi', 'hi'), etc. > > But that'd make the _ convention less useful unless it became more than > > a convention. > > > > All in all, I like this idea, but I think it might be a bit too heavy to > > get into Python. It has the feel of requiring quite a lot of new things. > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Daniel F. Moisset - UK Country Manager - Machinalis Limited www.machinalis.co.uk Skype: @dmoisset T: + 44 7398 827139 1 Fore St, London, EC2Y 9DT Machinalis Limited is a company registered in England and Wales. Registered number: 10574987. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmoisset at machinalis.com Fri May 4 10:45:26 2018 From: dmoisset at machinalis.com (Daniel Moisset) Date: Fri, 4 May 2018 15:45:26 +0100 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: <20180504121122.GI9562@ando.pearwood.info> References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <20180504121122.GI9562@ando.pearwood.info> Message-ID: This email from Steve has some good questions, let me try to help organize ideas: On 4 May 2018 at 13:11, Steven D'Aprano wrote: > I'll make a start, and you can correct me if I get any of it wrong. > > (1) Pattern matching takes a value, and compares it to a series of > *patterns* until the first match, at which point it returns a specified > value, skipping the rest of the patterns. > In a general sense based in most other languages, patterns are a syntactic construct that can be "matched" with a value in runtime. The matching process has two effects at once: 1) check that the value has some specific form dictated by the pattern (which can have a yes/no result) 2) bind some assignable targets referenced in the pattern to components of the value matched. The binding is usually done only if there is a match according to (1) Python actually has some form of patterns (called "target_list" in the formal syntax) that are used in assignments, for loops, and other places. As it is mostly restricted to assign single values, or decompose iterables, we normally say "tuple unpacking" instead of "pattern matching". And there's a second type of pattern which is included in the try/except statement, which matches by subtype (and also can bind a name) As a language designer, once you have your notion on matching defined, you can choose which kind of constructs use patterns (I just mentioned left of assignemnts, between "for" and "in", etc in python). Usual constructs are multi branch statement/expression that match a single value between several patterns, and run a branch of code depending on what pattern matched (After performing the corresponding bindings). That's not the only option, you could also implement patterns in other places, like regular assuments, or the conditions of loops and conditionals [resulting in an effect similar to some of the ones being discussed in the PEP572 thread]; although this last sentence is a beyond what the OP was suggesting and a generalization of the idea. (2) Patterns typically are single values, and the match is by equality, > although other kinds of patterns are available as well. > Typical patterns in other languages include: a) single values (matched by equality) b) assignables (names, stuff like mylist[0] or self.color) which match anything and bind the value to assignables c) type patterns (a value matches if the type of the value has a certain supertype) d) structure patterns (a value matches if it has certain structure. For example, being a dict with certain keys, or an iterable of certain amount of elements). These usually are recursive, and components of the structure can be also patterns e) arbitrary boolean conditions (that can use the names bound by other parts of the pattern) Python has support for (b) and (c) in both assignment and for loops. Python supports (b) and (c) in try statements. The proposal for the OP offers expanding to most of these patterns, and implement some sort of pattern matching expression. I argued in another email that a pattern matching statement feels more adequate to Python (I'm not arguing at this point if it's a good idea, just that IF any is a good idea, it's the statement) As an example, you could have a pattern (invented syntax) like "(1, 'foo', bar, z: int)" which would positively match 4-element tuples that have 1 in its first position, foo in its second, and an int instance in the last; when matching it would bind the names "bar" and "z" to the last 2 elements in the tuple. > (3) Unlike a case/switch statement, there's no implication that the > compiler could optimise the order of look-ups; it is purely top to > bottom. > [we are talking about a multi-branch pattern matching statement now, not just "apttern matching"] In most practical cases, a compiler can do relatively simple static analysis (even in python) that could result in performance improvements. One obvious improvement is that the matched expression can be evaluated once (you can achieve the same effect always with a dummy variable assignment right before the if/elif statement). But for multiple literal string patterns (a common scenario), you can compile a string matcher that is faster than a sequence of equality comparisons (either through hashing+comparison fallback or creating some decision tree that goes through the input string once). For integers you can make lookup tables. Even an ifinstance check choosing between several branches (a not so uncommon occurrence) could be implemented by a faster operation if somewhat considered that relevant. > (4) Unlike if...elif, each branch is limited to a single expression, not > a block. That's a feature: a match expression takes an input, and > returns a value, and typically we don't have to worry about it having > side-effects. > > So it is intentionally less general than a chain of if...elif blocks. > > That's the OP proposal, yes (as I mentioned, I argued with some simple data that a feature like that is of a more limited use. Of course, I'd love to see deeper analysis with data that proves me wrong, or arguing that what I looked that is irrelevant ;-) ) > (5) We could think of it as somewhat analogous to a case/switch > statement, a dict lookup, or if...elif, only better. > > (Why is it better?) > > I'll leave the OP to argue his side here. I've mentioned some opportunities for efficiency (which are IMO secondary) and I understand that there's an argument for readability, especially when using the binding feature. > Here is a list of patterns I would hope to support, off the top of my > head: > > * match by equality; > > * match by arbitrary predicates such as "greater than X" or > "between X and Y"; > > * match by string prefix, suffix, or substring; > > * match by type (isinstance). > > I think that before we start talking about syntax, we need to know what > features we need syntax for. > > > There's probably more to it, because so far it doesn't look like > anything but a restricted switch statement. Over to someone else with a > better idea of why pattern matching has become ubiquitous in functional > programming. > a multi branch statement tends to be present but it's not necessarily ubiquitous in FP. "pattern matching" as an idea is one of those pseudo-unviersal generalizations of computing that FP language designers love. Essentially it covers with a single thing what we do in python with several different features (assignment, argument passing, conditionals, exception catching, unpacking of data structures, instance checking). It works very well when you use algebraic data types (which are like unions of namedtuples)as your primary data structure, because there are very natural patterns to decompose those. In Python, there's less value to this because well... it already has all these features so adding a unifying concept after the fact doesn't make it simpler, but more complicated. So the main argument to talk about here is if the expressivity added can be of value (if we talk about pattern matching in many places of the language, it *might*) Best, -- Daniel F. Moisset - UK Country Manager - Machinalis Limited www.machinalis.co.uk Skype: @dmoisset T: + 44 7398 827139 1 Fore St, London, EC2Y 9DT Machinalis Limited is a company registered in England and Wales. Registered number: 10574987. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri May 4 11:43:23 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 5 May 2018 01:43:23 +1000 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <20180504121122.GI9562@ando.pearwood.info> Message-ID: On Sat, May 5, 2018 at 12:45 AM, Daniel Moisset wrote: >> (3) Unlike a case/switch statement, there's no implication that the >> compiler could optimise the order of look-ups; it is purely top to >> bottom. > > > [we are talking about a multi-branch pattern matching statement now, not > just "apttern matching"] In most practical cases, a compiler can do > relatively simple static analysis (even in python) that could result in > performance improvements. One obvious improvement is that the matched > expression can be evaluated once (you can achieve the same effect always > with a dummy variable assignment right before the if/elif statement). That one isn't an optimization, but part of the specification; it is an advantage of the fact that you're writing the match expression just once. But all the rest of your optimizations aren't trustworthy. > But > for multiple literal string patterns (a common scenario), you can compile a > string matcher that is faster than a sequence of equality comparisons > (either through hashing+comparison fallback or creating some decision tree > that goes through the input string once). Only if you're doing equality checks (no substrings or anything else where it might match more than one of them). And if you're doing "pattern matching" that's nothing more than string equality comparisons, a dict is a better way to spell it. > For integers you can make lookup tables. If they're just equality checks, again, a dict is better. If they're ranges, you would have to ensure that they don't overlap (no problem if they're all literals), and then you could potentially optimize it. > Even an ifinstance check choosing between several branches (a not so > uncommon occurrence) could be implemented by a faster operation if somewhat > considered that relevant. Only if you can guarantee that no single object can be an instance of more than one of the types. Otherwise, you still have to check in some particular order. In CPython, you can guarantee that isinstance(x, int) and isinstance(x, str) can't both be true, but that's basically a CPython implementation detail, due to the way C-implemented classes work. You can't use this to dispatch based on exception types, for instance. Let's say you try to separately dispatch ValueError, ZeroDivisionError, and OSError; and then you get this: >>> class DivisionByOSError(ZeroDivisionError, OSError, ValueError): pass ... >>> raise DivisionByOSError() Traceback (most recent call last): File "", line 1, in __main__.DivisionByOSError That thing really truly is all three of those types, and you have to decide how to dispatch that. So there needs to be an order to the checks, with no optimization. ChrisA From e+python-ideas at kellett.im Fri May 4 12:07:02 2018 From: e+python-ideas at kellett.im (Ed Kellett) Date: Fri, 4 May 2018 17:07:02 +0100 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> Message-ID: On 2018-05-04 08:26, Jacco van Dorp wrote: > Would this be valid? > > # Pattern matching with guards > x = 'three' > > number = match x: > 1 => "one" > y if y is str => f'The string is {y}' > z if z is int => f'the int is {z}' > _ => "anything" > > print(number) # The string is three > > If so, why are y and z both valid here ? Is the match variable rebound > to any other ? Or even to all names ? In the match case here: match x: y if y > 3 => f'{y} is >3' # to use an example that works there are three parts: "y" is a pattern. It specifies the shape of the value to match: in this case, anything at all. Nothing is bound yet. "if" is just the word if, used as a separator, nothing to do with "if" in expressions. "y > 3" is the guard expression for the match case. Iff the pattern matches, "y > 3" is evaluated, with names appearing in the pattern taking the values they matched. It's important to note that the thing on the left-hand side is explicitly *not* a variable. It's a pattern, which can look like a variable, but it could also be a literal or a display. > ofc, you could improve the clarity here with: > > number = match x as y: > > or any variant thereof. This way, you'd explicitely bind the variable > you use for testing. If you don't, the interpreter would never know > which ones to treat as rebindings and which to draw from surrounding > scopes, if any. I don't think anything in the pattern should be drawn from surrounding scopes. > I also haven't really seen a function where this would be better than > existing syntax, and the above is the only one to actually try > something not possible with dicts. The type checking one could better > be: > > [snip] > > The production datetime code could be: > > def convert_time_to_timedelta_with_match(unit:str, amount:int, now:date): > return { > "days":timedelta(**{unit: amount}), > "hours":timedelta(**{unit: amount}), > "weeks":timedelta(**{unit: amount}), > # why not something like subtracting two dates here to get an > accurate timedelta for your specific interval ? > "months":timedelta(days = 30*amount), # days = (365.25 / > 12)*amount ? Would be a lot more accurate for average month length. > (30.4375) > "years":timedelta(days=365*amount), # days = 365.25*amount ? > "cal_years":timedelta(now - now.replace(year=now.year - amount)), > }.get(unit) Don't you think the repetition of ``timedelta(**{unit: amount})'' sort of proves OP's point? Incidentally, there's no need to use the dict trick when the unit is known statically anyway. I can't decide if that would count as more reptition or less. > I honestly don't see the advantages of new syntax here. > Unless you hate the eager evaluation in the dict literal getting > indexed, so if it's performance critical an if/else might be better. > But I can't see a match statement outperforming if/else. (and if you > really need faster than if/else, you should perhaps move that bit of > code to C or something.) It's not really about performance. It's about power. A bunch of if statements can do many things--anything, arguably--but their generality translates into repetition when dealing with many instances of this family of cases. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From guido at python.org Fri May 4 12:36:24 2018 From: guido at python.org (Guido van Rossum) Date: Fri, 4 May 2018 09:36:24 -0700 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> Message-ID: Can I recommend going slow here? This is a very interesting topic where many languages have gone before. I liked Daniel F Moisset's analysis about the choices of a language designer and his conclusion that match should be a statement. I just noticed the very similar proposal for JavaScript linked to by the OP: https://github.com/tc39/proposal-pattern-matching -- this is more relevant than what's done in e.g. F# or Swift because Python and JS are much closer. (Possibly Elixir is also relevant, it seems the JS proposal borrows from that.) A larger topic may be how to reach decisions. If I've learned one thing from PEP 572 it's that we need to adjust how we discuss and evaluate proposals. I'll think about this and start a discussion at the Language Summit about this. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri May 4 12:52:30 2018 From: guido at python.org (Guido van Rossum) Date: Fri, 4 May 2018 09:52:30 -0700 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: Thanks! Perhaps the most useful bit of this post is the clear list of use cases (a useful summary of the motivational part of PEP 572). On Fri, May 4, 2018 at 5:06 AM, Nick Coghlan wrote: > (Note: Guido's already told me off-list that he doesn't like the way this > spelling reads, but I wanted to share it anyway since it addresses one of > the recurring requests in the PEP 572 discussions for a more targeted > proposal that focused specifically on the use cases that folks had agreed > were reasonable potential use cases for inline assignment expressions. > > I'll also note that another potential concern with this specific proposal > is that even though "given" wasn't used as a term in any easily discovered > Python APIs back when I first wrote PEP 3150, it's now part of the > Hypothesis testing API, so adopting it as a keyword now would be markedly > more disruptive than it might have been historically) > > Recapping the use cases where the inline assignment capability received > the most agreement regarding being potentially more readable than the > status quo: > > 1. Making an "exactly one branch is executed" construct clearer than is > the case for nested if statements: > > if m := pattern.search(data): > ... > elif m := other_pattern.search(data): > ... > else: > ... > > 2. Replacing a loop-and-a-half construct: > > while m := pattern.search(remaining_data): > ... > > 3. Sharing values between filtering clauses and result expressions in > comprehensions: > > result = [(x, y, x/y) for x in data if (y := f(x))] > > The essence of the given clause concept would be to modify *these specific > cases* (at least initially) to allow the condition expression to be > followed by an inline assignment, of the form "given TARGET = EXPR". (Note: > being able to implement such a syntactic constraint is a general > consequence of using a ternary notation rather than a binary one, since it > allows the construct to start with an arbitrary expression, without > requiring that expression to be both the result of the operation *and* the > value bound to a name - it isn't unique to the "given" keyword specifically) > > While the leading keyword would allow TARGET to be an arbitrary assignment > target without much chance for confusion, it could also be restricted to > simple names instead (as has been done for PEP 572. > > With that spelling, the three examples above would become: > > # Exactly one branch is executed here > if m given m = pattern.search(data): > ... > elif m given m = other_pattern.search(data)): > ... > else: > ... > > # This name is rebound on each trip around the loop > while m given m = pattern.search(remaining_data): > ... > > # "f(x)" is only evaluated once on each iteration > result = [(x, y, x/y) for x in data if y given y = f(x)] > > Constraining the syntax that way (at least initially) would avoid poking > into any dark corners of Python's current scoping and expression execution > ordering semantics, while still leaving the door open to later making > "result given NAME = expr" a general purpose ternary operator that returns > the LHS, while binding the RHS to the given name as a side effect. > > Using a new keyword (rather than a symbol) would make the new construct > easier to identify and search for, but also comes with all the downsides of > introducing a new keyword. (Hence the not-entirely-uncommon suggestion of > using "with" for a purpose along these lines, which runs into a different > set of problems related to trying to use "with" for two distinct and > entirely unrelated purposes). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Fri May 4 13:35:54 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 4 May 2018 20:35:54 +0300 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: 04.05.18 15:06, Nick Coghlan ????: > Recapping the use cases where the inline assignment capability received > the most agreement regarding being potentially more readable than the > status quo: Sorry, but these examples don't look as good examples for inline assignments to me. I think that all these cases can be written better without using the inline assignment. > 1. Making an "exactly one branch is executed" construct clearer than is > the case for nested if statements: > > ??? if m := pattern.search(data): > ??????? ... > ??? elif m := other_pattern.search(data): > ??????? ... > ??? else: > ??????? ... This case can be better handled by combining patterns in a single regular expression. pattern = re.compile('(?Ppattern1)|(?Ppattern2)|...') m = pattern.search(data) if not m: # this can be omitted if the pattern is always found ... elif m.group('foo'): ... elif m.group('bar'): ... See for example gettext.py where this pattern is used. > 2. Replacing a loop-and-a-half construct: > > ??? while m := pattern.search(remaining_data): > ??????? ... This case can be better handled by re.finditer(). for m in pattern.finditer(remaining_data): ... In more complex cases it is handy to write a simple generator function and iterate its result. The large number of similar cases are covered by a two-argument iter(). > 3. Sharing values between filtering clauses and result expressions in > comprehensions: > > ??? result = [(x, y, x/y) for x in data if (y := f(x))] There are a lot of ways of writing this. PEP 572 mentions them. Different ways are used in real code depending on preferences of the author. Actually the number of these cases is pretty low in comparison with the total number of comprehensions. It is possible to express an assignment in comprehensions with the "for var in [value]" idiom, and this idiom is more powerful than PEP 572 in this case because it allows to perform an assignment before the first 'for'. But really complex comprehensions could be better written as 'for' statements with explicit adding to the collection or yielding. From tim.peters at gmail.com Fri May 4 13:48:23 2018 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 4 May 2018 12:48:23 -0500 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> Message-ID: [Guido] > Can I recommend going slow here? This is a very interesting topic where many > languages have gone before. I liked Daniel F Moisset's analysis about the > choices of a language designer and his conclusion that match should be a > statement. Just to be annoying ;-) , I liked the way he _reached_ that conclusion: by looking at real-life Python code that may have been written instead to use constructs "like this". I find such examination far more persuasive than abstract arguments or made-up examples. An observation: syntax borrowed from functional languages often fails to work well in practice when grafted onto a language that's statement-oriented - it only works well for the expression subset of the language. and even then just for when that subset is being used in a functional way (e.g., the expression `object.method(arg)` is usually used for its side effects, not for its typically-None return value). OTOH, syntax borrowed from a statement-oriented language usually fails to work at all when grafted onto an "almost everything's an expression" language. So that's an abstract argument of my own, but - according to me - should be given almost no weight unless confirmed by examining realistic code. Daniel did some of both - great! > ... > A larger topic may be how to reach decisions. If I've learned one thing from > PEP 572 it's that we need to adjust how we discuss and evaluate proposals. > I'll think about this and start a discussion at the Language Summit about > this. Python needs something akin to a dictator, who tells people how things are going to be, like it or not. But a benevolent dictator, not an evil one. And to prevent palace intrigue, they should hold that position for life. Just thinking outside the box there ;-) From tim.peters at gmail.com Fri May 4 14:11:18 2018 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 4 May 2018 13:11:18 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: [Nick Coghlan ] > ... > Using a new keyword (rather than a symbol) would make the new construct > easier to identify and search for, but also comes with all the downsides of > introducing a new keyword. That deserves more thought. I started my paying career working on a Fortran compiler, a language which, by design, had no reserved words (although plenty of keywords). The language itself (and vendor-specific extensions) never had to suffer "but adding a new keyword could break old code!" consequences. In practice that worked out very well, Yes, you _could_ write hard-to-read code using language keywords as, e.g., identifier names too, but, no, absolutely nobody did that outside of "stupid Fortran tricks" posts on Usenet ;-) It had the _intended_ effect in practice: no breakage of old code just because the language grew new constructs. It's no longer the case that Python avoided that entirely, since "async def", "async for", and "async with" statements were added _without_ making "async" a new reserved word. It may require pain in the parser, but it's often doable anyway. At this stage in Python's life, adding new _reserved_ words "should be" an extremely high bar - but adding new non-reserved keywords (like "async") should be a much lower bar. That said, I expect it's easier in general to add a non-reserved keyword introducing a statement (like "async") than one buried inside expressions ("given"). From storchaka at gmail.com Fri May 4 14:32:05 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 4 May 2018 21:32:05 +0300 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> Message-ID: 04.05.18 20:48, Tim Peters ????: > [Guido] >> Can I recommend going slow here? This is a very interesting topic where many >> languages have gone before. I liked Daniel F Moisset's analysis about the >> choices of a language designer and his conclusion that match should be a >> statement. > > Just to be annoying ;-) , I liked the way he _reached_ that > conclusion: by looking at real-life Python code that may have been > written instead to use constructs "like this". I find such > examination far more persuasive than abstract arguments or made-up > examples. I would like to see such examination for PEP 572. And for all other syntax changing ideas. I withdrew some my ideas and patches when my examinations showed that the number of cases in the stdlib that will take a benefit from rewriting using a new feature or from applying a compiler optimization is not large enough. From guido at python.org Fri May 4 14:35:17 2018 From: guido at python.org (Guido van Rossum) Date: Fri, 4 May 2018 11:35:17 -0700 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On Fri, May 4, 2018 at 11:11 AM, Tim Peters wrote: > [Nick Coghlan ] > > ... > > Using a new keyword (rather than a symbol) would make the new construct > > easier to identify and search for, but also comes with all the downsides > of > > introducing a new keyword. > > That deserves more thought. I started my paying career working on a > Fortran compiler, a language which, by design, had no reserved words > (although plenty of keywords). The language itself (and > vendor-specific extensions) never had to suffer "but adding a new > keyword could break old code!" consequences. > > In practice that worked out very well, Yes, you _could_ write > hard-to-read code using language keywords as, e.g., identifier names > too, but, no, absolutely nobody did that outside of "stupid Fortran > tricks" posts on Usenet ;-) It had the _intended_ effect in practice: > no breakage of old code just because the language grew new > constructs. > > It's no longer the case that Python avoided that entirely, since > "async def", "async for", and "async with" statements were added > _without_ making "async" a new reserved word. It may require pain in > the parser, but it's often doable anyway. At this stage in Python's > life, adding new _reserved_ words "should be" an extremely high bar - > but adding new non-reserved keywords (like "async") should be a much > lower bar. > Do note that this was a temporary solution. In 3.5 we introduced this hack. In 3.6, other uses of `async` and `await` became deprecated (though you'd have to use `python -Wall` to get a warning). In 3.7, it's a syntax error. > That said, I expect it's easier in general to add a non-reserved > keyword introducing a statement (like "async") than one buried inside > expressions ("given"). > I'd also say that the difficulty of Googling for the meaning of ":=" shouldn't be exaggerated. Currently you can search for "python operators" and get tons of sites that list all operators. I also note that Google seems to be getting smarter about non-alphabetic searches -- I just searched for "python << operator" and the first hit was https://wiki.python.org/moin/BitwiseOperators -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcidy at gmail.com Fri May 4 15:27:28 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Fri, 04 May 2018 19:27:28 +0000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On Fri, May 4, 2018, 11:35 Guido van Rossum wrote: > On Fri, May 4, 2018 at 11:11 AM, Tim Peters wrote: > >> [Nick Coghlan ] >> > ... >> > Using a new keyword (rather than a symbol) would make the new construct >> > easier to identify and search for, but also comes with all the >> downsides of >> > introducing a new keyword. >> >> That deserves more thought. I started my paying career working on a >> Fortran compiler, a language which, by design, had no reserved words >> (although plenty of keywords). The language itself (and >> vendor-specific extensions) never had to suffer "but adding a new >> keyword could break old code!" consequences. >> >> In practice that worked out very well, Yes, you _could_ write >> hard-to-read code using language keywords as, e.g., identifier names >> too, but, no, absolutely nobody did that outside of "stupid Fortran >> tricks" posts on Usenet ;-) It had the _intended_ effect in practice: >> no breakage of old code just because the language grew new >> constructs. >> >> It's no longer the case that Python avoided that entirely, since >> "async def", "async for", and "async with" statements were added >> _without_ making "async" a new reserved word. It may require pain in >> the parser, but it's often doable anyway. At this stage in Python's >> life, adding new _reserved_ words "should be" an extremely high bar - >> but adding new non-reserved keywords (like "async") should be a much >> lower bar. >> > > Do note that this was a temporary solution. In 3.5 we introduced this > hack. In 3.6, other uses of `async` and `await` became deprecated (though > you'd have to use `python -Wall` to get a warning). In 3.7, it's a syntax > error. > > >> That said, I expect it's easier in general to add a non-reserved >> keyword introducing a statement (like "async") than one buried inside >> expressions ("given"). >> > > I'd also say that the difficulty of Googling for the meaning of ":=" > shouldn't be exaggerated. Currently you can search for "python operators" > and get tons of sites that list all operators. > Without adding hits to the search algorithm, this will remain the case. Google must have clicks to rank up. Right now there is no page, nothing on a high "Google juice" page like python.org, no one searching for it, and no mass of people clicking on it. no SO questions, etc. there is a transient response for all change. uniqueness and length of search term is just a faster one. All python syntax is findable eventually due to popularity. plus a better search is "why would I use...in python" or similar. = python also doesn't bring up anything interesting that wouldn't be had because of just "python". The details are too mundane and/or technical and everyone knows already. > that being said, if := had been (theoretically) included from the beginning, would people continue to have issues with it? unlikely, but I can't know. familiarity will cure many of these issues of readability or symbolic disagreement no matter what is chosen (well, to a point). it's unfortunate that changes have to be made up front with so little information like that, so I'm not advocating anything based on this, just pointing it out. I do think post hoc assignment will cause a cognitive load, like trying to figure out which variable is the iterator, and having to keep two contexts till the end of a comp with one given statement. [f(x) + a for all a in blah given x=1] not worse than a double nested comp though. > > I also note that Google seems to be getting smarter about non-alphabetic > searches -- I just searched for "python << operator" and the first hit was > https://wiki.python.org/moin/BitwiseOperators > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri May 4 15:30:39 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 5 May 2018 05:30:39 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On Sat, May 5, 2018 at 5:27 AM, Matt Arcidy wrote: >> I'd also say that the difficulty of Googling for the meaning of ":=" >> shouldn't be exaggerated. Currently you can search for "python operators" >> and get tons of sites that list all operators. > > > Without adding hits to the search algorithm, this will remain the case. > Google must have clicks to rank up. Right now there is no page, nothing on > a high "Google juice" page like python.org, no one searching for it, and no > mass of people clicking on it. no SO questions, etc. Did you try? I searched for 'python :=' and for 'python colon equals' and got this hit each time: https://stackoverflow.com/questions/26000198/what-does-colon-equal-in-python-mean Which, incidentally, now has a response to it citing PEP 572. Good ol' Stack Overflow. ChrisA From tim.peters at gmail.com Fri May 4 16:53:34 2018 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 4 May 2018 15:53:34 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: [Tim] >> ... >> It's no longer the case that Python avoided that entirely, since >> "async def", "async for", and "async with" statements were added >> _without_ making "async" a new reserved word. It may require pain in >> the parser, but it's often doable anyway. At this stage in Python's >> life, adding new _reserved_ words "should be" an extremely high bar - >> but adding new non-reserved keywords (like "async") should be a much >> lower bar. [Guido] > Do note that this was a temporary solution. In 3.5 we introduced this hack. > In 3.6, other uses of `async` and `await` became deprecated (though you'd > have to use `python -Wall` to get a warning). In 3.7, it's a syntax error. See my "that deserves more thought" at the start, but wrt future cases then ;-) In 3.5 and 3.6, "everything just works" for everyone. In 3.7 the implementation gets churned again, to go out of its way to break the handful of code using "async" as an identifier. It's obvious who that hurts, but who does that really benefit? My experience with Fortran convinces me nobody would _actually_ be confused even if they wrote code like: async def frobnicate(async=True): if async: async with ... But nobody would actually do that. Then again, "but people _could_ do that!" barely registers with me because the nobody-actually-does-it theoretical possibilities were so much worse in Fortran, so I tend to tune that kind of argument out reflexively. For example, whitespace was also irrelevant in Fortran, and these two statements mean radically different things: D O1 0I=1 00,30 0 D O1 0I=1 00.30 0 The first is like: for I in range(100, 301): # the block ends at the next statement with label 10 The seconds is like: DO10I = 100.300 All actual Fortran code spells them like this instead: DO 10 I = 100, 300 DO10I = 100.300 The differing intents are obvious at a glance then - although, yup, to the compiler the difference is solely due to that one uses a comma where the other a period. I'm not suggesting Python go anywhere near _that_ far ;-) Just as far as considering that there's no actual harm in Fortran allowing "DO" to be a variable name too. Nobody is even tempted to think that "DO" might mean "DO loop" in, e.g., DO = 4 X = FUNC(DO) X = DO(Y) IF (DO.OR.DONTDO) GOTO 10 etc. People generally _don't_ use Fortran keywords as identifiers despite that they can, but it's a real boon for the relatively rare older code that failed to anticipate keywords added after it was written. > ... > I'd also say that the difficulty of Googling for the meaning of ":=" > shouldn't be exaggerated. Currently you can search for "python operators" > and get tons of sites that list all operators. I've noted before that people don't seem to have trouble finding the meaning of Python's "is", "and", and "or" either. But Googling for "is" (etc) on its own isn't the way to do it ;-) > I also note that Google seems to be getting smarter about non-alphabetic > searches -- I just searched for "python << operator" and the first hit was > https://wiki.python.org/moin/BitwiseOperators Ya - most arguments are crap ;-) From njs at pobox.com Fri May 4 17:11:56 2018 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 4 May 2018 14:11:56 -0700 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On Fri, May 4, 2018 at 1:53 PM, Tim Peters wrote: > [Tim] >>> ... >>> It's no longer the case that Python avoided that entirely, since >>> "async def", "async for", and "async with" statements were added >>> _without_ making "async" a new reserved word. It may require pain in >>> the parser, but it's often doable anyway. At this stage in Python's >>> life, adding new _reserved_ words "should be" an extremely high bar - >>> but adding new non-reserved keywords (like "async") should be a much >>> lower bar. > > [Guido] >> Do note that this was a temporary solution. In 3.5 we introduced this hack. >> In 3.6, other uses of `async` and `await` became deprecated (though you'd >> have to use `python -Wall` to get a warning). In 3.7, it's a syntax error. > > See my "that deserves more thought" at the start, but wrt future cases > then ;-) In 3.5 and 3.6, "everything just works" for everyone. In > 3.7 the implementation gets churned again, to go out of its way to > break the handful of code using "async" as an identifier. It's > obvious who that hurts, but who does that really benefit? > > My experience with Fortran convinces me nobody would _actually_ be > confused even if they wrote code like: > > async def frobnicate(async=True): > if async: > async with ... IIUC, Javascript has also gone all-in on contextual keywords. The realities of browser deployment mean they simply cannot have flag days or break old code, ever, meaning that contextual keywords are really the only kind they can add at all. So their async/await uses the same kind of trick that Python 3.5 did, and I believe they plan to keep it that way forever. FWIW. -n -- Nathaniel J. Smith -- https://vorpus.org From guido at python.org Fri May 4 17:20:03 2018 From: guido at python.org (Guido van Rossum) Date: Fri, 4 May 2018 14:20:03 -0700 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: I hope Python never has to go there. It's a tooling nightmare. On Fri, May 4, 2018 at 2:11 PM, Nathaniel Smith wrote: > On Fri, May 4, 2018 at 1:53 PM, Tim Peters wrote: > > [Tim] > >>> ... > >>> It's no longer the case that Python avoided that entirely, since > >>> "async def", "async for", and "async with" statements were added > >>> _without_ making "async" a new reserved word. It may require pain in > >>> the parser, but it's often doable anyway. At this stage in Python's > >>> life, adding new _reserved_ words "should be" an extremely high bar - > >>> but adding new non-reserved keywords (like "async") should be a much > >>> lower bar. > > > > [Guido] > >> Do note that this was a temporary solution. In 3.5 we introduced this > hack. > >> In 3.6, other uses of `async` and `await` became deprecated (though > you'd > >> have to use `python -Wall` to get a warning). In 3.7, it's a syntax > error. > > > > See my "that deserves more thought" at the start, but wrt future cases > > then ;-) In 3.5 and 3.6, "everything just works" for everyone. In > > 3.7 the implementation gets churned again, to go out of its way to > > break the handful of code using "async" as an identifier. It's > > obvious who that hurts, but who does that really benefit? > > > > My experience with Fortran convinces me nobody would _actually_ be > > confused even if they wrote code like: > > > > async def frobnicate(async=True): > > if async: > > async with ... > > IIUC, Javascript has also gone all-in on contextual keywords. The > realities of browser deployment mean they simply cannot have flag days > or break old code, ever, meaning that contextual keywords are really > the only kind they can add at all. So their async/await uses the same > kind of trick that Python 3.5 did, and I believe they plan to keep it > that way forever. > > FWIW. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri May 4 17:41:25 2018 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 4 May 2018 14:41:25 -0700 Subject: [Python-ideas] Auto-wrapping coroutines into Tasks Message-ID: Hi all, This is a bit of a wacky idea, but I think it might be doable and have significant benefits, so throwing it out there to see what people think. In asyncio, there are currently three kinds of calling conventions for asynchronous functions: 1) Ones which return a Future 2) Ones which return a raw coroutine object 3) Ones which return a Future, but are documented to return a coroutine object, because we want to possibly switch to doing that in the future and are hoping people won't depend on them returning a Future In practice these have slightly different semantics. For example, types (1) and (3) start executing immediately, while type (2) doesn't start executing until passed to 'await' or some function like asyncio.gather. For type (1), you can immediately call .add_done_callback: func_returning_future().add_done_callback(...) while for type (2) and (3), you have to explicitly call ensure_future first: asyncio.ensure_future(func_returning_coro()).add_done_callback(...) In practice, these distinctions are mostly irrelevant and annoying to users; the only thing you can do with a raw coroutine is pass it to ensure_future() or equivalent, and the existence of type (3) functions means that you can't even assume that functions documented as returning raw coroutines actually return raw coroutines, or that these will stay the same across versions. But it is a source of confusion, see e.g. this thread on async-sig [1], or this one [2]. It also makes it harder to evolve asyncio, since any function documented as returning a Future cannot take advantage of async/await syntax. And it's forced the creation of awkward APIs like the "coroutine hook" used in asyncio's debug mode. Other languages with async/await, like C# and Javascript, don't have these problems, because they don't have raw coroutine objects at all: when you mark a function as async, that directly converts it into a function that returns a Future (or local equivalent). So the difference between async functions and Future-returning functions is only relevant to the person writing the function; callers don't have to care, and can assume that the full Future interface is always available. I think Python did a very smart thing in *not* hard-coding Futures into the language, like C#/JS do. But, I also think it would be nice if we didn't force regular asyncio users to be aware of all these details. So here's an idea: we add a new kind of hook that coroutine runners can set. In async_function.__call__, it creates a coroutine object, and then invokes this hook, which then can wrap the coroutine into a Task (or Deferred or whatever is appropriate for the current coroutine runner). This way, from the point of view of regular asyncio users, *all* async functions become functions-returning-Futures (type 1 above): async def foo(): pass # This returns a Task running on the current loop foo() Of course, async loops need a way to get at the actual coroutine objects, so we should also provide some method on async functions to do that: foo.__corocall__() -> returns a raw coroutine object And as an optimization, we can make 'await ' invoke this, so that in regular async function -> async function calls, we don't pay the cost of setting up an unnecessary Task object: # This await foo(*args, **kwargs) # Becomes sugar for: try: _callable = foo.__corocall__ except AttributeError: # Fallback, so 'await function_returning_promise()' still works: _callable = foo _awaitable = _callable(*args, **kwargs) await _awaitable (So this hook is actually quite similar to the existing coroutine hook, except that it's specifically only invoked on bare calls, not on await-calls.) Of course, if no coroutine runner hook is registered, then the default should remain the same as now. This also means that common idioms like: loop.run_until_complete(asyncfn()) still work, because at the time asyncfn() is called, no loop is running, asyncfn() silently returns a regular coroutine object, and then run_until_complete knows how to handle that. This would also help libraries like Trio that remove Futures altogether; in Trio, the convention is that 'await asyncfn()' is simply the only way to call asyncfn, and writing a bare 'asyncfn()' is always a mistake ? but one that is currently confusing and difficult to detect because all it does is produce a warning ("coroutine was never awaited") at some potentially-distant location that depends on what the GC does. In this proposal, Trio could register a hook that raises an immediate error on bare 'asyncfn()' calls. This would also allow libraries built on Trio-or-similar to migrate a function from sync->async or async->sync with a deprecation period. Since in Trio sync functions would always use __call__, and async functions would always use __corocall__, then during a transition period one could use a custom object that defines both, and has one of them emit a DeprecationWarning. This is a problem that comes up a lot in new libraries, and currently doesn't have any decent solution. (It's actually happened to Trio itself, and created the only case where Trio has been forced to break API without a deprecation period.) The main problem I can see here is cases where you have multiple incompatible coroutine runners in the same program. In this case, problems could arise when you call asyncfn() under runner A and pass it to runner B for execution, so you end up with a B-flavored coroutine getting passed to A's wrapping hook. The kinds of cases where this might happen are: - Using the trio-asyncio library to run asyncio libraries under trio - Using Twisted's Future<->Deferred conversion layer - Using async/await to implement an ad hoc coroutine runner (e.g. a state machine) inside a program that also uses an async library I'm not sure if this is OK or not, but I think it might be? trio-asyncio's API is already designed to avoid passing coroutine objects across the boundary ? to call an asyncio function from trio you write await run_asyncio(async_fn, *args) and then run_asyncio switches into asyncio context before actually calling async_fn, so that's actually totally fine. Twisted's API does not currently work this way, but I think it's still in enough of an early provisional state that it could be fixed. And for users implementing ad hoc coroutine runners, I think this is (a) rare, (b) using generators is probably better style, since the only difference between async/await and generators is that async/await is explicitly supposed to be opaque to users and signal "this is your async library", (c) if they're writing a coroutine runner then they can set up their coroutine runner hook appropriately anyway. But there would be some costs here; the trade-off would be a significant simplification and increase in usability, because regular users could simply stop having to know about 'coroutine objects' and 'awaitables' and all that entirely, and we'd be able to take more advantage of async/await in existing libraries. What do you think? -n [1] https://mail.python.org/pipermail/async-sig/2018-May/000484.html [2] https://mail.python.org/pipermail/async-sig/2018-April/000470.html -- Nathaniel J. Smith -- https://vorpus.org From guido at python.org Fri May 4 17:58:59 2018 From: guido at python.org (Guido van Rossum) Date: Fri, 4 May 2018 14:58:59 -0700 Subject: [Python-ideas] Auto-wrapping coroutines into Tasks In-Reply-To: References: Message-ID: First, "start executing immediately" is an overstatement, right? They won't run until the caller executes a (possibly unrelated) `await`. And I'm still unclear why anyone would care, *except* in the case where they've somehow learned by observation that "real" coroutines don't start immediately and build a dependency on this in their code. (The happy eyeballs use case that was brought up here earlier today seems like it would be better off not depending on this either way, and it wouldn't be hard to do this either.) Second, when adding callbacks (if you *have to* -- if you're not a framework author you're likely doing something wrong if you find yourself adding callbacks), the right thing to do is obviously to *always* call ensure_future() first. Third, hooks like this feel like a great way to create an even bigger mess -- it implicitly teaches users that all coroutines are Futures, which will then cause disappointments when they find themselves in an environment where the hook is not enabled. Perhaps we should go the other way and wrap most ways of creating Futures in coroutines? (Though there would have to be a way for ensure_future() to *unwrap* it instead of wrapping it in a second Future.) On Fri, May 4, 2018 at 2:41 PM, Nathaniel Smith wrote: > Hi all, > > This is a bit of a wacky idea, but I think it might be doable and have > significant benefits, so throwing it out there to see what people > think. > > In asyncio, there are currently three kinds of calling conventions for > asynchronous functions: > > 1) Ones which return a Future > 2) Ones which return a raw coroutine object > 3) Ones which return a Future, but are documented to return a > coroutine object, because we want to possibly switch to doing that in > the future and are hoping people won't depend on them returning a > Future > > In practice these have slightly different semantics. For example, > types (1) and (3) start executing immediately, while type (2) doesn't > start executing until passed to 'await' or some function like > asyncio.gather. For type (1), you can immediately call > .add_done_callback: > > func_returning_future().add_done_callback(...) > > while for type (2) and (3), you have to explicitly call ensure_future > first: > > asyncio.ensure_future(func_returning_coro()).add_done_callback(...) > > In practice, these distinctions are mostly irrelevant and annoying to > users; the only thing you can do with a raw coroutine is pass it to > ensure_future() or equivalent, and the existence of type (3) functions > means that you can't even assume that functions documented as > returning raw coroutines actually return raw coroutines, or that these > will stay the same across versions. But it is a source of confusion, > see e.g. this thread on async-sig [1], or this one [2]. It also makes > it harder to evolve asyncio, since any function documented as > returning a Future cannot take advantage of async/await syntax. And > it's forced the creation of awkward APIs like the "coroutine hook" > used in asyncio's debug mode. > > Other languages with async/await, like C# and Javascript, don't have > these problems, because they don't have raw coroutine objects at all: > when you mark a function as async, that directly converts it into a > function that returns a Future (or local equivalent). So the > difference between async functions and Future-returning functions is > only relevant to the person writing the function; callers don't have > to care, and can assume that the full Future interface is always > available. > > I think Python did a very smart thing in *not* hard-coding Futures > into the language, like C#/JS do. But, I also think it would be nice > if we didn't force regular asyncio users to be aware of all these > details. > > So here's an idea: we add a new kind of hook that coroutine runners > can set. In async_function.__call__, it creates a coroutine object, > and then invokes this hook, which then can wrap the coroutine into a > Task (or Deferred or whatever is appropriate for the current coroutine > runner). This way, from the point of view of regular asyncio users, > *all* async functions become functions-returning-Futures (type 1 > above): > > async def foo(): > pass > > # This returns a Task running on the current loop > foo() > > Of course, async loops need a way to get at the actual coroutine > objects, so we should also provide some method on async functions to > do that: > > foo.__corocall__() -> returns a raw coroutine object > > And as an optimization, we can make 'await ' invoke this, so > that in regular async function -> async function calls, we don't pay > the cost of setting up an unnecessary Task object: > > # This > await foo(*args, **kwargs) > # Becomes sugar for: > try: > _callable = foo.__corocall__ > except AttributeError: > # Fallback, so 'await function_returning_promise()' still works: > _callable = foo > _awaitable = _callable(*args, **kwargs) > await _awaitable > > (So this hook is actually quite similar to the existing coroutine > hook, except that it's specifically only invoked on bare calls, not on > await-calls.) > > Of course, if no coroutine runner hook is registered, then the default > should remain the same as now. This also means that common idioms > like: > > loop.run_until_complete(asyncfn()) > > still work, because at the time asyncfn() is called, no loop is > running, asyncfn() silently returns a regular coroutine object, and > then run_until_complete knows how to handle that. > > This would also help libraries like Trio that remove Futures > altogether; in Trio, the convention is that 'await asyncfn()' is > simply the only way to call asyncfn, and writing a bare 'asyncfn()' is > always a mistake ? but one that is currently confusing and difficult > to detect because all it does is produce a warning ("coroutine was > never awaited") at some potentially-distant location that depends on > what the GC does. In this proposal, Trio could register a hook that > raises an immediate error on bare 'asyncfn()' calls. > > This would also allow libraries built on Trio-or-similar to migrate a > function from sync->async or async->sync with a deprecation period. > Since in Trio sync functions would always use __call__, and async > functions would always use __corocall__, then during a transition > period one could use a custom object that defines both, and has one of > them emit a DeprecationWarning. This is a problem that comes up a lot > in new libraries, and currently doesn't have any decent solution. > (It's actually happened to Trio itself, and created the only case > where Trio has been forced to break API without a deprecation period.) > > The main problem I can see here is cases where you have multiple > incompatible coroutine runners in the same program. In this case, > problems could arise when you call asyncfn() under runner A and pass > it to runner B for execution, so you end up with a B-flavored > coroutine getting passed to A's wrapping hook. The kinds of cases > where this might happen are: > > - Using the trio-asyncio library to run asyncio libraries under trio > - Using Twisted's Future<->Deferred conversion layer > - Using async/await to implement an ad hoc coroutine runner (e.g. a > state machine) inside a program that also uses an async library > > I'm not sure if this is OK or not, but I think it might be? > trio-asyncio's API is already designed to avoid passing coroutine > objects across the boundary ? to call an asyncio function from trio > you write > > await run_asyncio(async_fn, *args) > > and then run_asyncio switches into asyncio context before actually > calling async_fn, so that's actually totally fine. Twisted's API does > not currently work this way, but I think it's still in enough of an > early provisional state that it could be fixed. And for users > implementing ad hoc coroutine runners, I think this is (a) rare, (b) > using generators is probably better style, since the only difference > between async/await and generators is that async/await is explicitly > supposed to be opaque to users and signal "this is your async > library", (c) if they're writing a coroutine runner then they can set > up their coroutine runner hook appropriately anyway. But there would > be some costs here; the trade-off would be a significant > simplification and increase in usability, because regular users could > simply stop having to know about 'coroutine objects' and 'awaitables' > and all that entirely, and we'd be able to take more advantage of > async/await in existing libraries. > > What do you think? > > -n > > [1] https://mail.python.org/pipermail/async-sig/2018-May/000484.html > [2] https://mail.python.org/pipermail/async-sig/2018-April/000470.html > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri May 4 21:16:03 2018 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 4 May 2018 18:16:03 -0700 Subject: [Python-ideas] Auto-wrapping coroutines into Tasks In-Reply-To: References: Message-ID: On Fri, May 4, 2018 at 2:58 PM, Guido van Rossum wrote: > First, "start executing immediately" is an overstatement, right? They won't > run until the caller executes a (possibly unrelated) `await`. Well, traditional Future-returning functions often do execute some logic immediately, but right, what I meant was something like "starts executing without further intervention". I'm sure you know what I mean, but here's a concrete example to make sure it's clear to everyone else. Say we write this code: async_log("it happened!") If async_log is a traditional Future-returning function, then this line is sufficient to cause the message to be logged (eventually). If async_log is an async coroutine-returning function, then it's a no-op (except for generating a "coroutine was never awaited" warning). With this proposal, it would always work. > And I'm still > unclear why anyone would care, *except* in the case where they've somehow > learned by observation that "real" coroutines don't start immediately and > build a dependency on this in their code. (The happy eyeballs use case that > was brought up here earlier today seems like it would be better off not > depending on this either way, and it wouldn't be hard to do this either.) > > Second, when adding callbacks (if you *have to* -- if you're not a framework > author you're likely doing something wrong if you find yourself adding > callbacks), the right thing to do is obviously to *always* call > ensure_future() first. Async/await often lets you avoid working with the Future API directly, but Futures are still a major part of asyncio's public API, and so are synchronous-flavored systems like protocols/transports, where you can't use 'await'. I've been told that we need to keep that in mind when thinking about asyncio extensions ;-). And if the right thing to do is to *always* call a function, that's a good argument that the library should call it for you, right? :-) In practice I think cases like my 'async_log' example are the main place where people are likely to run into this ? there are a lot of functions out there a bare call works to run something in the background, and a lot where it doesn't. (In particular, all existing Tornado and Twisted APIs are Future-returning, not async.) > Third, hooks like this feel like a great way to create an even bigger mess > -- it implicitly teaches users that all coroutines are Futures, which will > then cause disappointments when they find themselves in an environment where > the hook is not enabled. Switching between async libraries is always going to be a pretty messy. So I guess the only case people are likely to actually encounter an unexpected hook configuration is in the period before they enter asyncio (or whatever library they're using). Like, if you've learned that async functions always return Futures, you might expect this to work: fut = some_async_fun() # Error, 'fut' is actually a coroutine b/c the hook isn't set up yet fut.add_done_callback(...) asyncio.run(fut) That's a bit of a wart. But this is something that basically never worked and can't work, and very few people are likely to run into, so while it's sad that it's a wart I don't think it's an argument against fixing the other 99% of cases? (And of course this doesn't arise for libraries like Trio, where you just never call async functions outside of async context.) > Perhaps we should go the other way and wrap most ways of creating Futures in > coroutines? (Though there would have to be a way for ensure_future() to > *unwrap* it instead of wrapping it in a second Future.) So there's a few reasons I didn't suggest going this direction: - Just in practical terms, I don't know how we could make this change. There's one place that all coroutines are created, so we at least have the technical ability to change their behavior all at once. OTOH Future-returning functions are just regular functions that happen to return a Future, so we'd have to go fix them one at a time, right? - For regular asyncio users, the Future API is pretty much a superset of the coroutine API. (The only thing you can do with an coroutine is await it or call ensure_future, and Futures allow both of those.) That means that turning coroutines into Futures is mostly backwards compatible, but turning Futures into coroutines isn't. - Similarly, having coroutine-returning functions start running without further intervention is *mostly* backwards compatible, because it's very unusual to intentionally create a coroutine object and then never actually run it (via await or ensure_future or whatever). But I suspect it is fairly common to call Future-returning functions and never await them, like in the async_log example above. This is why we have the weird "category 3" in the first place: people would like to refactor Future-returning APIs to take advantage of async/await, but right now that's a compatibility-breaking change. - Exposing raw coroutine objects to users has led to various gross-ish hacks, like the hoops that asyncio debug mode has to jump through to try to give better warnings about missing 'await'. Eliminating raw coroutine objects from public APIs would remove the need for these hacks. Making coroutine objects more prominent would have the opposite effect :-). - And also of course it wouldn't have the benefits for Trio (better error messages for forgetting an 'await', ability to transition a function between sync and async with a deprecation period), for whatever that's worth. -n -- Nathaniel J. Smith -- https://vorpus.org From alexander.belopolsky at gmail.com Fri May 4 21:56:10 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 4 May 2018 21:56:10 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On Fri, May 4, 2018 at 8:06 AM, Nick Coghlan wrote: > ... > With that spelling, the three examples above would become: > > # Exactly one branch is executed here > if m given m = pattern.search(data): > ... > elif m given m = other_pattern.search(data)): > ... > else: > ... > > # This name is rebound on each trip around the loop > while m given m = pattern.search(remaining_data): > ... > > # "f(x)" is only evaluated once on each iteration > result = [(x, y, x/y) for x in data if y given y = f(x)] I think this is a step in the right direction. I stayed away from the PEP 572 discussions because while intuitively it felt wrong, I could not formulate what exactly was wrong with the assignment expressions proposals. This proposal has finally made me realize why I did not like PEP 572. The strong expression vs. statement dichotomy is one of the key features that set Python apart from many other languages and it makes Python programs much easier to understand. Right from the title, "Assignment Expressions", PEP 572 was set to destroy the very feature that in my view is responsible for much of Python's success. Unlike PEP 572, Nick's proposal does not feel like changing the syntax of Python expressions, instead it feels like an extension to the if-, while- and for-statements syntax. (While comprehensions are expressions, for the purposes of this proposal I am willing to view them as for-statements in disguise.) From tim.peters at gmail.com Fri May 4 23:36:03 2018 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 4 May 2018 22:36:03 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: [Nick Coghlan ] > ... > The essence of the given clause concept would be to modify *these specific > cases* (at least initially) to allow the condition expression to be followed > by an inline assignment, of the form "given TARGET = EXPR". I'm not clear on what "these specific cases" are, specifically ;-) Conditions in "if", "elif", and "while" statement expressions? Restricted to one "given" clause, or can they chain? In a listcomp, is it one "given" clause per "if", or after at most one "if"? Or is an "if" even needed at all in a listcomp? For example, [(f(x)**2, f(x)**3) for x in xs] has no conditions, and [(fx := f(x))**2, fx**3) for x in xs] is one reasonable use for binding expressions. [(fx**2, fx**3) for x in xs given fx = f(x)] reads better, although it's initially surprising (to my eyes) to find fx defined "at the end". But no more surprising than the current: [(fx**2, fx**3) for x in xs for fx in [f(x)]] trick. >.... > While the leading keyword would allow TARGET to be an arbitrary assignment > target without much chance for confusion, it could also be restricted to > simple names instead (as has been done for PEP 572. The problem with complex targets in general assignment expressions is that, despite trying, I found no plausible use case for (at least) unpacking syntax. As in, e.g., x, y = func_returning_twople() if x**2 + y**2 > 9: # i.e., distance > 3, but save expensive sqrt The names can be _unpacked_ in a general assignment expression, but there appears to be no sane way then to _use_ the names in the test. This may be as good as it gets: if [(x, y := func_returning_twople()). x**2 + y**2 > 9][-1]: That reminds me of the hideous (condition and [T] or [F])[0] idiom I "invented" long ago to get the effect (in all cases) of the current T if condition else F That was intended to be goofy fun at the time, but I was appalled to see people later use it ;-) It''s certain sanest as if x**2 + y**2 > 9 given x, y = func_returning_twople(): "given" really shines there! > With that spelling, the three examples above would become: > > # Exactly one branch is executed here > if m given m = pattern.search(data): > ... > elif m given m = other_pattern.search(data)): > ... > else: Which is OK. The one-letter variable name obscures that it doesn't actually reduce _redundancy_, though. That is, in the current match = pattern.search(data) if match: it's obviously less redundant typing as: if match := pattern.search(data): In if match given match = pattern.search(data): the annoying visual redundancy (& typing) persists. > # This name is rebound on each trip around the loop > while m given m = pattern.search(remaining_data): Also fine, but also doesn't reduce redundancy. > # "f(x)" is only evaluated once on each iteration > result = [(x, y, x/y) for x in data if y given y = f(x)] As above, the potential usefulness of "given" in a listcomp doesn't really depend on having a conditional. Or on having a listcomp either, for that matter ;-) r2, r3 = fx**2, fx**3 given fx = f(x) One more, a lovely (to my eyes) binding expression simplification requiring two bindings in an `if` test, taken from real-life code I happened to write during the PEP discussion: diff = x - x_base if diff: g = gcd(diff, n) if g > 1: return g collapsed to the crisp & clear: if (diff := x - x_base) and (g := gcd(diff, n)) > 1: return g If only one trailing "given" clause can be given per `if` test expression, presumably I couldn't do that without trickery. If it's more general, if (diff given diff = x _ xbase) and g > 1 given g = gcd(diff, n): reads worse to my eyes (perhaps because of the "visual redundancy" thing again), while if diff and g > 1 given diff = x - x_base given g = gcd(diff, n): has my eyes darting all over the place, and wondering which of the trailing `given` clauses executes first. > ... From njs at pobox.com Sat May 5 00:25:26 2018 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 4 May 2018 21:25:26 -0700 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On Fri, May 4, 2018 at 6:56 PM, Alexander Belopolsky wrote: > On Fri, May 4, 2018 at 8:06 AM, Nick Coghlan wrote: >> ... >> With that spelling, the three examples above would become: >> >> # Exactly one branch is executed here >> if m given m = pattern.search(data): >> ... >> elif m given m = other_pattern.search(data)): >> ... >> else: >> ... >> >> # This name is rebound on each trip around the loop >> while m given m = pattern.search(remaining_data): >> ... >> >> # "f(x)" is only evaluated once on each iteration >> result = [(x, y, x/y) for x in data if y given y = f(x)] > > I think this is a step in the right direction. I stayed away from the > PEP 572 discussions because while intuitively it felt wrong, I could > not formulate what exactly was wrong with the assignment expressions > proposals. This proposal has finally made me realize why I did not > like PEP 572. The strong expression vs. statement dichotomy is one of > the key features that set Python apart from many other languages and > it makes Python programs much easier to understand. Right from the > title, "Assignment Expressions", PEP 572 was set to destroy the very > feature that in my view is responsible for much of Python's success. This is what makes me uncomfortable too. As Dijkstra once wrote: "our intellectual powers are rather geared to master static relations and ... our powers to visualize processes evolving in time are relatively poorly developed. For that reason we should do (as wise programmers aware of our limitations) our utmost to shorten the conceptual gap between the static program and the dynamic process, to make the correspondence between the program (spread out in text space) and the process (spread out in time) as trivial as possible." [1] Normally, Python code strongly maps *time* onto *vertical position*: one side-effect per line. Of course there is some specific order-of-operations for everything inside an individual line that the interpreter has to keep track of, but I basically never have to care about that myself. But by definition, := involves embedding side-effects within expressions, so suddenly I do have to care after all. Except... for the three cases Nick wrote above, where the side-effect occurs at the very end of the evaluation. And these also seem to be the three cases that have the most compelling use cases anyway. So restricting to just those three cases makes it much more palatable to me. (I won't comment on Nick's actual proposal, which is a bit more complicated than those examples, since it allows things like 'if m.group(1) given m = ...'.) (And on another note, I also wonder if all this pent-up desire to enrich the syntax of comprehensions means that we should add some kind of multi-line version of comprehensions, that doesn't require the awkwardness of explicitly accumulating a list or creating a nested function to yield out of. Not sure what that would look like, but people sure seem to want it.) -n [1] This is from "Go to statement considered harmful". Then a few lines later he uses a sequence of assignment statements as an example, and says that the wonderful thing about this example is that there's a 1-1 correspondence between lines and distinguishable program states, which is also uncannily apropos. -- Nathaniel J. Smith -- https://vorpus.org From tim.peters at gmail.com Sat May 5 02:23:46 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 5 May 2018 01:23:46 -0500 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> Message-ID: [Tim] >> ... I liked the way he _reached_ that conclusion: by looking at real- >> life Python code that may have been written instead to use constructs >> "like this". I find such examination far more persuasive than abstract >> arguments or made-up examples. [Serhiy Storchaka ] > I would like to see such examination for PEP 572. And for all other syntax > changing ideas. I did it myself for 572, and posted several times about what I found. It was far more productive to me than arguing (and, indeed, I sat out of the first several hundred msgs on python-ideas entirely because I spent all my time looking at code instead). Short course: I found a small win frequently, a large win rarely, but in most cases wouldn't use it. In all I expect I'd use it significantly more often than ternary "if", but far less often than augmented assignment. But that's me - everybody needs to look at their own code to apply _their_ judgment. 572 is harder than a case/switch statement to consider this way, because virtually every assignment statement binding a name could _potentially_ be changed to a binding expression instead, and there are gazillions of those. For considering case/switch additions, you can automate searches to vastly whittle down the universe of places to look at (`elif` chains, and certain nested if/else if/else if/else ... patterns). > I withdrew some my ideas and patches when my examinations showed that the > number of cases in the stdlib that will take a benefit from rewriting using > a new feature or from applying a compiler optimization is not large enough. Good! I approve :-) From storchaka at gmail.com Sat May 5 02:54:21 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 5 May 2018 09:54:21 +0300 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> Message-ID: 05.05.18 09:23, Tim Peters ????: > [Tim] >>> ... I liked the way he _reached_ that conclusion: by looking at real- >>> life Python code that may have been written instead to use constructs >>> "like this". I find such examination far more persuasive than abstract >>> arguments or made-up examples. > > [Serhiy Storchaka ] >> I would like to see such examination for PEP 572. And for all other syntax >> changing ideas. > > I did it myself for 572, and posted several times about what I found. Could you please give links to these results? It is hard to find something in hundreds of messages. From greg.ewing at canterbury.ac.nz Sat May 5 04:52:20 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 05 May 2018 20:52:20 +1200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <5AED70C4.1030008@canterbury.ac.nz> Tim Peters wrote: > "async def", "async for", and "async with" statements were added > _without_ making "async" a new reserved word. It may require pain in > the parser, but it's often doable anyway. There would be less pain if the parser generator could deal with non-reserved keywords more directly. -- Greg From robertve92 at gmail.com Sat May 5 05:22:37 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Sat, 05 May 2018 09:22:37 +0000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AED70C4.1030008@canterbury.ac.nz> References: <5AED70C4.1030008@canterbury.ac.nz> Message-ID: I agree it would be useful to have new keywords without being reserved, and we could even go with mechanism like infix operators created by user. It would allow things like [given for x in range(5) given given = x+1] or even [given for given in range(given) given given = given + 1] haha, but as other pointed out, it would be called Bad practice ^^ By the way, I still prefer "where" to "given". Le sam. 5 mai 2018 ? 10:52, Greg Ewing a ?crit : > Tim Peters wrote: > > "async def", "async for", and "async with" statements were added > > _without_ making "async" a new reserved word. It may require pain in > > the parser, but it's often doable anyway. > > There would be less pain if the parser generator could > deal with non-reserved keywords more directly. > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Sat May 5 06:24:07 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Sat, 5 May 2018 06:24:07 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AED70C4.1030008@canterbury.ac.nz> Message-ID: I think that "expr as y" was discarded too quickly. It would be a syntax completely familiar to Python programmers, and the new semantics would be obvious. That choice would also be in line with the Zen of Python. The special cases that may arise over "except" and "with" can be worked out and documented. Cheers, On Sat, May 5, 2018 at 5:22 AM, Robert Vanden Eynde wrote: > I agree it would be useful to have new keywords without being reserved, > and we could even go with mechanism like infix operators created by user. > > It would allow things like [given for x in range(5) given given = x+1] or > even [given for given in range(given) given given = given + 1] haha, but as > other pointed out, it would be called Bad practice ^^ > > By the way, I still prefer "where" to "given". > > Le sam. 5 mai 2018 ? 10:52, Greg Ewing a > ?crit : > >> Tim Peters wrote: >> > "async def", "async for", and "async with" statements were added >> > _without_ making "async" a new reserved word. It may require pain in >> > the parser, but it's often doable anyway. >> >> There would be less pain if the parser generator could >> deal with non-reserved keywords more directly. >> >> -- >> Greg >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From Eloi.Gaudry at fft.be Sat May 5 04:04:45 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Sat, 5 May 2018 08:04:45 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active Message-ID: Hi folks, I intend to add a runtime assertion feature in python. Before submitting a PEP, I am sending a draft to this mailing list to discuss whether it would make sense (above this message). I have actually been using it for the last 2 years and it has proven to be robust and has achieved its primary goal. Briefly, the idea is to add a new assert that can be switch on/off depending on some variable/mechanism at runtime. The whole point of this assert is that it should not bring any overhead when off, i.e. by avoiding evaluating the expression enclosed in the runtime assert. It thus relies on Python grammar. Thanks for your feedback. Eloi Abstract This PEP aims at offering a runtime assert functionnality, extending the compiletime assert already available. Rationale There is no runtime assert relying on python grammar available. For diagnostics and/or debugging reasons, it would be valuable to add such a feature. A runtime assert makes sense when extra checks would be needed in a production environment (where non-debug builds are used). By extending the current python grammar, it would be possible to limit the overhead induces by those runtime asserts when running in a non "assertive" mode (-ed). The idea here is to avoid evaluating the expression when runtime assertion is not active. A brief example why avoiding evaluating the expression is needed to avoid any overhead in case the runtime assert should be ignored. :: runtime_assert( 999 in { i:None for i in range( 10000000 ) } ) Usage :: runtime_assert( expr ) #would result in if expr and runtime_assert_active: print RuntimeAssertionError() Implementation details There is already an implementation available, robust enough for production. The implementation is mainly grammar based, and thus the parser and the grammar needs to be updated: * Python/graminit.c * Python/ast.c * Python/symtable.c * Python/compile.c * Python/Python-ast.c * Python/sysmodule.c * Modules/parsermodule.c * Include/Python-ast.h * Include/graminit.h * Include/pydebug.h * Grammar/Grammar * Parser/Python.asdl * Lib/lib2to3/Grammar.txt * Lib/compiler/ast.py * Lib/compiler/pycodegen.py * Lib/compiler/transformer.py * Lib/symbol.py * Modules/parsermodule.c References [1] PEP 306, How to Change Python's Grammar (http://www.python.org/dev/peps/pep-0306) Copyright This document has been placed in the public domain. -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sat May 5 08:05:16 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 5 May 2018 15:05:16 +0300 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: Message-ID: 05.05.18 11:04, Eloi Gaudry ????: > Briefly, the idea is to add a new assert that can be switch on/off > depending on some variable/mechanism at runtime. The whole point of > this assert is that it should not bring any overhead when off, i.e. > by avoiding evaluating the expression enclosed in the runtime > assert. It thus relies on Python grammar. You should have an overhead for checking if it is switched on/off, isn't? And this overhead is virtually the same as testing the value of the global boolean variable. > runtime_assert( expr ) > > #would result in if expr and runtime_assert_active: > > print RuntimeAssertionError() How this will be different from if debug and not expr: print(RuntimeAssertionError()) ? From Eloi.Gaudry at fft.be Sat May 5 08:54:25 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Sat, 5 May 2018 12:54:25 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: Message-ID: I meant avoiding the overhead of the expression evaluation enclosed in the assert itself, not the active/disabled state (that comes at virtually no cost). ex: >>> runtime_assert( { i:None for i in range( 10000000 ) } ) By using the syntax you describe ('boolean and not expr'), you loose all the benefit of the Python grammar: - self-contained - able to catch semantic/syntax bugs ex: >>> f=1 >>> runtime_assert( f==1 ) >>> runtime_assert( f=1 ) File "", line 1 runtime_assert( f=1 ) ^ SyntaxError: invalid syntax -----Original Message----- From: Python-ideas On Behalf Of Serhiy Storchaka Sent: Saturday, May 5, 2018 2:05 PM To: python-ideas at python.org Subject: Re: [Python-ideas] Runtime assertion with no overhead when not active 05.05.18 11:04, Eloi Gaudry ????: > Briefly, the idea is to add a new assert that can be switch on/off > depending on some variable/mechanism at runtime. The whole point of > this assert is that it should not bring any overhead when off, i.e. > by avoiding evaluating the expression enclosed in the runtime > assert. It thus relies on Python grammar. You should have an overhead for checking if it is switched on/off, isn't? And this overhead is virtually the same as testing the value of the global boolean variable. > runtime_assert( expr ) > > #would result in if expr and runtime_assert_active: > > print RuntimeAssertionError() How this will be different from if debug and not expr: print(RuntimeAssertionError()) ? _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ From storchaka at gmail.com Sat May 5 09:09:39 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 5 May 2018 16:09:39 +0300 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: Message-ID: 05.05.18 15:54, Eloi Gaudry ????: > I meant avoiding the overhead of the expression evaluation enclosed in the assert itself, not the active/disabled state (that comes at virtually no cost). > ex: >>>> runtime_assert( { i:None for i in range( 10000000 ) } ) > > By using the syntax you describe ('boolean and not expr'), you loose all the benefit of the Python grammar: > - self-contained Could you please explain what you mean? > - able to catch semantic/syntax bugs > > ex: >>>> f=1 >>>> runtime_assert( f==1 ) >>>> runtime_assert( f=1 ) > File "", line 1 > runtime_assert( f=1 ) > ^ > SyntaxError: invalid syntax Until inline assignment expression is implemented, this is an error too: >>> if debug and not f=1: File "", line 1 if debug and not f=1: ^ SyntaxError: invalid syntax From mikhailwas at gmail.com Sat May 5 10:33:10 2018 From: mikhailwas at gmail.com (Mikhail V) Date: Sat, 5 May 2018 17:33:10 +0300 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On Fri, May 4, 2018 at 3:06 PM, Nick Coghlan wrote: > > With that spelling, the three examples above would become: > > # Exactly one branch is executed here > if m given m = pattern.search(data): > ... > elif m given m = other_pattern.search(data)): > ... > else: > ... > > # This name is rebound on each trip around the loop > while m given m = pattern.search(remaining_data): > ... > > # "f(x)" is only evaluated once on each iteration > result = [(x, y, x/y) for x in data if y given y = f(x)] > Well, I think it looks good. Nice job Nick! I like that it leaves the "=" alone finally (but it's maybe the reason why I am biased towards this syntax :) It has clear look so as not to confuse with conditionals and the variable is emphasized so it's well understood what it means. I wish I could like the whole idea of inline assignments more now - but well, it is just what it is. Something that can help on rare occasion. But as said, your variant feels right to me. Unlike with most other proposed constructs here my first impression - this is Python. If the new keyword is too much, there are some options too choose from. (though I've no idea whether this is plausible due to parsing ambiguities) if m given m = pattern.search(data): vs: if m with m = pattern.search(data): if m def m = pattern.search(data): if m as m = pattern.search(data): Yes, all these reads strange, but IMO understandable. BTW, using your keyword, in comprehensions vs the original idea of Chris then merely I suppose: result = [(x, y, x/y) given y = f(x) for x in data if y ] vs result = [(x, y, x/y) with y = f(x) for x in data if y ] result = [(x, y, x/y) def y = f(x) for x in data if y ] result = [(x, y, x/y) as y = f(x) for x in data if y ] vs result = [(x, y, x/y) for x in data if y given y = f(x)] vs result = [(x, y, x/y) for x in data if y with y = f(x)] result = [(x, y, x/y) for x in data if y def y = f(x)] result = [(x, y, x/y) for x in data if y as y = f(x)] I can't say for sure what is better but the former order seems to be slightly more appealing for some reson. [Tim] > it's obviously less redundant typing as: > if match := pattern.search(data): > In > if match given match = pattern.search(data): I disagree, I think it is not redundant but it's just how it is - namely it does not introduce 'implicitness' in the first place, and that is what I like about Nick's variant. But of course it is less compact. PS: recently I have discovered interesting fact: it seems one can already use 'inline assignment' with current syntax. E.g.: if exec("m = input()") or m: print (m) It seem to work as inline assignment correctly. Yes it is just coincidence, BUT: this has the wanted effect! - hides one line in "if" and "while" - leaves the assignment statement (or is it expression now? ;) - has explicit appearance So we have it already? Mikhail From Eloi.Gaudry at fft.be Sat May 5 11:04:06 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Sat, 5 May 2018 15:04:06 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: , , Message-ID: By 'self-contained', I meant that using the assert keyword and its expression is sufficient. An inline assertive expression as the one you describe does not fulfill this assert requirement. My proposal is simply to extend the current assert to non-debug builds and allow to switch it off/on at runtime. The syntax example I gave illustrated what I meant by syntax aware. De : Serhiy Storchaka Envoy? : samedi 5 mai ? 15:10 Objet : Re: [Python-ideas] Runtime assertion with no overhead when not active ? : python-ideas at python.org 05.05.18 15:54, Eloi Gaudry ????: > I meant avoiding the overhead of the expression evaluation enclosed in the assert itself, not the active/disabled state (that comes at virtually no cost). > ex: >>>> runtime_assert( { i:None for i in range( 10000000 ) } ) > > By using the syntax you describe ('boolean and not expr'), you loose all the benefit of the Python grammar: > - self-contained Could you please explain what you mean? > - able to catch semantic/syntax bugs > > ex: >>>> f=1 >>>> runtime_assert( f==1 ) >>>> runtime_assert( f=1 ) > File "", line 1 > runtime_assert( f=1 ) > ^ > SyntaxError: invalid syntax Until inline assignment expression is implemented, this is an error too: >>> if debug and not f=1: File "", line 1 if debug and not f=1: ^ SyntaxError: invalid syntax _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat May 5 11:26:31 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 6 May 2018 01:26:31 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On 6 May 2018 at 00:33, Mikhail V wrote: > recently I have discovered interesting fact: it seems one > can already use 'inline assignment' with current syntax. E.g.: > > if exec("m = input()") or m: > print (m) > > It seem to work as inline assignment correctly. > Yes it is just coincidence, BUT: this has the wanted effect! > - hides one line in "if" and "while" > - leaves the assignment statement (or is it expression now? ;) > - has explicit appearance > > So we have it already? > Not really, since: 1. exec with a plain string has a very high runtime cost (the compiler is pretty slow, since we assume the results will be cached) 2. exec'ed strings are generally opaque to static analysis tools 3. writing back to the local namespace via exec doesn't work consistently at function scope (and assuming PEP 558 is eventually accepted, will some day reliably *never* work) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat May 5 11:30:39 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 6 May 2018 01:30:39 +1000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: Message-ID: <20180505153039.GN9562@ando.pearwood.info> On Sat, May 05, 2018 at 08:04:45AM +0000, Eloi Gaudry wrote: > Hi folks, > I intend to add a runtime assertion feature in python. I'm very interested in this idea, but I'm afraid your draft PEP isn't very clear about this feature. Comments below. > Rationale > There is no runtime assert relying on python grammar available. For > diagnostics and/or debugging reasons, it would be valuable to add such > a feature. Why not this? if DEBUG_ENABLED: run_check(expression) where run_check is an ordinary function. If DEBUG_ENABLED is false, the only runtime overhead is the name lookup and bool test, the expression is not evaluated. What features does your suggestion offer over this simple technique? [...] > A brief example why avoiding evaluating the expression is needed to > avoid any overhead in case the runtime assert should be ignored. > :: > runtime_assert( 999 in { i:None for i in range( 10000000 ) } ) You say that you have been using this feature in production for two years. You should give real-life examples, not fake examples like the above. That example would be better written as: runtime_assert(True) since no part of the test actually depends on either the enormous dict or range objects that you build only to throw away. > Usage > :: > > runtime_assert( expr ) > > #would result in > if expr and runtime_assert_active: > print RuntimeAssertionError() Surely the test should be the other way around? if runtime_assert_active and expr: print(RuntimeAssertionError()) otherwise the expression is evaluated regardless of whether the runtime assertions are active or not. Please use Python 3 syntax for the PEP, as Python 2 is in maintenance mode and will absolutely not get any new language features. Some more observations: I like the idea of more control over runtime checking, but your PEP should justify why this is a good idea and why "just write more unit tests" isn't the solution. What happens if the caller has defined their own function or variable called runtime_assert? Can the caller customise the assertion error message? If you are changing the grammar, why not make runtime_assert a statement? Can the caller set an application-wide global across all modules, or do they have to set runtime_assert_active = True in every module they want to use runtime_assert? -- Steve From storchaka at gmail.com Sat May 5 11:35:10 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 5 May 2018 18:35:10 +0300 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: 05.05.18 18:26, Nick Coghlan ????: > 3. writing back to the local namespace via exec doesn't work > consistently at function scope (and assuming PEP 558 is eventually > accepted, will some day reliably *never* work) At end you can convert the source of the whole script to a string literal and write the whole script in one line. The problem solved! From storchaka at gmail.com Sat May 5 11:44:38 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 5 May 2018 18:44:38 +0300 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: Message-ID: 05.05.18 18:04, Eloi Gaudry ????: > By 'self-contained', I meant that using the assert keyword and its > expression is sufficient. An inline assertive expression as the one you > describe does not fulfill this assert requirement. Sufficient for what? And why writing with using the existing syntax is not sufficient? > My proposal is simply to extend the current assert to non-debug builds > and allow to switch it off/on at runtime. The main property of the assert statement is that has a zero overhead in non-debug run. If you remove this property, it will be not the assert statement, and you will not need a special syntax support for writing this runtime check. > The syntax example I gave illustrated what I meant by syntax aware. It doesn't illustrate why a new syntax is necessary. Or I can't understand this illustration. From ncoghlan at gmail.com Sat May 5 12:00:35 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 6 May 2018 02:00:35 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On 5 May 2018 at 13:36, Tim Peters wrote: > [Nick Coghlan ] > > ... > > The essence of the given clause concept would be to modify *these > specific > > cases* (at least initially) to allow the condition expression to be > followed > > by an inline assignment, of the form "given TARGET = EXPR". > > I'm not clear on what "these specific cases" are, specifically ;-) > Conditions in "if", "elif", and "while" statement expressions? > Exactly the 3 cases presented (if/elif/while conditions). The usage in comprehensions would mirror the usage in if statements, and avoid allowing name bindings in arbitrary locations due to the implicity nested scopes used by comprehensions. Conditional expressions would be initially omitted since including them would allow arbitrary name bindings in arbitrary locations via the quirky "x if x given x = expr else x" spelling, and because "else" isn't as distinctive an ending token as "given ... :", "given ... )", "given ... ]", or "given ... }". > Restricted to one "given" clause, or can they chain? In a listcomp, > is it one "given" clause per "if", or after at most one "if"? Or is > an "if" even needed at all in a listcomp? For example, > > [(f(x)**2, f(x)**3) for x in xs] > > has no conditions, and > > [(fx := f(x))**2, fx**3) for x in xs] > > is one reasonable use for binding expressions. > > [(fx**2, fx**3) for x in xs given fx = f(x)] > > reads better, although it's initially surprising (to my eyes) to find > fx defined "at the end". But no more surprising than the current: > > [(fx**2, fx**3) for x in xs for fx in [f(x)]] > > trick. > > There were a couple key reasons I left the "for x in y" case out of the initial proposal: 1. The "for x in y" header is already quite busy, especially when tuple unpacking is used in the assignment target 2. Putting the "given" clause at the end would make it ambiguous as to whether it's executed once when setting up the iterator, or on every iteration 3. You can stick in an explicit "if True" if you don't need the given variable in the filter condition [(fx**2, fx**3) for x in xs if True given fx = f(x)] And then once you've had an entire release where the filter condition was mandatory for the comprehension form, allowing the "if True" in "[(fx**2, fx**3) for x in xs given fx = f(x)]" to be implicit would be less ambiguous. [snip] > It''s certain sanest as > > if x**2 + y**2 > 9 given x, y = func_returning_twople(): > > "given" really shines there! > Yep, that's why I don't have the same immediate reaction of "It would need to be limited to simple names as targets" reaction as I do for assignment expressions. It might still be a good restriction to start out with, though (especially if we wanted to allow multiple name bindings in a single given clause). [snip] The one-letter variable name obscures that it doesn't > actually reduce _redundancy_, though. That is, in the current > > match = pattern.search(data) > if match: > > it's obviously less redundant typing as: > > if match := pattern.search(data): > > In > > if match given match = pattern.search(data): > > the annoying visual redundancy (& typing) persists. > Right, but that's specific to the case where the desired condition really is just "bool(target)". That's certainly likely to be a *common* use case, but if we decide that it's *that* particular flavour of redundancy that really bothers us, then there's always the "if expr as name:" spelling (similar to the way that Python had "a and b" and "a or b" logical control flow operators long before it got "a if c else b"). One more, a lovely (to my eyes) binding expression simplification > requiring two bindings in an `if` test, taken from real-life code I > happened to write during the PEP discussion: > > diff = x - x_base > if diff: > g = gcd(diff, n) > if g > 1: > return g > > collapsed to the crisp & clear: > > if (diff := x - x_base) and (g := gcd(diff, n)) > 1: > return g > > If only one trailing "given" clause can be given per `if` test > expression, presumably I couldn't do that without trickery. I was actually thinking that if we did want to allow multiple assignments, and we limited targets to single names, we could just use a comma as a separator: if diff and g > 1 given diff = x - x_base, g = gcd(diff, n): return g Similar to import statements, optional parentheses could be included in the grammar, allowing the name bindings to be split across multiple lines: if diff and g > 1 given ( diff = x - x_base, g = gcd(diff, n), ): return g (Other potential separators would be ";", but that reads weirdly to me since my brain expects the semi-colon to end the entire statement, and "and", but that feels overly verbose, while also being overly different from its regular meaning) > If it's > more general, > > if (diff given diff = x _ xbase) and g > 1 given g = gcd(diff, n): > > reads worse to my eyes (perhaps because of the "visual redundancy" > thing again), while > > if diff and g > 1 given diff = x - x_base given g = gcd(diff, n): > > has my eyes darting all over the place, and wondering which of the > trailing `given` clauses executes first. > I find that last effect is lessened when using the comma as a separator within the given clause rather than repeating the keyword itself. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat May 5 12:06:54 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 6 May 2018 02:06:54 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On 4 May 2018 at 22:06, Nick Coghlan wrote: > (Note: Guido's already told me off-list that he doesn't like the way this > spelling reads, but I wanted to share it anyway since it addresses one of > the recurring requests in the PEP 572 discussions for a more targeted > proposal that focused specifically on the use cases that folks had agreed > were reasonable potential use cases for inline assignment expressions. > > I'll also note that another potential concern with this specific proposal > is that even though "given" wasn't used as a term in any easily discovered > Python APIs back when I first wrote PEP 3150, it's now part of the > Hypothesis testing API, so adopting it as a keyword now would be markedly > more disruptive than it might have been historically) > Since I genuinely don't think this idea is important enough to disrupt hypothesis's public API, I've been pondering potential synonyms that are less likely to be common in real world code, while still being comprehensible and useful mnemonics for the proposed functionality. The variant I've most liked is the word "letting" (in the sense of "while letting these names have these values, do ..."): # Exactly one branch is executed here if m letting m = pattern.search(data): ... elif m letting m = other_pattern.search(data)): ... else: ... # This name is rebound on each trip around the loop while m letting m = pattern.search(remaining_data): ... # "f(x)" is only evaluated once on each iteration result = [(x, y, x/y) for x in data if y letting y = f(x)] # Tim's "bind two expressions" example if diff and g > 1 letting diff = x - x_base, g = gcd(diff, n): return g # The "bind two expressions" example across multiple lines while diff and g > 1 letting ( diff = x - x_base, g = gcd(diff, n), ): ... # Do something with diff and g Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Sat May 5 12:12:49 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 5 May 2018 11:12:49 -0500 Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> Message-ID: [Tim] >>>> ... I liked the way he _reached_ that conclusion: by looking at real- >>>> life Python code that may have been written instead to use constructs >>>> "like this". I find such examination far more persuasive than abstract >>>> arguments or made-up examples. [Serhiy] >>> I would like to see such examination for PEP 572. And for all other >>> syntax changing ideas. [Tim] >> I did it myself for 572, and posted several times about what I found. [Serhiy] > Could you please give links to these results? It is hard to find something > in hundreds of messages. It's no easier for me to find old messages, and you'd just ask for more & more anyway ;-) The "short course" I already gave didn't skip anything vital: Short course: I found a small win frequently, a large win rarely, but in most cases wouldn't use it. In all I expect I'd use it significantly more often than ternary "if", but far less often than augmented assignment. More importantly: But that's me - everybody needs to look at their own code to apply _their_ judgment. It's _applying_ the approach I find persuasive & productive, not someone else writing up the results of _their_ taking the approach. I'm not trying to change peoples' minds - just suggesting a more fruitful way (than abstract arguments, fashion, ...) to make up their minds to begin with. > I withdrew some my ideas and patches when my examinations showed that the > number of cases in the stdlib that will take a benefit from rewriting using > a new feature or from applying a compiler optimization is not large enough. Bingo! Note your "my examinations" in that. Someone who hasn't done their own examination is basically guessing. They may or may not reach the same conclusions if they did the work, but neither eloquence nor confidence is a reliable predictor of whether they would. Passion may even be negatively correlated ;-) From goosey15 at gmail.com Sat May 5 17:59:24 2018 From: goosey15 at gmail.com (Angus Hollands) Date: Sat, 05 May 2018 21:59:24 +0000 Subject: [Python-ideas] Python-ideas Digest, Vol 138, Issue 32 In-Reply-To: References: Message-ID: Hi all! Really interesting discussion on here. Personally I feel that PEP 572, as it stands, undermines the readability of the language, in particular for those new to the language, programming. I have issue with both the in-expression assignment, and the current proposed operator approach. Below is a modified version of a comment I made on a Reddit thread about this. I think the first point is that, beginners often read more code than they write - to build a mental picture of how the language works, and to build their solution from existing parts. Whether this is a good means of learning, or not, it's quite commonplace (and I learned using such a method). If this PEP passes, you will find people use the syntax. And it may well end up being disproportionately used in the early stages because of "new feature" semantics. Here is an extract from the Wikipedia A* search function reconstruct_path(cameFrom, current) total_path := [current] while current in cameFrom.Keys: current := cameFrom[current] total_path.append(current) return total_path In Python 3.7, that looks like this: def reconstruct_path(cameFrom, current): total_path = [current] while current in cameFrom.keys(): current = cameFrom[current] total_path.append(current) return total_path (of course it's not entirely Pythonic), but the point is - the pseudo code is designed to be semantically readable, and the Python code is very similar However with PEP572, now, the beginner will encounter both := assignments, and = assignments. When to use which one? Now they need to learn the edge cases / semantic differences between expression and statement explicitly, rather than picking this up as they go. I am not making this argument because I think that this is they best way to learn a programming language, I'm simply arguing that there is more cognitive overhead when unfamiliar with the language, in order to use the appropriate feature. In terms of implementation, it's especially odd that the current proposal (AFAICT) only binds to a name, rather than an assignment target. This feels very wrong, despite the fact that I can understand why it is suggested. I feel like the Python 3 series in particular has been making syntax more uniform, so that there aren't quirks and edge cases of "this only works in this context" (besides async, of course), which is one of the things that makes Python so expressive. Furthermore, with this PEP, assignment can now happen inside of expressions - and so one of the most fundamental benefits of expressions being effectively immutable in terms of local names is lost. In terms of prevalence, I don't think that these situations do occur all that often. I definitely agree that regex is the prime candidate for this kind of syntax. However, in the examples given (matching 3+ regexes), I would use a loop for simplicity anyway. When it comes to assigning to a result that is only used in the conditional block, this is certainly a case that benefits from the PEP. -------------------------------------------------------- If, however, the motivation for the PEP was deemed significant enough that warrant its inclusion in a future release, then I would like to suggest that the keyword approach is superior to the operator variant. In particular, I prefer the `where` to the `given` or 'let' candidates, as I think it is more descriptive and slightly shorter to type ;) Thanks! Angus Hollands -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Sat May 5 18:25:04 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sat, 5 May 2018 23:25:04 +0100 Subject: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter In-Reply-To: <20180503063208.GE9562@ando.pearwood.info> References: <20180503063208.GE9562@ando.pearwood.info> Message-ID: <2724634b-1ca8-fc78-8c98-f0a12046a4ae@btinternet.com> >> At some point, we're really better off just using a lambda. > Maybe I'm slow today, but I'm having trouble seeing how to write this as > a lambda. >>> values = {'x': 43, 'y': 55} >>> x, y, z = (lambda *args : tuple(values.get(arg,0) for arg in args))('x','y','z') >>> print(x, y, z) (43, 55, 0) >>> Rob Cliffe From ncoghlan at gmail.com Sat May 5 20:27:36 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 6 May 2018 10:27:36 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On 6 May 2018 at 02:06, Nick Coghlan wrote: > On 4 May 2018 at 22:06, Nick Coghlan wrote: > >> (Note: Guido's already told me off-list that he doesn't like the way this >> spelling reads, but I wanted to share it anyway since it addresses one of >> the recurring requests in the PEP 572 discussions for a more targeted >> proposal that focused specifically on the use cases that folks had agreed >> were reasonable potential use cases for inline assignment expressions. >> >> I'll also note that another potential concern with this specific proposal >> is that even though "given" wasn't used as a term in any easily discovered >> Python APIs back when I first wrote PEP 3150, it's now part of the >> Hypothesis testing API, so adopting it as a keyword now would be markedly >> more disruptive than it might have been historically) >> > > Since I genuinely don't think this idea is important enough to disrupt > hypothesis's public API, I've been pondering potential synonyms that are > less likely to be common in real world code, while still being > comprehensible and useful mnemonics for the proposed functionality. > I've also been pondering Tim's suggestion of instead enhancing the code generation pipeline's native support for pseudo-keywords in a way that can be explicitly represented in the Grammar and AST, rather than having to be hacked in specifically every time we want to introduce a new keyword without a mandatory __future__ statement (and without creating major compatibility headaches for code that uses those new keywords as attribute or variable names). While I haven't actually tried this out yet, the way I'm thinking that might look is to add the following nodes to the grammar: name_plus: NAME | pseudo_keyword pseudo_keyword: 'given' and the replace all direct uses of 'NAME' in the grammar with 'name_plus'. That way, to allow a new keyword to be used as a name in addition to its syntactic use case, we'd add it to the pseudo_keyword list in addition to adding it to . To avoid ambiguities in the grammar, this could only be done for keywords that *can't* be used to start a new expression or statement (so it wouldn't have been sufficient for the async/await case, since 'async' can start statements, and 'await' can start both statements and expressions). So if Guido's view on the out-of-order execution approach to inline name binding softens, I think this would be a better approach to pursue than making a more awkward keyword choice. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat May 5 20:36:03 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 6 May 2018 10:36:03 +1000 Subject: [Python-ideas] Python-ideas Digest, Vol 138, Issue 32 In-Reply-To: References: Message-ID: On 6 May 2018 at 07:59, Angus Hollands wrote: > If, however, the motivation for the PEP was deemed significant enough that > warrant its inclusion in a future release, then I would like to suggest > that the keyword approach is superior to the operator variant. In > particular, I prefer the `where` to the `given` or 'let' candidates, as I > think it is more descriptive and slightly shorter to type ;) > Aside from an API naming conflict with NumPy, the key problem with using "where" for this purpose is that "where" is explicitly used to name *filtering* clauses in SQL, NumPy, and other contexts ("having" is used in a similar way for filtering on SQL aggregate groups). So in "[(x, y, x/y) for x in data if y where y = f(x)]", having both an "if" clause and a "where" clause makes it look like there are two filters being defined (and a mispelt "==" in the second one), rather than a filter and a name binding. The virtue of "given" is that in any context which uses it, the intended meaning is to associate a name with a value (either directly, as in the mathematical usage, or indirectly, as in the hypothesis API usage), which is exactly the meaning we're interested in here. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat May 5 22:13:09 2018 From: guido at python.org (Guido van Rossum) Date: Sat, 5 May 2018 19:13:09 -0700 Subject: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter In-Reply-To: References: Message-ID: Hi Vincent, Your idea is interesting but we are worried that there are not enough real use cases where it would be useful. Have you encountered situations yourself where this would make a difference? I am asking not for clarifying examples (you already provided one and from that it's perfectly clear to me what you are proposing) but for real-world code that would benefit from this addition to the itemgetter API. --Guido On Wed, May 2, 2018 at 1:08 AM, Vincent Maillol wrote: > Hi everybody, > > Our PEP idea would be to purpose to add a global default value for > itemgeet and attrgetter method. > > This was inspired from bug 14384 (https://bugs.python.org/issue14384); > opened by Miki TEBEKA. > > For example, we could do: > > p1 = {'x': 43; 'y': 55} > x, y, z = itemgetter('x', 'y', 'z', default=0)(values) > print(x, y, z) > 43, 55, 0 > > instead of: > > values = {'x': 43; 'y': 55} > x = values.get('x', 0) > y = values.get('y', 0) > z = values.get('z', 0) > print(x, y, z) > 43, 55, 0 > > The goal is to have have concise code and improve consistency with > getattr, attrgetter and itemgetter > > What are you thinking about this? > > MAILLOL Vincent > GALODE Alexandre > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Sun May 6 01:51:48 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 6 May 2018 00:51:48 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: [Nick] >... > There were a couple key reasons I left the "for x in y" case out of the > initial proposal: > > 1. The "for x in y" header is already quite busy, especially when tuple > unpacking is used in the assignment target > 2. Putting the "given" clause at the end would make it ambiguous as to > whether it's executed once when setting up the iterator, or on every > iteration > 3. You can stick in an explicit "if True" if you don't need the given > variable in the filter condition > > [(fx**2, fx**3) for x in xs if True given fx = f(x)] > > And then once you've had an entire release where the filter condition was > mandatory for the comprehension form, allowing the "if True" in "[(fx**2, > fx**3) for x in xs given fx = f(x)]" to be implicit would be less ambiguous. And some people claim ":=" would make Python harder to teach ;-) [Tim] >> ... >> It''s certain sanest as >> >> if x**2 + y**2 > 9 given x, y = func_returning_twople(): >> >> "given" really shines there! > Yep, that's why I don't have the same immediate reaction of "It would need > to be limited to simple names as targets" reaction as I do for assignment > expressions. It might still be a good restriction to start out with, though I contrived that specific "use case", of course - I actually didn't stumble into any real code where multiple targets would benefit to my eyes. Perhaps because, as you noted above of `"for x in y" headers`, multiple-target assignment statements are often quite busy already too (I have no interest in cramming as much logic as possible into each line - but "sparse is better than dense" doesn't also mean "almost empty is better than sparse" ;-) ). > (especially if we wanted to allow multiple name bindings in a single given > clause). >> ... >> The one-letter variable name obscures that it doesn't >> actually reduce _redundancy_, though. That is, in the current >> >> match = pattern.search(data) >> if match: >> >> it's obviously less redundant typing as: >> >> if match := pattern.search(data): >> >> In >> >> if match given match = pattern.search(data): >> >> the annoying visual redundancy (& typing) persists. > Right, but that's specific to the case where the desired condition really is > just "bool(target)". Not only. If the result _needs_ to be used N times in total in the test, binding expressions allow for that, but `given` requires N+1 instances of the name (the "extra one" to establish the name to begin with). For example, where `probable_prime()` returns `True` or `False`, and `bool(candidate)` is irrelevant: # highbit is a power of 2 >= 2; create a random prime # whose highest bit is highbit while (not probable_prime(candidate) given candidate = highbit | randrange(1, highbit, 2)): pass versus while not probable_prime(candidate := highbit | randrange(1, highbit, 2)): pass There I picked a "long" name to make the redundancy visually annoying ;-) > That's certainly likely to be a *common* use case, In all the code I looked at where I believe a gimmick like this would actually help, it was indeed by far _most_ common that the result only needed to be used once in the test. In all such cases, the binding expression spelling of the test requires one instance of the name, and the `given` spelling two. > but if we decide that it's *that* particular flavour of redundancy that really > bothers us, then there's always the "if expr as name:" spelling (similar to > the way that Python had "a and b" and "a or b" logical control flow > operators long before it got "a if c else b"). Reducing each redundancy is a small win to me, but reaches "importance" because it's so frequent. Binding expressions have more uses than _just_ that, though. But I'm sure I don't know what they all are. When a _general_ feature is added, people find surprising uses for it. For example, at times I'd love to write code like this, but can't: while any(n % p == 0 for p in small_primes): # divide p out - but what is p? Generator expressions prevent me from seeing which value of `p` succeeded. While that's often "a feature", sometimes it's a PITA. I don't know whether this binding-expression stab would work instead (I'm not sure the PEP realized there's "an issue" here, about the intended scope for `thisp`): while any(n % (thisp := p) == 0 for p in small_primes): n //= thisp If that is made to work, I think that counts as "a surprising use" (capturing a witness for `any(genexp)` and a counterexample for `all(genexp)`, both of which are wanted at times, but neither of which `any()`/`all()` will ever support on their own).. I suppose I could do it with `given` like so: while p is not None given p = next( (p for p in small_primes if n % p == 0), None): n //= p but at that point I'd pay to go back to the original loop-and-a-half ;-) >> One more, a lovely (to my eyes) binding expression simplification >> requiring two bindings in an `if` test, taken from real-life code I >> happened to write during the PEP discussion: >> >> diff = x - x_base >> if diff: >> g = gcd(diff, n) >> if g > 1: >> return g >> >> collapsed to the crisp & clear: >> >> if (diff := x - x_base) and (g := gcd(diff, n)) > 1: >> return g >> >> If only one trailing "given" clause can be given per `if` test >> expression, presumably I couldn't do that without trickery. > I was actually thinking that if we did want to allow multiple assignments, > and we limited targets to single names, we could just use a comma as a > separator: > > if diff and g > 1 given diff = x - x_base, g = gcd(diff, n): > return g I expect that 's bound to be confusing, because the assignment _statement_ diff = x - x_base, g = gcd(diff, n) groups very differently than intended: diff = (x - x_base, g) = gcd(diff, n) And that's a syntax error. With enclosing parens, expectations change, and then people would expect it to work like specifying keyword arguments instead: > Similar to import statements, optional parentheses could be included in the > grammar, allowing the name bindings to be split across multiple lines: > > if diff and g > 1 given ( > diff = x - x_base, > g = gcd(diff, n), > ): > return g Keyword arguments work as a syntactic model (group as intended), but not semantically: if they really were keyword arguments, `x - x_base` and `gcd(diff, n)` would both be evaluated _before_ any bindings occurred. So it's more quirky `given`-specific rules no matter how you cut it. The closest bit of Python syntax that captures the grouping (but only partially), and the "left-to-right, with each binding in turn visible to later expressions" semantics, is the semicolon. Which would create even weirder expectations :-( > (Other potential separators would be ";", but that reads weirdly to me since > my brain expects the semi-colon to end the entire statement, and "and", but > that feels overly verbose, while also being overly different from its > regular meaning) Yup. >> If it's more general, >> >> if (diff given diff = x _ xbase) and g > 1 given g = gcd(diff, n): >> >> reads worse to my eyes (perhaps because of the "visual redundancy" >> thing again), while >> >> if diff and g > 1 given diff = x - x_base given g = gcd(diff, n): >> >> has my eyes darting all over the place, and wondering which of the >> trailing `given` clauses executes first. > I find that last effect is lessened when using the comma as a separator > within the given clause rather than repeating the keyword itself. Definitely. I tend to believe Python has "slightly more than enough" meanings for commas already, though. But using commas and _requiring_ parens for more than one `given` binding seems least surprising to me overall. Then again, everyone already knows what ":=" means. They just dislike it because so many major languages already have it -) From ubershmekel at gmail.com Sun May 6 02:53:11 2018 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Sun, 06 May 2018 06:53:11 +0000 Subject: [Python-ideas] __dir__ in which folder is this py file Message-ID: Hi Ideas, I often need to reference a script's current directory. I end up writing: import os SRC_DIR = os.path.dirname(__file__) But I would prefer to have a new dunder for that. I propose: "__dir__". I was wondering if others would find it convenient to include such a shortcut. Here are some examples of dirname(__file__) in prominent projects. https://github.com/tensorflow/models/search?l=Python&q=dirname&type= https://github.com/django/django/search?l=Python&q=dirname&type= https://github.com/nose-devs/nose/search?l=Python&q=dirname&type= Reasons not to add __dir__: * There already is one way to do it and it's clear and fairly short. * Avoid the bikeshed discussion of __dir__, __folder__, and other candidates. Reasons to add it: * os.path.dirname(__file__) returns the empty string when you're in the same directory as the script. Luckily, os.path.join understands an empty string as a ".", but this still is suboptimal for logging where it might be surprising to find the empty string. __dir__ could be implemented to contain a "." in that case. * I would save about 20 characters and a line from 50% of my python scripts. * This is such a common construct that everyone giving it their own name seems suboptimal for communicating. Common names include: here, path, dirname, module_dir. Cheers, Yuval Greenfield P.s. nodejs has it - https://nodejs.org/docs/latest/api/modules.html#modules_dirname also I apologize if this has been suggested before - my googling didn't find a previous thread. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun May 6 06:23:47 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 6 May 2018 20:23:47 +1000 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: <20180506102347.GR9562@ando.pearwood.info> On Sun, May 06, 2018 at 06:53:11AM +0000, Yuval Greenfield wrote: > Hi Ideas, > > I often need to reference a script's current directory. I end up writing: > > import os > SRC_DIR = os.path.dirname(__file__) > > > But I would prefer to have a new dunder for that. I propose: "__dir__". I > was wondering if others would find it convenient to include such a shortcut. Not really, no. If I'm doing file name processing such that I need the script's directory, I already need to import os, so providing this pre-calculated would only save at most a single line. Not every one-liner needs to be a built-in. I don't strongly oppose this, but given how easy it is, I don't really see the point. New features add a cost, and while these costs are individually tiny: - one more thing to document - one more thing for people to learn and memorise - every script pays the cost of calculating this dirname whether it is needed or not etc, the corresponding benefit is also tiny, so it is not clear to me that the benefit from having this is greater than the cost of having it. If you can demonstrate a clear, significant benefit, the balance would shift in favour of this proposal, but as it stands, it seems like a mere matter of personal taste. So in the absence of any clear, non-trivial benefit, I'm vaguely -0 on this. However, I am opposed to the use of __dir__ as the dunder name, since __dir__ is already used as the dunder method for the dir() builtin. Even though strictly speaking there is no conflict between a method and a module global, conceptually they would be better kept distinct. If this is approved, I suggest __dirname__ instead. > Reasons not to add __dir__: > * There already is one way to do it and it's clear and fairly short. Indeed. > Reasons to add it: > * os.path.dirname(__file__) returns the empty string when you're in the > same directory as the script. Luckily, os.path.join understands an empty > string as a ".", but this still is suboptimal for logging where it might be > surprising to find the empty string. Can you give an example where the empty string actually is a real problem, rather than just "might be"? Bonus points if it was an actual problem in real code, not just a hypothetical problem made up as an example. For what it's worth, if there is such a genuine problem, that would shift me to +0.5 on the proposal: SRC_DIR = os.path.dirname(__file__) or '.' versus __dirname__ -- Steve From steve at pearwood.info Sun May 6 07:00:27 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 6 May 2018 21:00:27 +1000 Subject: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter In-Reply-To: <20180503063208.GE9562@ando.pearwood.info> References: <20180503063208.GE9562@ando.pearwood.info> Message-ID: <20180506110027.GS9562@ando.pearwood.info> On Thu, May 03, 2018 at 04:32:09PM +1000, Steven D'Aprano wrote: > Maybe I'm slow today, but I'm having trouble seeing how to write this as > a lambda. Yes, I was definitely having a "cannot brain, I have the dumb" day, because it is not that hard to write using lambda. See discussion here: https://mail.python.org/pipermail/python-list/2018-May/732795.html If anything, the problem is a plethora of choices, where it isn't clear which if any is the best way, or the One Obvious Way. -- Steve From cody.piersall at gmail.com Sun May 6 09:28:16 2018 From: cody.piersall at gmail.com (Cody Piersall) Date: Sun, 06 May 2018 13:28:16 +0000 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: With PEP 562, the name __dir__ is off limits for this. Cody On Sun, May 6, 2018, 1:54 AM Yuval Greenfield wrote: > Hi Ideas, > > I often need to reference a script's current directory. I end up writing: > > import os > SRC_DIR = os.path.dirname(__file__) > > > But I would prefer to have a new dunder for that. I propose: "__dir__". I > was wondering if others would find it convenient to include such a shortcut. > > > Here are some examples of dirname(__file__) in prominent projects. > > https://github.com/tensorflow/models/search?l=Python&q=dirname&type= > https://github.com/django/django/search?l=Python&q=dirname&type= > https://github.com/nose-devs/nose/search?l=Python&q=dirname&type= > > Reasons not to add __dir__: > * There already is one way to do it and it's clear and fairly short. > * Avoid the bikeshed discussion of __dir__, __folder__, and other > candidates. > > Reasons to add it: > * os.path.dirname(__file__) returns the empty string when you're in the > same directory as the script. Luckily, os.path.join understands an empty > string as a ".", but this still is suboptimal for logging where it might be > surprising to find the empty string. __dir__ could be implemented to > contain a "." in that case. > * I would save about 20 characters and a line from 50% of my python > scripts. > * This is such a common construct that everyone giving it their own name > seems suboptimal for communicating. Common names include: here, path, > dirname, module_dir. > > Cheers, > > Yuval Greenfield > > P.s. nodejs has it - > https://nodejs.org/docs/latest/api/modules.html#modules_dirname also I > apologize if this has been suggested before - my googling didn't find a > previous thread. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From george at fischhof.hu Sun May 6 11:05:02 2018 From: george at fischhof.hu (George Fischhof) Date: Sun, 6 May 2018 17:05:02 +0200 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: 2018-05-06 15:28 GMT+02:00 Cody Piersall : > With PEP 562, the name __dir__ is off limits for this. > > Cody > > On Sun, May 6, 2018, 1:54 AM Yuval Greenfield > wrote: > >> Hi Ideas, >> >> I often need to reference a script's current directory. I end up writing: >> >> import os >> SRC_DIR = os.path.dirname(__file__) >> >> >> But I would prefer to have a new dunder for that. I propose: "__dir__". I >> was wondering if others would find it convenient to include such a shortcut. >> >> >> Here are some examples of dirname(__file__) in prominent projects. >> >> https://github.com/tensorflow/models/search?l=Python&q=dirname&type= >> https://github.com/django/django/search?l=Python&q=dirname&type= >> https://github.com/nose-devs/nose/search?l=Python&q=dirname&type= >> >> Reasons not to add __dir__: >> * There already is one way to do it and it's clear and fairly short. >> * Avoid the bikeshed discussion of __dir__, __folder__, and other >> candidates. >> >> Reasons to add it: >> * os.path.dirname(__file__) returns the empty string when you're in the >> same directory as the script. Luckily, os.path.join understands an empty >> string as a ".", but this still is suboptimal for logging where it might be >> surprising to find the empty string. __dir__ could be implemented to >> contain a "." in that case. >> * I would save about 20 characters and a line from 50% of my python >> scripts. >> * This is such a common construct that everyone giving it their own name >> seems suboptimal for communicating. Common names include: here, path, >> dirname, module_dir. >> >> Cheers, >> >> Yuval Greenfield >> >> P.s. nodejs has it - https://nodejs.org/docs/latest/api/modules.html# >> modules_dirname also I apologize if this has been suggested before - my >> googling didn't find a previous thread. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > Hi, I would give +1 for __dirname__ George -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun May 6 13:44:07 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 7 May 2018 03:44:07 +1000 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: On Mon, May 7, 2018 at 1:05 AM, George Fischhof wrote: >> On Sun, May 6, 2018, 1:54 AM Yuval Greenfield >> wrote: >>> >>> Hi Ideas, >>> >>> I often need to reference a script's current directory. I end up writing: >>> >>> import os >>> SRC_DIR = os.path.dirname(__file__) > I would give +1 for __dirname__ Something to keep in mind: making this available to every module, whether it's wanted or not, means that the Python interpreter has to prepare that just in case it's wanted. That's extra work as part of setting up a module. Which, in turn, means it's extra work for EVERY import, and consequently, slower Python startup. It might only be a small slowdown, but it's also an extremely small benefit. -1. ChrisA From tim.peters at gmail.com Sun May 6 21:32:47 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 6 May 2018 20:32:47 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 Message-ID: In a different thread I noted that I sometimes want to write code like this: while any(n % p == 0 for p in small_primes): # divide p out - but what's p? But generator expressions hide the value of `p` that succeeded, so I can't. `any()` and `all()` can't address this themselves - they merely work with an iterable of objects to evaluate for truthiness, and know nothing about how they're computed. If you want to identify a witness (for `any()` succeeding) or a counterexample (for `all()` failing), you need to write a workalike loop by hand. So would this spelling work using binding expressions? while any(n % (thisp := p) == 0 for p in small_primes): n //= thisp I'm not entirely clear from what the PEP says, but best guess is "no", from this part of the discussion[1]: """ It would be convenient to use this feature to create rolling or self-effecting data streams: progressive_sums = [total := total + value for value in data] This will fail with UnboundLocalError due to total not being initalized. Simply initializing it outside of the comprehension is insufficient - unless the comprehension is in class scope: ... """ So, in my example above, I expect that `thisp` is viewed as being local to the created-by-magic lexically nested function implementing the generator expression. `thisp` would be bound on each iteration, but would vanish when `any()` finished and the anonymous function vanished with it. I'd get a NameError on "n //= thisp" (or pick up whatever object it was bound to before the loop). I have a long history of arguing that magically created lexically nested anonymous functions try too hard to behave exactly like explicitly typed lexically nested functions, but that's the trendy thing to do so I always lose ;-) The problem: in a magically created nested function, you have no possibility to say _anything_ about scope; at least when you type it by hand, you can add `global` and/or `nonlocal` declarations to more-or-less say what you want. Since there's no way to explicitly identify the desired scope, I suggest that ":=" inside magically created nested functions do the more-useful-more-often thing: treat the name being bound as if the binding had been spelled in its enclosing context instead. So, in the above, if `thisp` was declared `global`, also `global` in the genexp; if `nonlocal`, also `nonlocal`; else (almost always the case in real life) local to the containing code (meaning it would be local to the containing code, but nonlocal in the generated function). Then my example would work fine, and the PEP's would too just by adding total = 0 before it. Funny: before `nonlocal` was added, one of the (many) alternative suggestions was that binding a name in an enclosing scope use ":=" instead of "=". No, I didn't have much use for `for` target names becoming magically local to invisible nested functions either, but I appreciate that it's less surprising overall. Using ":=" is much more strongly screaming "I'm going way out of my way to give a name to this thing, so please don't fight me by assuming I need to be protected from the consequences of what I explicitly asked for". [1] https://www.python.org/dev/peps/pep-0572/ From rosuav at gmail.com Sun May 6 22:02:56 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 7 May 2018 12:02:56 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: On Mon, May 7, 2018 at 11:32 AM, Tim Peters wrote: > In a different thread I noted that I sometimes want to write code like this: > > while any(n % p == 0 for p in small_primes): > # divide p out - but what's p? > > But generator expressions hide the value of `p` that succeeded, so I > can't. `any()` and `all()` can't address this themselves - they > merely work with an iterable of objects to evaluate for truthiness, > and know nothing about how they're computed. If you want to identify > a witness (for `any()` succeeding) or a counterexample (for `all()` > failing), you need to write a workalike loop by hand. > > So would this spelling work using binding expressions? > > while any(n % (thisp := p) == 0 for p in small_primes): > n //= thisp > > So, in my example above, I expect that `thisp` is viewed as being > local to the created-by-magic lexically nested function implementing > the generator expression. `thisp` would be bound on each iteration, > but would vanish when `any()` finished and the anonymous function > vanished with it. I'd get a NameError on "n //= thisp" (or pick up > whatever object it was bound to before the loop). You're correct. The genexp is approximately equivalent to: def genexp(): for p in small_primes: thisp = p yield n % thisp == 0 while any(genexp()): n //= thisp With generator expressions, since they won't necessarily be iterated over immediately, I think it's correct to create an actual nested function; you need the effects of closures. With comprehensions, it's less obvious, and what you're asking for might be more plausible. The question is, how important is the parallel between list(x for x in iter) and [x for x in iter] ? Guido likes it, and to create that parallel, list comps MUST be in their own functions too. > I have a long history of arguing that magically created lexically > nested anonymous functions try too hard to behave exactly like > explicitly typed lexically nested functions, but that's the trendy > thing to do so I always lose ;-) The problem: in a magically created > nested function, you have no possibility to say _anything_ about > scope; at least when you type it by hand, you can add `global` and/or > `nonlocal` declarations to more-or-less say what you want. That's a fair point. But there is another equally valid use-case for assignment expressions inside list comps: values = [y + 2 for x in iter if (y := f(x)) > 0] In this case, it's just as obvious that the name 'y' should be local to the comprehension, as 'x' is. Since there's no way to declare "nonlocal y" inside the comprehension, you're left with a small handful of options: 1) All names inside list comprehensions are common with their surrounding scope. The comprehension isn't inside a function, the iteration variable leaks, you can share names easily. Or if it *is* inside a function, all its names are implicitly "nonlocal" (in which case there's not much point having the function). 2) All names are local to their own scope. No names leak, and that includes names made with ":=". 3) Some sort of rule like "iteration variables don't leak, but those used with := are implicitly nonlocal". Would create odd edge cases eg [x for x in iter if x := x] and that would probably result in x leaking. 4) A special adornment on local names if you don't want them to leak 5) A special adornment on local names if you DO want them to leak 6) A combination of #3 and #4: "x := expr" will be nonlocal, ".x := expr" will be local, "for x in iter" will be local. Backward compatible but a pain to explain. I can't say I'm a fan of any of the complicated ones (3 through 6). Option #2 is current status - the name binding is part of the expression, the expression is inside an implicit function, so the name is bound within the function. Option 1 is plausible, but would be a backward compatibility break, with all the consequences thereof. It'd also be hard to implement cleanly with genexps, since they MUST be functions. (Unless they're an entirely new concept of callable block that doesn't include its own scope, which could work, but would be a boatload of new functionality.) > Since there's no way to explicitly identify the desired scope, I > suggest that ":=" inside magically created nested functions do the > more-useful-more-often thing: treat the name being bound as if the > binding had been spelled in its enclosing context instead. So, in the > above, if `thisp` was declared `global`, also `global` in the genexp; > if `nonlocal`, also `nonlocal`; else (almost always the case in real > life) local to the containing code (meaning it would be local to the > containing code, but nonlocal in the generated function). Is it really more useful more often? > No, I didn't have much use for `for` target names becoming magically > local to invisible nested functions either, but I appreciate that it's > less surprising overall. Using ":=" is much more strongly screaming > "I'm going way out of my way to give a name to this thing, so please > don't fight me by assuming I need to be protected from the > consequences of what I explicitly asked for". Personally, I'd still like to go back to := creating a statement-local name, one that won't leak out of ANY statement. But the tide was against that one, so I gave up on it. ChrisA From ncoghlan at gmail.com Sun May 6 22:13:13 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 May 2018 12:13:13 +1000 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: On 7 May 2018 at 03:44, Chris Angelico wrote: > On Mon, May 7, 2018 at 1:05 AM, George Fischhof > wrote: > >> On Sun, May 6, 2018, 1:54 AM Yuval Greenfield > >> wrote: > >>> > >>> Hi Ideas, > >>> > >>> I often need to reference a script's current directory. I end up > writing: > >>> > >>> import os > >>> SRC_DIR = os.path.dirname(__file__) > > I would give +1 for __dirname__ > > Something to keep in mind: making this available to every module, > whether it's wanted or not, means that the Python interpreter has to > prepare that just in case it's wanted. That's extra work as part of > setting up a module. Which, in turn, means it's extra work for EVERY > import, and consequently, slower Python startup. It might only be a > small slowdown, but it's also an extremely small benefit. > It also makes the name show up in dir(mod) for every module, and we're currently looking for ways to make that list *shorter*, not longer. So I have a different suggestion: perhaps it might make sense to propose promoting a key handful of path manipulation operations to the status of being builtins? Specifically, the ones I'd have in mind would be: - dirname (aka os.path.dirname) - joinpath (aka os.path.join) - abspath (aka os.path.abspath) Why those 3? Because with just those three operations you can locate other files relative to `__file__`, the current working directory [1], and arbitrary absolute paths, as well as remove path traversal notation like ".." and "." from the resulting paths (since abspath() internally calls normpath()). _launch_dir = abspath('') def open_from_launch_dir(relpath, mode='r'): return open(abspath(joinpath(_launch_dir, relpath)), mode) _script_dir = dirname(abspath(__file__)) def open_from_script_dir(relpath, mode='r'): return open(abspath(joinpath(_script_dir, relpath)), mode) You'd still need to import pathlib or os.path for more complex path manipulations, but they generally wouldn't be needed any more if all you're doing is reading and/or writing a handful of specific files. Cheers, Nick. [1] abspath can stand in for os.getcwd(), since you can spell the latter as abspath('.') or abspath(''), and we could potentially even make it so you can retrieve the cwd via just abspath() -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Sun May 6 22:26:19 2018 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Sun, 06 May 2018 21:26:19 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: <163386bccf8.2837.db5b03704c129196a4e9415e55413ce6@gmail.com> On May 6, 2018 8:41:26 PM Tim Peters wrote: > In a different thread I noted that I sometimes want to write code like this: > > while any(n % p == 0 for p in small_primes): > # divide p out - but what's p? > > But generator expressions hide the value of `p` that succeeded, so I > can't. `any()` and `all()` can't address this themselves - they > merely work with an iterable of objects to evaluate for truthiness, > and know nothing about how they're computed. If you want to identify > a witness (for `any()` succeeding) or a counterexample (for `all()` > failing), you need to write a workalike loop by hand. > Couldn't you just do: def first(it): return next(it, None) while (item := first(p for p in small_primes if n % p == 0)): # ... IMO for pretty much anything more complex, it should probably be a loop in its own function. > So would this spelling work using binding expressions? > > while any(n % (thisp := p) == 0 for p in small_primes): > n //= thisp > > I'm not entirely clear from what the PEP says, but best guess is "no", > from this part of the discussion[1]: > > """ > It would be convenient to use this feature to create rolling or > self-effecting data streams: > > progressive_sums = [total := total + value for value in data] > > This will fail with UnboundLocalError due to total not being > initalized. Simply initializing it outside of the comprehension is > insufficient - unless the comprehension is in class scope: ... > """ > > So, in my example above, I expect that `thisp` is viewed as being > local to the created-by-magic lexically nested function implementing > the generator expression. `thisp` would be bound on each iteration, > but would vanish when `any()` finished and the anonymous function > vanished with it. I'd get a NameError on "n //= thisp" (or pick up > whatever object it was bound to before the loop). > > I have a long history of arguing that magically created lexically > nested anonymous functions try too hard to behave exactly like > explicitly typed lexically nested functions, but that's the trendy > thing to do so I always lose ;-) The problem: in a magically created > nested function, you have no possibility to say _anything_ about > scope; at least when you type it by hand, you can add `global` and/or > `nonlocal` declarations to more-or-less say what you want. > > Since there's no way to explicitly identify the desired scope, I > suggest that ":=" inside magically created nested functions do the > more-useful-more-often thing: treat the name being bound as if the > binding had been spelled in its enclosing context instead. So, in the > above, if `thisp` was declared `global`, also `global` in the genexp; > if `nonlocal`, also `nonlocal`; else (almost always the case in real > life) local to the containing code (meaning it would be local to the > containing code, but nonlocal in the generated function). > > Then my example would work fine, and the PEP's would too just by adding > > total = 0 > > before it. > > Funny: before `nonlocal` was added, one of the (many) alternative > suggestions was that binding a name in an enclosing scope use ":=" > instead of "=". > > No, I didn't have much use for `for` target names becoming magically > local to invisible nested functions either, but I appreciate that it's > less surprising overall. Using ":=" is much more strongly screaming > "I'm going way out of my way to give a name to this thing, so please > don't fight me by assuming I need to be protected from the > consequences of what I explicitly asked for". > > > [1] https://www.python.org/dev/peps/pep-0572/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Ryan (????) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else https://refi64.com/ From tim.peters at gmail.com Sun May 6 22:34:13 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 6 May 2018 21:34:13 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: [Chris Angelico ] > ... > You're correct. The genexp is approximately equivalent to: > > def genexp(): > for p in small_primes: > thisp = p > yield n % thisp == 0 > while any(genexp()): > n //= thisp > > With generator expressions, since they won't necessarily be iterated > over immediately, I think it's correct to create an actual nested > function; you need the effects of closures. With comprehensions, it's > less obvious, and what you're asking for might be more plausible. The > question is, how important is the parallel between list(x for x in > iter) and [x for x in iter] ? Guido likes it, and to create that > parallel, list comps MUST be in their own functions too. I don't care how they're implemented here; I only care here about the visible semantics. >> I have a long history of arguing that magically created lexically >> nested anonymous functions try too hard to behave exactly like >> explicitly typed lexically nested functions, but that's the trendy >> thing to do so I always lose ;-) The problem: in a magically created >> nested function, you have no possibility to say _anything_ about >> scope; at least when you type it by hand, you can add `global` and/or >> `nonlocal` declarations to more-or-less say what you want. > That's a fair point. But there is another equally valid use-case for > assignment expressions inside list comps: > > values = [y + 2 for x in iter if (y := f(x)) > 0] > > In this case, it's just as obvious that the name 'y' should be local > to the comprehension, as 'x' is. There's a difference, though: if `y` "leaks", BFD. Who cares? ;-) If `y` remains inaccessible, there's no way around that. > Since there's no way to declare "nonlocal y" inside the > comprehension, you're left with a small handful of options: i leapt straight to #3: > 1) All names inside list comprehensions are common with their > surrounding scope. The comprehension isn't inside a function, the > iteration variable leaks, you can share names easily. Or if it *is* > inside a function, all its names are implicitly "nonlocal" (in which > case there's not much point having the function). DOA. Breaks old code. > 2) All names are local to their own scope. No names leak, and that > includes names made with ":=". Saying "local to their own scope" _assumes_ what you're trying to argue _for_ - it's circular. In fact it's impossible to know what the user intends the scope to be. > 3) Some sort of rule like "iteration variables don't leak, but those > used with := are implicitly nonlocal". Explicitly, because "LHS inherits scope from its context" (whether global, nonlocal, or local) is part of what ":=" is defined to _mean_ then. > Would create odd edge cases eg [x for x in iter if x := x] and that would > probably result in x leaking. Don't care. > 4) A special adornment on local names if you don't want them to leak > > 5) A special adornment on local names if you DO want them to leak Probably also DOA. > 6) A combination of #3 and #4: "x := expr" will be nonlocal, ".x := > expr" will be local, "for x in iter" will be local. Backward > compatible but a pain to explain. Definitely DOA. ... >> Since there's no way to explicitly identify the desired scope, I >> suggest that ":=" inside magically created nested functions do the >> more-useful-more-often thing: treat the name being bound as if the >> binding had been spelled in its enclosing context instead. So, in the >> above, if `thisp` was declared `global`, also `global` in the genexp; >> if `nonlocal`, also `nonlocal`; else (almost always the case in real >> life) local to the containing code (meaning it would be local to the >> containing code, but nonlocal in the generated function). > Is it really more useful more often? I found no comprehensions of any kind in my code where binding expressions would actually be of use unless the name "leaked". Other code bases may, of course, yield different guesses. I'm not a "cram a lot of stuff on each line" kind of coder. But the point above remains: if they don't leak, contexts that want them to leak have no recourse. If they do leak, then the other uses would still work fine, but they'd possibly be annoyed by a leak they didn't want. >> No, I didn't have much use for `for` target names becoming magically >> local to invisible nested functions either, but I appreciate that it's >> less surprising overall. Using ":=" is much more strongly screaming >> "I'm going way out of my way to give a name to this thing, so please >> don't fight me by assuming I need to be protected from the >> consequences of what I explicitly asked for". > Personally, I'd still like to go back to := creating a statement-local > name, one that won't leak out of ANY statement. But the tide was > against that one, so I gave up on it. Part of that is because - as the existence of this thread attests to - we can't even control all the scopes gimmicks Python already has. So people are understandably terrified of adding even more ;-) From rosuav at gmail.com Sun May 6 22:35:22 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 7 May 2018 12:35:22 +1000 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: On Mon, May 7, 2018 at 12:13 PM, Nick Coghlan wrote: > So I have a different suggestion: perhaps it might make sense to propose > promoting a key handful of path manipulation operations to the status of > being builtins? > > Specifically, the ones I'd have in mind would be: > > - dirname (aka os.path.dirname) > - joinpath (aka os.path.join) These two are the basics of path manipulation. +1 for promoting to builtins, unless pathlib becomes core (which I suspect isn't happening). > - abspath (aka os.path.abspath) Only +0.5 on this, as it has to do file system operations. It may be worthwhile, instead, to promote os.path.normpath, which (like the others) is purely processing the string form of the path. It'll return the same value regardless of the file system. But yes, I'd much rather see path manipulation based on __file__ and builtins rather than injecting yet another module-level attribute that's derived from what we already have. ChrisA From marcidy at gmail.com Sun May 6 22:37:27 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Sun, 6 May 2018 19:37:27 -0700 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: > Personally, I'd still like to go back to := creating a statement-local > name, one that won't leak out of ANY statement. But the tide was > against that one, so I gave up on it. yes. I have some probably tangential to bad arguments but I'm going to make them anyways, because I think := makes the most sense along with SLNB. first, := vs post-hoc (e.g. where or given) base case: [ x for x in range(1) ] while obvious to all of us, reading left to right does not yield what x is till later. [ (x, y) for x in range(1) for y in range(1) ] doubly so. If x or y were defined above, it would not be clear until the right end if what contex they had. [ (x, y) for x in range(n) given y = f(n) ] i dont know what's the iterator till after 'for' [ (x, y:=f(n) for x in range(n) ] At a minimum, I learn immediately that y is not the iterator. Slightly less cognitive load. it's not that one is better, or that either is unfamiliar, it's about having to hold a "promise" in my working memory, vs getting an immediate assignment earlier. (it's a metric!) now my silly argument. ":" is like a "when" operator. if y==x: > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From marcidy at gmail.com Sun May 6 22:38:57 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Sun, 6 May 2018 19:38:57 -0700 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: On Sun, May 6, 2018 at 7:37 PM, Matt Arcidy wrote: >> Personally, I'd still like to go back to := creating a statement-local >> name, one that won't leak out of ANY statement. But the tide was >> against that one, so I gave up on it. > > yes. > > I have some probably tangential to bad arguments but I'm going to make > them anyways, because I think := makes the most sense along with SLNB. > > first, := vs post-hoc (e.g. where or given) > > base case: > [ x for x in range(1) ] > while obvious to all of us, reading left to right does not yield what > x is till later. > [ (x, y) for x in range(1) for y in range(1) ] doubly so. > If x or y were defined above, it would not be clear until the right > end if what contex they had. > > [ (x, y) for x in range(n) given y = f(n) ] > i dont know what's the iterator till after 'for' > > [ (x, y:=f(n) for x in range(n) ] > At a minimum, I learn immediately that y is not the iterator. > Slightly less cognitive load. > > it's not that one is better, or that either is unfamiliar, it's about > having to hold a "promise" in my working memory, vs getting an > immediate assignment earlier. (it's a metric!) > > now my silly argument. > ":" is like a "when" operator. > > if y==x: keyboard fail. if y == x: # execute when y == x for x in y: # execute when x in y class/def f: # execute when in f x := 1 # x = 1 when in local scope Stretch! told you ti was silly. > > > >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ From tim.peters at gmail.com Sun May 6 22:49:39 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 6 May 2018 21:49:39 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: <163386bccf8.2837.db5b03704c129196a4e9415e55413ce6@gmail.com> References: <163386bccf8.2837.db5b03704c129196a4e9415e55413ce6@gmail.com> Message-ID: [Tim] >> In a different thread I noted that I sometimes want to write code like >> this: >> ... >> while any(n % (thisp := p) == 0 for p in small_primes): >> n //= thisp >> ... [Ryan Gonzalez ] > Couldn't you just do: > > def first(it): > return next(it, None) > > while (item := first(p for p in small_primes if n % p == 0)): > # ... In the "different thread" I mentioned above, I already noted that kind of spelling. I'm not at a loss to think of many ways to spell it ;-) The point of this thread was intended to be about the semantics of binding expressions in comprehensions. For that purpose, the PEP noting that total = 0 progressive_sums = [total := total + value for value in data] fails too is equally relevant. Of course there are many possible ways to rewrite that too that would work. That doesn't change that the failing attempts "look like they should work", but don't, but could if the semantics of ":=" were defined differently inside magically-created anonymous lexically nested functions From ncoghlan at gmail.com Sun May 6 22:51:03 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 May 2018 12:51:03 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: On 7 May 2018 at 11:32, Tim Peters wrote: > I have a long history of arguing that magically created lexically > nested anonymous functions try too hard to behave exactly like > explicitly typed lexically nested functions, but that's the trendy > thing to do so I always lose ;-) You have the reasoning there backwards: implicitly nested scopes behave like explicitly nested scopes because that was the *easy* way for me to implement them in Python 3.0 (since I got to re-use all the pre-existing compile time and runtime machinery that was built to handle explicit lexical scopes). Everything else I tried (including any suggestions made by others on the py3k mailing list when I discussed the problems I was encountering) ran into weird corner cases at either compile time or run time, so I eventually gave up and proposed that the implicit scope using to hide the iteration variable name binding be a full nested closure, and we'd just live with the consequences of that. The sublocal scoping proposal in the earlier drafts of PEP 572 was our first serious attempt at defining a different way of doing things that would allow names to be hidden from surrounding code while still being visible in nested suites, and it broke people's brains to the point where Guido explicitly asked Chris to take it out of the PEP :) However, something I *have* been wondering is whether or not it might make sense to allow inline scoping declarations in comprehension name bindings. Then your example could be written: def ...: p = None while any(n % p for nonlocal p in small_primes): # p was declared as nonlocal in the nested scope, so our p points to the last bound value Needing to switch from "nonlocal p" to "global p" at module level would likely be slightly annoying, but also a reminder that the bound name is now visible as a module attribute. If any other form of comprehension level name binding does eventually get accepted, then inline scope declarations could similarly be used to hoist values out into the surrounding scope: rem = None while any((nonlocal rem := n % p) for nonlocal p in small_primes): # p and rem were declared as nonlocal in the nested scope, so our rem and p point to the last bound value Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun May 6 22:48:53 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 7 May 2018 12:48:53 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: On Mon, May 7, 2018 at 12:34 PM, Tim Peters wrote: >> That's a fair point. But there is another equally valid use-case for >> assignment expressions inside list comps: >> >> values = [y + 2 for x in iter if (y := f(x)) > 0] >> >> In this case, it's just as obvious that the name 'y' should be local >> to the comprehension, as 'x' is. > > There's a difference, though: if `y` "leaks", BFD. Who cares? ;-) > If `y` remains inaccessible, there's no way around that. That's Steve D'Aprano's view - why not just let them ALL leak? I don't like it though. >> 2) All names are local to their own scope. No names leak, and that >> includes names made with ":=". > > Saying "local to their own scope" _assumes_ what you're trying to > argue _for_ - it's circular. In fact it's impossible to know what the > user intends the scope to be. Sorry, I meant "local to the comprehension's scope". We can't know the user's intention. We have to create semantics before the user's intention even exists. > But the point above remains: if they don't leak, contexts that want > them to leak have no recourse. If they do leak, then the other uses > would still work fine, but they'd possibly be annoyed by a leak they > didn't want. Then let's revert the Py3 change that put comprehensions into functions, and put them back to the vanilla transformation: stuff = [x + 1 for x in iter if x % 3] stuff = [] for x in iter: if x % 3: stuff.append(x + 1) Now 'x' leaks as well, and it's more consistent with how people explain comprehensions. Is that a good thing? I don't think so. Having the iteration variable NOT leak means it's a self-contained unit that simply says "that thing we're iterating over". > Part of that is because - as the existence of this thread attests to - > we can't even control all the scopes gimmicks Python already has. So > people are understandably terrified of adding even more ;-) Part of it is just that people seem to be fighting for the sake of fighting. I'm weary of it, and I'm not going to debate this point with you. You want 'em to leak? No problem. Implement it that way and I'm not going to argue it. ChrisA From ncoghlan at gmail.com Sun May 6 23:11:48 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 May 2018 13:11:48 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: On 7 May 2018 at 12:51, Nick Coghlan wrote: > If any other form of comprehension level name binding does eventually get > accepted, then inline scope declarations could similarly be used to hoist > values out into the surrounding scope: > > rem = None > while any((nonlocal rem := n % p) for nonlocal p in small_primes): > # p and rem were declared as nonlocal in the nested scope, so > our rem and p point to the last bound value > Thinking about it a little further, I suspect the parser would reject "nonlocal name := ..." as creating a parsing ambiguity at statement level (where it would conflict with the regular nonlocal declaration statement). The extra keyword in the given clause would avoid that ambiguity problem: p = rem = None while any(rem for nonlocal p in small_primes given nonlocal rem = n % p): # p and rem were declared as nonlocal in the nested scope, so our p and rem refer to their last bound values Such a feature could also be used to make augmented assignments do something useful at comprehension scope: input_tally = 0 process_inputs(x for x in input_iter given nonlocal input_tally += x) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Sun May 6 23:15:46 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 6 May 2018 22:15:46 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: [Tim] >> There's a difference, though: if `y` "leaks", BFD. Who cares? ;-) >> If `y` remains inaccessible, there's no way around that. [Chris] > That's Steve D'Aprano's view - why not just let them ALL leak? I don't > like it though. I didn't suggest that. I'm not suggesting changing _any_ existing behavior (quite the contrary). Since ":=" would be brand new, there is no existing behavior for it. > ... > Sorry, I meant "local to the comprehension's scope". We can't know the > user's intention. We have to create semantics before the user's > intention even exists. Exactly. That's why I would like ":-=" to be defined from the start in a way that does least damage ;-) >> But the point above remains: if they don't leak, contexts that want >> them to leak have no recourse. If they do leak, then the other uses >> would still work fine, but they'd possibly be annoyed by a leak they >> didn't want. > Then let's revert the Py3 change that put comprehensions into > functions, and put them back to the vanilla transformation: Again, I'm not suggesting changing any existing behavior. > stuff = [x + 1 for x in iter if x % 3] > > stuff = [] > for x in iter: > if x % 3: > stuff.append(x + 1) > > Now 'x' leaks as well, and it's more consistent with how people > explain comprehensions. Is that a good thing? I don't think so. Having > the iteration variable NOT leak means it's a self-contained unit that > simply says "that thing we're iterating over". It's fine by me that for-target names don't leak. I didn't suggest changing that. >> Part of that is because - as the existence of this thread attests to - >> we can't even control all the scopes gimmicks Python already has. So >> people are understandably terrified of adding even more ;-) > Part of it is just that people seem to be fighting for the sake of > fighting. I'm weary of it, and I'm not going to debate this point with > you. You want 'em to leak? No problem. Implement it that way and I'm > not going to argue it. I'm more interested in real-life use cases than in arguments. My suggestion came from staring at my real-life use cases, where binding expressions in comprehensions would clearly be more useful if the names bound leaked. Nearly (but not all) of the time,, they're quite happy with that for-target names don't leak. Those are matters of observation rather than of argument. From ncoghlan at gmail.com Sun May 6 23:29:11 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 May 2018 13:29:11 +1000 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: On 7 May 2018 at 12:35, Chris Angelico wrote: > On Mon, May 7, 2018 at 12:13 PM, Nick Coghlan wrote: > > So I have a different suggestion: perhaps it might make sense to propose > > promoting a key handful of path manipulation operations to the status of > > being builtins? > > > > Specifically, the ones I'd have in mind would be: > > > > - dirname (aka os.path.dirname) > > - joinpath (aka os.path.join) > > These two are the basics of path manipulation. +1 for promoting to > builtins, unless pathlib becomes core (which I suspect isn't > happening). > pathlib has too many dependencies to ever make the type available as a builtin: $ ./python -X importtime -c pass 2>&1 | wc -l 25 $ ./python -X importtime -c "import pathlib" 2>&1 | wc -l 53 It's a good way of unifying otherwise scattered standard library APIs, but it's overkill if all you want to do is to calculate and resolve some relative paths. > > - abspath (aka os.path.abspath) > > Only +0.5 on this, as it has to do file system operations. It may be > worthwhile, instead, to promote os.path.normpath, which (like the > others) is purely processing the string form of the path. It'll return > the same value regardless of the file system. > My rationale for suggesting abspath() over any of its component parts is based on a few key considerations: - "make the given path absolute" is a far more common path manipulation activitity than "normalise the given path" (most users wouldn't even know what the latter means - the only reason *I* know what it means is because I looked up the docs for abspath while writing my previous comment) - __file__ isn't always absolute (especially in __main__), so you need to be able to do abspath(__file__) in order to reliably apply dirname() more than once - it can stand in for both os.getcwd() (when applied to the empty string or os.curdir) and os.path.normpath() (when the given path is already absolute), so we get 3 new bits of builtin functionality for the price of one new builtin name - I don't want to read "normpath(joinpath(getcwd(), relpath))" when I could be reading "abspath(relpath)" instead Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun May 6 23:33:33 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 6 May 2018 20:33:33 -0700 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: Spit-balling: how about __filepath__ as a lazily-created-on-first-access pathlib.Path(__file__)? Promoting os.path stuff to builtins just as pathlib is emerging as TOOWTDI makes me a bit uncomfortable. On Sun, May 6, 2018 at 8:29 PM, Nick Coghlan wrote: > On 7 May 2018 at 12:35, Chris Angelico wrote: >> >> On Mon, May 7, 2018 at 12:13 PM, Nick Coghlan wrote: >> > So I have a different suggestion: perhaps it might make sense to propose >> > promoting a key handful of path manipulation operations to the status of >> > being builtins? >> > >> > Specifically, the ones I'd have in mind would be: >> > >> > - dirname (aka os.path.dirname) >> > - joinpath (aka os.path.join) >> >> These two are the basics of path manipulation. +1 for promoting to >> builtins, unless pathlib becomes core (which I suspect isn't >> happening). > > > pathlib has too many dependencies to ever make the type available as a > builtin: > > $ ./python -X importtime -c pass 2>&1 | wc -l > 25 > $ ./python -X importtime -c "import pathlib" 2>&1 | wc -l > 53 > > It's a good way of unifying otherwise scattered standard library APIs, but > it's overkill if all you want to do is to calculate and resolve some > relative paths. > >> >> > - abspath (aka os.path.abspath) >> >> Only +0.5 on this, as it has to do file system operations. It may be >> worthwhile, instead, to promote os.path.normpath, which (like the >> others) is purely processing the string form of the path. It'll return >> the same value regardless of the file system. > > > My rationale for suggesting abspath() over any of its component parts is > based on a few key considerations: > > - "make the given path absolute" is a far more common path manipulation > activitity than "normalise the given path" (most users wouldn't even know > what the latter means - the only reason *I* know what it means is because I > looked up the docs for abspath while writing my previous comment) > - __file__ isn't always absolute (especially in __main__), so you need to be > able to do abspath(__file__) in order to reliably apply dirname() more than > once > - it can stand in for both os.getcwd() (when applied to the empty string or > os.curdir) and os.path.normpath() (when the given path is already absolute), > so we get 3 new bits of builtin functionality for the price of one new > builtin name > - I don't want to read "normpath(joinpath(getcwd(), relpath))" when I could > be reading "abspath(relpath)" instead > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Sun May 6 23:38:25 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 May 2018 13:38:25 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: On 7 May 2018 at 13:15, Tim Peters wrote: > [Tim] > >> There's a difference, though: if `y` "leaks", BFD. Who cares? ;-) > >> If `y` remains inaccessible, there's no way around that. > > Part of it is just that people seem to be fighting for the sake of > > fighting. I'm weary of it, and I'm not going to debate this point with > > you. You want 'em to leak? No problem. Implement it that way and I'm > > not going to argue it. > > I'm more interested in real-life use cases than in arguments. My > suggestion came from staring at my real-life use cases, where binding > expressions in comprehensions would clearly be more useful if the > names bound leaked. Nearly (but not all) of the time,, they're quite > happy with that for-target names don't leak. Those are matters of > observation rather than of argument. > The issue is that because name binding expressions are just ordinary expressions, they can't be defined as "in comprehension scope they do X, in other scopes they do Y" - they have to have consistent scoping semantics regardless of where they appear. However, it occurs to me that a nonlocal declaration clause could be allowed in comprehension syntax, regardless of how any nested name bindings are spelt: p = rem = None while any((rem := n % p) for p in small_primes nonlocal (p, rem)): # p and rem were declared as nonlocal in the nested scope, so our rem and p point to the last bound value I don't really like that though, since it doesn't read as nicely as being able to put the nonlocal declaration inline. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Sun May 6 23:46:00 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 6 May 2018 22:46:00 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: [Tim] >> I have a long history of arguing that magically created lexically >> nested anonymous functions try too hard to behave exactly like >> explicitly typed lexically nested functions, but that's the trendy >> thing to do so I always lose ;-) [Nick Coghlan ] > You have the reasoning there backwards: That's easy to believe - I also had a long history of resisting nested scopes at all ;-) > implicitly nested scopes behave like explicitly nested scopes because > that was the *easy* way for me to implement them in Python 3.0 (since > I got to re-use all the pre-existing compile time and runtime machinery > that was built to handle explicit lexical scopes). > Everything else I tried (including any suggestions made by others on the > py3k mailing list when I discussed the problems I was encountering) ran into > weird corner cases at either compile time or run time, so I eventually gave > up and proposed that the implicit scope using to hide the iteration variable > name binding be a full nested closure, and we'd just live with the > consequences of that. It's unfortunate that there are "consequences" at all. That kind of thing is done all the time in Lisp-ish languages, but they require explicitly declaring names' scopes. Python's "infer scope instead by looking for bindings" worked great when it had 3 scopes total, but keeps forcing "consequences" that may or may not be desired in a generally-nested-scopes world. > The sublocal scoping proposal in the earlier drafts of PEP 572 was our first > serious attempt at defining a different way of doing things that would allow > names to be hidden from surrounding code while still being visible in nested > suites, and it broke people's brains to the point where Guido explicitly > asked Chris to take it out of the PEP :) To which removal I was sympathetic, BTW. > However, something I *have* been wondering is whether or not it might make > sense to allow inline scoping declarations in comprehension name bindings. > Then your example could be written: > > def ...: > p = None > while any(n % p for nonlocal p in small_primes): > # p was declared as nonlocal in the nested scope, so our p > points to the last bound value Which more directly addresses the underlying problem: not really "binding expressions" per se, but the lack of control over scope decisions in comprehensions period. It's not at all that nested scopes are a poor model, it's that we have no control over what's local _to_ nested scopes the language creates. I'd say that's worth addressing in its own right, regardless of PEP 572's fate. BTW, the "p = None" there is annoying too ;-) > Needing to switch from "nonlocal p" to "global p" at module level would > likely be slightly annoying, but also a reminder that the bound name is now > visible as a module attribute. Or `nonlocal` could be taught that its use one level below `global` has an obvious meaning: global. > If any other form of comprehension level name binding does eventually get > accepted, then inline scope declarations could similarly be used to hoist > values out into the surrounding scope: > > rem = None > while any((nonlocal rem := n % p) for nonlocal p in small_primes): > # p and rem were declared as nonlocal in the nested scope, so > our rem and p point to the last bound value Right - as above, inline scope declarations would be applicable to all forms of comprehension-generated code. And to any other future construct that creates lexically nested functions. From ncoghlan at gmail.com Sun May 6 23:47:34 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 May 2018 13:47:34 +1000 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: On 7 May 2018 at 13:33, Nathaniel Smith wrote: > Spit-balling: how about __filepath__ as a > lazily-created-on-first-access pathlib.Path(__file__)? > > Promoting os.path stuff to builtins just as pathlib is emerging as > TOOWTDI makes me a bit uncomfortable. > pathlib *isn't* TOOWTDI, since it takes almost 10 milliseconds to import it, and it introduces a higher level object-oriented abstraction that's genuinely distracting when you're using Python as a replacement for shell scripting. While lazy imports could likely help with the import time problem (since 6.5 of those milliseconds are from importing fnmatch), I think there's also a legitimate argument for a two tier system here, where we say "If you can't handle your filesystem manipulation task with just open, dirname, abspath, and joinpath, then reach for the higher level pathlib abstraction". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Mon May 7 00:04:41 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 6 May 2018 23:04:41 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: [Nick Coghlan ] > The issue is that because name binding expressions are just ordinary > expressions, they can't be defined as "in comprehension scope they do X, in > other scopes they do Y" - they have to have consistent scoping semantics > regardless of where they appear. While I'm not generally a fan of arguments, I have to concede that's a really good argument :-) Of course their definition _could_ be context-dependent, but even I'll agree they shouldn't be. Never mind! > However, it occurs to me that a nonlocal declaration clause could be allowed > in comprehension syntax, regardless of how any nested name bindings are > spelt: > > p = rem = None > while any((rem := n % p) for p in small_primes nonlocal (p, rem)): > # p and rem were declared as nonlocal in the nested scope, so our > rem and p point to the last bound value > > I don't really like that though, since it doesn't read as nicely as being > able to put the nonlocal declaration inline. If the idea gets traction, I'm sure we'll see 100 other syntax ideas by the time I wake up again. From raymond.hettinger at gmail.com Mon May 7 00:07:01 2018 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 6 May 2018 23:07:01 -0500 Subject: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter In-Reply-To: <20180506110027.GS9562@ando.pearwood.info> References: <20180503063208.GE9562@ando.pearwood.info> <20180506110027.GS9562@ando.pearwood.info> Message-ID: <5AB92C0D-F9DC-4D15-8158-B06AD1BC8DA8@gmail.com> > On May 6, 2018, at 6:00 AM, Steven D'Aprano wrote: > > On Thu, May 03, 2018 at 04:32:09PM +1000, Steven D'Aprano wrote: > >> Maybe I'm slow today, but I'm having trouble seeing how to write this as >> a lambda. > > Yes, I was definitely having a "cannot brain, I have the dumb" day, > because it is not that hard to write using lambda. See discussion here: > > https://mail.python.org/pipermail/python-list/2018-May/732795.html > > If anything, the problem is a plethora of choices, where it isn't clear > which if any is the best way, or the One Obvious Way At one time, lambda was the one obvious way. Later, partial, itemgetter, attrgetter, and methodcaller were added to express common patterns for key-functions and map(). If needed, the zoo of lambda alternatives could be further extended to add a rpartial() function that partials from the right. That would have helped with Miki's example. Instead of: get = attrgetter('foo', None) return get(args) or get(config) or get(env) He could've written: get = rpartial(getattr, 'foo', None) return get(args) or get(config) or get(env) If itemgetter and attrgetter only did a single lookup, a default might make sense. However, that doesn't fit well with multiple and/or chained lookups where are number of options are possible. (See https://bugs.python.org/issue14384#msg316222 for examples and alternatives). Raymond From njs at pobox.com Mon May 7 00:33:03 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 6 May 2018 21:33:03 -0700 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: On Sun, May 6, 2018 at 8:47 PM, Nick Coghlan wrote: > On 7 May 2018 at 13:33, Nathaniel Smith wrote: >> >> Spit-balling: how about __filepath__ as a >> lazily-created-on-first-access pathlib.Path(__file__)? >> >> Promoting os.path stuff to builtins just as pathlib is emerging as >> TOOWTDI makes me a bit uncomfortable. > > pathlib *isn't* TOOWTDI, since it takes almost 10 milliseconds to import it, > and it introduces a higher level object-oriented abstraction that's > genuinely distracting when you're using Python as a replacement for shell > scripting. Hmm, the feedback I've heard from at least some folks teaching intro-python-for-scientists is like, "pathlib is so great for scripting that it justifies upgrading to python 3". How is data_path = __filepath__.parent / "foo.txt" more distracting than data_path = joinpath(dirname(__file__), "foo.txt") ? And the former gives you far more power: the full Path interface, not just 2-3 common operations. Import times are certainly a consideration, but I'm uncomfortable with jumping straight to adding things to builtins based on current import times, without at least exploring options for speeding that up... -n -- Nathaniel J. Smith -- https://vorpus.org From python-ideas at mgmiller.net Mon May 7 00:30:04 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Sun, 6 May 2018 21:30:04 -0700 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: <89bfbc03-a544-9b72-9045-2fe55c2aca01@mgmiller.net> On 2018-05-06 19:13, Nick Coghlan wrote: > Specifically, the ones I'd have in mind would be: > > - dirname (aka os.path.dirname) > - joinpath (aka os.path.join) > - abspath (aka os.path.abspath) Yes, I end up importing those in most scripts currently. Just "join" has worked fine, although I could imagine someone getting confused about it. -Mike From ncoghlan at gmail.com Mon May 7 01:36:08 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 May 2018 15:36:08 +1000 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: On 7 May 2018 at 14:33, Nathaniel Smith wrote: > On Sun, May 6, 2018 at 8:47 PM, Nick Coghlan wrote: > > On 7 May 2018 at 13:33, Nathaniel Smith wrote: > >> > >> Spit-balling: how about __filepath__ as a > >> lazily-created-on-first-access pathlib.Path(__file__)? > >> > >> Promoting os.path stuff to builtins just as pathlib is emerging as > >> TOOWTDI makes me a bit uncomfortable. > > > > pathlib *isn't* TOOWTDI, since it takes almost 10 milliseconds to import > it, > > and it introduces a higher level object-oriented abstraction that's > > genuinely distracting when you're using Python as a replacement for shell > > scripting. > > Hmm, the feedback I've heard from at least some folks teaching > intro-python-for-scientists is like, "pathlib is so great for > scripting that it justifies upgrading to python 3". > > How is > > data_path = __filepath__.parent / "foo.txt" > > more distracting than > > data_path = joinpath(dirname(__file__), "foo.txt") > Fair point :) In that case, perhaps the right answer here would be to adjust the opening examples section in the pathlib docs, showing some additional common operations like: _script_dir = Path(__file__).parent _launch_dir = Path.cwd() _home_dir = Path.home() And perhaps in a recipes section: def open_file_from_dir(dir_path, rel_path, *args, **kwds): return open(Path(dir_path) / rel_path, *args, **kwds) (Now that open() accepts path objects natively, I'm inclined to recommend that over the pathlib-specific method spelling) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.van.dorp at deonet.nl Mon May 7 05:33:48 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Mon, 7 May 2018 11:33:48 +0200 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: For what it's worth, i'm totally +1 on inline uses of global and nonlocal. As a related improvement, i'd also like it if "global x = 5" would be a legal statement. As a noob learning python, I was suprised to find out I couldn't and had to split it on two lines.(aside from a 9-hour course of C and some labview (which I totally hate), python was my first language and still the one im by far most proficient with.) 2018-05-07 6:04 GMT+02:00 Tim Peters : > [Nick Coghlan ] >> The issue is that because name binding expressions are just ordinary >> expressions, they can't be defined as "in comprehension scope they do X, in >> other scopes they do Y" - they have to have consistent scoping semantics >> regardless of where they appear. > > While I'm not generally a fan of arguments, I have to concede that's a > really good argument :-) > > Of course their definition _could_ be context-dependent, but even I'll > agree they shouldn't be. Never mind! > > >> However, it occurs to me that a nonlocal declaration clause could be allowed >> in comprehension syntax, regardless of how any nested name bindings are >> spelt: >> >> p = rem = None >> while any((rem := n % p) for p in small_primes nonlocal (p, rem)): >> # p and rem were declared as nonlocal in the nested scope, so our >> rem and p point to the last bound value >> >> I don't really like that though, since it doesn't read as nicely as being >> able to put the nonlocal declaration inline. > > If the idea gets traction, I'm sure we'll see 100 other syntax ideas > by the time I wake up again. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From Eloi.Gaudry at fft.be Mon May 7 05:58:37 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Mon, 7 May 2018 09:58:37 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: , Message-ID: I didn't mean to replace the current (debug) assert but I wanted to add another one that would allow to be switch on or off on production builds. The need for a new keyword (not syntax) comes from this difference. I cannot think of another example that would convince you of the benefit of having a specific keyword for such a runtime assert. I do believe that having such a feature in non-debug build is more than interesting but indeed needed. At some point, there should be a trade-off between to relying on a simple keyword/mechanism (for a runtime assert) versus the development cost of adding such a feature and maintaining it. Anyway, thanks for your feedback Serhiy, it helps. ________________________________ From: Python-ideas on behalf of Serhiy Storchaka Sent: Saturday, May 5, 2018 17:44 To: python-ideas at python.org Subject: Re: [Python-ideas] Runtime assertion with no overhead when not active 05.05.18 18:04, Eloi Gaudry ????: > By 'self-contained', I meant that using the assert keyword and its > expression is sufficient. An inline assertive expression as the one you > describe does not fulfill this assert requirement. Sufficient for what? And why writing with using the existing syntax is not sufficient? > My proposal is simply to extend the current assert to non-debug builds > and allow to switch it off/on at runtime. The main property of the assert statement is that has a zero overhead in non-debug run. If you remove this property, it will be not the assert statement, and you will not need a special syntax support for writing this runtime check. > The syntax example I gave illustrated what I meant by syntax aware. It doesn't illustrate why a new syntax is necessary. Or I can't understand this illustration. _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon May 7 06:44:36 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 7 May 2018 20:44:36 +1000 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: <20180507104436.GU9562@ando.pearwood.info> On Sun, May 06, 2018 at 09:33:03PM -0700, Nathaniel Smith wrote: > How is > > data_path = __filepath__.parent / "foo.txt" > > more distracting than > > data_path = joinpath(dirname(__file__), "foo.txt") Why are you dividing by a string? That's weird. [looks up the pathlib docs] Oh, that's why. It's still weird. So yes, its very distracting. First I have to work out what __filepath__ is, then I have to remember the differences between all the various flavours of pathlib.Path and suffer a moment or two of existential dread as I try to work out whether or not *this* specific flavour is the one I need. This might not matter for heavy users of pathlib, but for casual users, it's a big, intimidating API with: - an important conceptual difference between pure paths and concrete paths; - at least six classes; - about 50 or so methods and properties As far as performance goes, I don't think it matters that we could technically make pathlib imported lazily. Many people put all their pathname manipulations at the beginning of their script, so lazy or not, the pathlib module is going to be loaded *just after* startup, . For many scripts, this isn't going to matter, but for those who want to avoid the overhead of pathlib, making it lazy doesn't help. That just delays the overhead, it doesn't remove it. -- Steve From njs at pobox.com Mon May 7 07:42:00 2018 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 07 May 2018 11:42:00 +0000 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: <20180507104436.GU9562@ando.pearwood.info> References: <20180507104436.GU9562@ando.pearwood.info> Message-ID: On Mon, May 7, 2018, 03:45 Steven D'Aprano wrote: > On Sun, May 06, 2018 at 09:33:03PM -0700, Nathaniel Smith wrote: > > > How is > > > > data_path = __filepath__.parent / "foo.txt" > > > > more distracting than > > > > data_path = joinpath(dirname(__file__), "foo.txt") > > > Why are you dividing by a string? That's weird. > > [looks up the pathlib docs] > > Oh, that's why. It's still weird. > > So yes, its very distracting. > Well, yes, you do have to know the API to use it, and if you happen to have learned the os.path API but not the pathlib API then of course the os.path API will look more familiar. I'm not sure what this is supposed to prove. > First I have to work out what __filepath__ is, then I have to remember > the differences between all the various flavours of pathlib.Path > and suffer a moment or two of existential dread as I try to work out > whether or not *this* specific flavour is the one I need. This might not > matter for heavy users of pathlib, but for casual users, it's a big, > intimidating API with: > > - an important conceptual difference between pure paths and > concrete paths; > - at least six classes; > The docs could perhaps be more beginner friendly. For casual users, the answer is always "you want pathlib.Path". - about 50 or so methods and properties > Yeah, filesystems have lots of operations. That's why before pathlib users had to learn about os and os.path and shutil and glob and maybe some more I'm forgetting. > As far as performance goes, I don't think it matters that we could > technically make pathlib imported lazily. Many people put all their > pathname manipulations at the beginning of their script, so lazy or not, > the pathlib module is going to be loaded *just after* startup, . > > For many scripts, this isn't going to matter, but for those who want to > avoid the overhead of pathlib, making it lazy doesn't help. That just > delays the overhead, it doesn't remove it. > AFAIK were two situations where laziness has been mentioned in this thread: - my suggestion that we delay loading pathlib until someone accesses __filepath__. I don't actually know how to implement this so it was mostly intended to try to spur new ideas, but if we could do it, the point of the laziness would be so that scripts that didn't use __filepath__ wouldn't pay for it. - Nick's observation that pathlib could load faster if it loaded fnmatch lazily. Since this is only used for a few methods, this would benefit any script that didn't use those methods. (And for scripts that do need fnmatch's functionality, without pathlib they'd just be importing it directly, so pathlib importing it isn't really an extra cost.) It's true that laziness isn't a silver bullet, though, yeah. We should also look for ways to speed things up. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon May 7 07:42:10 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 7 May 2018 21:42:10 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: <20180507114208.GV9562@ando.pearwood.info> On Mon, May 07, 2018 at 12:48:53PM +1000, Chris Angelico wrote: > On Mon, May 7, 2018 at 12:34 PM, Tim Peters wrote: > > There's a difference, though: if `y` "leaks", BFD. Who cares? ;-) > > If `y` remains inaccessible, there's no way around that. > > That's Steve D'Aprano's view - why not just let them ALL leak? I don't > like it though. I know popular opinion is against me, and backward compatibility and all that, but I wish that generator expressions and comprehensions ran in their surrounding scope, like regular for statements. (Yes, I know that makes generator expressions tricky to implement. As the guy who doesn't have to implement it, I don't have to care :-) Calling it a "leak" assumes that it is a bad thing. I don't think it is a bad thing. It's not often that I want to check the value of a comprehension loop, but when I do, I have to tear the comprehension apart into a for-loop. Even if it is only temporarily, for debugging, then put the comprehension back together. The only time I can see it is a bad thing is if I blindly copy and paste a comprehension out of one piece of code and dump it into another piece of code without checking to see that it slots in nicely without blowing away existing variables. But if you're in the habit of blindly and carelessly pasting into your code base without looking it over, this is probably the least of your worries... *wink* But what's done is done, and while there are plenty of windmills I am willing to tilt at, reversing the comprehensions scope decision is not one of them. [...] > Sorry, I meant "local to the comprehension's scope". We can't know the > user's intention. We have to create semantics before the user's > intention even exists. Surely that's backwards? We ought to find out what people want before telling them that they can't have it :-) > > But the point above remains: if they don't leak, contexts that want > > them to leak have no recourse. If they do leak, then the other uses > > would still work fine, but they'd possibly be annoyed by a leak they > > didn't want. Indeed. > Then let's revert the Py3 change that put comprehensions into > functions, and put them back to the vanilla transformation: You know we can't do that. But we do have a choice with binding expressions. *Either way*, whatever we do, we're going to upset somebody, so we simply have to decide who that will be. Actually we have at least three choices: (1) Consistency ?ber Alles (whether foolish or not) Now that comprehensions are their own scope, be consistent about it. Binding expressions inside the comprehension will be contained to the comprehension. I'll hate it, but at least it is consistent and easy to remember: the entities which create a new scope are modules, classes, functions, plus comprehensions. That's going to cut out at least one motivating example though. See below. (2) Binding assignments are *defined* as "leaking", or as I prefer, defined as existing in the lexical scope that contains the comprehension. Hence: # module level [(x := a) for a in [98, 99]] assert x == 99 # class level class X: [(x := a) for a in [98, 99]] assert X.x == 99 # function level def func(): [(x := a) for a in [98, 99]] assert x == 99 Note that in *all* of these cases, the variable a does not "leak". This will helpfully support the "running total" use-case that began this whole saga: total = 0 running_totals = [(total := total + x) for x in [98, 99]] assert total == 197 (I have to say, given that this was THE motivating use-case that began this discussion, I think it is ironic and not a little sad that the PEP has evolved in a direction that leaves this use-case unsatisfied.) (3) A compromise: binding assignments are scoped local to the comprehension, but they are initialised from their surrounding scope. This would be similar to the way Lua works, as well as function parameter defaults. I have some vague ideas about implementation, but there's no point discussing that unless people actually are interested in this option. This will *half* satisfy the running-total example: total = 0 running_totals = [(total := total + x) for x in [98, 99]] assert total == 0 Guaranteed to generate at least two Stackoverflow posts a month complaining about it, but better than nothing :-) > Having > the iteration variable NOT leak means it's a self-contained unit that > simply says "that thing we're iterating over". Assuming there are no side-effects to any of the operations inside the comprehension. > Part of it is just that people seem to be fighting for the sake of > fighting. Them's fightin' words! *wink* Honestly Chris, I know this must be frustrating, but I'm not fighting for the sake of it, and I doubt Tim is either. I'm arguing because there are real use-cases which remain unmet if binding-variables inside comprehensions are confined to the comprehension. -- Steve From ncoghlan at gmail.com Mon May 7 07:48:46 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 May 2018 21:48:46 +1000 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: <20180507104436.GU9562@ando.pearwood.info> References: <20180507104436.GU9562@ando.pearwood.info> Message-ID: On 7 May 2018 at 20:44, Steven D'Aprano wrote: > First I have to work out what __filepath__ is, then I have to remember > the differences between all the various flavours of pathlib.Path > and suffer a moment or two of existential dread as I try to work out > whether or not *this* specific flavour is the one I need. This might not > matter for heavy users of pathlib, but for casual users, it's a big, > intimidating API with: > > - an important conceptual difference between pure paths and > concrete paths; > - at least six classes; > - about 50 or so methods and properties > Right, but that's why I think this may primarily be a docs issue, as for simple use cases, only one pathlib class matters, and that's "pathlib.Path" (which is the appropriate concrete path type for the running platform), together with its alternate constructors "Path.cwd()" and "Path.home()". So if you spell out the OP's original example with pathlib instead of os.path, you get: from pathlib import Path SRC_DIR = Path(__file__).parent And then SRC_DIR is a rich path object that will mostly let you avoid importing any of: - os - os.path - stat - glob - fnmatch > As far as performance goes, I don't think it matters that we could > technically make pathlib imported lazily. Many people put all their > pathname manipulations at the beginning of their script, so lazy or not, > the pathlib module is going to be loaded *just after* startup, . > It's the fnmatch and re module imports *inside* pathlib that may be worth making lazy, as those currently account for a reasonable chunk of the import time but are only used to implement PurePath.match and _WildcardSelector. That means making them lazy may allow folks to avoid those imports if they don't use any of the wildcard matching features. > For many scripts, this isn't going to matter, but for those who want to > avoid the overhead of pathlib, making it lazy doesn't help. That just > delays the overhead, it doesn't remove it. > That's fine - it's not uncommon for folks looking to minimise startup overhead to have to opt in to using a lower level API for exactly that reason. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon May 7 07:54:30 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 7 May 2018 21:54:30 +1000 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: <20180507104436.GU9562@ando.pearwood.info> Message-ID: On 7 May 2018 at 21:42, Nathaniel Smith wrote: > On Mon, May 7, 2018, 03:45 Steven D'Aprano wrote: > >> On Sun, May 06, 2018 at 09:33:03PM -0700, Nathaniel Smith wrote: >> >> > How is >> > >> > data_path = __filepath__.parent / "foo.txt" >> > >> > more distracting than >> > >> > data_path = joinpath(dirname(__file__), "foo.txt") >> >> >> Why are you dividing by a string? That's weird. >> >> [looks up the pathlib docs] >> >> Oh, that's why. It's still weird. >> >> So yes, its very distracting. >> > > Well, yes, you do have to know the API to use it, and if you happen to > have learned the os.path API but not the pathlib API then of course the > os.path API will look more familiar. I'm not sure what this is supposed to > prove. > I think it strongly suggests that *magically* introducing a path object into a module's namespace would be a bad idea, since it harms readability (since merely having `path` in the name isn't a strong enough hint that the object in question is a `pathlib.Path` instance). Your original point is still valid though: given the boilerplate reduction already available via "from pathlib import Path; _this_dir = Path(__file__).parent", it's the pathlib version that needs to be taken as the baseline for how verbose the status quo really is, not the lower level os.path API (no matter how accustomed some of us may still be to using the latter). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon May 7 08:38:51 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 7 May 2018 22:38:51 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: <20180507114208.GV9562@ando.pearwood.info> References: <20180507114208.GV9562@ando.pearwood.info> Message-ID: On Mon, May 7, 2018 at 9:42 PM, Steven D'Aprano wrote: > On Mon, May 07, 2018 at 12:48:53PM +1000, Chris Angelico wrote: >> On Mon, May 7, 2018 at 12:34 PM, Tim Peters wrote: > >> > There's a difference, though: if `y` "leaks", BFD. Who cares? ;-) >> > If `y` remains inaccessible, there's no way around that. >> >> That's Steve D'Aprano's view - why not just let them ALL leak? I don't >> like it though. > > I know popular opinion is against me, and backward compatibility and all > that, but I wish that generator expressions and comprehensions ran > in their surrounding scope, like regular for statements. > > (Yes, I know that makes generator expressions tricky to implement. As > the guy who doesn't have to implement it, I don't have to care :-) Yeah, it's really easy when you don't have to worry about how on earth you can implement the concept of "unexecuted block of code that can be executed later even after the surrounding context has returned, but which isn't a function". :) > (3) A compromise: binding assignments are scoped local to the > comprehension, but they are initialised from their surrounding scope. > > This would be similar to the way Lua works, as well as function > parameter defaults. > > I have some vague ideas about implementation, but there's no point > discussing that unless people actually are interested in this option. > > This will *half* satisfy the running-total example: > > total = 0 > running_totals = [(total := total + x) for x in [98, 99]] > assert total == 0 > > > Guaranteed to generate at least two Stackoverflow posts a month > complaining about it, but better than nothing :-) Does it HAVE to be initialised from the surrounding scope? What if the surrounding scope doesn't have that variable? stuff = [spam for x in items if (spam := f(x)) < 0] Is this going to NameError if you haven't defined spam? Or is the compiler to somehow figure out whether or not to pull in a value? Never mind about implementation - what are the semantics? >> Having >> the iteration variable NOT leak means it's a self-contained unit that >> simply says "that thing we're iterating over". > > Assuming there are no side-effects to any of the operations inside the > comprehension. Which is often the case. >> Part of it is just that people seem to be fighting for the sake of >> fighting. > > Them's fightin' words! *wink* > > Honestly Chris, I know this must be frustrating, but I'm not fighting > for the sake of it, and I doubt Tim is either. I'm arguing because there > are real use-cases which remain unmet if binding-variables inside > comprehensions are confined to the comprehension. I don't think you are and I don't think Tim is. But do you honestly want to say that about EVERY person in these threads? ChrisA From rosuav at gmail.com Mon May 7 08:45:30 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 7 May 2018 22:45:30 +1000 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: <20180507104436.GU9562@ando.pearwood.info> References: <20180507104436.GU9562@ando.pearwood.info> Message-ID: On Mon, May 7, 2018 at 8:44 PM, Steven D'Aprano wrote: > On Sun, May 06, 2018 at 09:33:03PM -0700, Nathaniel Smith wrote: > >> How is >> >> data_path = __filepath__.parent / "foo.txt" >> >> more distracting than >> >> data_path = joinpath(dirname(__file__), "foo.txt") > > > Why are you dividing by a string? That's weird. > > [looks up the pathlib docs] > > Oh, that's why. It's still weird. > > So yes, its very distracting. Isn't it strange how we can divide a path by a string, and that works, and we can take the remainder after you divide a string by a string, and that works as long as there's exactly one "%s" in the string, but nobody's interested in having "foo bar spam ham"/" " ==> ["foo","bar","spam","ham"] ? Just sayin', it ain't all that strange. ChrisA From storchaka at gmail.com Mon May 7 09:14:43 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 7 May 2018 16:14:43 +0300 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: 06.05.18 09:53, Yuval Greenfield ????: > I often need to reference a script's current directory. I end up writing: > > import os > SRC_DIR = os.path.dirname(__file__) > > > But I would prefer to have a new dunder for that. I propose: "__dir__". > I was wondering if others would find it convenient to include such a > shortcut. > > Here are some examples of dirname(__file__) in prominent projects. > > https://github.com/tensorflow/models/search?l=Python&q=dirname&type= > https://github.com/django/django/search?l=Python&q=dirname&type= > https://github.com/nose-devs/nose/search?l=Python&q=dirname&type= > > Reasons not to add __dir__: > * There already is one way to do it and it's clear and fairly short.. > * Avoid the bikeshed discussion of __dir__, __folder__, and other > candidates. * Additional burden on maintainers of import machinery. It is already too complex, and __file__ is set in multiple places. Don't forgot about third-party implementations. See also issue33277: "Deprecate __loader__, __package__, __file__, and __cached__ on modules" (https://bugs.python.org/issue33277). * More complex user code, because you have to handle different cases: - __file__ is set, but __dir__ is not set. - __file__ and __dir__ are set, but are not consistent. > Reasons to add it: Are you aware of importlib.resources? https://docs.python.org/3.7/whatsnew/3.7.html#importlib-resources From Eloi.Gaudry at fft.be Mon May 7 09:18:22 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Mon, 7 May 2018 13:18:22 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <20180505153039.GN9562@ando.pearwood.info> References: <20180505153039.GN9562@ando.pearwood.info> Message-ID: <1525699094.30503.5.camel@fft.be> On Sun, 2018-05-06 at 01:30 +1000, Steven D'Aprano wrote: > On Sat, May 05, 2018 at 08:04:45AM +0000, Eloi Gaudry wrote: > > > Hi folks, > > I intend to add a runtime assertion feature in python. > > I'm very interested in this idea, but I'm afraid your draft PEP > isn't? > very clear about this feature. Comments below. Thanks, I'll try to clarify this here. > > Rationale > > There is no runtime assert relying on python grammar available. > > For? > > diagnostics and/or debugging reasons, it would be valuable to add > > such? > > a feature. > > Why not this? > > ????if DEBUG_ENABLED: > ????????run_check(expression) > > where run_check is an ordinary function. If DEBUG_ENABLED is false, > the? > only runtime overhead is the name lookup and bool test, the > expression? > is not evaluated. for the sake of clarity across any Python application/extension for instance, and because having run_check defined as a statement seems actually the good thing to do. > > What features does your suggestion offer over this simple technique? Code readiness I think (kiss) : simpler and shorter. The simple technique would still be used for other purpose (assert that are context based for instance). > > [...] > > A brief example why avoiding evaluating the expression is needed > > to? > > avoid any overhead in case the runtime assert should be ignored. > > :: > > runtime_assert( 999 in { i:None for i in range( 10000000 ) } ) > > You say that you have been using this feature in production for two? > years. You should give real-life examples, not fake examples like > the? > above. That example would be better written as: > > ????runtime_assert(True) > > since no part of the test actually depends on either the enormous > dict? > or range objects that you build only to throw away. Real life examples would be more application-based I think. In my case, I used this for switching off some heaving computing expression evaluation at runtime. Those expression would have been responsible for sorting vectors and communicating them through MPI using some method from our framework, in order to finally return a boolean. > > Usage > > :: > > > > runtime_assert( expr ) > > > > #would result in? > > if expr and runtime_assert_active: > > ????print RuntimeAssertionError() > > Surely the test should be the other way around? > > ????if runtime_assert_active and expr: > ????????print(RuntimeAssertionError()) otherwise the expression is evaluated regardless of whether the runtime? > assertions are active or not. You are absolutely right :) > Please use Python 3 syntax for the PEP, as Python 2 is in > maintenance? > mode and will absolutely not get any new language features. Sure > > Some more observations: > > I like the idea of more control over runtime checking, but your PEP? > should justify why this is a good idea and why "just write more unit? > tests" isn't the solution. My main reasons is the one I gave in the context description of the PEP. Assert remains an efficient way for controlling your application behavior when running in production (as opposed to a debug build). Writing more unit tests is always a good practice, but : -?it does not necessarily cover all possible paths that are runtime dependent (an application relies on various app-parameters and architecture-parameters for instance) - it does not offer a solution for your extension/application consumers (that might be using your application in a very specific set of paths/set of decision/etc.). > What happens if the caller has defined their own function or > variable? > called runtime_assert? making runtime_assert a statement (as assert is already) would forbid such a definition, thus it would break backward compatibility (in those specific cases) for sure. > Can the caller customise the assertion error message? At this stage, no, but I don't know if this would be something difficult to add. > If you are changing the grammar, why not make runtime_assert a > statement? I'm not really changing the grammar, just making 'runtime_assert' a statement as 'assert' already is (and IMHO could have been named as debug_assert). There is no significant difference between both assert in term of implementation. > Can the caller set an application-wide global across all modules, or > do? > they have to set > > ????runtime_assert_active = True > > in every module they want to use runtime_assert? I would vote in favor of having two different ways to activate or not the runtime assertion evaluation: - the first one would consist of controlling?a global variable at the C-level (the current recipe in cPython) and would be used by python extension. This is the solution I am using personnaly. - the second would use a set method available in pure python (not implemented, to be done at C-level too I think) to easily (de)activate all the runtime asserts. From storchaka at gmail.com Mon May 7 09:24:05 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 7 May 2018 16:24:05 +0300 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: Message-ID: 07.05.18 12:58, Eloi Gaudry ????: > I didn't mean to replace the current (debug) assert but I wanted to add > another one that would allow to be switch on or off on production builds. > > The need for a new keyword (not syntax) comes from this difference. > > > ?I cannot think of another example that would convince you of the > benefit of having a specific keyword for such a runtime assert. ?I do > believe that?having such a feature in non-debug build is more than > interesting but indeed needed. I just don't understand why you need a new keyword for writing runtime checks. From steve at pearwood.info Mon May 7 10:26:36 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 8 May 2018 00:26:36 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> Message-ID: <20180507142636.GW9562@ando.pearwood.info> On Mon, May 07, 2018 at 10:38:51PM +1000, Chris Angelico wrote: > On Mon, May 7, 2018 at 9:42 PM, Steven D'Aprano wrote: [...] > > (3) A compromise: binding assignments are scoped local to the > > comprehension, but they are initialised from their surrounding scope. [...] > Does it HAVE to be initialised from the surrounding scope? What if the > surrounding scope doesn't have that variable? No. Then it's just an uninitialised local, and it is a NameError to try to evaluate it before something gets bound to it. > stuff = [spam for x in items if (spam := f(x)) < 0] > > Is this going to NameError if you haven't defined spam? It shouldn't be an error, because by the time the comprehension looks up the value of "spam", it has been bound by the binding-expression. > Or is the > compiler to somehow figure out whether or not to pull in a value? > Never mind about implementation - what are the semantics? Variable "spam" is not defined in any surrounding scope, so these ought to all be NameError (or UnboundLocalError): [spam for a in it] # obviously [(spam := spam + a) for a in it] [spam if True else (spam := a) for a in it] [spam for a in it if True or (spam := a)] They are errors because the name "spam" is unbound when you do a lookup. This is not an error, because the name is never looked up: [True or spam for a in it if True or (spam := a)] Although "spam" never gets bound, neither does it get looked up, so no error. The next one is also okay, because "spam" gets bound before the first lookup: [(spam := spam+1) for a in it if (spam := a*2) > 0] Here's a sketch of how I think locals are currently handled: 1. When a function is compiled, the compiler does a pass over the source and determines which locals are needed. 2. The compiler builds an array of slots, one for each local, and sets the initial value of the slot to "empty" (undefined). 3. When the function is called, if it tries reading from a local's slot which is still empty, it raises UnboundLocalError. (am I close?) Here's the change I would suggest: 2. The compiler builds an array of slots, one for each local: 2a. For locals that are the target of binding-expression only: - look up the target in the current scope (that is, not the comprehension's scope, but the scope that the comprehension is inside) using the normal lookup rules, as if you were compiling "lambda x=x: None" and needed the value of x; - if the target is undefined, then swallow the error and leave the slot as empty; - otherwise store a reference to that value in the slot. 2b. For all other locals, proceed as normal. > >> Part of it is just that people seem to be fighting for the sake of > >> fighting. > > > > Them's fightin' words! *wink* > > > > Honestly Chris, I know this must be frustrating, but I'm not fighting > > for the sake of it, and I doubt Tim is either. I'm arguing because there > > are real use-cases which remain unmet if binding-variables inside > > comprehensions are confined to the comprehension. > > I don't think you are and I don't think Tim is. But do you honestly > want to say that about EVERY person in these threads? I'm going to assume good faith, no matter the evidence :-) -- Steve From ericsnowcurrently at gmail.com Mon May 7 10:42:20 2018 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 7 May 2018 08:42:20 -0600 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: On Mon, May 7, 2018 at 7:14 AM, Serhiy Storchaka wrote: > * Additional burden on maintainers of import machinery. It is already too > complex, and __file__ is set in multiple places. Don't forgot about > third-party implementations. > > See also issue33277: "Deprecate __loader__, __package__, __file__, and > __cached__ on modules" (https://bugs.python.org/issue33277). Thanks for mentioning all this, Serhiy. :) That said, it *may* be worth considering a method on ModuleSpec (aka "dir()['__spec__']"). One (rough) possibility: def dirname(self): """Return the absolute path to the directory the module is in. This will return None for modules that do not have __file__ or where "directory" does not make sense (e.g. extension modules). """ if self.origin is None: # XXX ...or self.origin isn't a filename. return None import os.path # This "lazy" import is necessary in this case. filename = os.path.abspath(self.origin) return os.path.dirname(filename) Putting this on the module spec has several advantages: 1. __spec__ is a single source of truth (though tied to how a module was "found" rather than to anything that happened when "loaded") 2. encourages folks to rely on __spec__ (where we'd like to head, as demonstrated by the issue Serhiy referenced above) 3. does not add any overhead to import performance (i.e. cost only incurred when needed) 4. does not add complexity to any other part of the import machinery I'm not necessarily saying we should add ModuleSpec.dirname(), but it (or something like it) is what I'd advocate for *if* we were to add a convenient shortcut to the directory a module is in. FWIW, I'd probably use it. -eric From storchaka at gmail.com Mon May 7 11:15:59 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 7 May 2018 18:15:59 +0300 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: 07.05.18 17:42, Eric Snow ????: > I'm not necessarily saying we should add ModuleSpec.dirname(), but it > (or something like it) is what I'd advocate for *if* we were to add a > convenient shortcut to the directory a module is in. FWIW, I'd > probably use it. The question is *why* you need the absolute path to the directory the module is in? Taking into account the availability of importlib.resources etc. From Eloi.Gaudry at fft.be Mon May 7 11:37:59 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Mon, 7 May 2018 15:37:59 +0000 Subject: [Python-ideas] Allow mutable builtin types (optionally) Message-ID: <1525707472.12114.1.camel@fft.be> Hi, I'd like to bring back this discussion (from 2005, by Greg): https://bugs.python.org/issue1229239 Briefly, non-heap types cannot have their attributes changed by Python code. This makes sense for python builtin types, but not for the types defined in extension/modules. As we have been using this patch for the very same reasons and for more than 10 years, I think it might make sense to reconsider the discussion that Greg started. The main question at that time was "why not using a heap type instead ?" (because heap-type do not have this limitation). But I think that the right question could have been "why imposing such a restriction on non-heap types as defined in (non Python core) extensions ?". I mean, to my knowledge, there is no reason why a type should be allocated on the heap (https://docs.python.org/2/c-api/typeobj.html) to be able to change its attributes at Python level. I'm not saying that all non-heap types should be able to do so, just that it would make sense to allow this behavior, as an option (bit flag). At the implementation level, the changes needed are really limited (about a few lines): - Include/object.h - Objects/typeobject.c: Eloi From steve at pearwood.info Mon May 7 12:17:33 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 8 May 2018 02:17:33 +1000 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: <20180507104436.GU9562@ando.pearwood.info> Message-ID: <20180507161732.GX9562@ando.pearwood.info> On Mon, May 07, 2018 at 11:42:00AM +0000, Nathaniel Smith wrote: > On Mon, May 7, 2018, 03:45 Steven D'Aprano wrote: [...] > > So yes, its very distracting. > > Well, yes, you do have to know the API to use it, and if you happen to have > learned the os.path API but not the pathlib API then of course the os.path > API will look more familiar. I'm not sure what this is supposed to prove. Apologies for not being more clear. I'm arguing that for some people, your preferred syntax *is* more distracting and hard to comprehend than the more self-descriptive version with named functions. And its not just a matter of *learning* the API, it is a matter of using it so often that it ceases to look weird and looks natural.[1] There's a school of thought that says that operator overloading is a bad idea, that operators should never be overridden to do something aside from their common meaning (e.g. + should always mean plus, / should always mean numeric division, etc). >From that perspective, using / to mean something kinda-sorta like string concatenation, only path separator aware, is precisely the sort of thing that makes some people dislike operator overloading. http://cafe.elharo.com/programming/operator-overloading-considered-harmful/ https://blog.jooq.org/2014/02/10/why-everyone-hates-operator-overloading/ I am not going to go quite that far. I think operator overloading has its uses. I'm not going to argue that pathlib's use of / was "bad" or a mistake or harmful. I called it *weird* and that's as far as I'll go. I use lots of weird things, and I even like some of them. But if you think it isn't distracting, I think you are mistaken, and I think we ought to show caution in making it a built-in or an offical part of the module API. Your earlier comment (which I redacted): Hmm, the feedback I've heard from at least some folks teaching intro-python-for-scientists is like, "pathlib is so great for scripting that it justifies upgrading to python 3". felt to me awfully close to "pathlib! it's the future!" I know that's not what you said, or even meant, but I felt it was important to remind people that not everyone knows pathlib or finds its API clearer than the explicitly named functions of os.path. joinpath() may be longer than / but it is self-descriptive and easier to look up. help(joinpath) will tell you exactly what it does. help("/") is almost surely going to talk about numeric division, and it probably won't even mention strings or path objects at all. I say that because we've had + for string concatenation since way back in Python 1.5 or older, and yet as late as 3.6 help("+") still doesn't say a thing about string, list or tuple concatenation. As a Linux user, I'm used to paths containing slashes: $HOMEDIR/spam/eggs but putting quotes around the components looks unusual and is a hint that something usual is going on (namely a shell escape). But writing something like: HOMEDIR / "spam" / "eggs" doesn't even look like a path, just looks *wrong*. It looks like I'm escaping the wrong parts of the path: instead of escaping the spaces, I've escaped the parts with no spaces. It looks wrong as a Linux path, it looks wrong as a Windows path, and it looks wrong as division. So, yes, it is distracting. I'm not saying that everyone will feel the same way, or that I cannot or will not learn to accept / as I've learned to accept % for string interpolation despite it looking like percentage. But I'm saying it's not a slam-dunk useability win to move to pathlib. > > First I have to work out what __filepath__ is, then I have to remember > > the differences between all the various flavours of pathlib.Path > > and suffer a moment or two of existential dread as I try to work out > > whether or not *this* specific flavour is the one I need. This might not > > matter for heavy users of pathlib, but for casual users, it's a big, > > intimidating API with: > > > > - an important conceptual difference between pure paths and > > concrete paths; > > - at least six classes; > > > > The docs could perhaps be more beginner friendly. For casual users, the > answer is always "you want pathlib.Path". That might be what I want, but it isn't what I get: py> p = pathlib.Path('/') py> p PosixPath('/') I know what a PosixPath is. But the point is, even beginners have to deal with the complexity of the pathlib API the moment they print a path object in the interactive interpreter. [1] I've been using % for string interpolation for two decades now, and it still looks like a misplaced percentage sign every single time. -- Steve From rosuav at gmail.com Mon May 7 12:18:47 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 8 May 2018 02:18:47 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: <20180507142636.GW9562@ando.pearwood.info> References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: On Tue, May 8, 2018 at 12:26 AM, Steven D'Aprano wrote: > Here's a sketch of how I think locals are currently handled: > > 1. When a function is compiled, the compiler does a pass over the > source and determines which locals are needed. > > 2. The compiler builds an array of slots, one for each local, > and sets the initial value of the slot to "empty" (undefined). > > 3. When the function is called, if it tries reading from a local's > slot which is still empty, it raises UnboundLocalError. > > (am I close?) Yeah, I'm pretty sure that's all correct. > Here's the change I would suggest: > > 2. The compiler builds an array of slots, one for each local: > > 2a. For locals that are the target of binding-expression only: > > - look up the target in the current scope (that is, not the > comprehension's scope, but the scope that the comprehension > is inside) using the normal lookup rules, as if you were > compiling "lambda x=x: None" and needed the value of x; > > - if the target is undefined, then swallow the error and leave > the slot as empty; > > - otherwise store a reference to that value in the slot. > > 2b. For all other locals, proceed as normal. It's easy when you're not implementing things. I'm going to just say "sure, go for it", and also not implement it. Have fun, whoever goes in and tries to actually do the work... I don't think there's any other construct in Python that can replicate "a thing or the absence of a thing" in that way. For instance, there's no way to copy a value from one dict into another, or delete from the target dict if it isn't in the source (other than a long-hand). I've no idea how it would be implemented, and don't feel like finding out. ChrisA From brett at python.org Mon May 7 12:52:52 2018 From: brett at python.org (Brett Cannon) Date: Mon, 07 May 2018 16:52:52 +0000 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: On Mon, 7 May 2018 at 08:17 Serhiy Storchaka wrote: > 07.05.18 17:42, Eric Snow ????: > > I'm not necessarily saying we should add ModuleSpec.dirname(), but it > > (or something like it) is what I'd advocate for *if* we were to add a > > convenient shortcut to the directory a module is in. FWIW, I'd > > probably use it. > > The question is *why* you need the absolute path to the directory the > module is in? Taking into account the availability of > importlib.resources etc. > And just "why", and "how often"? I'm sure we have all done it before, but it isn't something that comes up *constantly*. And duplicating part of the details what __spec__.location contains just to save an import and a line to strip off the file seems unnecessary. Plus, this doesn't take into consideration the fact that not every module is going to exist in a directory (e.g. what if I loaded from a sqlite database?). IOW I'm -1 on this addition to modules as I don't think it's difficult enough or used enough to warrant adding the overhead of providing it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericfahlgren at gmail.com Mon May 7 13:02:17 2018 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Mon, 7 May 2018 10:02:17 -0700 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: <89bfbc03-a544-9b72-9045-2fe55c2aca01@mgmiller.net> References: <89bfbc03-a544-9b72-9045-2fe55c2aca01@mgmiller.net> Message-ID: On Sun, May 6, 2018 at 9:30 PM, Mike Miller wrote: > On 2018-05-06 19:13, Nick Coghlan wrote: > >> Specifically, the ones I'd have in mind would be: >> >> - dirname (aka os.path.dirname) >> - joinpath (aka os.path.join) >> - abspath (aka os.path.abspath) >> > Yes, I end up importing those in most scripts currently. Just "join" has > worked fine, although I could imagine someone getting confused about it. Our homebuilt pre-pathlib package has an 'abs_path' parameter in join, so that could easily eliminate the abspath function itself: >>> joinpath('.', abs_path=True) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon May 7 12:56:37 2018 From: brett at python.org (Brett Cannon) Date: Mon, 07 May 2018 16:56:37 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: Message-ID: On Mon, 7 May 2018 at 06:28 Serhiy Storchaka wrote: > 07.05.18 12:58, Eloi Gaudry ????: > > I didn't mean to replace the current (debug) assert but I wanted to add > > another one that would allow to be switch on or off on production builds. > > > > The need for a new keyword (not syntax) comes from this difference. > > > > > > I cannot think of another example that would convince you of the > > benefit of having a specific keyword for such a runtime assert. I do > > believe that having such a feature in non-debug build is more than > > interesting but indeed needed. > > I just don't understand why you need a new keyword for writing runtime > checks. > My question is how is this different to running with -O which leaves the assert statement out of the bytecode and so you avoid any run-time cost of the statement entirely? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 7 13:38:09 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 7 May 2018 10:38:09 -0700 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: I am convinced by Tim's motivation. I hadn't thought of this use case before -- I had mostly thought "local scope in a comprehension or generator expression is the locals of the synthetic function". But Tim's reasoning feels right. The only solution that makes sense to me is Steven's (2). (1) is what the PEP currently says and what Tim doesn't want; (3) has no precedent (function defaults don't really work like this) and just gets my hackles all up. (I can't even tell if Steven meant it as a serious proposal.) So let's see if we can change PEP 572 so that := inside a comprehension or generator expression al ways assigns to a variable in the containing scope. It may be inconsistent with the scope of the loop control variable, but it's consistent with uses of := outside comprehensions: [x := 0] [x := i for i in range(1)] both have the side effect of setting x to zero. I like that. There's one corner case (again) -- class scopes. If the containing scope is a function, everything's fine, we can use the existing closure machinery. If the containing scope is global, everything's fine too, we can treat it as a global. But if the containing scope is a class, we can't easily extend the current machinery. But this breakage is similar to the existing breakage with comprehensions in class scope that reference class variables: class C: hosts = ['boring', 'haring', 'tering'] full_hosts = [host + suffix for suffix in ('.cwi.nl', '.com') for host in hosts] Traceback (most recent call last): File "", line 1, in File "", line 3, in C File "", line 3, in NameError: name 'hosts' is not defined I propose to punt on this case. If we want to fix it we can fix it in a number of ways and the fix can easily apply to both getting and setting -- but this is a separate fix (and we should take it out of PEP 572). PS1. The various proposals that add random extra keywords to the syntax (like 'for nonlocal i') don't appeal to me at all. PS2. IIRC the reason we gave loop control variables their own scope was the poor imagination of many people when it comes to choosing loop control variable names. We had seen just too many examples of for x in something: ...lots of code using x... blah blah [x+1 for x in something else] ...some more code using x, broken... It's true that this can also happen with a for-loop statement nested inside the outer loop (and it does) but the case of a comprehension was easier to miss. I've never looked back. PS3. Comprehensions and generator expressions should be interchangeable. They just look too similar to have different semantics (and the possibly delayed execution of generator expressions is not an issue -- it's rare, and closure semantics make it work just fine). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 7 13:52:09 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 7 May 2018 10:52:09 -0700 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: Message-ID: On Mon, May 7, 2018 at 6:24 AM, Serhiy Storchaka wrote: > I just don't understand why you need a new keyword for writing runtime > checks. > Oh, that's pretty clear. The OP wants to be able to turn these checks off with some flag he can set/clear at runtime, and when it's off he doesn't want to incur the overhead of evaluating the check. The assert statement has the latter property, but you have to use -O to turn it off. He basically wants a macro so that runtime_assert() expands to if and (): raise AssertionError In Lisp this would be easy. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon May 7 14:19:58 2018 From: brett at python.org (Brett Cannon) Date: Mon, 07 May 2018 18:19:58 +0000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On Fri, 4 May 2018 at 05:07 Nick Coghlan wrote: > (Note: Guido's already told me off-list that he doesn't like the way this > spelling reads, but I wanted to share it anyway since it addresses one of > the recurring requests in the PEP 572 discussions for a more targeted > proposal that focused specifically on the use cases that folks had agreed > were reasonable potential use cases for inline assignment expressions. > > I'll also note that another potential concern with this specific proposal > is that even though "given" wasn't used as a term in any easily discovered > Python APIs back when I first wrote PEP 3150, it's now part of the > Hypothesis testing API, so adopting it as a keyword now would be markedly > more disruptive than it might have been historically) > > Recapping the use cases where the inline assignment capability received > the most agreement regarding being potentially more readable than the > status quo: > > 1. Making an "exactly one branch is executed" construct clearer than is > the case for nested if statements: > > if m := pattern.search(data): > ... > elif m := other_pattern.search(data): > ... > else: > ... > > 2. Replacing a loop-and-a-half construct: > > while m := pattern.search(remaining_data): > ... > > 3. Sharing values between filtering clauses and result expressions in > comprehensions: > > result = [(x, y, x/y) for x in data if (y := f(x))] > > The essence of the given clause concept would be to modify *these specific > cases* (at least initially) to allow the condition expression to be > followed by an inline assignment, of the form "given TARGET = EXPR". (Note: > being able to implement such a syntactic constraint is a general > consequence of using a ternary notation rather than a binary one, since it > allows the construct to start with an arbitrary expression, without > requiring that expression to be both the result of the operation *and* the > value bound to a name - it isn't unique to the "given" keyword specifically) > > While the leading keyword would allow TARGET to be an arbitrary assignment > target without much chance for confusion, it could also be restricted to > simple names instead (as has been done for PEP 572. > > With that spelling, the three examples above would become: > > # Exactly one branch is executed here > if m given m = pattern.search(data): > ... > elif m given m = other_pattern.search(data)): > ... > else: > ... > > # This name is rebound on each trip around the loop > while m given m = pattern.search(remaining_data): > ... > > # "f(x)" is only evaluated once on each iteration > result = [(x, y, x/y) for x in data if y given y = f(x)] > My brain wants to drop the variable name in front of 'given': if given m = pattern.search(data): while given m = pattern.search(remaining_data): Maybe it's because the examples use such a short variable name? if match given match = pattern.search(data): vs. if given match = pattern.search(data); Nope, I still like mine more. ;) -Brett > > Constraining the syntax that way (at least initially) would avoid poking > into any dark corners of Python's current scoping and expression execution > ordering semantics, while still leaving the door open to later making > "result given NAME = expr" a general purpose ternary operator that returns > the LHS, while binding the RHS to the given name as a side effect. > > Using a new keyword (rather than a symbol) would make the new construct > easier to identify and search for, but also comes with all the downsides of > introducing a new keyword. (Hence the not-entirely-uncommon suggestion of > using "with" for a purpose along these lines, which runs into a different > set of problems related to trying to use "with" for two distinct and > entirely unrelated purposes). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Mon May 7 14:23:41 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Mon, 07 May 2018 11:23:41 -0700 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: <20180507161732.GX9562@ando.pearwood.info> References: <20180507104436.GU9562@ando.pearwood.info> <20180507161732.GX9562@ando.pearwood.info> Message-ID: <5AF099AD.5040608@brenbarn.net> On 2018-05-07 09:17, Steven D'Aprano wrote: > I'm arguing that for some people, your preferred syntax*is* more > distracting and hard to comprehend than the more self-descriptive > version with named functions. And its not just a matter of*learning* > the API, it is a matter of using it so often that it ceases to look > weird and looks natural.[1] > But if you think it isn't distracting, I think you are mistaken, and I > think we ought to show caution in making it a built-in or an offical > part of the module API. As an aside, this has some parallels with the recent thread about "objectively quantifying readability". Saying things like "you are mistaken" implies that there is an objective ground truth about what is distracting and what is not. And personally I agree that there is such an objective ground truth, and that it is based on facts about human pyschology (although I don't think I agree with you about this particular case). Of course, there may be differences in how individuals react to things, but there is a real sense in which different syntaxes, constructs, etc., have something like a "mean level of confusion" which represents how easy to deal with people in general find them on average, and by which they can be meaningfully compared. I'm not sure how to proceed to uncover this (unless the PSF starts funding psychological experiments!), but I do think it would be good if we could find ways to get at something like hard evidence for claims about whether things "really are" distracting, readable, unreadable, intuitive, etc. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From brenbarn at brenbarn.net Mon May 7 14:33:43 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Mon, 07 May 2018 11:33:43 -0700 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: <5AF09C07.6060500@brenbarn.net> On 2018-05-06 18:32, Tim Peters wrote: > In a different thread I noted that I sometimes want to write code like this: > > while any(n % p == 0 for p in small_primes): > # divide p out - but what's p? > > But generator expressions hide the value of `p` that succeeded, so I > can't. `any()` and `all()` can't address this themselves - they > merely work with an iterable of objects to evaluate for truthiness, > and know nothing about how they're computed. If you want to identify > a witness (for `any()` succeeding) or a counterexample (for `all()` > failing), you need to write a workalike loop by hand. I agree that is a limitation, and I see from a later message in the thread that Guido finds it compelling, but personally I don't find that that particular case such a showstopper that it would tip the scales for me either way. If you have to write the workalike look that iterates and returns the missing value, so be it. That's not a big deal. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From vincent.maillol at gmail.com Mon May 7 14:42:37 2018 From: vincent.maillol at gmail.com (Vincent Maillol) Date: Mon, 7 May 2018 20:42:37 +0200 Subject: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter In-Reply-To: <5AB92C0D-F9DC-4D15-8158-B06AD1BC8DA8@gmail.com> References: <20180503063208.GE9562@ando.pearwood.info> <20180506110027.GS9562@ando.pearwood.info> <5AB92C0D-F9DC-4D15-8158-B06AD1BC8DA8@gmail.com> Message-ID: Hi everybody, > Have you encountered situations yourself where this would make a difference ? I need to get multiple objects in several dicts using the same set of keys. I wanted to use itemgetter to reduce the number of lines of code but I have mixed up getattr/dict.get that have default parameter with attrgetter/itemgetter. > At some point, we're really better off just using a lambda. I kept severals line with `.get('key', value)`, I find that is more readable. > Also, I'm concerned that about increasing the complexity of itemgetter() API to serve an occasional exotic use case rather that being easy to learn and remember for the common cases. I understand you, each additional parameter increases the cost of maintenance and update operator module will take a lot of work we should update c and python module, and ensure compatibility with pickle. I did it just to try it with itemgetter https://github.com/Maillol/cpython/compare/master...Add-default-parameter-to-operator-itemgetter I don't know if we add parameter `default` to itemgetter, getitem, attrgetter the API will become more complexe or more consistency, but I agree with you, it is an occasional use case and we can always use `my_dict.get`. 2018-05-07 6:07 GMT+02:00 Raymond Hettinger : > >> On May 6, 2018, at 6:00 AM, Steven D'Aprano wrote: >> >> On Thu, May 03, 2018 at 04:32:09PM +1000, Steven D'Aprano wrote: >> >>> Maybe I'm slow today, but I'm having trouble seeing how to write this as >>> a lambda. >> >> Yes, I was definitely having a "cannot brain, I have the dumb" day, >> because it is not that hard to write using lambda. See discussion here: >> >> https://mail.python.org/pipermail/python-list/2018-May/732795.html >> >> If anything, the problem is a plethora of choices, where it isn't clear >> which if any is the best way, or the One Obvious Way > > At one time, lambda was the one obvious way. Later, partial, itemgetter, attrgetter, and methodcaller were added to express common patterns for key-functions and map(). If needed, the zoo of lambda alternatives could be further extended to add a rpartial() function that partials from the right. That would have helped with Miki's example. Instead of: > > get = attrgetter('foo', None) > return get(args) or get(config) or get(env) > > He could've written: > > get = rpartial(getattr, 'foo', None) > return get(args) or get(config) or get(env) > > If itemgetter and attrgetter only did a single lookup, a default might make sense. However, that doesn't fit well with multiple and/or chained lookups where are number of options are possible. (See https://bugs.python.org/issue14384#msg316222 for examples and alternatives). > > > Raymond > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From tim.peters at gmail.com Mon May 7 14:48:19 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 7 May 2018 13:48:19 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: [Guido] > I am convinced by Tim's motivation. I hadn't thought of this use case before > -- I had mostly thought "local scope in a comprehension or generator > expression is the locals of the synthetic function". But Tim's reasoning > feels right. I'm trying very hard _not_ to reason. That is, I'm looking at code and trying to figure out what would actually work well, logic & consistency & preconceptions be damned. "Reasons" can be made up after the fact - which is how the world actually works regardless ;-) > The only solution that makes sense to me is Steven's (2). (1) is what the > PEP currently says and what Tim doesn't want; (3) has no precedent (function > defaults don't really work like this) and just gets my hackles all up. (I > can't even tell if Steven meant it as a serious proposal.) He doesn't want (3) either. I can channel him on that. > So let's see if we can change PEP 572 so that := inside a comprehension or > generator expression always assigns to a variable in the containing scope. While I don't have real use cases beyond that, given that much, "consistency" kicks in to suggest that: def f(): [x := 42 for x in range(1)] makes `x` local to `f` despite that x wasn't bound elsewhere in f's body. def f(): global x [x := 42 for x in range(1)] binds the global `x`. def f(): nonlocal x [x := 42 for x in range(1)] binds `x` in the closest-containing scope in which `x` is local. The last two act as if the declaration of `x` in `f` were duplicated at the start of the synthetic function. More succinctly, that `x := RHS` in a synthetic function "act the same" as `x = RHS` appearing in the scope directly containing the synthetic function. Does that generalize to class scope too? I don't know. I never use fancy stuff in class scopes, and have no idea how they work anymore. So long as "def method(self, ...)" continues to work, I'm happy ;-) > It may be inconsistent with the scope of the loop control variable, but it's > consistent with uses of := outside comprehensions: > > [x := 0] > [x := i for i in range(1)] > > both have the side effect of setting x to zero. I like that. And is what anyone would expect if they didn't think too much about it. ... [snipping stuff about class scope - nothing to add] ... > PS1. The various proposals that add random extra keywords to the syntax > (like 'for nonlocal i') don't appeal to me at all. They're appealing to the extent that "explicit is better than implicit" for people who actually understand how all this stuff is implemented. I don't know what percentage of Python programmers that is, though. I've certainly, e.g., seen many on Stackoverflow who can make a list comprehension work who couldn't distinguish a lexical scope from an avocado. The ":= is treated specially in comprehensions" idea is aimed more at them than at people who think invoking a synthetic anonymous lambda "is obvious". > PS2. IIRC the reason we gave loop control variables their own scope was the > poor imagination of many people when it comes to choosing loop control > variable names. We had seen just too many examples of > > for x in something: > ...lots of code using x... > blah blah [x+1 for x in something else] > ...some more code using x, broken... > > It's true that this can also happen with a for-loop statement nested inside > the outer loop (and it does) but the case of a comprehension was easier to > miss. I've never looked back. I don't want to change anything about any of that - already believe Python struck the best balance possible. > PS3. Comprehensions and generator expressions should be interchangeable. > They just look too similar to have different semantics (and the possibly > delayed execution of generator expressions is not an issue -- it's rare, and > closure semantics make it work just fine). Wholly agreed. From tim.peters at gmail.com Mon May 7 15:02:36 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 7 May 2018 14:02:36 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: <5AF09C07.6060500@brenbarn.net> References: <5AF09C07.6060500@brenbarn.net> Message-ID: [Tim] >> In a different thread I noted that I sometimes want to write code like >> this: >> >> while any(n % p == 0 for p in small_primes): >> # divide p out - but what's p? >> >> But generator expressions hide the value of `p` that succeeded, so I >> can't. `any()` and `all()` can't address this themselves - they >> merely work with an iterable of objects to evaluate for truthiness, >> and know nothing about how they're computed. If you want to identify >> a witness (for `any()` succeeding) or a counterexample (for `all()` >> failing), you need to write a workalike loop by hand. [Brendan Barnwell ] > I agree that is a limitation, and I see from a later message in the > thread that Guido finds it compelling, but personally I don't find that that > particular case such a showstopper that it would tip the scales for me > either way. If you have to write the workalike look that iterates and > returns the missing value, so be it. That's not a big deal. Guido didn't find it compelling: for that specific example to show `p` would require that for-loop targets "leak", and he remains opposed to that. I don't want that changed either. The issue instead is what the brand-new proposed ":=" should do, which isn't used in that example at all. Whether that specific example can be written in 500 other ways (of course it can) isn't really relevant. One of the ironies already noted is that PEP 572 gives an example of something that _won't_ work ("progressive_sums") which happens to be the very use case that started the current debate about assignment expressions to begin with. That raises the very same issue about ":=" that "the obvious" rewrite of my example at the top raises. Which suggests to me (& apparently to Guido too) that there may be a real issue here worth addressing. There are many use cases for binding expressions outside of synthetically generated functions. For PEP 572, it's the totality that will be judged, not just how they might work inside list comprehensions and generator expressions (the only topics in _this_ particular thread), let alone how they work in one specific example. From tim.peters at gmail.com Mon May 7 15:21:26 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 7 May 2018 14:21:26 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: [Tim] > While I don't have real use cases beyond that, given that much, > "consistency" kicks in to suggest that: > > def f(): > [x := 42 for x in range(1)] > > makes `x` local to `f` despite that x wasn't bound elsewhere in f's body. > > def f(): > global x > [x := 42 for x in range(1)] > > binds the global `x`. > > def f(): > nonlocal x > [x := 42 for x in range(1)] > > binds `x` in the closest-containing scope in which `x` is local. The > last two act as if the declaration of `x` in `f` were duplicated at > the start of the synthetic function. > > More succinctly, that `x := RHS` in a synthetic function "act the > same" as `x = RHS` appearing in the scope directly containing the > synthetic function. Oh, fudge - I wasn't trying to make a silly subtle point by reusing `x` as the `for` variable too. Pretend those all said "for i in range(1)" instead. Of course what happens if `x` is used in both places needs to be defined, but that's entirely beside the intended point _here_. From encukou at gmail.com Mon May 7 15:23:11 2018 From: encukou at gmail.com (Petr Viktorin) Date: Mon, 7 May 2018 15:23:11 -0400 Subject: [Python-ideas] Allow mutable builtin types (optionally) In-Reply-To: <1525707472.12114.1.camel@fft.be> References: <1525707472.12114.1.camel@fft.be> Message-ID: On 05/07/18 11:37, Eloi Gaudry wrote: > Hi, > > I'd like to bring back this discussion (from 2005, by Greg): > https://bugs.python.org/issue1229239 > > Briefly, non-heap types cannot have their > attributes changed by Python code. This makes sense for python builtin > types, but not for the types defined in extension/modules. > > As we have been using this patch for the very same reasons and for more > than 10 years, I think it might make sense to reconsider the discussion > that Greg started. > > The main question at that time was "why not using a heap type instead > ?" (because heap-type do not have this limitation). > > But I think that the right question could have been "why imposing such > a restriction on non-heap types as defined in (non Python core) > extensions ?". > > I mean, to my knowledge, there is no reason why a type should be > allocated on the heap (https://docs.python.org/2/c-api/typeobj.html) to > be able to change its attributes at Python level. One reason is sub-interpreter support: you can have multiple interpreters per process, and those shouldn't influence each other. (see https://docs.python.org/3/c-api/init.html#sub-interpreter-support) With heap types, each sub-interpreter can have its own copy of the type object. But with builtins, changes done in one interpreter would be visible in all the others. > I'm not saying that all non-heap types should be able to do so, just > that it would make sense to allow this behavior, as an option (bit > flag). > > At the implementation level, the changes needed are really limited > (about a few lines): > - Include/object.h > - Objects/typeobject.c: > > Eloi From mistersheik at gmail.com Mon May 7 15:56:01 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 7 May 2018 12:56:01 -0700 (PDT) Subject: [Python-ideas] string method count() In-Reply-To: References: <20180425213353.GA7400@ando.pearwood.info> Message-ID: Regular expressions are not just "an order of magnitude better"?they're asymptotically faster. See https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm for a non-regular-expression algorithm. On Thursday, April 26, 2018 at 5:45:20 AM UTC-4, Jacco van Dorp wrote: > > or build it yourself... > > def str_count(string, sub): > c = 0 > for c in range(len(string)-len(sub)): > if string[c:].startswith(sub): > c += 1 > return c > > (probably some optimizations possible...) > > Or in one line with a generator expression: > def str_count(string, sub): > return sum(string[c:].startswith(sub) for c in > range(len(string)-len(sub))) > > regular expressions would probably be at least an order of magnitude > better in speed, if it's a bottleneck to you. But pure python > implementation for this is a lot easier than it would be for the > current string.count(). > > 2018-04-26 8:57 GMT+02:00 Wes Turner >: > > > > > > On Wednesday, April 25, 2018, Steven D'Aprano > wrote: > >> > >> On Wed, Apr 25, 2018 at 11:22:24AM -0700, Julia Kim wrote: > >> > Hi, > >> > > >> > There?s an error with the string method count(). > >> > > >> > x = ?AAA? > >> > y = ?AA? > >> > print(x.count(y)) > >> > > >> > The output is 1, instead of 2. > >> > >> Are you proposing that there ought to be a version of count that looks > >> for *overlapping* substrings? > >> > >> When will this be useful? > > > > > > "Finding a motif in DNA" > > http://rosalind.info/problems/subs/ > > > > This is possible with re.find, re.finditer, re.findall, regex.findall(, > > overlapped=True), sliding window > > > https://stackoverflow.com/questions/2970520/string-count-with-overlapping-occurrences > > > > n-grams can be by indices or by value. > > count = len(indices) > > https://en.wikipedia.org/wiki/N-gram#Examples > > > > > https://en.wikipedia.org/wiki/String_(computer_science)#String_processing_algorithms > > > > https://en.wikipedia.org/wiki/Sequential_pattern_mining > > > >> > >> > >> -- > >> Steve > >> _______________________________________________ > >> Python-ideas mailing list > >> Python... at python.org > >> https://mail.python.org/mailman/listinfo/python-ideas > >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python... at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon May 7 16:35:54 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 7 May 2018 16:35:54 -0400 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: On 5/7/2018 1:38 PM, Guido van Rossum wrote: > I am convinced by Tim's motivation. I hadn't thought of this use case > before -- I had mostly thought "local scope in a comprehension or > generator expression is the locals of the synthetic function". But Tim's > reasoning feels right. > > The only solution that makes sense to me is Steven's (2). (1) is what > the PEP currently says and what Tim doesn't want; (3) has no precedent > (function defaults don't really work like this) and just gets my hackles > all up. (I can't even tell if Steven meant it as a serious proposal.) > > So let's see if we can change PEP 572 so that := inside a comprehension > or generator expression al ways assigns to a variable in the containing > scope. > > It may be inconsistent with the scope of the loop control variable, but > it's consistent with uses of := outside comprehensions: > > ? [x := 0] > ? [x := i for i in range(1)] > > both have the side effect of setting x to zero. I like that. If I am understanding correctly, this would also let one *intentionally 'leak' (export) the last value of the loop variable when wanted. [math.log(xlast:=x) for x in it if x > 0] print(xlast) > There's one corner case (again) -- class scopes. If the containing scope > is a function, everything's fine, we can use the existing closure > machinery. If the containing scope is global, everything's fine too, we > can treat it as a global. But if the containing scope is a class, we > can't easily extend the current machinery. But this breakage is similar > to the existing breakage with comprehensions in class scope that > reference class variables: > > ? class C: > ????? hosts = ['boring', 'haring', 'tering'] > ????? full_hosts = [host + suffix for suffix in ('.cwi.nl > ', '.com') for host in hosts] > > Traceback (most recent call last): > ? File "", line 1, in > ? File "", line 3, in C > ? File "", line 3, in > NameError: name 'hosts' is not defined This is a special case of the fact that no function called in class scope can access class variables, even if defined in the class scope. >>> class C: x = 0 def f(): return x z = f() Traceback (most recent call last): File "", line 1, in class C: File "", line 5, in C z = f() File "", line 4, in f return x NameError: name 'x' is not defined I would find it strange if only functions defined by a comprehension were given new class scope access. > PS1. The various proposals that add random extra keywords to the syntax > (like 'for nonlocal i') don't appeal to me at all. > > PS2. IIRC the reason we gave loop control variables their own scope was > the poor imagination of many people when it comes to choosing loop > control variable names. We had seen just too many examples of > > ? for x in something: > ????? ...lots of code using x... > ????? blah blah [x+1 for x in something else] > ????? ...some more code using x, broken... > > It's true that this can also happen with a for-loop statement nested > inside the outer loop (and it does) but the case of a comprehension was > easier to miss. I've never looked back. > > PS3. Comprehensions and generator expressions should be interchangeable. > They just look too similar to have different semantics (and the possibly > delayed execution of generator expressions is not an issue -- it's rare, > and closure semantics make it work just fine). To me, this is the prime justification for the 3.0 comprehension change. I currently see a comprehension as a specialized generator expression. A generator expression generalizes math set builder notation. If left 'raw', the implied function yields the values generated (what else could it do?). If a collection type is indicated by the fences and expression form, values are instead added to an anonymous instance thereof. -- Terry Jan Reedy From tim.peters at gmail.com Mon May 7 17:41:40 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 7 May 2018 16:41:40 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: [Terry Reedy ] > ... > If I am understanding correctly, this would also let one *intentionally > 'leak' (export) the last value of the loop variable when wanted. > > [math.log(xlast:=x) for x in it if x > 0] > print(xlast) Yup! You can do that today by emulating a nonlocal "cell" reference, but I don't recommend it ;-) xlast_cell = [None] [math.log(x) for x in it if x > 0 for xlast_cell[0] in [x]] print(xlast_cell[0]) From danilo.bellini at gmail.com Mon May 7 19:37:02 2018 From: danilo.bellini at gmail.com (Danilo J. S. Bellini) Date: Mon, 7 May 2018 20:37:02 -0300 Subject: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter In-Reply-To: <5AB92C0D-F9DC-4D15-8158-B06AD1BC8DA8@gmail.com> References: <20180503063208.GE9562@ando.pearwood.info> <20180506110027.GS9562@ando.pearwood.info> <5AB92C0D-F9DC-4D15-8158-B06AD1BC8DA8@gmail.com> Message-ID: (1) On 7 May 2018 at 01:07, Raymond Hettinger wrote: > He could've written: > > get = rpartial(getattr, 'foo', None) > return get(args) or get(config) or get(env) > That's somewhat hybrid, it would behave like: lambda obj: getattr(obj, "foo", None), but one might expect a "reversed partial" to have its arguments reversed as well, like: lambda obj: getattr(obj, None, "foo"). (2) Why getattr can't accept keyword arguments? >>> getattr_zero = partial(getattr, default=0) >>> getattr_zero({}, "blah") Traceback (most recent call last): File "", line 1, in TypeError: getattr() takes no keyword arguments Since partial/partialmethod can apply keyword arguments, allowing the "default=" keyword argument in getattr would be an alternative to the "rpartial". (3) At one time, lambda was the one obvious way. [...] > I really like lambdas, but there are some "closure gotchas" like: >>> for idx in indices: ... do_something(lambda seq: seq[idx]) If the lambda is stored somewhere to be called after the loop ends (or after its iteration), the seq[idx] would load the item with the last iterated index. This: >>> for idx in indices: ... do_something(itemgetter(idx)) would behave more like this instead: >>> for idx in indices: ... do_something((lambda i: (lambda seq: seq[i]))(idx)) Which I wouldn't call "the one obvious way". (4) If itemgetter and attrgetter only did a single lookup, a default might make > sense. However, that doesn't fit well with multiple and/or chained lookups > where are number of options are possible. (See https://bugs.python.org/ > issue14384#msg316222 for examples and alternatives). As attrgetter/itemgetter might get heterogeneus data, I would expect a per-get default, not [just] a global default. Perhaps something like: >>> itemgetter(-1, 0, -2, ... default0="first argument default", ... default1=["second", "argument", "default"], ... default={"global": "default"}, ... ) -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon May 7 20:15:52 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 8 May 2018 10:15:52 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: <20180508001552.GY9562@ando.pearwood.info> On Mon, May 07, 2018 at 10:38:09AM -0700, Guido van Rossum wrote: > The only solution that makes sense to me is Steven's (2). (1) is what the > PEP currently says and what Tim doesn't want; (3) has no precedent > (function defaults don't really work like this) and just gets my hackles > all up. (I can't even tell if Steven meant it as a serious proposal.) It doesn't get my hackles up as much as you, but its not really what I want. It's just a compromise between what I *don't* want (1), which fails to solve the original motivating example that started this discussion, and what Chris was pushing back against (2). > There's one corner case (again) -- class scopes. If the containing scope is > a function, everything's fine, we can use the existing closure machinery. > If the containing scope is global, everything's fine too, we can treat it > as a global. But if the containing scope is a class, we can't easily extend > the current machinery. But this breakage is similar to the existing > breakage with comprehensions in class scope that reference class variables: [...] > I propose to punt on this case. If we want to fix it we can fix it in a > number of ways and the fix can easily apply to both getting and setting -- > but this is a separate fix (and we should take it out of PEP 572). +1 Whether the current class behaviour is "broken" or desirable or somewhere in between, it is what we have now and its okay if binding expressions have the same behaviour. -- Steve From tim.peters at gmail.com Mon May 7 20:33:47 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 7 May 2018 19:33:47 -0500 Subject: [Python-ideas] A cute Python implementation of itertools.tee In-Reply-To: References: Message-ID: I posted this several weeks ago, just for fun - an obscure but surprisingly brief Python implementation of itertools.tee, sharing a single stream (as a singly linked list) among all the returned iterables. Didn't think about it again until today, when recent discussions of lexical scoping made me wonder "hmm - when's the last time I even used nonlocal?". Well, this was. And shortly thereafter, "but there's no need to keep track of `last` at all!" I have no idea where that thought came from :-) > def mytee(xs, n): > last = [None, None] > > def gen(it, mylast): > nonlocal last > while True: > mylast = mylast[1] > if not mylast: > mylast = last[1] = last = [next(it), None] > yield mylast[0] > > it = iter(xs) > return tuple(gen(it, last) for _ in range(n)) So, in fact, the inner function there can be replaced by the even briefer: def gen(it, mylast): while True: if mylast[1] is None: mylast[1] = [next(it), None] mylast = mylast[1] yield mylast[0] To make it more relevant to current discussions, collapsing the last two lines using a binding expression: yield (mylast := mylast[1])[0] isn't even slightly tempting. Most of the cases where it would be _possible_ to use binding expressions don't strike me as being _sensible_ places to use them - slamming together conceptually different tasks just because it's possible to do so. But because name bindings are so very common, that still leaves plenty where the transformation leaves the code clearer to my eyes (usually just a little, rarely a lot). There are no such cases in the code above. From tim.peters at gmail.com Mon May 7 20:54:17 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 7 May 2018 19:54:17 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: [Tim] > ... The ":= is treated specially in comprehensions" idea is aimed > more at them than at people who think invoking a synthetic > anonymous lambda "is obvious". It occurs to me that, while suggestive, this is an unhelpful way to express it. It's not at all that the semantics of ":=" change inside a listcomp/genexp, it's that the latter's idea of intended _scopes_ for names is made more nuanced (inside a synthetic function created to implement a listcomp/genexp, names bound by "=" are local; names bound by ":=" are nonlocal; names bound by both are "who cares?"- compiler-time error would be fine by me, or the first person to show a real use case wins). Regardless, the runtime implementation of ":=" remains the same everywhere. From barry at python.org Mon May 7 22:09:20 2018 From: barry at python.org (Barry Warsaw) Date: Mon, 7 May 2018 19:09:20 -0700 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: Yuval Greenfield wrote: > I often need to reference a script's current directory. I end up writing: > > import os > SRC_DIR = os.path.dirname(__file__) The question I have is, why do you want to reference the script's current directory? If the answer is to access other files in that directory, then please consider using importlib.resources (for Python 3.7) and importlib_resources (for Python 2.7, 3.4-3.6). __file__ simply isn't safe, and pkg_resources can be a performance killer. The problem of course is that if you're writing an application and *any* of your dependencies use either technique, you are going to pay for it. This is exactly why Brett and I wrote importlib.resources. We wanted a consistent API, that allows custom loaders to play along, and which is about as efficient as possible, uses Python's import machinery, and is safe for uses like zipapps. now-you-don't-have-to-attend-my-pycon-talk-ly y'rs, -Barry From gadgetsteve at live.co.uk Mon May 7 02:05:15 2018 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Mon, 7 May 2018 06:05:15 +0000 Subject: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc. Message-ID: In a lot of uses of os.walk it is desirable to skip version control directories, (which are usually hidden directories), to the point that almost all of the examples given look like: import os for root, dirs, files in os.walk(some_dir): if 'CVS' in dirs: dirs.remove('CVS') # or .svn or .hg etc. # do something... But of course there are many version control systems to the point that much of my personal code looks like, (note that I have to use a multitude of version control systems due to project requirements): import os vcs_dirs = ['.hg', '.svn', 'CSV', '.git', '.bz'] # Version control directory names I know for root, dirs, files in os.walk(some_dir): for dirname in vcs_dirs: dirs.remove(dirname) I am sure that I am missing many other version control systems but the one thing that all of the ones that I am familiar with default to creating their files in hidden directories. I know that the above sometimes hits problems on Windows if someone manually created a directory and you end up with abortions such as Csv\ or .SVN .... Since it could be argued that hidden directories are possibly more common than simlinks, (especially in the Windows world of course), and that hidden directories have normally been hidden by someone for a reason it seems to make sense to me to normally ignore them in directory traversal. Obviously there are also occasions when it makes sense to include VCS, or other hidden, directories files, (e.g. "Where did all of my disk space go?" or "delete recursively"), so I would like to suggest including in the os.walk family of functions an additional parameter to control skipping all hidden directories - either positively or negatively. Names that spring to mind include: * nohidden * nohidden_dirs * hidden * hidden_dirs This change could be made with no impact on current behaviour by defaulting to hidden=True (or nohidden=False) which would just about ensure that no existing code is broken or quite a few bugs in existing code could be quietly fixed, (and some new ones introduced), by defaulting to this behaviour. Since the implementation of os.walk has changed to use os.scandir which exposes the returned file statuses in the os.DirEntry.stat() the overhead should be minimal. An alternative would be to add another new function, say os.vwalk(), to only walk visible entries. Note that a decision would have to be made on whether to include such filtering when topdown is False, personally I am tempted to include the filtering so as to maintain consistency but ignoring the filter when topdown is False, (or if topdown is False and the hidden behaviour is unspecified), might make sense if the skipping of hidden directories becomes the new default (then recursively removing files & directories would still include processing hidden items by default). If this receives a positive response I would be happy to undertake the effort involved in producing a PR. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From j.van.dorp at deonet.nl Tue May 8 02:51:48 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Tue, 8 May 2018 08:51:48 +0200 Subject: [Python-ideas] Delivery Status Notification (Failure) In-Reply-To: <00000000000017568b056bac3125@google.com> References: <20180425213353.GA7400@ando.pearwood.info> <00000000000017568b056bac3125@google.com> Message-ID: 2018-05-07 21:56 GMT+02:00 Neil Girdhar : > Regular expressions are not just "an order of magnitude better"?they're > asymptotically faster. See > https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm > for a non-regular-expression algorithm. Hence my >> [Jacco wrote, capitalized important words] >> regular expressions would probably be AT LEAST an order of magnitude >> better in speed, if it's a bottleneck to you. But pure python >> implementation for this is a lot easier than it would be for the >> current string.count(). >> But I think my point stands that that's what you need to do if speed is an issue, and python code is fine when it isn't. Also, intersting read. Thanks. Jacco From steve at pearwood.info Tue May 8 03:05:23 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 8 May 2018 17:05:23 +1000 Subject: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc. In-Reply-To: References: Message-ID: <20180508070522.GA9562@ando.pearwood.info> On Mon, May 07, 2018 at 06:05:15AM +0000, Steve Barnes wrote: > In a lot of uses of os.walk it is desirable to skip version control > directories, (which are usually hidden directories), to the point that > almost all of the examples given look like: > > import os > for root, dirs, files in os.walk(some_dir): > if 'CVS' in dirs: > dirs.remove('CVS') # or .svn or .hg etc. > # do something... I would write something like: for root, dirs, files in filter(ignorable, os.walk(some_dir)): ... where ignorable() is a function that returns False for whatever you want to ignore. I don't think we can possibly agree on a single definition of "ignorable". This could include any combination of: - dot files; - files with the invisible bit set, for file systems which support that; - files within certain directories; - files ending in ~ (backup files); - files with certain extensions; or more. Possibly this is a good use-case for composible functions, so we could have a set of pre-built filters: ignorable = invisible + dotfiles + directories('.git', '.hg') + extensions('~', '.pdf') but that sounds like it ought to be a separate module, not built in. -- Steve From ubershmekel at gmail.com Tue May 8 03:12:35 2018 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Tue, 08 May 2018 07:12:35 +0000 Subject: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc. In-Reply-To: References: Message-ID: On Mon, May 7, 2018 at 9:44 PM Steve Barnes wrote: > Since the implementation of os.walk has changed to use os.scandir which > exposes the returned file statuses in the os.DirEntry.stat() the > overhead should be minimal. > > An alternative would be to add another new function, say os.vwalk(), to > only walk visible entries. > On Tue, May 8, 2018 at 12:06 AM Steven D'Aprano wrote: > I would write something like: > for root, dirs, files in filter(ignorable, os.walk(some_dir)): I agree with Steven with regards to `filter` needing to be flexible. If you want to avoid duplicate `stat` calls, you'll probably write: import os import stat def is_hidden(st): return bool(st.st_file_attributes & stat.FILE_ATTRIBUTE_HIDDEN) def visible_walk(path): for entry in os.scandir(path): if entry.is_dir(): if not is_hidden(entry.stat()): yield from visible_walk(entry.path) else: if not is_hidden(entry.stat()): yield entry.path Then you can decide whether you want to ignore hidden files or just hidden directories. The variations for such a need are many. So it makes sense to leave any specific filtering need outside of the standard library. A PyPI package with a few standard filtered walks could be a nice exploration for this idea. Cheers, Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.van.dorp at deonet.nl Tue May 8 03:17:13 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Tue, 8 May 2018 09:17:13 +0200 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: > [Tim] > (inside a synthetic function created to > implement a listcomp/genexp, names bound by "=" are local; names bound > by ":=" are nonlocal; names bound by both are "who cares?"- > compiler-time error would be fine by me, or the first person to show a > real use case wins). Wait, you can't use = in a listcomp, right ? Or are you talking about the implementation hidden to the casual user ? I thought letting := bind to the surrounding scope was fine basically because it's currently not possible, so therefore there would be no syntactic ambiguity, and it'd actually do what people would expect. Jacco From Eloi.Gaudry at fft.be Tue May 8 03:29:26 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Tue, 8 May 2018 07:29:26 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: Message-ID: <1525764564.24469.3.camel@fft.be> On Mon, 2018-05-07 at 10:52 -0700, Guido van Rossum wrote: > On Mon, May 7, 2018 at 6:24 AM, Serhiy Storchaka > wrote: > > I just don't understand why you need a new keyword for writing > > runtime checks. > > Oh, that's pretty clear. The OP wants to be able to turn these checks > off with some flag he can set/clear at runtime, and when it's off he > doesn't want to incur the overhead of evaluating the check. The > assert statement has the latter property, but you have to use -O to > turn it off. He basically wants a macro so that > > ? runtime_assert() > > expands to > > ? if and (): > ????? raise AssertionError > > In Lisp this would be easy. :-) and he already has a diff ready for review if needed (basically very similar to the current 'assert' implementation :) From vincent.maillol at gmail.com Tue May 8 04:13:43 2018 From: vincent.maillol at gmail.com (Vincent Maillol) Date: Tue, 8 May 2018 10:13:43 +0200 Subject: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter In-Reply-To: References: <20180503063208.GE9562@ando.pearwood.info> <20180506110027.GS9562@ando.pearwood.info> <5AB92C0D-F9DC-4D15-8158-B06AD1BC8DA8@gmail.com> Message-ID: > > As attrgetter/itemgetter might get heterogeneus data, I would expect a > per-get default, not [just] a global default. > +1 Perhaps something like: > > >>> itemgetter(-1, 0, -2, > ... default0="first argument default", > ... default1=["second", "argument", "default"], > ... default={"global": "default"}, > ... ) > The keywords suffixed by indice are uncommon. Maybe we can use dedicated object. >>> itemgetter( ... itemgetter.WithDefault(-1, "first argument default"), ... itemgetter.WithDefault(0, ["second", "argument", "default"]) ... -2 # has no default value ... ) Another possibility is that itemgetter becomes an object with `add_default` method. >>> getter = itemgetter(-1, 0, 2) >>> getter.add_default(0, "first argument default") >>> getter.add_default(2, "third argument default") or simply default parameter should be a list >>> itemgetter(-1, 0, 2, ... default=["first", itemgetter.NoDefault, "third"]) 2018-05-08 1:37 GMT+02:00 Danilo J. S. Bellini : > (1) > > On 7 May 2018 at 01:07, Raymond Hettinger > wrote: > >> He could've written: >> >> get = rpartial(getattr, 'foo', None) >> return get(args) or get(config) or get(env) >> > > That's somewhat hybrid, it would behave like: > > lambda obj: getattr(obj, "foo", None), > > but one might expect a "reversed partial" to have its arguments reversed > as well, like: > > lambda obj: getattr(obj, None, "foo"). > > (2) > > Why getattr can't accept keyword arguments? > > >>> getattr_zero = partial(getattr, default=0) > >>> getattr_zero({}, "blah") > Traceback (most recent call last): > File "", line 1, in > TypeError: getattr() takes no keyword arguments > > Since partial/partialmethod can apply keyword arguments, allowing the > "default=" keyword argument in getattr would be an alternative to the > "rpartial". > > (3) > > At one time, lambda was the one obvious way. [...] >> > > I really like lambdas, but there are some "closure gotchas" like: > > >>> for idx in indices: > ... do_something(lambda seq: seq[idx]) > > If the lambda is stored somewhere to be called after the loop ends (or > after its iteration), the seq[idx] would load the item with the last > iterated index. This: > > >>> for idx in indices: > ... do_something(itemgetter(idx)) > > would behave more like this instead: > > >>> for idx in indices: > ... do_something((lambda i: (lambda seq: seq[i]))(idx)) > > Which I wouldn't call "the one obvious way". > > (4) > > If itemgetter and attrgetter only did a single lookup, a default might >> make sense. However, that doesn't fit well with multiple and/or chained >> lookups where are number of options are possible. (See >> https://bugs.python.org/issue14384#msg316222 for examples and >> alternatives). > > As attrgetter/itemgetter might get heterogeneus data, I would expect a > per-get default, not [just] a global default. > Perhaps something like: > > >>> itemgetter(-1, 0, -2, > ... default0="first argument default", > ... default1=["second", "argument", "default"], > ... default={"global": "default"}, > ... ) > > -- > Danilo J. S. Bellini > --------------- > "*It is not our business to set up prohibitions, but to arrive at > conventions.*" (R. Carnap) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Eloi.Gaudry at fft.be Tue May 8 03:26:47 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Tue, 8 May 2018 07:26:47 +0000 Subject: [Python-ideas] Allow mutable builtin types (optionally) In-Reply-To: References: <1525707472.12114.1.camel@fft.be> Message-ID: <1525764405.24469.1.camel@fft.be> On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote: > On 05/07/18 11:37, Eloi Gaudry wrote: > > I mean, to my knowledge, there is no reason why a type should be > > allocated on the heap (https://docs.python.org/2/c-api/typeobj.html > > ) to > > be able to change its attributes at Python level. > > One reason is sub-interpreter support: you can have multiple? > interpreters per process, and those shouldn't influence each other. > (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor > t) > > With heap types, each sub-interpreter can have its own copy of the > type? > object. But with builtins, changes done in one interpreter would be? > visible in all the others. Yes, this could be a reason, but if you don't rely on such a feature neither implicitly nor explicitly ? I mean, our types are built-in and should be considered as immutable across interpreters. And we (as most users I guess) are only running one interpreter. In case several intepreters are used, it would make sense to have a non-heap type that would be seen as a singleton across all of them, no ? From phd at phdru.name Tue May 8 05:31:20 2018 From: phd at phdru.name (Oleg Broytman) Date: Tue, 8 May 2018 11:31:20 +0200 Subject: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc. In-Reply-To: References: Message-ID: <20180508093120.w7hkdzemzldf74tc@phdru.name> Hi! On Tue, May 08, 2018 at 07:12:35AM +0000, Yuval Greenfield wrote: > If you > want to avoid duplicate `stat` calls, you'll probably write: > > import os > import stat > def is_hidden(st): > return bool(st.st_file_attributes & stat.FILE_ATTRIBUTE_HIDDEN) > def visible_walk(path): > for entry in os.scandir(path): > if entry.is_dir(): > if not is_hidden(entry.stat()): > yield from visible_walk(entry.path) > else: > if not is_hidden(entry.stat()): > yield entry.path So anyone who wants to filter os.walk() must reimplement os.walk() themselves instead of passing something like filter_dir and filter_file (or accept_dir/accept_file) to os.walk()? Kind of painful, no? > Cheers, > Yuval Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From steve at pearwood.info Tue May 8 06:17:32 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 8 May 2018 20:17:32 +1000 Subject: [Python-ideas] Add "default" kw argument to operator.itemgetter and operator.attrgetter In-Reply-To: References: <20180503063208.GE9562@ando.pearwood.info> <20180506110027.GS9562@ando.pearwood.info> <5AB92C0D-F9DC-4D15-8158-B06AD1BC8DA8@gmail.com> Message-ID: <20180508101732.GB9562@ando.pearwood.info> On Tue, May 08, 2018 at 10:13:43AM +0200, Vincent Maillol wrote: > Perhaps something like: > > > > > >>> itemgetter(-1, 0, -2, > > ... default0="first argument default", > > ... default1=["second", "argument", "default"], > > ... default={"global": "default"}, > > ... ) Sorry Vincent, but I think that's precisely the sort of overly complicated API that Raymond was worried about when he voted against this proposal. A single default value that applies when the item (or key) isn't present is easy to comprehend. But this extention is, I think, a case of an over-engineered, excessively intricate solution, and I doubt it solves any real problem. And I'm not even sure I know what it means. You have keyword parameters default0, default1, and then *default* with no suffix. Is that a typo for default2, or is it supposed to be a default default, the default to use when no default is specified? > The keywords suffixed by indice are uncommon. Maybe we can use dedicated > object. > > >>> itemgetter( > ... itemgetter.WithDefault(-1, "first argument default"), > ... itemgetter.WithDefault(0, ["second", "argument", "default"]) > ... -2 # has no default value > ... ) I'm afraid there's no elegance to this design either. I've had many occasions where I want to get an item, falling back on a default if it is not present. With dicts this is common enough that we have dict.get(). It is less convenient with lists, and I'd like to see itemgetter support that case. But I cannot think of any case where I have needed or wanted to extract item 5 with "spam" as the default item 2 with no default item 3 with "eggs" as the default (for example), let alone that this is *so common* that I'd rather read the complicated documenation to work out how to use the function, rather than just write a short helper: def extract(sequence): return ("spam" if len(sequence) < 5 else sequence[5], sequence[2], "eggs" if len(sequence) < 5 else sequence[3], ) Raymond and Serhey are right: beyond a certain point, we should just write a custom function (whether lambda or not). We just disagree on which side of that point the single-default value case is, but I think we will agree that your example is so far past the point that we cannot even see it from here :-) -- Steve From mertz at gnosis.cx Tue May 8 08:00:51 2018 From: mertz at gnosis.cx (David Mertz) Date: Tue, 08 May 2018 12:00:51 +0000 Subject: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc. In-Reply-To: <20180508093120.w7hkdzemzldf74tc@phdru.name> References: <20180508093120.w7hkdzemzldf74tc@phdru.name> Message-ID: I like the idea. I think an argument to os.walk() is the simplest option for most users. But per some comments, "hidden" is actually more subtle than the filesystem bit sometimes. I.e. dot-files, ~ suffix, maybe .bak, etc. I'd suggest meeting the ideas slightly and making the new argument 'filter' or 'skip' that takes a callable. Default to None, but provide an os.is_hidden that users don't need to figure out how to implement. E.g. os.walk(PATH, skip=os.is_hidden) os.walk(PATH, skip=lambda entry: entry.name.endswith(('~', '.bak', '.tmp'))) On Tue, May 8, 2018, 5:47 AM Oleg Broytman wrote: > Hi! > > On Tue, May 08, 2018 at 07:12:35AM +0000, Yuval Greenfield < > ubershmekel at gmail.com> wrote: > > If you > > want to avoid duplicate `stat` calls, you'll probably write: > > > > import os > > import stat > > def is_hidden(st): > > return bool(st.st_file_attributes & stat.FILE_ATTRIBUTE_HIDDEN) > > def visible_walk(path): > > for entry in os.scandir(path): > > if entry.is_dir(): > > if not is_hidden(entry.stat()): > > yield from visible_walk(entry.path) > > else: > > if not is_hidden(entry.stat()): > > yield entry.path > > So anyone who wants to filter os.walk() must reimplement os.walk() > themselves instead of passing something like filter_dir and filter_file > (or accept_dir/accept_file) to os.walk()? Kind of painful, no? > > > Cheers, > > Yuval > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Eloi.Gaudry at fft.be Tue May 8 03:37:55 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Tue, 8 May 2018 07:37:55 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: Message-ID: <1525765072.24469.5.camel@fft.be> On Mon, 2018-05-07 at 16:56 +0000, Brett Cannon wrote: > > My question is how is this different to running with -O which leaves > the assert statement out of the bytecode and so you avoid any run- > time cost of the statement entirely?? Not so much different, except that: - the switch won't need to be on the command line - the 2 different asserts would be considered differently by developers/users : * debug assert, for helping developing a new features, used with debug- builds * runtime assert, for ensuring correctness and/or diagnosing issue with production/release-builds on-site. From eric at trueblade.com Tue May 8 09:35:49 2018 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 8 May 2018 09:35:49 -0400 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <1525765072.24469.5.camel@fft.be> References: <1525765072.24469.5.camel@fft.be> Message-ID: <0dd9cf67-08cc-6c57-0570-98507290c483@trueblade.com> On 5/8/2018 3:37 AM, Eloi Gaudry wrote: > On Mon, 2018-05-07 at 16:56 +0000, Brett Cannon wrote: >> >> My question is how is this different to running with -O which leaves >> the assert statement out of the bytecode and so you avoid any run- >> time cost of the statement entirely? > > Not so much different, except that: > - the switch won't need to be on the command line So, is the switch set in code? If so, what would that look like? Eric From eric at trueblade.com Tue May 8 09:58:51 2018 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 8 May 2018 09:58:51 -0400 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <1525787426.5522.1.camel@fft.be> References: <1525765072.24469.5.camel@fft.be> <0dd9cf67-08cc-6c57-0570-98507290c483@trueblade.com> <1525787426.5522.1.camel@fft.be> Message-ID: On 5/8/2018 9:50 AM, Eloi Gaudry wrote: > On Tue, 2018-05-08 at 09:35 -0400, Eric V. Smith wrote: >> On 5/8/2018 3:37 AM, Eloi Gaudry wrote: >>> On Mon, 2018-05-07 at 16:56 +0000, Brett Cannon wrote: >>>> >>>> My question is how is this different to running with -O which >>>> leaves >>>> the assert statement out of the bytecode and so you avoid any >>>> run- >>>> time cost of the statement entirely? >>> >>> Not so much different, except that: >>> - the switch won't need to be on the command line >> >> So, is the switch set in code? If so, what would that look like? >> >> Eric > So far, I only have the "extension" method, where a global variable is > set directly from within the C-interface, as currently done with the > debug assert: > > Include/pydebug.h : int Py_RuntimeAssertFlag = 1; > Your_extension/main.c :?SetFlag(Py_RuntimeAssertFlag); I think what's confusing to me (and maybe others) is that we haven't seen your vision on how this would look in Python code. An example that would throw runtime assertions and the same example where it wouldn't (after a global switch is set?) would be helpful. Eric From Eloi.Gaudry at fft.be Tue May 8 09:50:29 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Tue, 8 May 2018 13:50:29 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <0dd9cf67-08cc-6c57-0570-98507290c483@trueblade.com> References: <1525765072.24469.5.camel@fft.be> <0dd9cf67-08cc-6c57-0570-98507290c483@trueblade.com> Message-ID: <1525787426.5522.1.camel@fft.be> On Tue, 2018-05-08 at 09:35 -0400, Eric V. Smith wrote: > On 5/8/2018 3:37 AM, Eloi Gaudry wrote: > > On Mon, 2018-05-07 at 16:56 +0000, Brett Cannon wrote: > > > > > > My question is how is this different to running with -O which > > > leaves > > > the assert statement out of the bytecode and so you avoid any > > > run- > > > time cost of the statement entirely? > > > > Not so much different, except that: > > - the switch won't need to be on the command line > > So, is the switch set in code? If so, what would that look like? > > Eric So far, I only have the "extension" method, where a global variable is set directly from within the C-interface, as currently done with the debug assert: Include/pydebug.h : int Py_RuntimeAssertFlag = 1; Your_extension/main.c :?SetFlag(Py_RuntimeAssertFlag); From ericfahlgren at gmail.com Tue May 8 10:37:23 2018 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Tue, 8 May 2018 07:37:23 -0700 Subject: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc. In-Reply-To: <20180508093120.w7hkdzemzldf74tc@phdru.name> References: <20180508093120.w7hkdzemzldf74tc@phdru.name> Message-ID: On Tue, May 8, 2018 at 2:31 AM, Oleg Broytman wrote: > So anyone who wants to filter os.walk() must reimplement os.walk() > themselves instead of passing something like filter_dir and filter_file > (or accept_dir/accept_file) to os.walk()? Kind of painful, no? > ?Not really. It's pretty simple code so you put it in your 'usual suspects' module and just forget about it. Here's our version, maybe 10 years old (reworked last whenever scandir came out)?: def _compiled_patterns(patterns, globs=True, flags=0): """ $uuid:95a9b8e2-fb6a-59be-b9c2-da0e6e12f8d3$ Compile a list of patterns into regex patterns. If ``globs`` is true, use ``fnmatch`` to convert the patterns into regular expressions prior to compilation. ``flags`` is any of the ``re`` module's regular expression flags. """ if globs: patterns = list(_fnmatch.translate(glob) for glob in patterns) return list(_re.compile(regex, flags=flags) for regex in patterns) def walk(root_path, ignore_directories=[], show_directories=False): """ $uuid:f77197cd-239b-5d93-9253-c3eb7439d720$ Walk the directory tree and return all the file entries, trimming directories as we go. ``ignore_directories`` is a list of Unix file globs. """ ignore_directories = _compiled_patterns(ignore_directories) def do_walk(top): """ $uuid:e6a4f789-5b5f-56a2-8551-297c142c3e17$ """ for entry in _scandir.scandir(top): if not entry.is_dir(): yield entry elif entry.is_symlink(): pass # Ignore links. elif not any(ignore.match(entry.name) for ignore in ignore_directories): if show_directories: yield entry for entry in do_walk(entry.path): yield entry return do_walk(root_path) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue May 8 10:37:30 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 8 May 2018 07:37:30 -0700 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <1525764564.24469.3.camel@fft.be> References: <1525764564.24469.3.camel@fft.be> Message-ID: On Tue, May 8, 2018 at 12:29 AM, Eloi Gaudry wrote: > On Mon, 2018-05-07 at 10:52 -0700, Guido van Rossum wrote: > > On Mon, May 7, 2018 at 6:24 AM, Serhiy Storchaka > > wrote: > > > I just don't understand why you need a new keyword for writing > > > runtime checks. > > > > Oh, that's pretty clear. The OP wants to be able to turn these checks > > off with some flag he can set/clear at runtime, and when it's off he > > doesn't want to incur the overhead of evaluating the check. The > > assert statement has the latter property, but you have to use -O to > > turn it off. He basically wants a macro so that > > > > runtime_assert() > > > > expands to > > > > if and (): > > raise AssertionError > > > > In Lisp this would be easy. :-) > > and he already has a diff ready for review if needed (basically very > similar to the current 'assert' implementation :) That seems premature. There is not even a hint of agreement that such a feature would be useful *in general* (I'm not doubting your situation) and worth our limited volunteer developer resources to maintain, document etc. for decades to come. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Eloi.Gaudry at fft.be Tue May 8 11:01:46 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Tue, 8 May 2018 15:01:46 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: <1525764564.24469.3.camel@fft.be> Message-ID: <1525791703.5522.3.camel@fft.be> On Tue, 2018-05-08 at 07:37 -0700, Guido van Rossum wrote: > > > In Lisp this would be easy. :-) > > > > and he already has a diff ready for review if needed (basically > > very > > similar to the current 'assert' implementation :) > > That seems premature. There is not even a hint of agreement that such > a feature would be useful *in general* (I'm not doubting your > situation) and worth our limited volunteer developer resources to > maintain, document etc. for decades to come. I didn't mean to push too fast, just to show how things could be implemented and maybe clarify my comments/statements. From g.rodola at gmail.com Tue May 8 10:53:26 2018 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Tue, 8 May 2018 16:53:26 +0200 Subject: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc. In-Reply-To: References: <20180508093120.w7hkdzemzldf74tc@phdru.name> Message-ID: On Tue, May 8, 2018 at 2:00 PM, David Mertz wrote: > I like the idea. I think an argument to os.walk() is the simplest option > for most users. But per some comments, "hidden" is actually more subtle > than the filesystem bit sometimes. I.e. dot-files, ~ suffix, maybe .bak, > etc. > > I'd suggest meeting the ideas slightly and making the new argument > 'filter' or 'skip' that takes a callable. Default to None, but provide an > os.is_hidden that users don't need to figure out how to implement. E.g. > > os.walk(PATH, skip=os.is_hidden) > > os.walk(PATH, skip=lambda entry: entry.name.endswith(('~', '.bak', > '.tmp'))) > I think this would be a good addition because it gives direct access to the underlying os.scandir() objects which are currently inaccessible and discarded (if os.walk() were to be written today it'd probably yield (root, os.DirEntry) instead of (root, dirs, files)). As such one can implement advanced filtering logic without having to call os.stat() for each path string yielded by os.walk() (faster). IMO the callback should accept a (root, os.DirEntry) pair though, because the "root" path can also be part of the filtering logic. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue May 8 11:34:34 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 9 May 2018 01:34:34 +1000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <1525765072.24469.5.camel@fft.be> References: <1525765072.24469.5.camel@fft.be> Message-ID: <20180508153434.GC9562@ando.pearwood.info> On Tue, May 08, 2018 at 07:37:55AM +0000, Eloi Gaudry wrote: > * debug assert, for helping developing a new features, used with debug- > builds > * runtime assert, for ensuring correctness and/or diagnosing issue with > production/release-builds on-site. I don't think that is a useful distinction to make. I use `assert` all the time, and I have never once cared whether it is a "debug build" or a "production build". I use assert in my code, and then people (including me) can use it with whatever build of Python they like. I don't even know which builds of Python I'm running. My *guess* is that the OS-provided Python is probably a non-debug build, and the ones I've compiled from source will be whatever the default build settings are, but I don't know what that is or how to find out. And if there was some sort of runtime_assert that could be enabled and disabled individually, why wouldn't I use it with debug builds as well as non-debug builds? -- Steve From steve at pearwood.info Tue May 8 11:39:49 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 9 May 2018 01:39:49 +1000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: <1525765072.24469.5.camel@fft.be> <0dd9cf67-08cc-6c57-0570-98507290c483@trueblade.com> <1525787426.5522.1.camel@fft.be> Message-ID: <20180508153948.GD9562@ando.pearwood.info> On Tue, May 08, 2018 at 09:58:51AM -0400, Eric V. Smith wrote: > I think what's confusing to me (and maybe others) is that we haven't > seen your vision on how this would look in Python code. > > An example that would throw runtime assertions and the same example > where it wouldn't (after a global switch is set?) would be helpful. In Eloi's first post in this thread, he gave the example: runtime_assert( expr ) Although it is written as a function call, he refers to it as a statement, so I guess he means: runtime_assert expr He says that would compile to the equivalent of: if runtime_assert_active and expr: print(RuntimeAssertionError()) but he gives no idea of what runtime_assert_active is (is it a per-module global variable? a single application-wide super-global? a local variable? something else?) or how we are supposed to set it. Nor does he explain why failed assertions merelt *print* an error message, rather than raising an exception. -- Steve From random832 at fastmail.com Tue May 8 11:45:17 2018 From: random832 at fastmail.com (Random832) Date: Tue, 08 May 2018 11:45:17 -0400 Subject: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc. In-Reply-To: References: Message-ID: <1525794317.827268.1364967608.08D3936B@webmail.messagingengine.com> On Mon, May 7, 2018, at 02:05, Steve Barnes wrote: > In a lot of uses of os.walk it is desirable to skip version control > directories, (which are usually hidden directories), to the point that > almost all of the examples given look like: CVS isn't a hidden directory on Linux. Maybe it can be on windows, but it probably won't be if it's manually created, which you mentioned issues with below. There's probably a discussion we should be having about exposing these system-specific attributes, but they really can't be a general solution for the problem you have. MacOS, incidentally, has two distinct attributes for hiding files [chflags hidden and setfile -a V], along with a ".private" file that can be in a directory containing a list of filenames to hide. From storchaka at gmail.com Tue May 8 11:52:02 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 8 May 2018 18:52:02 +0300 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: Message-ID: 07.05.18 20:52, Guido van Rossum ????: > He basically wants a macro so that > > ? runtime_assert() > > expands to > > ? if and (): > ????? raise AssertionError > > In Lisp this would be easy. :-) Python is not Lisp (still). But there is the MacroPy project. And at end you always can use an external tool for code generation. For example the old good cpp. From Eloi.Gaudry at fft.be Tue May 8 11:38:38 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Tue, 8 May 2018 15:38:38 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: <1525765072.24469.5.camel@fft.be> <0dd9cf67-08cc-6c57-0570-98507290c483@trueblade.com> <1525787426.5522.1.camel@fft.be> Message-ID: <1525793916.5522.5.camel@fft.be> On Tue, 2018-05-08 at 09:58 -0400, Eric V. Smith wrote: > I think what's confusing to me (and maybe others) is that we haven't? > seen your vision on how this would look in Python code. > > An example that would throw runtime assertions and the same example? > where it wouldn't (after a global switch is set?) would be helpful. > > Eric Eric, I can only agree to your comment. As a matter of fact, I have only used the case where the runtime assert is activated through an extension. In this case, adding a set method is simple : in our framework, we would set the variable Py_RuntimeAssertFlag to 1, depending on some command line parameters, or settings found in a configuration file). In pure python, if something as having a method registered in __builtins__ make sense, it could be used to trigger the assertive behavior. In this example, let's assume that you may want to perform some basic check on a given file in diagnostics mode: >>> def check_mounts_size(): ... ??return len( open( '/proc/self/mounts', 'r' ).readlines() )>1024 ... >>>?runtime_assert( len( open( '/proc/self/mounts', 'r' ).readlines()>1024 ) ) >>>?runtime_assert(?check_mounts_size() ) >>> __builtins__.set_runtime_assert( True ) >>>?runtime_assert( len( open( '/proc/self/mounts', 'r' ).readlines()>1024 ) ) Traceback (most recent call last): ??File "", line 1, in AssertionError >>>?runtime_assert(?check_mounts_size() ) Traceback (most recent call last): ??File "", line 1, in AssertionError From storchaka at gmail.com Tue May 8 12:08:59 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 8 May 2018 19:08:59 +0300 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <20180508153434.GC9562@ando.pearwood.info> References: <1525765072.24469.5.camel@fft.be> <20180508153434.GC9562@ando.pearwood.info> Message-ID: 08.05.18 18:34, Steven D'Aprano ????: > I don't even know which builds of Python I'm running. My *guess* is that > the OS-provided Python is probably a non-debug build, and the ones I've > compiled from source will be whatever the default build settings are, > but I don't know what that is or how to find out. There are different means of "debug build". It may mean that binaries are build with enabling runtime checks in C code. When you build from sources you will get a non-debug build by default. You need to pass a special option --with-pydebug to ./configure for getting a debug build. The OS-provided Python is a non-debug build too. It may means that the "assert" statement in Python code is not a no-op and the building __debug__ constant is True. Python is ran in debug mode by default. You have to pass the -O option to the python command for running Python in non-debug mode. ISTM the OP uses the term "debug build" in this meaning. Finally, a special "development mode" mode was introduced in 3.7. It is enabled by the command line option -X dev. It switches on several expensive runtime checks in C code (if they are not switched off by compiling binaries in "release build" in the first meaning). https://docs.python.org/3.8/whatsnew/3.7.html#new-development-mode-x-dev All these things are virtually orthogonal and can be combined arbitrary. From tim.peters at gmail.com Tue May 8 12:18:16 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 8 May 2018 11:18:16 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: [Tim] >> (inside a synthetic function created to >> implement a listcomp/genexp, names bound by "=" are local; names bound >> by ":=" are nonlocal; names bound by both are "who cares?"- >> compiler-time error would be fine by me, or the first person to show a >> real use case wins). [Jacco van Dorp ] > Wait, you can't use = in a listcomp, right ? Or are you talking about > the implementation hidden to the casual user ? Sorry, I was too obscure there - I intended "=" to mean "name binding by any means other than :=". Off the top of my head, I believe that - today - the only "any means other than :=" possible in a listcomp/genexp is appearing as a target in a `for` clause (like the `i` in "[i+1 for i in iterable]`). If there's some other way I missed, I meant to cover that too. But, yes, you're right, `names bound by "="` makes no literal sense at all there. > I thought letting := bind to the surrounding scope was fine basically > because it's currently not possible, so therefore there would be no > syntactic ambiguity, and it'd actually do what people would expect. It's not really about the semantics of `:=` so much as about how synthetic functions are defined. In most cases, it amounts to saying "in the nested function synthesized for a listcomp/genexp, if a name `x` appears as the target of a binding expression in the body, a `nonlocal x` declaration is generated near the top of the synthetic function". For example, if this block appears inside a function: it = (i for i in range(10)) total = 0 for psum in (total := total + value for value in it): print(psum} under the current PEP meaning it blows up in the same way this code blows up today: it = (i for i in range(10)) total = 0 def _func(it): for value in it: total = total + value # blows up here yield total for psum in _func(it): print(psum) with UnboundLocalError: local variable 'total' referenced before assignment But add nonlocal total at the top of `_func()` and it works fine (displays 0, 1, 3, 6, 10, 15, ...). So it's not really about what ":=" does, but about how ":=" affects scope in synthesized nested functions. But if you wrote a nested function yourself? There's no suggestion here that ":=" have any effect on scope decisions in explicitly given nested functions (same as for "=", it would imply "the target is local"), just on those generated "by magic" for listcomps/genexps. Maybe there should be, though. My initial thought was "no, because the user has total control over scope decisions in explicitly given functions today, but if something was magically made nonlocal they would have no way to override that". From Eloi.Gaudry at fft.be Tue May 8 11:51:12 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Tue, 8 May 2018 15:51:12 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <20180508153434.GC9562@ando.pearwood.info> References: <1525765072.24469.5.camel@fft.be> <20180508153434.GC9562@ando.pearwood.info> Message-ID: <1525794670.5522.7.camel@fft.be> On Wed, 2018-05-09 at 01:34 +1000, Steven D'Aprano wrote: > On Tue, May 08, 2018 at 07:37:55AM +0000, Eloi Gaudry wrote: > > > * debug assert, for helping developing a new features, used with > > debug- > > builds > > * runtime assert, for ensuring correctness and/or diagnosing issue > > with > > production/release-builds on-site. > > I don't think that is a useful distinction to make. > > I use `assert` all the time, and I have never once cared whether it > is a? > "debug build" or a "production build". I use assert in my code, and > then? > people (including me) can use it with whatever build of Python they? > like. I do understand your point but I don't share your opinion. I think that is a difference between: - the current 'assert' which usage seems (to me) to focus on development correctness (I think of it as the C-assert enabled in any C program in debug build ) - the runtime_assert that I submitted on the list, which would be focusing on usage correctness (hence runtime), and easily disabled at runtime (when the python command line options parsing is not an option, for instance when the python interpreter is not python itself and/or when the consumer/extension wants to behave differently). > I don't even know which builds of Python I'm running. My *guess* is > that? > the OS-provided Python is probably a non-debug build, and the ones > I've? > compiled from source will be whatever the default build settings > are,? > but I don't know what that is or how to find out. > > And if there was some sort of runtime_assert that could be enabled > and? > disabled individually, why wouldn't I use it with debug builds as > well? > as non-debug builds? There would be no reason why, but you would benefit from being able to easily activate/deactivate the runtime assert. From guido at python.org Tue May 8 13:06:42 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 8 May 2018 10:06:42 -0700 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: So the way I envision it is that *in the absence of a nonlocal or global declaration in the containing scope*, := inside a comprehension or genexpr causes the compiler to assign to a local in the containing scope, which is elevated to a cell (if it isn't already). If there is an explicit nonlocal or global declaration in the containing scope, that is honored. Examples: # Simplest case, neither nonlocal nor global declaration def foo(): [p := q for q in range(10)] # Creates foo-local variable p print(p) # Prints 9 # There's a nonlocal declaration def bar(): p = 42 # Needed to determine its scope def inner(): nonlocal p [p := q for q in range(10)] # Assigns to p in bar's scope inner() print(p) # Prints 9 # There's a global declaration def baz(): global p [p := q for q in range(10)] baz() print(p) # Prints 9 All these would work the same way if you wrote list(p := q for q in range(10)) instead of the comprehension. We should probably define what happens when you write [p := p for p in range(10)]. I propose that this overwrites the loop control variable rather than creating a second p in the containing scope -- either way it's probably a typo anyway. := outside a comprehension/genexpr is treated just like any other assignment (other than in-place assignment), i.e. it creates a local unless a nonlocal or global declaration exists. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Tue May 8 13:28:59 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Tue, 8 May 2018 13:28:59 -0400 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: So the way I envision it is that *in the absence of a nonlocal or global > declaration in the containing scope*, := inside a comprehension or genexpr > causes the compiler to assign to a local in the containing scope, which is > elevated to a cell (if it isn't already). If there is an explicit nonlocal > or global declaration in the containing scope, that is honored. > This seems to be getting awfully complicated. Proof? Try to write the docs for the proposed semantics. I don't understand why we went so astray from the original requirements, which could all be met by having `if` and `while` accept `as` to bind an expression to a variable that would be local to the structured statement. Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue May 8 13:38:16 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 May 2018 03:38:16 +1000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <1525794670.5522.7.camel@fft.be> References: <1525765072.24469.5.camel@fft.be> <20180508153434.GC9562@ando.pearwood.info> <1525794670.5522.7.camel@fft.be> Message-ID: On Wed, May 9, 2018 at 1:51 AM, Eloi Gaudry wrote: > I think that is a difference between: > - the current 'assert' which usage seems (to me) to focus on > development correctness (I think of it as the C-assert enabled in any C > program in debug build ) > - the runtime_assert that I submitted on the list, which would be > focusing on usage correctness (hence runtime), and easily disabled at > runtime (when the python command line options parsing is not an option, > for instance when the python interpreter is not python itself and/or > when the consumer/extension wants to behave differently). What's the difference between "development correctness" and "usage correctness"? Does the latter depend on user input at run time? I still don't understand the distinction you're trying to make here. ChrisA From tim.peters at gmail.com Tue May 8 13:57:33 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 8 May 2018 12:57:33 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: These all match my expectations. Some glosses: [Guido] > So the way I envision it is that *in the absence of a nonlocal or global > declaration in the containing scope*, := inside a comprehension or genexpr > causes the compiler to assign to a local in the containing scope, which is > elevated to a cell (if it isn't already). If there is an explicit nonlocal > or global declaration in the containing scope, that is honored. If the genexp/listcomp is at module level, then "assign to a local in the containing scope" still makes sense ("locals" and "globals" mean the same thing at module level), but "elevated to a cell" doesn't then - it's just a plain global. In absolutely all cases, what I expect is that NAME := EXPR in a genexp/listcomp do the binding _as if_ NAME = object_EXPR_evaluates_to were executed in the immediately containing scope. Describing the goal instead of part of the implementation may be easier to grasp ;-) > Examples: > > # Simplest case, neither nonlocal nor global declaration > def foo(): > [p := q for q in range(10)] # Creates foo-local variable p > print(p) # Prints 9 > > # There's a nonlocal declaration > def bar(): > p = 42 # Needed to determine its scope > def inner(): > nonlocal p > [p := q for q in range(10)] # Assigns to p in bar's scope > inner() > print(p) # Prints 9 > > # There's a global declaration > def baz(): > global p > [p := q for q in range(10)] > baz() > print(p) # Prints 9 > > All these would work the same way if you wrote list(p := q for q in > range(10)) instead of the comprehension. 100% agreed. Add at module scope: [p := q for q in range(10)] print(p) # Prints 9 But uou're on your own for class scope, because I never did anything fancy enough at class scope to need to learn how it works ;-) > We should probably define what happens when you write [p := p for p in > range(10)]. I propose that this overwrites the loop control variable rather > than creating a second p in the containing scope -- either way it's probably > a typo anyway. A compile-time error would be fine by me too. Creating two meanings for `p` is nuts - pick one in case of conflict. I suggested before that the first person with a real use case for this silliness should get the meaning their use case needs, but nobody bit, so "it's local then" is fine. > := outside a comprehension/genexpr is treated just like any other assignment > (other than in-place assignment), i.e. it creates a local unless a nonlocal > or global declaration exists. Also agreed. People have total control over scopes in explicitly given functions now, and if the compiler magically made anything nonlocal they would have no way to stop it. Well, I suppose we could add a "non_nonlocal" declaration, but I'd rather not ;-) From steve at pearwood.info Tue May 8 14:19:32 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 9 May 2018 04:19:32 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507142636.GW9562@ando.pearwood.info> Message-ID: <20180508181931.GG9562@ando.pearwood.info> On Tue, May 08, 2018 at 01:28:59PM -0400, Juancarlo A?ez wrote: > So the way I envision it is that *in the absence of a nonlocal or global > > declaration in the containing scope*, := inside a comprehension or genexpr > > causes the compiler to assign to a local in the containing scope, which is > > elevated to a cell (if it isn't already). If there is an explicit nonlocal > > or global declaration in the containing scope, that is honored. > > > > This seems to be getting awfully complicated. Proof? Try to write the docs > for the proposed semantics. Okay, I'll bite. I don't know why you think its complicated: it is precisely the same as ordinary ``=`` assignment scoping rules. It is comprehensions that are the special case. * * * The binding expression `` := `` evaluates the right hand side , binds it to , and then returns that value. Unless explicitly declared nonlocal or global (in which case that declaration is honoured), will belong to the current scope, the same as other assignments such ``name = value``, with one difference. Inside comprehensions and generator expressions, variables created with ``for name in ...`` exist in a separate scope distinct from the usual local/nonlocal/global/builtin scopes, and are inaccessible from outside the comprehension. (They do not "leak".) That is not the case for those created with ``:=``, which belong to the scope containing the comprehension. To give an example: a = 0 x = [b := 10*a for a in (1, 2, 3)] assert x == [10, 20, 30] assert a = 0 assert b = 30 > I don't understand why we went so astray from the original requirements, > which could all be met by having `if` and `while` accept `as` to bind an > expression to a variable that would be local to the structured statement. That is not the original motivation for binding expressions. The original requirements were specifically for comprehensions. https://mail.python.org/pipermail/python-ideas/2018-February/048971.html This is hardly the only time that something similar has been raised. -- Steve From tim.peters at gmail.com Tue May 8 14:23:15 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 8 May 2018 13:23:15 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: [Guido] >> So the way I envision it is that *in the absence of a nonlocal or global >> declaration in the containing scope*, := inside a comprehension or genexpr >> causes the compiler to assign to a local in the containing scope, which is >> elevated to a cell (if it isn't already). If there is an explicit nonlocal >> or global declaration in the containing scope, that is honored. [Juancarlo A?ez ] > This seems to be getting awfully complicated. Proof? Try to write the docs > for the proposed semantics. Implementation details - even just partial sketches - are always "busy". Think of it this way instead: it's _currently_ the case that listcomps & genexps run in a scope S that's the same as the scope C that contains them, _except_ that names appearing as `for` targets are local to S. All other names in S resolve to exactly the same scopes they resolved to in C (local in C, global in C, nonlocal in C - doesn't matter). What changes now? Nothing in that high-level description, except that a name appearing as a binding expression target in S that's otherwise unknown in C establishes that the name is local to C. That's nothing essentially new, though - bindings _always_ establish scopes for otherwise-unknown names in Python. > I don't understand why we went so astray from the original requirements, > which could all be met by having `if` and `while` accept `as` to bind an > expression to a variable that would be local to the structured statement. "Original" depends on when you first jumped into this ;-) From rosuav at gmail.com Tue May 8 15:40:44 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 May 2018 05:40:44 +1000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: <1525765072.24469.5.camel@fft.be> <20180508153434.GC9562@ando.pearwood.info> <1525794670.5522.7.camel@fft.be> Message-ID: On Wed, May 9, 2018 at 5:27 AM, Eloi Gaudry wrote: > My choice of words might not be the best, yes. > I do see to different meanings and/or context for the historical assert and the one I propose: > > 1/ > the first one would be something saying that, as a developer, when writing >>>> assert (expr) > in my python code, I mean that all my unit tests and real life tests I could think of should succeed the test. I do mean "don't go further, I might not know where you come from or where you intend to go or why you are behaving as such, but you failed to meet this and/or this criteria/condition". > > 2/ > the second one is there to activate some other checks, not while developing, just at runtime when the user uses my extension and want to get some diagnostics/enforcing checks to happen, because he/she is using something I couldn't think of in the first place, something that would not have been checked before. > > Yes, those checks might be considered as identical in a language sense, but then : as an extension/interpreter writer, why should I only rely on the debug assert available today? Why would it not make sense to offer another assert, semantically different, aiming at runtime checks issues and this time where control is indeed by the consumer/the extension? > No, they're not identical. The first one is an assertion; the second is simply an 'if' and a 'raise'. It doesn't need any special syntax - all you need is standard exception creation. def average(values): if not values: raise ValueError("Cannot calculate average of empty collection") This should not be an assertion, "run-time" or otherwise. You never want to disable it. ChrisA From Eloi.Gaudry at fft.be Tue May 8 15:27:33 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Tue, 8 May 2018 19:27:33 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: <1525765072.24469.5.camel@fft.be> <20180508153434.GC9562@ando.pearwood.info> <1525794670.5522.7.camel@fft.be> Message-ID: My choice of words might not be the best, yes. I do see to different meanings and/or context for the historical assert and the one I propose: 1/ the first one would be something saying that, as a developer, when writing >>> assert (expr) in my python code, I mean that all my unit tests and real life tests I could think of should succeed the test. I do mean "don't go further, I might not know where you come from or where you intend to go or why you are behaving as such, but you failed to meet this and/or this criteria/condition". 2/ the second one is there to activate some other checks, not while developing, just at runtime when the user uses my extension and want to get some diagnostics/enforcing checks to happen, because he/she is using something I couldn't think of in the first place, something that would not have been checked before. Yes, those checks might be considered as identical in a language sense, but then : as an extension/interpreter writer, why should I only rely on the debug assert available today? Why would it not make sense to offer another assert, semantically different, aiming at runtime checks issues and this time where control is indeed by the consumer/the extension? -----Original Message----- From: Python-ideas On Behalf Of Chris Angelico Sent: Tuesday, May 8, 2018 7:38 PM To: python-ideas at python.org Subject: Re: [Python-ideas] Runtime assertion with no overhead when not active On Wed, May 9, 2018 at 1:51 AM, Eloi Gaudry wrote: > I think that is a difference between: > - the current 'assert' which usage seems (to me) to focus on > development correctness (I think of it as the C-assert enabled in any > C program in debug build ) > - the runtime_assert that I submitted on the list, which would be > focusing on usage correctness (hence runtime), and easily disabled at > runtime (when the python command line options parsing is not an > option, for instance when the python interpreter is not python itself > and/or when the consumer/extension wants to behave differently). What's the difference between "development correctness" and "usage correctness"? Does the latter depend on user input at run time? I still don't understand the distinction you're trying to make here. ChrisA From apalala at gmail.com Tue May 8 20:24:05 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Tue, 8 May 2018 20:24:05 -0400 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: > Implementation details - even just partial sketches - are always > "busy". Think of it this way instead: it's _currently_ the case that > listcomps & genexps run in a scope S that's the same as the scope C > that contains them, _except_ that names appearing as `for` targets are > local to S. All other names in S resolve to exactly the same scopes > they resolved to in C (local in C, global in C, nonlocal in C - > doesn't matter). > > What changes now? Nothing in that high-level description, except that > a name appearing as a binding expression target in S that's otherwise > unknown in C establishes that the name is local to C. That's nothing > essentially new, though - bindings _always_ establish scopes for > otherwise-unknown names in Python. That's a very nice (and short) explanation! Maybe my distrust is just don't like the new syntax, or that I'am biased towards using "as". -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.van.dorp at deonet.nl Wed May 9 03:15:49 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Wed, 9 May 2018 09:15:49 +0200 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: > [Tim] > {About binding the for loop variable} Yeah, that binding is the one I attempted to refer to. So I did understand you after all. > We should probably define what happens when you write [p := p for p in > range(10)]. I propose that this overwrites the loop control variable rather > than creating a second p in the containing scope -- either way it's probably > a typo anyway. My naive assumption would be both. If it's just the insertion of a nonlocal statement like Tim suggested, wouldn't the comprehension blow up to: def implicitfunc() nonlocal p templist = [] for p in range(10): p = p templist.append(p) return templist ? If it were [q := p for p in range(10)], it would be: def implicitfunc() nonlocal q templist = [] for p in range(10): q = p templist.append(q) return templist Why would it need to be treated differently ? (type checkers probably should, though.) > [Tim] > A compile-time error would be fine by me too. Creating two meanings > for `p` is nuts - pick one in case of conflict. I suggested before > that the first person with a real use case for this silliness should > get the meaning their use case needs, but nobody bit, so "it's local > then" is fine. x = x is legal. Why wouldn't p := p be ? > [Juancarlo] > Maybe my distrust is just don't like the new syntax, or that I'am biased towards using "as". I share your bias, but not your distrust. (go as !) From gadgetsteve at live.co.uk Wed May 9 01:22:30 2018 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Wed, 9 May 2018 05:22:30 +0000 Subject: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc. In-Reply-To: References: <20180508093120.w7hkdzemzldf74tc@phdru.name> Message-ID: On 08/05/2018 15:53, Giampaolo Rodola' wrote: > > > On Tue, May 8, 2018 at 2:00 PM, David Mertz > wrote: > > I like the idea. I think an argument to os.walk() is the simplest > option for most users. But per some comments, "hidden" is actually > more subtle than the filesystem bit sometimes. I.e. dot-files, ~ > suffix, maybe .bak, etc. > > I'd suggest meeting the ideas slightly and making the new argument > 'filter' or 'skip' that takes a callable. Default to None, but > provide an os.is_hidden that users don't need to figure out how to > implement. E.g. > > os.walk(PATH, skip=os.is_hidden) > > os.walk(PATH, skip=lambda entry: entry.name.endswith(('~', '.bak', > '.tmp'))) > > > I think this would be a good addition because it gives direct access to > the underlying os.scandir() objects which are currently inaccessible and > discarded (if os.walk() were to be written today it'd probably yield > (root, os.DirEntry) instead of (root, dirs, files)). As such one can > implement advanced filtering logic without having to call os.stat() for > each path string yielded by os.walk() (faster). > > IMO the callback should accept a (root, os.DirEntry) pair though, > because the "root" path can also be part of the filtering logic. > > -- > > Giampaolo - http://grodola.blogspot.com > > I like the idea of extending the original idea to a filtered walk possibly with some predefined filters. As there does not seem to be a lot of strong opposition so far to the basic idea, (other than a some "why bother it is too easy to do yourself"), it seems like there is a choice now is between: a) raising an enhancement request on the tracker (I am not sure if this is major enough to require a PEP) or b) setting up a new library on PyPi and putting it out there to see if it sinks or swims. What is the general feeling between the two options? -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From j.van.dorp at deonet.nl Wed May 9 04:14:25 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Wed, 9 May 2018 10:14:25 +0200 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: My apologies for something unclear in my previous mail - the second block I quoted (the one without a name) originated from Guido, not from Tim. From wes.turner at gmail.com Wed May 9 08:29:16 2018 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 9 May 2018 08:29:16 -0400 Subject: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc. In-Reply-To: References: <20180508093120.w7hkdzemzldf74tc@phdru.name> Message-ID: fnmatch.filter does Unix filename pattern matching. https://docs.python.org/3/library/fnmatch.html#fnmatch.filter grin and grind are like grep and find with options to filter hidden files and VCS directories by default. https://pypi.org/project/grin/ There's an example of using the Python API here: https://github.com/rkern/grin/blob/master/examples/grinpython.py - grin.get_regex(args) - grin.get_filenames(args) https://github.com/rkern/grin/blob/master/grin.py On Wednesday, May 9, 2018, Steve Barnes wrote: > > > On 08/05/2018 15:53, Giampaolo Rodola' wrote: > > > > > > On Tue, May 8, 2018 at 2:00 PM, David Mertz > > wrote: > > > > I like the idea. I think an argument to os.walk() is the simplest > > option for most users. But per some comments, "hidden" is actually > > more subtle than the filesystem bit sometimes. I.e. dot-files, ~ > > suffix, maybe .bak, etc. > > > > I'd suggest meeting the ideas slightly and making the new argument > > 'filter' or 'skip' that takes a callable. Default to None, but > > provide an os.is_hidden that users don't need to figure out how to > > implement. E.g. > > > > os.walk(PATH, skip=os.is_hidden) > > > > os.walk(PATH, skip=lambda entry: entry.name.endswith(('~', '.bak', > > '.tmp'))) > > > > > > I think this would be a good addition because it gives direct access to > > the underlying os.scandir() objects which are currently inaccessible and > > discarded (if os.walk() were to be written today it'd probably yield > > (root, os.DirEntry) instead of (root, dirs, files)). As such one can > > implement advanced filtering logic without having to call os.stat() for > > each path string yielded by os.walk() (faster). > > > > IMO the callback should accept a (root, os.DirEntry) pair though, > > because the "root" path can also be part of the filtering logic. > > > > -- > > > > Giampaolo - http://grodola.blogspot.com > > > > > > I like the idea of extending the original idea to a filtered walk > possibly with some predefined filters. > > As there does not seem to be a lot of strong opposition so far to the > basic idea, (other than a some "why bother it is too easy to do > yourself"), it seems like there is a choice now is between: > > a) raising an enhancement request on the tracker (I am not sure if > this is major enough to require a PEP) or > b) setting up a new library on PyPi and putting it out there to see > if it sinks or swims. > > What is the general feeling between the two options? > > -- > Steve (Gadget) Barnes > Any opinions in this message are my personal opinions and do not reflect > those of my employer. > > --- > This email has been checked for viruses by AVG. > http://www.avg.com > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From facundobatista at gmail.com Wed May 9 08:39:08 2018 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 9 May 2018 09:39:08 -0300 Subject: [Python-ideas] Have a "j" format option for lists Message-ID: This way, I could do: >>> authors = ["John", "Mary", "Estela"] >>> "Authors: {:, j}".format(authors) 'Authors: John, Mary, Estela' In this case the join can be made in the format yes, but this proposal would be very useful when the info to format comes inside a structure together with other stuff, like... >>> info = { ... 'title': "A book", ... 'price': Decimal("2.34"), ... 'authors: ["John", "Mary", "Estela"], ... } ... >>> print("{title!r} (${price}) by {authors:, j}".format(**info)) "A book" ($2.34) by John, Mary, Estela What do you think? -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From jsbueno at python.org.br Wed May 9 08:43:24 2018 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Wed, 9 May 2018 09:43:24 -0300 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: Message-ID: On 9 May 2018 at 09:39, Facundo Batista wrote: > This way, I could do: > >>>> authors = ["John", "Mary", "Estela"] >>>> "Authors: {:, j}".format(authors) > 'Authors: John, Mary, Estela' +1. I bit concerned about the relevant whitespace in there, but just a little bit - it is already inside a string-literal anyway. > > In this case the join can be made in the format yes, but this proposal > would be very useful when the info to format comes inside a structure > together with other stuff, like... > >>>> info = { > ... 'title': "A book", > ... 'price': Decimal("2.34"), > ... 'authors: ["John", "Mary", "Estela"], > ... } > ... >>>> print("{title!r} (${price}) by {authors:, j}".format(**info)) > "A book" ($2.34) by John, Mary, Estela > > What do you think? > > -- > . Facundo > From steve at pearwood.info Wed May 9 09:06:10 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 9 May 2018 23:06:10 +1000 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: Message-ID: <20180509130610.GK9562@ando.pearwood.info> On Wed, May 09, 2018 at 09:39:08AM -0300, Facundo Batista wrote: > This way, I could do: > > >>> authors = ["John", "Mary", "Estela"] > >>> "Authors: {:, j}".format(authors) > 'Authors: John, Mary, Estela' Looks interesting, but I think we need to know the semantics in more detail. For example: - if the items of the list aren't already strings, how are they converted? - do you truly mean lists *only*, or is any iterable acceptible? Here's a tiny proof-of-concept for the feature: import string class Template(string.Formatter): def format_field(self, value, spec): if spec.endswith('j'): value = ', '.join(map(str, value)) spec = spec[:-1] + 's' return super(Template, self).format_field(value, spec) Template().format('{:j} => {:d}', ['alpha', 'beta', 42, 'delta'], 99) # returns 'alpha, beta, 42, delta => 99' -- Steve From wolfgang.maier at biologie.uni-freiburg.de Wed May 9 09:20:40 2018 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Wed, 9 May 2018 15:20:40 +0200 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: Message-ID: <795cca5b-4d00-722c-3b4c-ca0e34993def@biologie.uni-freiburg.de> On 05/09/2018 02:39 PM, Facundo Batista wrote: > This way, I could do: > >>>> authors = ["John", "Mary", "Estela"] >>>> "Authors: {:, j}".format(authors) > 'Authors: John, Mary, Estela' > > In this case the join can be made in the format yes, but this proposal > would be very useful when the info to format comes inside a structure > together with other stuff, like... > >>>> info = { > ... 'title': "A book", > ... 'price': Decimal("2.34"), > ... 'authors: ["John", "Mary", "Estela"], > ... } > ... >>>> print("{title!r} (${price}) by {authors:, j}".format(**info)) > "A book" ($2.34) by John, Mary, Estela > > What do you think? > For reference (first message of a rather long previous thread): https://mail.python.org/pipermail/python-ideas/2015-September/035787.html From rosuav at gmail.com Wed May 9 09:28:41 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 May 2018 23:28:41 +1000 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: <20180509130610.GK9562@ando.pearwood.info> References: <20180509130610.GK9562@ando.pearwood.info> Message-ID: On Wed, May 9, 2018 at 11:06 PM, Steven D'Aprano wrote: > On Wed, May 09, 2018 at 09:39:08AM -0300, Facundo Batista wrote: >> This way, I could do: >> >> >>> authors = ["John", "Mary", "Estela"] >> >>> "Authors: {:, j}".format(authors) >> 'Authors: John, Mary, Estela' > > > > Looks interesting, but I think we need to know the semantics in more > detail. For example: > > - if the items of the list aren't already strings, how are they > converted? I'd expect that they'd be converted using format(), which by default would just call str(). How you'd go about specifying a format string, though, I'm not sure. > - do you truly mean lists *only*, or is any iterable acceptible? With the letter being "j" and the semantics being lifted from str.join(), I would guess the latter. >From the sound of it, this would be a change made to format(), or rather the underlying C level function, PyObject_Format(). If done there, it would also automatically apply to f-strings and anything else that calls format(). Perhaps the right way is not a colon marker, but an enhancement to the ! notation? We currently have !s and !r to do str() and repr(), and this could be !j followed by a join string. Combining this with a colon would allow the individual elements to be formatted with the given string, and then joined. For instance: x = [1,2,3] msg = '#{x:3d!j, }#'.format(x=x) # or equivalently msg = f'#{x:3d!j, }#' assert msg == '# 1, 2, 3#' +0.5 on this. I don't currently yearn for it, but I'd probably use it if it were available. ChrisA From eric at trueblade.com Wed May 9 09:42:49 2018 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 9 May 2018 09:42:49 -0400 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: Message-ID: On 5/9/18 8:39 AM, Facundo Batista wrote: > This way, I could do: > >>>> authors = ["John", "Mary", "Estela"] >>>> "Authors: {:, j}".format(authors) > 'Authors: John, Mary, Estela' > > In this case the join can be made in the format yes, but this proposal > would be very useful when the info to format comes inside a structure > together with other stuff, like... > >>>> info = { > ... 'title': "A book", > ... 'price': Decimal("2.34"), > ... 'authors: ["John", "Mary", "Estela"], > ... } > ... >>>> print("{title!r} (${price}) by {authors:, j}".format(**info)) > "A book" ($2.34) by John, Mary, Estela > > What do you think? Among other things, I think you should use .format_map(info) instead of .format(**info)! Eric From eric at trueblade.com Wed May 9 09:49:12 2018 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 9 May 2018 09:49:12 -0400 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: <20180509130610.GK9562@ando.pearwood.info> Message-ID: <072125aa-6278-2ee1-d8d3-a4b417faf2bd@trueblade.com> On 5/9/18 9:28 AM, Chris Angelico wrote: > On Wed, May 9, 2018 at 11:06 PM, Steven D'Aprano wrote: >> On Wed, May 09, 2018 at 09:39:08AM -0300, Facundo Batista wrote: >>> This way, I could do: >>> >>>>>> authors = ["John", "Mary", "Estela"] >>>>>> "Authors: {:, j}".format(authors) >>> 'Authors: John, Mary, Estela' >> >> >> >> Looks interesting, but I think we need to know the semantics in more >> detail. For example: >> >> - if the items of the list aren't already strings, how are they >> converted? > > I'd expect that they'd be converted using format(), which by default > would just call str(). How you'd go about specifying a format string, > though, I'm not sure. > >> - do you truly mean lists *only*, or is any iterable acceptible? > > With the letter being "j" and the semantics being lifted from > str.join(), I would guess the latter. Since '{:spec}'.format(obj) basically becomes obj.__format__('spec'), this would have to be implemented on a concrete type (in the above example, list). > From the sound of it, this would be a change made to format(), or > rather the underlying C level function, PyObject_Format(). If done > there, it would also automatically apply to f-strings and anything > else that calls format(). Perhaps the right way is not a colon marker, > but an enhancement to the ! notation? We currently have !s and !r to > do str() and repr(), and this could be !j followed by a join string. > Combining this with a colon would allow the individual elements to be > formatted with the given string, and then joined. For instance: I would object to changing the format machinery. Any format spec should be interpreted by some object's __format__ method. Eric > x = [1,2,3] > msg = '#{x:3d!j, }#'.format(x=x) > # or equivalently > msg = f'#{x:3d!j, }#' > assert msg == '# 1, 2, 3#' > > +0.5 on this. I don't currently yearn for it, but I'd probably use it > if it were available. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From p.f.moore at gmail.com Wed May 9 10:01:21 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 May 2018 15:01:21 +0100 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: <072125aa-6278-2ee1-d8d3-a4b417faf2bd@trueblade.com> References: <20180509130610.GK9562@ando.pearwood.info> <072125aa-6278-2ee1-d8d3-a4b417faf2bd@trueblade.com> Message-ID: On 9 May 2018 at 14:49, Eric V. Smith wrote: > On 5/9/18 9:28 AM, Chris Angelico wrote: >> On Wed, May 9, 2018 at 11:06 PM, Steven D'Aprano >> wrote: >> >>> - do you truly mean lists *only*, or is any iterable acceptible? >> >> With the letter being "j" and the semantics being lifted from >> str.join(), I would guess the latter. > > Since '{:spec}'.format(obj) basically becomes obj.__format__('spec'), this > would have to be implemented on a concrete type (in the above example, > list). [...] > I would object to changing the format machinery. Any format spec should be > interpreted by some object's __format__ method. Agreed. In theory this is a nice idea, but the way formatting is implemented (and the fact that join is a method on strings taking an arbitrary iterable as an argument) means that it's a bad fit for the format mini-language. I don't think the improved convenience is sufficient to warrant the change that would be required in practice. (But if someone found a way to make it work *without* changes to the underlying format machinery, that would be a different matter...) Paul From eric at trueblade.com Wed May 9 10:29:22 2018 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 9 May 2018 10:29:22 -0400 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: <20180509130610.GK9562@ando.pearwood.info> <072125aa-6278-2ee1-d8d3-a4b417faf2bd@trueblade.com> Message-ID: <8348b129-7b9d-383f-8e62-a1c32c855ad2@trueblade.com> On 5/9/18 10:01 AM, Paul Moore wrote: > On 9 May 2018 at 14:49, Eric V. Smith wrote: >> I would object to changing the format machinery. Any format spec should be >> interpreted by some object's __format__ method. > > Agreed. In theory this is a nice idea, but the way formatting is > implemented (and the fact that join is a method on strings taking an > arbitrary iterable as an argument) means that it's a bad fit for the > format mini-language. > > I don't think the improved convenience is sufficient to warrant the > change that would be required in practice. (But if someone found a way > to make it work *without* changes to the underlying format machinery, > that would be a different matter...) Well, since you asked, let's combine this with dataclasses, because we can! from dataclasses import dataclass from typing import List @dataclass class Join: o: List[str] # or similar def __format__(self, spec): return spec.join(self.o) l = ['a', 'b'] print('{:, }'.format(Join(l))) print(f'{Join(l):-}') Gives: a, b a-b Eric From p.f.moore at gmail.com Wed May 9 10:41:44 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 May 2018 15:41:44 +0100 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: <8348b129-7b9d-383f-8e62-a1c32c855ad2@trueblade.com> References: <20180509130610.GK9562@ando.pearwood.info> <072125aa-6278-2ee1-d8d3-a4b417faf2bd@trueblade.com> <8348b129-7b9d-383f-8e62-a1c32c855ad2@trueblade.com> Message-ID: On 9 May 2018 at 15:29, Eric V. Smith wrote: > On 5/9/18 10:01 AM, Paul Moore wrote: >> >> On 9 May 2018 at 14:49, Eric V. Smith wrote: >>> >>> I would object to changing the format machinery. Any format spec should >>> be >>> interpreted by some object's __format__ method. >> >> >> Agreed. In theory this is a nice idea, but the way formatting is >> implemented (and the fact that join is a method on strings taking an >> arbitrary iterable as an argument) means that it's a bad fit for the >> format mini-language. >> >> I don't think the improved convenience is sufficient to warrant the >> change that would be required in practice. (But if someone found a way >> to make it work *without* changes to the underlying format machinery, >> that would be a different matter...) > > > Well, since you asked, let's combine this with dataclasses, because we can! > > from dataclasses import dataclass > from typing import List > > @dataclass > class Join: > o: List[str] # or similar > def __format__(self, spec): > return spec.join(self.o) > > l = ['a', 'b'] > > print('{:, }'.format(Join(l))) > print(f'{Join(l):-}') > > Gives: > a, b > a-b :-) Sorry, I should have been clearer. Given that print('{}'.format(', '.join(l))) isn't that difficult, at least for me the attraction of the proposal was the possibility of not having to wrap the argument (whether in a function call, or in a custom class). But the print(f'{Join(l):-}') approach is certainly cleaner looking than print(f'{", ".join(l)}'). Paul From njs at pobox.com Wed May 9 11:04:22 2018 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 09 May 2018 15:04:22 +0000 Subject: [Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc. In-Reply-To: References: Message-ID: There are hidden directories, and then there are hidden directories :-). It makes sense to me to add an option to the stdlib functions to skip directories (and files) that the system considers hidden, so I guess that means dotfiles on Unix and files with the hidden attribute on Windows. But if you want "smart" matching that has special knowledge of CVS directories and so forth, then that seems like something that would fit better as a library on PyPI. The rust "ignore" crate has a pretty good set of semantics, for reference. It's not trivial, but it sure is handy :-): https://docs.rs/ignore/0.4.2/ignore/struct.WalkBuilder.html -n On Tue, May 8, 2018, 00:43 Steve Barnes wrote: > In a lot of uses of os.walk it is desirable to skip version control > directories, (which are usually hidden directories), to the point that > almost all of the examples given look like: > > import os > for root, dirs, files in os.walk(some_dir): > if 'CVS' in dirs: > dirs.remove('CVS') # or .svn or .hg etc. > # do something... > > But of course there are many version control systems to the point that > much of my personal code looks like, (note that I have to use a > multitude of version control systems due to project requirements): > > > import os > vcs_dirs = ['.hg', '.svn', 'CSV', '.git', '.bz'] # Version control > directory names I know > > > for root, dirs, files in os.walk(some_dir): > for dirname in vcs_dirs: > dirs.remove(dirname) > > I am sure that I am missing many other version control systems but the > one thing that all of the ones that I am familiar with default to > creating their files in hidden directories. I know that the above > sometimes hits problems on Windows if someone manually created a > directory and you end up with abortions such as Csv\ or .SVN .... > > Since it could be argued that hidden directories are possibly more > common than simlinks, (especially in the Windows world of course), and > that hidden directories have normally been hidden by someone for a > reason it seems to make sense to me to normally ignore them in directory > traversal. > > Obviously there are also occasions when it makes sense to include VCS, > or other hidden, directories files, (e.g. "Where did all of my disk > space go?" or "delete recursively"), so I would like to suggest > including in the os.walk family of functions an additional parameter to > control skipping all hidden directories - either positively or negatively. > > Names that spring to mind include: > * nohidden > * nohidden_dirs > * hidden > * hidden_dirs > > This change could be made with no impact on current behaviour by > defaulting to hidden=True (or nohidden=False) which would just about > ensure that no existing code is broken or quite a few bugs in existing > code could be quietly fixed, (and some new ones introduced), by > defaulting to this behaviour. > > Since the implementation of os.walk has changed to use os.scandir which > exposes the returned file statuses in the os.DirEntry.stat() the > overhead should be minimal. > > An alternative would be to add another new function, say os.vwalk(), to > only walk visible entries. > > Note that a decision would have to be made on whether to include such > filtering when topdown is False, personally I am tempted to include the > filtering so as to maintain consistency but ignoring the filter when > topdown is False, (or if topdown is False and the hidden behaviour is > unspecified), might make sense if the skipping of hidden directories > becomes the new default (then recursively removing files & directories > would still include processing hidden items by default). > > If this receives a positive response I would be happy to undertake the > effort involved in producing a PR. > -- > Steve (Gadget) Barnes > Any opinions in this message are my personal opinions and do not reflect > those of my employer. > > --- > This email has been checked for viruses by AVG. > http://www.avg.com > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Wed May 9 12:18:07 2018 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 9 May 2018 11:18:07 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: ... [Guido] >> We should probably define what happens when you write [p := p for p in >> range(10)]. I propose that this overwrites the loop control variable rather >> than creating a second p in the containing scope -- either way it's probably >> a typo anyway. [Jacco van Dorp ] > My naive assumption would be both. Since this is all about scope, while I'm not 100% sure of what Guido meant, I assumed he was saying "p can only have one scope in the synthetic function: local or non-local, not both, and local is what I propose". For example, let's flesh out his example a bit more: p = 42 [p := p for p in range(10) if p == 3] print(p) # 42? 3? 9? If `p` is local to the listcomp, it must print 42. If `p` is not-local, it must print 9. If it's some weird mixture of both, 3 makes most sense (the only time `p := p` is executed is when the `for` target `p` is 3). > If it's just the insertion of a nonlocal statement like Tim suggested, Then all occurrences of `p` in the listcomp are not-local, and the example above prints 9.. > wouldn't the comprehension blow up to: > > def implicitfunc() > nonlocal p > templist = [] > for p in range(10): > p = p > templist.append(p) > return templist > > ? Yes. > If it were [q := p for p in range(10)], it would be: > > def implicitfunc() > nonlocal q > templist = [] > for p in range(10): > q = p > templist.append(q) > return templist There's no question about that one, because `q` isn't _also_ used as a `for` target. There are two "rules" here: 1. A name appearing as a `for` target is local. That's already the case. 2. All other names (including a name appearing as a binding-expression target) are not local. Clearer? If a name appears as both, which rule applies? "Both" is likely the worst possible answer, since it's incoherent ;-) If a name appears as both a `for` target and as a binding-expression target, that particular way of phrasing "the rules" suggests #1 (it's local, period) is the more natural choice. And, whether Guido consciously knows it or not, that's why he suggested it ;-) > Why would it need to be treated differently ? Because it's incoherent. It's impossible to make the example above print 3 _merely_ by fiddling the scope of `p`. Under the covers, two distinct variables would need to be created, both of which are named `p` as far as the user can see. For my extension of Guido's example: def implicitfunc() nonlocal p templist = [] for hidden_loop_p in range(10): if hidden_loop_p == 3: p = hidden_loop_p templist.append(hidden_loop_p) return templist [Tim] >> A compile-time error would be fine by me too. Creating two meanings >> for `p` is nuts - pick one in case of conflict. I suggested before >> that the first person with a real use case for this silliness should >> get the meaning their use case needs, but nobody bit, so "it's local >> then" is fine. > x = x is legal. Why wouldn't p := p be ? It's easy to make it "legal": just say `p is local, period` or `p is not local, period`. The former will confuse people who think "but names appearing as binding-expression targets are not local", and the latter will confuse people who think "but names appearing as `for` targets are local". Why bother? In the absence of an actual use case (still conspicuous by absence), I'd be happiest refusing to compile such pathological code. Else `p is local, period` is the best pointless choice ;-) From rhodri at kynesim.co.uk Wed May 9 12:48:37 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Wed, 9 May 2018 17:48:37 +0100 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: Message-ID: On 09/05/18 13:39, Facundo Batista wrote: > This way, I could do: > >>>> authors = ["John", "Mary", "Estela"] >>>> "Authors: {:, j}".format(authors) > 'Authors: John, Mary, Estela' > > In this case the join can be made in the format yes, but this proposal > would be very useful when the info to format comes inside a structure > together with other stuff, like... > >>>> info = { > ... 'title': "A book", > ... 'price': Decimal("2.34"), > ... 'authors: ["John", "Mary", "Estela"], > ... } > ... >>>> print("{title!r} (${price}) by {authors:, j}".format(**info)) > "A book" ($2.34) by John, Mary, Estela > > What do you think? -1 until you give me an actual spec rather than a curious example. Sorry if that sounds a bit rude, but I spend most of my time trying to find polite ways to inform people that I'm perfectly capable of implementing the fluffy collection of vague aspirations and inconsistent terminology they have given me, but the results won't be what they expect and probably won't be what they want either. And that's just talking to my boss :-) If you want something specific you'll have to specify it, otherwise you'll get something else. -- Rhodri James *-* Kynesim Ltd From facundobatista at gmail.com Wed May 9 15:56:38 2018 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 9 May 2018 16:56:38 -0300 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: Message-ID: 2018-05-09 13:48 GMT-03:00 Rhodri James : > -1 until you give me an actual spec rather than a curious example. > > Sorry if that sounds a bit rude, but I spend most of my time trying to find Be sorry, it was rude. This list is for throwing ideas and see if they gain momentum... if yes, it will be time for a PEP. I know how to do a PEP, believe me. But if we'll be asking for a Spec for every idea we think may be valuable, we'll stall. Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From steve at pearwood.info Wed May 9 20:45:14 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 10 May 2018 10:45:14 +1000 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: Message-ID: <20180510004513.GM9562@ando.pearwood.info> On Wed, May 09, 2018 at 04:56:38PM -0300, Facundo Batista wrote: > 2018-05-09 13:48 GMT-03:00 Rhodri James : > > > -1 until you give me an actual spec rather than a curious example. > > > > Sorry if that sounds a bit rude, but I spend most of my time trying to find > > Be sorry, it was rude. I do not agree that Rhodri's comments were rude. They were frank and honest constructive criticism of the presentation of your idea. I have no patience with hair-trigger accusations of "rudeness" as a debating tactic to curtail criticism, and your response to Rhodri's mild and fair words looks exactly like that. Whether you intended it that way or not, that is what it looks like to me. I too spent some time scratching my head trying to guess what you expect this "j" format code to do, including the fundamental question "why j and not some other letter?". Even after I worked out some of it, I was still left with questions. So I think that Rhodri's criticism is completely valid: your post was vague, with a lack of detail and not even an attempt to provide a partial specification. That doesn't have to be a formal specification, but it should be at least an attempt to say "this is what the format code means", rather than expect people to guess from a single example. (By the way, if you answered my earlier questions, I haven't seen the answer.) Dealing with under-specified, vague instructions, and the invariable failure to guess correctly what was intended, is frustrating, annoying and painful. We should not be surprised that vague proposals can trigger an angry response, but Rhodri's post showed great restraint. It isn't just the waste of time and effort when when we guess wrong, it is also the implied disrespect: "I am too important to bother explaining myself to minions like you. Work out what I mean." If an idea is important enough to post here, we should be willing to spend some time presenting the idea in sufficient detail so that people don't have to guess what we mean. As the Zen says, "Explicit is better than implicit". No, that doesn't mean that we have to write a PEP before posting. There is a middle-ground between giving a single example and expecting people to extrapolate from it, and writing a full PEP. -- Steve From rosuav at gmail.com Wed May 9 20:50:13 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 10 May 2018 10:50:13 +1000 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: <20180510004513.GM9562@ando.pearwood.info> References: <20180510004513.GM9562@ando.pearwood.info> Message-ID: On Thu, May 10, 2018 at 10:45 AM, Steven D'Aprano wrote: > I too spent some time scratching my head trying to guess what you expect > this "j" format code to do, including the fundamental question "why j > and not some other letter?". Interestingly, that question didn't faze me in the slightest. But maybe my instant and automatic "oh you mean join" coloured my expectations elsewhere, and possibly not correctly. ChrisA From steve at pearwood.info Wed May 9 21:02:37 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 10 May 2018 11:02:37 +1000 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: <20180510004513.GM9562@ando.pearwood.info> Message-ID: <20180510010237.GN9562@ando.pearwood.info> On Thu, May 10, 2018 at 10:50:13AM +1000, Chris Angelico wrote: > On Thu, May 10, 2018 at 10:45 AM, Steven D'Aprano wrote: > > I too spent some time scratching my head trying to guess what you expect > > this "j" format code to do, including the fundamental question "why j > > and not some other letter?". > > Interestingly, that question didn't faze me in the slightest. But > maybe my instant and automatic "oh you mean join" coloured my > expectations elsewhere, and possibly not correctly. Oh, I didn't think of that! I was thinking i, j, k as integers, and thinking "i is used, and j is the next letter". -- Steve From python-ideas at shalmirane.com Wed May 9 21:22:56 2018 From: python-ideas at shalmirane.com (Ken Kundert) Date: Wed, 9 May 2018 18:22:56 -0700 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: Message-ID: <20180510012256.GA25502@kundert.designers-guide.com> Facundo, I think this is the beginning of a great feature. And it fills a hole in the current string formatting. Specifically, we can carefully control the formatting of the base data types, but not collections. I would like to see you flesh out the idea. In particular, I'd like to see you address cases where: 1. The underlying members in the collection are not strings. Besides the basic types such as numbers, it would also be nice to be able to apply formats recursively so that one can construct a string using the attributes of members that are objects or items or other collections. 2. The ability to handle collections other than simple lists or iterables, such as dictionaries. -Ken On Wed, May 09, 2018 at 09:39:08AM -0300, Facundo Batista wrote: > This way, I could do: > > >>> authors = ["John", "Mary", "Estela"] > >>> "Authors: {:, j}".format(authors) > 'Authors: John, Mary, Estela' > > In this case the join can be made in the format yes, but this proposal > would be very useful when the info to format comes inside a structure > together with other stuff, like... > > >>> info = { > ... 'title': "A book", > ... 'price': Decimal("2.34"), > ... 'authors: ["John", "Mary", "Estela"], > ... } > ... > >>> print("{title!r} (${price}) by {authors:, j}".format(**info)) > "A book" ($2.34) by John, Mary, Estela > > What do you think? > > -- . Facundo From tjreedy at udel.edu Wed May 9 21:57:49 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 9 May 2018 21:57:49 -0400 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: <20180510010237.GN9562@ando.pearwood.info> References: <20180510004513.GM9562@ando.pearwood.info> <20180510010237.GN9562@ando.pearwood.info> Message-ID: On 5/9/2018 9:02 PM, Steven D'Aprano wrote: > On Thu, May 10, 2018 at 10:50:13AM +1000, Chris Angelico wrote: >> On Thu, May 10, 2018 at 10:45 AM, Steven D'Aprano wrote: >>> I too spent some time scratching my head trying to guess what you expect >>> this "j" format code to do, including the fundamental question "why j >>> and not some other letter?". >> >> Interestingly, that question didn't faze me in the slightest. But >> maybe my instant and automatic "oh you mean join" coloured my >> expectations elsewhere, and possibly not correctly. > > Oh, I didn't think of that! I was thinking i, j, k as integers, and > thinking "i is used, and j is the next letter". I took me several minutes to *guess* that j abbreviated join. It would have been better if it were given in the proposal. -- Terry Jan Reedy From guido at python.org Wed May 9 23:33:05 2018 From: guido at python.org (Guido van Rossum) Date: Wed, 9 May 2018 20:33:05 -0700 Subject: [Python-ideas] PEP 572: about the operator precedence of := Message-ID: (I vaguely recall this has been brought up before, but I'm too lazy to find the subtread. So it goes.) PEP 572 currently seems to specify that when used in expressions, the precedence of `:=` is lower (i.e. it binds more tightly) than all operators except for the comma. I derive this from the single example `stuff = [[y := f(x), x/y] for x in range(5)]`. >From this it would follow that `f(a := 1, a)` is equivalent to `a = 1; f(1, 1)`, and also that `(a := 1, a)` is equivalent to `a = 1; (1, 1)`. (Although M.A.L. objected to this.) But what should `a := 1, 1` at the top level (as a statement) do? On the one hand, analogy with the above suggest that it is equivalent to `a = 1; (1, 1)`. But on the other hand, it would be really strange if the following two lines had different meanings: a = 1, 1 # a = (1, 1) a := 1, 1 # a = 1; (1, 1) I now think that the best way out is to rule `:=` in the top level expression of an expression statement completely (it would still be okay inside parentheses, where it would bind tighter than comma). An alternative would be to make `:=` bind less tight than comma (like `=`) everywhere, so that `a := 1, 1` indeed meant the same as `a = 1, 1`. But that feels very wrong at least for the case `f(a := 1, 1)` -- I believe Tim already mentioned that we've been conditioned by keyword arguments to parse this as `f((a := 1), 1)`. (I could add to this that we have done various things to generator expression syntax to avoid ever having to deal with ambiguities like `a, a+1 for a in range(10)` or `a for a in x, y`.) Another alternative would be to always require parentheses around `:=`, so that we would have to write `f((a := 1), 1)`. That's unambiguous, but otherwise just gets on the nerves. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Wed May 9 23:35:05 2018 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 9 May 2018 22:35:05 -0500 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: Message-ID: <67A74EA6-DA5B-492C-9811-F995EF90997A@gmail.com> On May 9, 2018, at 7:39 AM, Facundo Batista wrote: > > This way, I could do: > >>>> authors = ["John", "Mary", "Estela"] >>>> "Authors: {:, j}".format(authors) > 'Authors: John, Mary, Estela' > ... > > What do you think? That is an inspired idea. I like it :-) Raymond From rosuav at gmail.com Wed May 9 23:42:05 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 10 May 2018 13:42:05 +1000 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: References: Message-ID: On Thu, May 10, 2018 at 1:33 PM, Guido van Rossum wrote: > (I vaguely recall this has been brought up before, but I'm too lazy to find > the subtread. So it goes.) > > PEP 572 currently seems to specify that when used in expressions, the > precedence of `:=` is lower (i.e. it binds more tightly) than all operators > except for the comma. I derive this from the single example `stuff = [[y := > f(x), x/y] for x in range(5)]`. > > From this it would follow that `f(a := 1, a)` is equivalent to `a = 1; f(1, > 1)`, and also that `(a := 1, a)` is equivalent to `a = 1; (1, 1)`. (Although > M.A.L. objected to this.) > > But what should `a := 1, 1` at the top level (as a statement) do? On the one > hand, analogy with the above suggest that it is equivalent to `a = 1; (1, > 1)`. But on the other hand, it would be really strange if the following two > lines had different meanings: > > a = 1, 1 # a = (1, 1) > a := 1, 1 # a = 1; (1, 1) > > I now think that the best way out is to rule `:=` in the top level > expression of an expression statement completely (it would still be okay > inside parentheses, where it would bind tighter than comma). I would have := bind more tightly than the comma. Consider: a = 1, x := 2, 3 IMO the only sane interpretation is "x = 2; a = 1, 2, 3". Effectively, the := operator does not like to play with commas; we've already ruled out "a, b := range(2)" as a means of unpacking, so it makes more sense to have that simply mean "b = range(2); a, b". ChrisA From guido at python.org Wed May 9 23:52:40 2018 From: guido at python.org (Guido van Rossum) Date: Wed, 9 May 2018 20:52:40 -0700 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: References: Message-ID: On Wed, May 9, 2018 at 8:42 PM, Chris Angelico wrote: > On Thu, May 10, 2018 at 1:33 PM, Guido van Rossum > wrote: > > (I vaguely recall this has been brought up before, but I'm too lazy to > find > > the subtread. So it goes.) > > > > PEP 572 currently seems to specify that when used in expressions, the > > precedence of `:=` is lower (i.e. it binds more tightly) than all > operators > > except for the comma. I derive this from the single example `stuff = [[y > := > > f(x), x/y] for x in range(5)]`. > > > > From this it would follow that `f(a := 1, a)` is equivalent to `a = 1; > f(1, > > 1)`, and also that `(a := 1, a)` is equivalent to `a = 1; (1, 1)`. > (Although > > M.A.L. objected to this.) > > > > But what should `a := 1, 1` at the top level (as a statement) do? On the > one > > hand, analogy with the above suggest that it is equivalent to `a = 1; (1, > > 1)`. But on the other hand, it would be really strange if the following > two > > lines had different meanings: > > > > a = 1, 1 # a = (1, 1) > > a := 1, 1 # a = 1; (1, 1) > > > > I now think that the best way out is to rule `:=` in the top level > > expression of an expression statement completely (it would still be okay > > inside parentheses, where it would bind tighter than comma). > > I would have := bind more tightly than the comma. Consider: > > a = 1, x := 2, 3 > > IMO the only sane interpretation is "x = 2; a = 1, 2, 3". Effectively, > the := operator does not like to play with commas; we've already ruled > out "a, b := range(2)" as a means of unpacking, so it makes more sense > to have that simply mean "b = range(2); a, b". > Oh, I hadn't even though of combining the two in one statement. That example is truly horrible (on first skim I didn't even notice it used two different assignment operators!) and strengthens my confidence that we should just disallow an un-parenthesized `:=` operator at the top level, where now the top level includes the RHS of a classic assignment. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Thu May 10 00:46:40 2018 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 9 May 2018 23:46:40 -0500 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: References: Message-ID: [Guido] > (I vaguely recall this has been brought up before, but I'm too lazy to find > the subtread. So it goes.) > > PEP 572 currently seems to specify that when used in expressions, the > precedence of `:=` is lower (i.e. it binds more tightly) Umm ... that's the opposite of what the Reference Manual says "lower":means: """ 6.16. Operator precedence The following table summarizes the operator precedence in Python, from lowest precedence (least binding) to highest precedence (most binding). """ > than all operators except for the comma. Which gets more potentially confusing become the comma isn't listed at all as "an operator". Instead the meaning of commas for building tuples is captured by the higher-level `expression-list` grammar production: expression_list ::= expression ( "," expression )* [","] So _every_ mere operator "binds more tightly" (if you view it in those terms) than a comma in an expression list. What was mostly discussed before - after giving up on fully generally "assignment expressions" - was whether ":=" should get a precedence between comparison and bitwise OR operators. So that, e.g., parens wouldn't be needed in if x := match() is not None: to get the intended if (x := match()) is not None: But that never gained much traction, and it was dropped quickly. It was left with a (possibly unwritten!) consensus that ":=" should bind very weakly as an operator. Of the binary infix operators, the most weakly binding (the weakest of which now is Boolean OR). > I derive this from the single example > `stuff = [[y := f(x), x/y] for x in range(5)]`.` As above, the part you're looking at there falls under the expression_list part of the grammar, and no binary operator could group as y OP (f(x), x/y) instead. All binary operators group as (y OP f(x)), (x/y) > From this it would follow that `f(a := 1, a)` And now you're throwing in the entirely different meaning of commas in argument lists ;-) But same thing: no operator can cross comma boundaries in argument lists, because the grammar of argument lists doesn't allow for that either. Even; e.g., f(lambda x: x, 2) groups as f((lambda x: x), 2) > is equivalent to `a = 1; f(1, 1)`, Yup. > and also that `(a := 1, a)` is equivalent to `a = 1; (1, 1)`. (Although > M.A.L. objected to this.) That's just another instance of expression_list. Nothing so far has surprised me, because I have been viewing ":=" as an operator for some time now. > But what should `a := 1, 1` at the top level (as a statement) do? On the one > hand, analogy with the above suggest that it is equivalent to >`a = 1; (1, 1)`. If it's an operator, there's really no choice about that. > But on the other hand, it would be really strange if the following two > lines had different meanings: > > a = 1, 1 # a = (1, 1) > a := 1, 1 # a = 1; (1, 1) Agreed. > I now think that the best way out is to rule `:=` in the top level > expression of an expression statement completely (it would still be okay > inside parentheses, where it would bind tighter than comma). Probably prudent, and no objections here. > An alternative would be to make `:=` bind less tight than comma (like `=`) > everywhere, so that `a := 1, 1` indeed meant the same as `a = 1, 1`. But > that feels very wrong at least for the case `f(a := 1, 1)` -- I believe Tim > already mentioned that we've been conditioned by keyword arguments to parse > this as `f((a := 1), 1)`. (I could add to this that we have done various > things to generator expression syntax to avoid ever having to deal with > ambiguities like `a, a+1 for a in range(10)` or `a for a in x, y`.) Offhand, since comma is not formally "an operator" now, I expect it would require major surgery to the grammar to have a := 1, 1 group as a := (1, 1) in any context. At least if ";=" is treated as "just another operator" and doesn't grow its own unique-to-it pile of grammar rules. > Another alternative would be to always require parentheses around `:=`, so > that we would have to write `f((a := 1), 1)`. That's unambiguous, but > otherwise just gets on the nerves. I hoped that rigidly calling these "binding expressions" would break the connection in peoples' minds that these somehow "should behave" like assignment statements, but that seems futile. There's really nothing surprising here _if_ it's viewed as just another operator. Nobody would be tempted to, e.g., add parens to f(a + 1, 1), or if `+` were any other binary operator either. So, over time, it would be just as annoying need to type them for the `:=` operator. From goosey15 at gmail.com Thu May 10 03:30:10 2018 From: goosey15 at gmail.com (Angus Hollands) Date: Thu, 10 May 2018 07:30:10 +0000 Subject: [Python-ideas] PEP 572: about the operator precedence of := (Guido van Rossum) In-Reply-To: References: Message-ID: > > Is there a more appropriate mechanism to showing support for a 'EXPR given x = EXPR' approach, as suggested by Nick Coghlan? Then, keeping the binding rules the same for statement assign, requiring parenthesis, would keep things consistent. I personally prefer it to := for the reasons I mentioned previously. Angus Hollands Message: 4 > Date: Wed, 9 May 2018 20:33:05 -0700 > From: Guido van Rossum > To: Python-Ideas > Subject: [Python-ideas] PEP 572: about the operator precedence of := > Message-ID: > < > CAP7+vJJSDeL0FWiYDhRfkgfxGX-oKKynfzyF3fHS+kkjb6DePA at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > (I vaguely recall this has been brought up before, but I'm too lazy to find > the subtread. So it goes.) > > PEP 572 currently seems to specify that when used in expressions, the > precedence of `:=` is lower (i.e. it binds more tightly) than all operators > except for the comma. I derive this from the single example `stuff = [[y := > f(x), x/y] for x in range(5)]`. > > >From this it would follow that `f(a := 1, a)` is equivalent to `a = 1; > f(1, > 1)`, and also that `(a := 1, a)` is equivalent to `a = 1; (1, 1)`. > (Although M.A.L. objected to this.) > > But what should `a := 1, 1` at the top level (as a statement) do? On the > one hand, analogy with the above suggest that it is equivalent to `a = 1; > (1, 1)`. But on the other hand, it would be really strange if the following > two lines had different meanings: > > a = 1, 1 # a = (1, 1) > a := 1, 1 # a = 1; (1, 1) > > I now think that the best way out is to rule `:=` in the top level > expression of an expression statement completely (it would still be okay > inside parentheses, where it would bind tighter than comma). > > An alternative would be to make `:=` bind less tight than comma (like `=`) > everywhere, so that `a := 1, 1` indeed meant the same as `a = 1, 1`. But > that feels very wrong at least for the case `f(a := 1, 1)` -- I believe Tim > already mentioned that we've been conditioned by keyword arguments to parse > this as `f((a := 1), 1)`. (I could add to this that we have done various > things to generator expression syntax to avoid ever having to deal with > ambiguities like `a, a+1 for a in range(10)` or `a for a in x, y`.) > > Another alternative would be to always require parentheses around `:=`, so > that we would have to write `f((a := 1), 1)`. That's unambiguous, but > otherwise just gets on the nerves. > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/python-ideas/attachments/20180509/41b36ded/attachment-0001.html > > > > ------------------------------ > > Message: 5 > Date: Wed, 9 May 2018 22:35:05 -0500 > From: Raymond Hettinger > To: Facundo Batista > Cc: Python-Ideas > Subject: Re: [Python-ideas] Have a "j" format option for lists > Message-ID: <67A74EA6-DA5B-492C-9811-F995EF90997A at gmail.com> > Content-Type: text/plain; charset=us-ascii > > On May 9, 2018, at 7:39 AM, Facundo Batista > wrote: > > > > This way, I could do: > > > >>>> authors = ["John", "Mary", "Estela"] > >>>> "Authors: {:, j}".format(authors) > > 'Authors: John, Mary, Estela' > > > ... > > > > What do you think? > > That is an inspired idea. I like it :-) > > > Raymond > > > > > ------------------------------ > > Message: 6 > Date: Thu, 10 May 2018 13:42:05 +1000 > From: Chris Angelico > To: Python-Ideas > Subject: Re: [Python-ideas] PEP 572: about the operator precedence of > := > Message-ID: > < > CAPTjJmrWHhmidGQ-H4FcBoJqrP69eVfkF2MBG1KYrh_W6adFKQ at mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > On Thu, May 10, 2018 at 1:33 PM, Guido van Rossum > wrote: > > (I vaguely recall this has been brought up before, but I'm too lazy to > find > > the subtread. So it goes.) > > > > PEP 572 currently seems to specify that when used in expressions, the > > precedence of `:=` is lower (i.e. it binds more tightly) than all > operators > > except for the comma. I derive this from the single example `stuff = [[y > := > > f(x), x/y] for x in range(5)]`. > > > > From this it would follow that `f(a := 1, a)` is equivalent to `a = 1; > f(1, > > 1)`, and also that `(a := 1, a)` is equivalent to `a = 1; (1, 1)`. > (Although > > M.A.L. objected to this.) > > > > But what should `a := 1, 1` at the top level (as a statement) do? On the > one > > hand, analogy with the above suggest that it is equivalent to `a = 1; (1, > > 1)`. But on the other hand, it would be really strange if the following > two > > lines had different meanings: > > > > a = 1, 1 # a = (1, 1) > > a := 1, 1 # a = 1; (1, 1) > > > > I now think that the best way out is to rule `:=` in the top level > > expression of an expression statement completely (it would still be okay > > inside parentheses, where it would bind tighter than comma). > > I would have := bind more tightly than the comma. Consider: > > a = 1, x := 2, 3 > > IMO the only sane interpretation is "x = 2; a = 1, 2, 3". Effectively, > the := operator does not like to play with commas; we've already ruled > out "a, b := range(2)" as a means of unpacking, so it makes more sense > to have that simply mean "b = range(2); a, b". > > ChrisA > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > > ------------------------------ > > End of Python-ideas Digest, Vol 138, Issue 65 > ********************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu May 10 03:32:32 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 10 May 2018 03:32:32 -0400 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: References: Message-ID: On 5/9/2018 11:33 PM, Guido van Rossum wrote: > I now think that the best way out is to rule `:=` in the top level > expression of an expression statement completely I would like to be able to interactively enter >>> a: = f(2,4) to have 'a' echoed as well as bound. -- Terry Jan Reedy From barry at barrys-emacs.org Thu May 10 03:55:44 2018 From: barry at barrys-emacs.org (Barry Scott) Date: Thu, 10 May 2018 08:55:44 +0100 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: References: Message-ID: <64D2E0DB-8176-4968-99F4-1403261CEB42@barrys-emacs.org> > On 7 May 2018, at 18:52, Guido van Rossum wrote: > > On Mon, May 7, 2018 at 6:24 AM, Serhiy Storchaka > wrote: > I just don't understand why you need a new keyword for writing runtime checks. > > Oh, that's pretty clear. The OP wants to be able to turn these checks off with some flag he can set/clear at runtime, and when it's off he doesn't want to incur the overhead of evaluating the check. The assert statement has the latter property, but you have to use -O to turn it off. He basically wants a macro so that > > runtime_assert() > > expands to > > if and (): > raise AssertionError > > In Lisp this would be easy. :-) This idea requires the same sort of machinery in python that I was hoping for to implement the short circuit logging. My logging example would be log( control_flag, msg_expr ) expanding to: if : log_function( ) Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Thu May 10 05:04:04 2018 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 10 May 2018 11:04:04 +0200 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: References: Message-ID: <17376e38-c17c-5249-b4cb-6b8e1d441403@egenix.com> On 10.05.2018 05:52, Guido van Rossum wrote: >> I would have := bind more tightly than the comma. Consider: >> >> a = 1, x := 2, 3 >> >> IMO the only sane interpretation is "x = 2; a = 1, 2, 3". Effectively, >> the := operator does not like to play with commas; we've already ruled >> out "a, b := range(2)" as a means of unpacking, so it makes more sense >> to have that simply mean "b = range(2); a, b". >> > > Oh, I hadn't even though of combining the two in one statement. That > example is truly horrible (on first skim I didn't even notice it used two > different assignment operators!) and strengthens my confidence that we > should just disallow an un-parenthesized `:=` operator at the top level, > where now the top level includes the RHS of a classic assignment. Yes, please, and ideally not only at the top level, but everywhere. To a (former Pascal) programmer, a := 1 doesn't read like an operator. It's an assignment expression. If embedded expressions is where Python is heading, it should be made very clear where the embedded expression starts and where it ends on a line. Anything else will just result in hard to debug code with subtle errors... oops = 1, a := 2 * 3, 4, 5 - 6 By having to write: ahhh = 1, (a := 2) * 3, 4, 5 - 6 you at least know that "(a := 2)" will evaluate to 2 in the tuple on the right side. Not sure whether this was discussed before (I'm not really following the discussion), but what happens if you write: check = 0 and (a := 1) ? Will "a" get assigned or not ? All that said, I sometimes do wonder what people dislike so much about explicit for-loops which make all of these things obvious even to a complete Python newbie reading the code. IMO, line noise is not something Python should strive for. Other languages such as APL are much better at that - just look at the beauty of "(~R?R?.?R)/R?1??R" ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 10 2018) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From rosuav at gmail.com Thu May 10 05:08:59 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 10 May 2018 19:08:59 +1000 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: <17376e38-c17c-5249-b4cb-6b8e1d441403@egenix.com> References: <17376e38-c17c-5249-b4cb-6b8e1d441403@egenix.com> Message-ID: On Thu, May 10, 2018 at 7:04 PM, M.-A. Lemburg wrote: > Not sure whether this was discussed before (I'm not really > following the discussion), but what happens if you write: > > check = 0 and (a := 1) > > ? Will "a" get assigned or not ? No, it won't. It's the same as any other side effects after an 'and' - that expression is completely not evaluated. ChrisA From ncoghlan at gmail.com Thu May 10 06:39:47 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 May 2018 20:39:47 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On 8 May 2018 at 04:19, Brett Cannon wrote: > My brain wants to drop the variable name in front of 'given': > > if given m = pattern.search(data): > > while given m = pattern.search(remaining_data): > > Maybe it's because the examples use such a short variable name? > Does that change if the condition isn't just "bool(name)"? For example: if y > 0 given y = f(x): ... That's the situation where I strongly prefer the postfix operator spelling, since it's pretty clear how I should pronounce it (i.e. "if y is greater than zero, given y is set to f-of-x, then ..."). By contrast, while a variety of plausible suggestions have been made, I still don't really know how to pronounce "if (y := f(x)) > 0:)" in a way that's going to be clear to an English-speaking listener (aside from pronouncing it the same way as I'd pronounce the version using "given", but that then raises the question of "Why isn't it written the way it is pronounced?"). I do agree with Tim that the name repetition would strongly encourage the use of short names rather than long ones (since you're always typing them at least twice), such that we'd probably see code like: while not probable_prime(n) given (n = highbit | randrange(1, highbit, 2)): pass Rather than the more explicit: while not probable_prime(candidate) given (candidate = highbit | randrange(1, highbit, 2)): pass However, I'd still consider both of those easier to follow than: while not probable_prime(candidate := highbit | randrange(1, highbit, 2)): pass since it's really unclear to me that "candidate" in the latter form is a positional argument being bound to a name in the local environment, and *not* a keyword argument being passed to "probable_prime". I've also been pondering what the given variant might look like as a generally available postfix operator, rather than being restricted to if/elif/while clauses, and I think that would have interesting implications for the flexibility of its usage in comprehensions, since there would now be *three* places where "given" could appear (as is already the case for the inline binding operator spelling): - in the result expression - in the iterable expression - in the filter expression That is: [(x, y, x - y) given y = f(x) for x in data] [(x, data) for x in data given data = get_data()] [(x, y, x/y) for x in data if y given y = f(x)] Rather than: [(x, y := f(x), x - y) for x in data] [(x, data) for x in data := get_data()] [(x, y, x/y) for x in data if y := f(x)] Opening it up that way would allow for some odd usages that might need to be discouraged in PEP 8 (like explicitly preferring "probable_prime(n) given n = highbit | randrange(1, highbit, 2)" to "probable_prime(n given n = highbit | randrange(1, highbit, 2))"), but it would probably still be simpler overall than attempting to restrict the construct purely to if/elif/while. Even as a generally available postfix keyword, "given" should still be amenable to the treatment where it could be allowed as a variable name in a non-operator context (since we don't allow two adjacent expressions to imply a function call, it's only prefix keywords that have to be disallowed as names to avoid ambiguity in the parser). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu May 10 07:02:40 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 May 2018 21:02:40 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: On 9 May 2018 at 03:57, Tim Peters wrote: > These all match my expectations. Some glosses: > > [Guido] > > We should probably define what happens when you write [p := p for p in > > range(10)]. I propose that this overwrites the loop control variable > rather > > than creating a second p in the containing scope -- either way it's > probably > > a typo anyway. > > A compile-time error would be fine by me too. Creating two meanings > for `p` is nuts - pick one in case of conflict. I suggested before > that the first person with a real use case for this silliness should > get the meaning their use case needs, but nobody bit, so "it's local > then" is fine. > I'd suggest that the handling of conflicting global and nonlocal declarations provides a good precedent here: >>> def f(): ... global x ... nonlocal x ... x = 1 ... File "", line 2 SyntaxError: name 'x' is nonlocal and global Since using a name as a binding target *and* as the iteration variable would effectively be declaring it as both local and nonlocal, or as local and global. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Thu May 10 07:04:37 2018 From: mertz at gnosis.cx (David Mertz) Date: Thu, 10 May 2018 11:04:37 +0000 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: References: Message-ID: On Wed, May 9, 2018, 11:53 PM Guido van Rossum wrote: > Oh, I hadn't even though of combining the two in one statement. That > example is truly horrible (on first skim I didn't even notice it used two > different assignment operators!) and strengthens my confidence that we > should just disallow an un-parenthesized `:=` operator at the top level, > where now the top level includes the RHS of a classic assignment. > IMO, I would strongly prefer calling an unparenthesized top-level ':=' a SyntaxError. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhodri at kynesim.co.uk Thu May 10 07:02:19 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Thu, 10 May 2018 12:02:19 +0100 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: Message-ID: On 09/05/18 20:56, Facundo Batista wrote: > 2018-05-09 13:48 GMT-03:00 Rhodri James : > >> -1 until you give me an actual spec rather than a curious example. >> >> Sorry if that sounds a bit rude, but I spend most of my time trying to find > > Be sorry, it was rude. On reflection, I'm not sorry at all. It needed to be said, and attempting to deduce a spec from a single example has already caused people to go off in a variety of different directions. If provoking you into being even a bit more specific gets our collective cat herd moving in something more like the same direction, it'll be worth it. In that spirit, and trying not to take this as a challenge to see how rude I can be without actually being rude: ;-) I too didn't equate "j" with "join". I was somewhat expecting "l" for "list" given that "s" for "sequence" is taken. Should I take it that everything from the colon to the "j" is an implicitly delimited string to use for joining? If so, does anything in that string need escaping, like say braces or other "j"s? I'm not sure if format specifiers currently allow arbitrary text (I don't think they do, but I'm not sure), which might make this a more serious undertaking than it first looks. Alternatively, are certain characters forbidden in this implicit join string? I presume also that no other formatting is possible with this specifier: no field widths, no justification, none of the other stuff applicable to strings. I'm not talking about formatting the original strings (or whatever) in the list, that would clearly involve crazy amounts of sub-formatting and look like a complete mess in the code, but I can see a need for formatting the final joined string, and I can't see a way of doing that from what you've given us. I'm not convinced, I admit. Your use case >>> print("{title!r} (${price}) by {authors:, j}".format(**info)) "A book" ($2.34) by John, Mary, Estela is moderately compelling, but is starting to look too magic for my tastes. I think it's the implicitly delimited join string; it isn't obvious at first sight that the comma and space aren't independently meaningful in some obscure way, like everything else in that position in a format would be. > This list is for throwing ideas and see if they gain momentum... if > yes, it will be time for a PEP. > > I know how to do a PEP, believe me. > > But if we'll be asking for a Spec for every idea we think may be > valuable, we'll stall. Raising the barrier a bit is no bad thing in my opinion; it shouldn't be easy to add things to Python, otherwise we'd have a much uglier language. Regardless, there is a difference between a full-blown PEP and an outline spec. Much as I would like to have every jot and tittle in place for a first draft, I know that's not realistic and I don't expect it. On the other hand, bitter experience has taught me that when the first draft of a proposal is as nebulous as what you gave us, I should walk away from the incipient disaster right now. -- Rhodri James *-* Kynesim Ltd From ncoghlan at gmail.com Thu May 10 07:16:55 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 May 2018 21:16:55 +1000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <64D2E0DB-8176-4968-99F4-1403261CEB42@barrys-emacs.org> References: <64D2E0DB-8176-4968-99F4-1403261CEB42@barrys-emacs.org> Message-ID: On 10 May 2018 at 17:55, Barry Scott wrote: > On 7 May 2018, at 18:52, Guido van Rossum wrote: > > On Mon, May 7, 2018 at 6:24 AM, Serhiy Storchaka > wrote: > >> I just don't understand why you need a new keyword for writing runtime >> checks. >> > > Oh, that's pretty clear. The OP wants to be able to turn these checks off > with some flag he can set/clear at runtime, and when it's off he doesn't > want to incur the overhead of evaluating the check. The assert statement > has the latter property, but you have to use -O to turn it off. He > basically wants a macro so that > > runtime_assert() > > expands to > > if and (): > raise AssertionError > > In Lisp this would be easy. :-) > > > This idea requires the same sort of machinery in python that I was hoping > for to implement the short circuit logging. > > My logging example would be > > log( control_flag, msg_expr ) > > expanding to: > > if : > log_function( ) > Logging is the example that came to mind for me as well - https://docs.python.org/3/howto/logging.html#optimization discusses the "logging.isEnabledFor" API, and how it can be used to avoid the overhead of expensive state queries when the result log message would just be discarded anyway. Generally speaking though, the spelling for that kind of lazy evaluation is to accept a zero-argument callable, which can then be used with "lambda: " at the call site: runtime_assert(lambda: ) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu May 10 07:31:41 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 May 2018 21:31:41 +1000 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: References: Message-ID: On 10 May 2018 at 13:33, Guido van Rossum wrote: > (I vaguely recall this has been brought up before, but I'm too lazy to > find the subtread. So it goes.) > > PEP 572 currently seems to specify that when used in expressions, the > precedence of `:=` is lower (i.e. it binds more tightly) than all operators > except for the comma. I derive this from the single example `stuff = [[y := > f(x), x/y] for x in range(5)]`. > > From this it would follow that `f(a := 1, a)` is equivalent to `a = 1; > f(1, 1)`, and also that `(a := 1, a)` is equivalent to `a = 1; (1, 1)`. > (Although M.A.L. objected to this.) > > But what should `a := 1, 1` at the top level (as a statement) do? On the > one hand, analogy with the above suggest that it is equivalent to `a = 1; > (1, 1)`. But on the other hand, it would be really strange if the following > two lines had different meanings: > > a = 1, 1 # a = (1, 1) > a := 1, 1 # a = 1; (1, 1) > > I now think that the best way out is to rule `:=` in the top level > expression of an expression statement completely (it would still be okay > inside parentheses, where it would bind tighter than comma). > FWIW, this is one of the ambiguities that the generalised postfix expression form of the given clause would reduce fairly significantly by separating the result expression from the bound expression: a = 1, 1 a given a = 1, 1 # As above, but also returns a a = 1, x, 3 given x = 2 They also compose fairly nicely as an essentially right associative operator: a given a = 1, x, 3 given x = 2 a given a = (1, x, 3 given x = 2) # Same as previous (a given a = 1), x, 3 given x = 2 # Forcing left-associativity While you do have to repeat the bound name at least once, you gain a much clearer order of execution (while it's an order of execution that's right-to-left rather than left-to-right, "rightmost expression first" is the normal way assignments get evaluated anyway). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu May 10 07:47:44 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 10 May 2018 21:47:44 +1000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <64D2E0DB-8176-4968-99F4-1403261CEB42@barrys-emacs.org> References: <64D2E0DB-8176-4968-99F4-1403261CEB42@barrys-emacs.org> Message-ID: <20180510114740.GP9562@ando.pearwood.info> On Thu, May 10, 2018 at 08:55:44AM +0100, Barry Scott wrote: > This idea requires the same sort of machinery in python that I was > hoping for to implement the short circuit logging. I'm working on an idea for delayed evaluation of expressions, and I think your idea of logging below would make an excellent use-case. > My logging example would be > > log( control_flag, msg_expr ) > > expanding to: > > if : > log_function( ) The idea isn't yet ready for Python-Ideas, but if you're interested feel free to contact me off-list. -- Steve From eric at trueblade.com Thu May 10 08:09:21 2018 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 10 May 2018 08:09:21 -0400 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: Message-ID: <5e55bdf8-5e56-43c3-a7a1-dc258b47592a@trueblade.com> On 5/10/18 7:02 AM, Rhodri James wrote: > If so, does anything in that string need escaping, like say braces or > other "j"s? I'm not sure if format specifiers currently allow arbitrary > text (I don't think they do, but I'm not sure), which might make this a > more serious undertaking than it first looks. Alternatively, are > certain characters forbidden in this implicit join string? A format spec can contain anything. Its interpretation is completely left to the type being formatted. For example, a datetime supports a strftime() string, which can be anything: >>> '{:today is %Y-%m-%d}'.format(datetime.datetime.now()) 'today is 2018-05-10' In str.format(), there's a restriction that the format spec can't contain a closing brace }, but that's solely a restriction of the string scanning that's going on, and not a function of the __format__ protocol itself. For example: >>> format(datetime.datetime.now(), 'today is %Y-%m-%d {tada!}') 'today is 2018-05-10 {tada!}' I always suggest to people that they don't look at f-strings or str.format() when coming up with examples, but instead use format() or obj.__format__() to more clearly see what's happening. In Facundo's original example: authors = ["John", "Mary", "Estela"] "Authors: {:, j}".format(authors) it's best to think of this as: "Authors: " + format(authors, ', j') or the equivalent (in this case): "Authors: " + authors.__format__(', j') authors is a list in this example, so list.__format__() would have to understand the format spec ', j'. Okay, that's probably doable (although I'm not advocating it!). But you really don't want this to work just for lists. What about: def authors1(): yield "John" yield "Mary" yield "Estela" format(authors1(), ', j') How would that work? Would generators grow a __format__()? It's not possible to add this functionality to every iterator past, present, and future. This is why I think a wrapper that adds a __format__(), which itself calls str.join(), is the only way to handle this. This is what I showed in my earlier response as the Join class. And at that point, I don't think it's really worth the hassle over just calling str.join(). Eric From ncoghlan at gmail.com Thu May 10 08:17:38 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 May 2018 22:17:38 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: On 9 May 2018 at 03:06, Guido van Rossum wrote: > So the way I envision it is that *in the absence of a nonlocal or global > declaration in the containing scope*, := inside a comprehension or genexpr > causes the compiler to assign to a local in the containing scope, which is > elevated to a cell (if it isn't already). If there is an explicit nonlocal > or global declaration in the containing scope, that is honored. > > Examples: > > # Simplest case, neither nonlocal nor global declaration > def foo(): > [p := q for q in range(10)] # Creates foo-local variable p > print(p) # Prints 9 > > # There's a nonlocal declaration > def bar(): > p = 42 # Needed to determine its scope > def inner(): > nonlocal p > [p := q for q in range(10)] # Assigns to p in bar's scope > inner() > print(p) # Prints 9 > > # There's a global declaration > def baz(): > global p > [p := q for q in range(10)] > baz() > print(p) # Prints 9 > > All these would work the same way if you wrote list(p := q for q in > range(10)) instead of the comprehension. > How would you expect this to work in cases where the generator expression isn't immediately consumed? If "p" is nonlocal (or global) by default, then that opens up the opportunity for it to be rebound between generator steps. That gets especially confusing if you have multiple generator expressions in the same scope iterating in parallel using the same binding target: # This is fine gen1 = (p for p in range(10)) gen2 = (p for p in gen1) print(list(gen2)) # This is not (given the "let's reintroduce leaking from comprehensions" proposal) p = 0 gen1 = (p := q for q in range(10)) gen2 = (p, p := q for q in gen1) print(list(gen2)) It also reintroduces the original problem that comprehension scopes solved, just in a slightly different form: # This is fine for x in range(10): for y in range(10): transposed_related_coords = [y, x for x, y in related_coords(x, y)] # This is not (given the "let's reintroduce leaking from comprehensions" proposal) for x in range(10): for y in range(10): related_interesting_coords = [x, y for x in related_x_coord(x, y) if is_interesting(y := f(x))] Deliberately reintroducing stateful side effects into a nominally functional construct seems like a recipe for significant confusion, even if there are some cases where it might arguably be useful to folks that don't want to write a named function that returns multiple values instead. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From facundobatista at gmail.com Thu May 10 09:08:12 2018 From: facundobatista at gmail.com (Facundo Batista) Date: Thu, 10 May 2018 10:08:12 -0300 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: <20180510004513.GM9562@ando.pearwood.info> References: <20180510004513.GM9562@ando.pearwood.info> Message-ID: 2018-05-09 21:45 GMT-03:00 Steven D'Aprano : > On Wed, May 09, 2018 at 04:56:38PM -0300, Facundo Batista wrote: >> 2018-05-09 13:48 GMT-03:00 Rhodri James : >> >> > -1 until you give me an actual spec rather than a curious example. >> > >> > Sorry if that sounds a bit rude, but I spend most of my time trying to find >> >> Be sorry, it was rude. > > I do not agree that Rhodri's comments were rude. They were frank and > honest constructive criticism of the presentation of your idea. > > I have no patience with hair-trigger accusations of "rudeness" as a > debating tactic to curtail criticism, and your response to Rhodri's mild > and fair words looks exactly like that. Whether you intended it that way > or not, that is what it looks like to me. I totally apologize if I made you, Rhodri or anybody to feel bad about this. It was not my intention to do that. It felt rude to me and I reacted badly, and too fast. I should have behaved differently, sorry again. Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From facundobatista at gmail.com Thu May 10 09:13:00 2018 From: facundobatista at gmail.com (Facundo Batista) Date: Thu, 10 May 2018 10:13:00 -0300 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: <20180510012256.GA25502@kundert.designers-guide.com> References: <20180510012256.GA25502@kundert.designers-guide.com> Message-ID: > I would like to see you flesh out the idea. In particular, I'd like to see you > address cases where: > 1. The underlying members in the collection are not strings. Besides the basic > types such as numbers, it would also be nice to be able to apply formats > recursively so that one can construct a string using the attributes of > members that are objects or items or other collections. I didn't think about this at the time. I have the feeling (not really well thought yet) that "recursive formatting" will complicate everything too much, that doing str() (as {} defaults to) on every object would fulfill most of the basic cases and keep this simple. > 2. The ability to handle collections other than simple lists or iterables, such > as dictionaries. Ideally, it will handle *any* iterable. Thanks! -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From facundobatista at gmail.com Thu May 10 09:21:54 2018 From: facundobatista at gmail.com (Facundo Batista) Date: Thu, 10 May 2018 10:21:54 -0300 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: Message-ID: 2018-05-10 8:02 GMT-03:00 Rhodri James : > On 09/05/18 20:56, Facundo Batista wrote: >> >> 2018-05-09 13:48 GMT-03:00 Rhodri James : >> >>> -1 until you give me an actual spec rather than a curious example. >>> >>> Sorry if that sounds a bit rude, but I spend most of my time trying to >>> find >> >> >> Be sorry, it was rude. > > > On reflection, I'm not sorry at all. It needed to be said, and attempting > to deduce a spec from a single example has already caused people to go off > in a variety of different directions. If provoking you into being even a > bit more specific gets our collective cat herd moving in something more like > the same direction, it'll be worth it. I apologize if *I* was rude or made feel you badly in any way. > I too didn't equate "j" with "join". I was somewhat expecting "l" for > "list" given that "s" for "sequence" is taken. Should I take it that > everything from the colon to the "j" is an implicitly delimited string to > use for joining? Yes. > If so, does anything in that string need escaping, like say braces or other > "j"s? I'm not sure if format specifiers currently allow arbitrary text (I > don't think they do, but I'm not sure), which might make this a more serious > undertaking than it first looks. Alternatively, are certain characters > forbidden in this implicit join string? Don't know. I think I need to read this for ideas: https://mail.python.org/pipermail/python-ideas/2015-September/035789.html > I presume also that no other formatting is possible with this specifier: no > field widths, no justification, none of the other stuff applicable to > strings. I'm not talking about formatting the original strings (or > whatever) in the list, that would clearly involve crazy amounts of > sub-formatting and look like a complete mess in the code, but I can see a > need for formatting the final joined string, and I can't see a way of doing > that from what you've given us. I concur, joining *and* do special formatting for the elements would be a mess. That's way in other mail I said that probably cover the basic case (doing just str()) is better; if you want something more complex you always can do it as a separate step :/ -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From guido at python.org Thu May 10 09:22:17 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 10 May 2018 06:22:17 -0700 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: On Thu, May 10, 2018 at 5:17 AM, Nick Coghlan wrote: > On 9 May 2018 at 03:06, Guido van Rossum wrote: > >> So the way I envision it is that *in the absence of a nonlocal or global >> declaration in the containing scope*, := inside a comprehension or genexpr >> causes the compiler to assign to a local in the containing scope, which is >> elevated to a cell (if it isn't already). If there is an explicit nonlocal >> or global declaration in the containing scope, that is honored. >> >> Examples: >> >> # Simplest case, neither nonlocal nor global declaration >> def foo(): >> [p := q for q in range(10)] # Creates foo-local variable p >> print(p) # Prints 9 >> >> # There's a nonlocal declaration >> def bar(): >> p = 42 # Needed to determine its scope >> def inner(): >> nonlocal p >> [p := q for q in range(10)] # Assigns to p in bar's scope >> inner() >> print(p) # Prints 9 >> >> # There's a global declaration >> def baz(): >> global p >> [p := q for q in range(10)] >> baz() >> print(p) # Prints 9 >> >> All these would work the same way if you wrote list(p := q for q in >> range(10)) instead of the comprehension. >> > > How would you expect this to work in cases where the generator expression > isn't immediately consumed? If "p" is nonlocal (or global) by default, then > that opens up the opportunity for it to be rebound between generator steps. > That gets especially confusing if you have multiple generator expressions > in the same scope iterating in parallel using the same binding target: > > # This is fine > gen1 = (p for p in range(10)) > gen2 = (p for p in gen1) > print(list(gen2)) > > # This is not (given the "let's reintroduce leaking from > comprehensions" proposal) > p = 0 > gen1 = (p := q for q in range(10)) > gen2 = (p, p := q for q in gen1) > print(list(gen2)) > That's just one of several "don't do that" situations. *What will happen* is perhaps hard to see at a glance, but it's perfectly well specified. Not all legal code does something useful though, and in this case the obvious advice should be to use different variables. > It also reintroduces the original problem that comprehension scopes > solved, just in a slightly different form: > > # This is fine > for x in range(10): > for y in range(10): > transposed_related_coords = [y, x for x, y in > related_coords(x, y)] > > # This is not (given the "let's reintroduce leaking from > comprehensions" proposal) > for x in range(10): > for y in range(10): > related_interesting_coords = [x, y for x in related_x_coord(x, > y) if is_interesting(y := f(x))] > > Deliberately reintroducing stateful side effects into a nominally > functional construct seems like a recipe for significant confusion, even if > there are some cases where it might arguably be useful to folks that don't > want to write a named function that returns multiple values instead. > You should really read Tim's initial post in this thread, where he explains his motivation. It sounds like you're not buying it, but your example is just a case where the user is shooting themselves in the foot by reusing variable names. When writing `:=` you should always keep the scope of the variable in mind -- it's no different when using `:=` outside a comprehension. PS. Thanks for the suggestion about conflicting signals about scope; that's what we'll do. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu May 10 09:34:52 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 10 May 2018 23:34:52 +1000 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: <20180510012256.GA25502@kundert.designers-guide.com> Message-ID: On Thu, May 10, 2018 at 11:13 PM, Facundo Batista wrote: >> I would like to see you flesh out the idea. In particular, I'd like to see you >> address cases where: >> 1. The underlying members in the collection are not strings. Besides the basic >> types such as numbers, it would also be nice to be able to apply formats >> recursively so that one can construct a string using the attributes of >> members that are objects or items or other collections. > > I didn't think about this at the time. I have the feeling (not really > well thought yet) that "recursive formatting" will complicate > everything too much, that doing str() (as {} defaults to) on every > object would fulfill most of the basic cases and keep this simple. > > >> 2. The ability to handle collections other than simple lists or iterables, such >> as dictionaries. > > Ideally, it will handle *any* iterable. If it's to handle arbitrary iterables, it can't be the normal style of "take this string, pass it to the object's __format__ method, and let it interpret it". That's why I suggested a bang notation instead. We have some already: >>> x = "O'Really?" >>> print(f"!s: {x!s} !r: {x!r}") !s: O'Really? !r: "O'Really?" Those markers apply to ANY object, and pass it through str() or repr() respectively, before using any provided format string. A "!j" flag could take an iterable, format each element using the given format, and then join them. The letter "j" makes good sense then, as it parallels str.join() - this would be broadly similar to "...".join(format(...) for x in iter). ChrisA From guido at python.org Thu May 10 09:44:41 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 10 May 2018 09:44:41 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: I'm sorry, but unless there's a sudden landslide of support for 'given' in favor of ':=', I'm really not going to consider it. I'd pronounce "if (x := y) > 0" as either "if y (assigned to x) is greater than zero" or "if x (assigned from y) is greater than zero". On Thu, May 10, 2018 at 6:39 AM, Nick Coghlan wrote: > On 8 May 2018 at 04:19, Brett Cannon wrote: > >> My brain wants to drop the variable name in front of 'given': >> >> if given m = pattern.search(data): >> >> while given m = pattern.search(remaining_data): >> >> Maybe it's because the examples use such a short variable name? >> > > Does that change if the condition isn't just "bool(name)"? For example: > > if y > 0 given y = f(x): > ... > > That's the situation where I strongly prefer the postfix operator > spelling, since it's pretty clear how I should pronounce it (i.e. "if y is > greater than zero, given y is set to f-of-x, then ..."). By contrast, while > a variety of plausible suggestions have been made, I still don't really > know how to pronounce "if (y := f(x)) > 0:)" in a way that's going to be > clear to an English-speaking listener (aside from pronouncing it the same > way as I'd pronounce the version using "given", but that then raises the > question of "Why isn't it written the way it is pronounced?"). > > I do agree with Tim that the name repetition would strongly encourage the > use of short names rather than long ones (since you're always typing them > at least twice), such that we'd probably see code like: > > while not probable_prime(n) given (n = > highbit | randrange(1, highbit, 2)): > pass > > Rather than the more explicit: > > while not probable_prime(candidate) given (candidate = > highbit | randrange(1, highbit, 2)): > pass > > However, I'd still consider both of those easier to follow than: > > while not probable_prime(candidate := highbit | randrange(1, highbit, > 2)): > pass > > since it's really unclear to me that "candidate" in the latter form is a > positional argument being bound to a name in the local environment, and > *not* a keyword argument being passed to "probable_prime". > > I've also been pondering what the given variant might look like as a > generally available postfix operator, rather than being restricted to > if/elif/while clauses, and I think that would have interesting implications > for the flexibility of its usage in comprehensions, since there would now > be *three* places where "given" could appear (as is already the case for > the inline binding operator spelling): > > - in the result expression > - in the iterable expression > - in the filter expression > > That is: > > [(x, y, x - y) given y = f(x) for x in data] > [(x, data) for x in data given data = get_data()] > [(x, y, x/y) for x in data if y given y = f(x)] > > Rather than: > > [(x, y := f(x), x - y) for x in data] > [(x, data) for x in data := get_data()] > [(x, y, x/y) for x in data if y := f(x)] > > Opening it up that way would allow for some odd usages that might need to > be discouraged in PEP 8 (like explicitly preferring "probable_prime(n) > given n = highbit | randrange(1, highbit, 2)" to "probable_prime(n given n > = highbit | randrange(1, highbit, 2))"), but it would probably still be > simpler overall than attempting to restrict the construct purely to > if/elif/while. > > Even as a generally available postfix keyword, "given" should still be > amenable to the treatment where it could be allowed as a variable name in a > non-operator context (since we don't allow two adjacent expressions to > imply a function call, it's only prefix keywords that have to be disallowed > as names to avoid ambiguity in the parser). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu May 10 09:53:40 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 10 May 2018 09:53:40 -0400 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: References: Message-ID: On Thu, May 10, 2018 at 3:32 AM, Terry Reedy wrote: > On 5/9/2018 11:33 PM, Guido van Rossum wrote: > > I now think that the best way out is to rule `:=` in the top level >> expression of an expression statement completely >> > > I would like to be able to interactively enter > > >>> a: = f(2,4) > > to have 'a' echoed as well as bound. > I hope that's a typo (the can be no space between `:` and `=`, since `:=` is a single token, just like `<='). We *could* make this work while still ruling out the ambiguous cases (which involve top-level commas on either side of the assignment expression). OTOH I worry that this particular feature would cause `:=` to become part of many a teacher's bag of tricks to show off, exposing users to it way too early for any curriculum, and it might then elicit complaints that >>> def f(): ... a := 5 ... >>> f() >>> doesn't print `5`. So all in all I'm not sure I think this is important enough to support, and the rule "Use `:=` in expressions, not as a top level assignment" seems easier to explain and understand. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu May 10 09:57:22 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 10 May 2018 09:57:22 -0400 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: <17376e38-c17c-5249-b4cb-6b8e1d441403@egenix.com> References: <17376e38-c17c-5249-b4cb-6b8e1d441403@egenix.com> Message-ID: On Thu, May 10, 2018 at 5:04 AM, M.-A. Lemburg wrote: > To a (former Pascal) programmer, a := 1 doesn't read like an > operator. It's an assignment expression. If embedded expressions > is where Python is heading, it should be made very clear where > the embedded expression starts and where it ends on a line. > The rules we've arrived at are about as straightforward as it gets: the RHS of `:=` ends at the nearest comma or close parenthesis/bracket/brace. OT about the name: despite Tim's relentless pushing of "binding expressions" in the end I think they should be called "assignment expressions" just like in C. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhodri at kynesim.co.uk Thu May 10 10:23:38 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Thu, 10 May 2018 15:23:38 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <47bbb292-362a-f6e0-f82d-eacc6ddd6df4@kynesim.co.uk> On 10/05/18 14:44, Guido van Rossum wrote: > I'm sorry, but unless there's a sudden landslide of support for 'given' in > favor of ':=', I'm really not going to consider it. OK, this is my ha'p'th in favour of 'given', for what little it's worth. The more I see of general assignment expressions, the less I like them. All I really want is a less clumsy way to write while true: thing_to_do = get_something_to_do() if thing_to_do == GET_OUT_OF_HERE: break # else do stuff -- Rhodri James *-* Kynesim Ltd From marky1991 at gmail.com Thu May 10 10:26:48 2018 From: marky1991 at gmail.com (marky1991 .) Date: Thu, 10 May 2018 10:26:48 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: If it just needs a stream of +1s, I personally like the "given" approach much more than the ":=" approach, for all of the many reasons repeated many times in the various email chains. (I preferred it as "as", but that's been struck down already) (and if it's between ":=" and not having them at all, I would rather just not have them) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu May 10 11:10:50 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 10 May 2018 11:10:50 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: Please no, it's not that easy. I can easily generate a stream of +1s or -1s for any proposal. I'd need well-reasoned explanations and it would have to come from people who are willing to spend significant time writing it up eloquently. Nick has tried his best and failed to convince me. So the bar is high. (Also note that most of the examples that have been brought up lately were meant to illustrate the behavior in esoteric corner cases while I was working out the fine details of the semantics. Users should use this feature sparingly and stay very far away of those corner cases -- but they have to be specified in order to be able to implement this thing.) On Thu, May 10, 2018 at 10:26 AM, marky1991 . wrote: > If it just needs a stream of +1s, I personally like the "given" approach > much more than the ":=" approach, for all of the many reasons repeated many > times in the various email chains. (I preferred it as "as", but that's been > struck down already) (and if it's between ":=" and not having them at all, > I would rather just not have them) > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From e+python-ideas at kellett.im Thu May 10 11:49:00 2018 From: e+python-ideas at kellett.im (Ed Kellett) Date: Thu, 10 May 2018 16:49:00 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <14d15390-fba5-023d-12f5-274d884b88a0@kellett.im> On 2018-05-10 16:10, Guido van Rossum wrote: > Please no, it's not that easy. I can easily generate a stream of +1s or -1s > for any proposal. I'd need well-reasoned explanations and it would have to > come from people who are willing to spend significant time writing it up > eloquently. Nick has tried his best and failed to convince me. So the bar > is high. > > (Also note that most of the examples that have been brought up lately were > meant to illustrate the behavior in esoteric corner cases while I was > working out the fine details of the semantics. Users should use this > feature sparingly and stay very far away of those corner cases -- but they > have to be specified in order to be able to implement this thing.) Poor prospects, then, but I'll do my best. I think the most obvious argument (to me) favouring `given` over `:=` is that it separates the two things it's doing: if m.group(2) given m = pattern.search(data): as opposed to the more-nested := version: if (m := pattern.search(data)).group(2): which, at least to me, is more complicated to think about because it feels like it's making the .group() something to do with the assignment. Put another way, I think your use of parentheses when discussing the *pronunciation* of this thing is telling. It feels as though one needs to start in the middle and then go in both directions at once, first explaining the origin (or destination) of the operand in question in a parenthesized offshoot, and then switching context and describing what is done to it. It's midly mentally taxing. I'm sure we can all live with that, but I don't want to: Python's exceptionally-readable syntax is one of the bigger reasons I choose it. There's a striking parallel in C, where the well-known idiom: while ((c = getchar()) != EOF) ... has an obviously-nicer alternative: while (c = getchar(), c != EOF) ... Most people I show this to agree that it's nicer, despite the fact that it manages to repeat a variable name *and* use the comma operator. I don't have proof, but I'd suggest that unwrapping that layer of context for the reader imparts a significant benefit. The C example also provides a convenient test: if you think the former example is nicer, I can just give up now ;) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From gjcarneiro at gmail.com Thu May 10 12:10:02 2018 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Thu, 10 May 2018 16:10:02 +0000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <14d15390-fba5-023d-12f5-274d884b88a0@kellett.im> References: <14d15390-fba5-023d-12f5-274d884b88a0@kellett.im> Message-ID: On Thu, 10 May 2018 at 16:49, Ed Kellett wrote: > On 2018-05-10 16:10, Guido van Rossum wrote: > > Please no, it's not that easy. I can easily generate a stream of +1s or > -1s > > for any proposal. I'd need well-reasoned explanations and it would have > to > > come from people who are willing to spend significant time writing it up > > eloquently. Nick has tried his best and failed to convince me. So the bar > > is high. > > > > (Also note that most of the examples that have been brought up lately > were > > meant to illustrate the behavior in esoteric corner cases while I was > > working out the fine details of the semantics. Users should use this > > feature sparingly and stay very far away of those corner cases -- but > they > > have to be specified in order to be able to implement this thing.) > > Poor prospects, then, but I'll do my best. > > I think the most obvious argument (to me) favouring `given` over `:=` is > that it separates the two things it's doing: > > if m.group(2) given m = pattern.search(data): > > as opposed to the more-nested := version: > > if (m := pattern.search(data)).group(2): > > which, at least to me, is more complicated to think about because it > feels like it's making the .group() something to do with the assignment. > > Put another way, I think your use of parentheses when discussing the > *pronunciation* of this thing is telling. It feels as though one needs > to start in the middle and then go in both directions at once, first > explaining the origin (or destination) of the operand in question in a > parenthesized offshoot, and then switching context and describing what > is done to it. It's midly mentally taxing. I'm sure we can all live with > that, but I don't want to: Python's exceptionally-readable syntax is one > of the bigger reasons I choose it. > > There's a striking parallel in C, where the well-known idiom: > > while ((c = getchar()) != EOF) ... > > has an obviously-nicer alternative: > > while (c = getchar(), c != EOF) ... > > Most people I show this to agree that it's nicer, despite the fact that > it manages to repeat a variable name *and* use the comma operator. I > don't have proof, but I'd suggest that unwrapping that layer of context > for the reader imparts a significant benefit. > > The C example also provides a convenient test: if you think the former > example is nicer, I can just give up now ;) > IMHO, all these toy examples don't translate well to the real world because they tend to use very short variable names while in real world [good written code] tends to select longer more descriptive variable names. Try replacing "c" with a longer name, like input_command, then it becomes: while ((input_command = getchar()) != EOF) ... while (input_command = getchar(), input_command != EOF) ... In the second example, having to type the variable name twice is an annoyance that adds almost nothing to readability, so I would definitely prefer the first one. The "given" proposals have the same issue. (a shame we can't use "as", for reasons already stated, it would have been perfect otherwise) -- Gustavo J. A. M. Carneiro Gambit Research "The universe is always one step beyond logic." -- Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From facundobatista at gmail.com Thu May 10 12:28:49 2018 From: facundobatista at gmail.com (Facundo Batista) Date: Thu, 10 May 2018 13:28:49 -0300 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: <20180510012256.GA25502@kundert.designers-guide.com> Message-ID: 2018-05-10 10:34 GMT-03:00 Chris Angelico : >> >> Ideally, it will handle *any* iterable. > > If it's to handle arbitrary iterables, it can't be the normal style of > "take this string, pass it to the object's __format__ method, and let > it interpret it". That's why I suggested a bang notation instead. We > have some already: Yes, I think it fits better, as it's not just a formatting, it's more an "operation"... > A "!j" flag > could take an iterable, format each element using the given format, > and then join them. The letter "j" makes good sense then, as it Where would you indicate the separator if we use the bang notation? Thanks! -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From mal at egenix.com Thu May 10 12:29:33 2018 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 10 May 2018 18:29:33 +0200 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: References: <17376e38-c17c-5249-b4cb-6b8e1d441403@egenix.com> Message-ID: <69ef44b1-d35f-d8f9-c329-655e060e4baf@egenix.com> On 10.05.2018 15:57, Guido van Rossum wrote: > On Thu, May 10, 2018 at 5:04 AM, M.-A. Lemburg wrote: > >> To a (former Pascal) programmer, a := 1 doesn't read like an >> operator. It's an assignment expression. If embedded expressions >> is where Python is heading, it should be made very clear where >> the embedded expression starts and where it ends on a line. >> > > The rules we've arrived at are about as straightforward as it gets: the RHS > of `:=` ends at the nearest comma or close parenthesis/bracket/brace. That may be easy for a computer to parse, but it's not for a programmer. It would be better to contain such expressions inside a safe container which is clearly visible to a human eye. ohoh = a := (1, 2, 3), 4, a * 2 vs. aha = ((a := (1, 2, 3)), 4, a * 2) You'd simplify the above logic to: the RHS of ":=" ends at the nearest closing parenthesis. > OT about the name: despite Tim's relentless pushing of "binding > expressions" in the end I think they should be called "assignment > expressions" just like in C. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 10 2018) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From ethan at stoneleaf.us Thu May 10 12:38:17 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 10 May 2018 09:38:17 -0700 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: <69ef44b1-d35f-d8f9-c329-655e060e4baf@egenix.com> References: <17376e38-c17c-5249-b4cb-6b8e1d441403@egenix.com> <69ef44b1-d35f-d8f9-c329-655e060e4baf@egenix.com> Message-ID: <5AF47579.1070203@stoneleaf.us> On 05/10/2018 09:29 AM, M.-A. Lemburg wrote: > On 10.05.2018 15:57, Guido van Rossum wrote: >> On Thu, May 10, 2018 at 5:04 AM, M.-A. Lemburg wrote: >>> To a (former Pascal) programmer, a := 1 doesn't read like an >>> operator. It's an assignment expression. If embedded expressions >>> is where Python is heading, it should be made very clear where >>> the embedded expression starts and where it ends on a line. >>> >> >> The rules we've arrived at are about as straightforward as it gets: the RHS >> of `:=` ends at the nearest comma or close parenthesis/bracket/brace. > > That may be easy for a computer to parse, but it's not for > a programmer. It would be better to contain such expressions > inside a safe container which is clearly visible to a human > eye. > > ohoh = a := (1, 2, 3), 4, a * 2 I have no problem reading that. > vs. > > aha = ((a := (1, 2, 3)), 4, a * 2) The extra parens are unneeded line noise (at least for me). -- ~Ethan~ From ncoghlan at gmail.com Thu May 10 12:38:52 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 11 May 2018 02:38:52 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: On 10 May 2018 at 23:22, Guido van Rossum wrote: > On Thu, May 10, 2018 at 5:17 AM, Nick Coghlan wrote: > >> How would you expect this to work in cases where the generator expression >> isn't immediately consumed? If "p" is nonlocal (or global) by default, then >> that opens up the opportunity for it to be rebound between generator steps. >> That gets especially confusing if you have multiple generator expressions >> in the same scope iterating in parallel using the same binding target: >> >> # This is fine >> gen1 = (p for p in range(10)) >> gen2 = (p for p in gen1) >> print(list(gen2)) >> >> # This is not (given the "let's reintroduce leaking from >> comprehensions" proposal) >> p = 0 >> gen1 = (p := q for q in range(10)) >> gen2 = (p, p := q for q in gen1) >> print(list(gen2)) >> > > That's just one of several "don't do that" situations. *What will happen* > is perhaps hard to see at a glance, but it's perfectly well specified. Not > all legal code does something useful though, and in this case the obvious > advice should be to use different variables. > I can use that *exact same argument* to justify the Python 2 comprehension variable leaking behaviour. We decided that was a bad idea based on ~18 years of experience with it, and there hasn't been a clear justification presented for going back on that decision presented beyond "Tim would like using it sometimes". PEP 572 was on a nice trajectory towards semantic simplification (removing sublocal scoping, restricting to name targets only, prohibiting name binding expressions in the outermost iterable of comprehensions to avoid exposing the existing scoping quirks any more than they already are), and then we suddenly had this bizarre turn into "and they're going to be implicitly nonlocal or global when used in comprehension scope". > It also reintroduces the original problem that comprehension scopes > solved, just in a slightly different form: > > # This is fine >> for x in range(10): >> for y in range(10): >> transposed_related_coords = [y, x for x, y in >> related_coords(x, y)] >> >> # This is not (given the "let's reintroduce leaking from >> comprehensions" proposal) >> for x in range(10): >> for y in range(10): >> related_interesting_coords = [x, y for x in >> related_x_coord(x, y) if is_interesting(y := f(x))] >> >> Deliberately reintroducing stateful side effects into a nominally >> functional construct seems like a recipe for significant confusion, even if >> there are some cases where it might arguably be useful to folks that don't >> want to write a named function that returns multiple values instead. >> > > You should really read Tim's initial post in this thread, where he > explains his motivation. > I did, and then I talked him out of it by pointing out how confusing it would be to have the binding semantics of "x := y" be context dependent. > It sounds like you're not buying it, but your example is just a case where > the user is shooting themselves in the foot by reusing variable names. When > writing `:=` you should always keep the scope of the variable in mind -- > it's no different when using `:=` outside a comprehension. > It *is* different, because ":=" normally binds the same as any other name binding operation including "for x in y:" (i.e. it creates a local variable), while at comprehension scope, the proposal has now become for "x := y" to create a local variable in the containing scope, while "for x in y" doesn't. Comprehension scoping is already hard to explain when its just a regular nested function that accepts a single argument, so I'm not looking forward to having to explain that "x := y" implies "nonlocal x" at comprehension scope (except that unlike a regular nonlocal declaration, it also implicitly makes it a local in the immediately surrounding scope). It isn't reasonable to wave this away as "It's only confusing to Nick because he's intimately familiar with how comprehensions are implemented", as I also wrote some of the language reference docs for the current (already complicated) comprehension scoping semantics, and I can't figure out how we're going to document the proposed semantics in a way that will actually be reasonably easy for readers to follow. The best I've been able to come up with is: - for comprehensions at function scope (including in a lambda expression inside a comprehension scope), a binding expression targets the nearest function scope, not the comprehension scope, or any intervening comprehension scope. It will appear in locals() the same way nonlocal references usually do. - for comprehensions at module scope, a binding expression targets the global scope, not the comprehension scope, or any intervening comprehension scope. It will not appear in locals() (as with any other global reference). - for comprehensions at class scope, the class scope is ignored for purposes of determining the target binding scope (and hence will implicitly create a new global variable when used in a top level class definition, and new function local when used in a class definition nested inside a function) Sublocal scopes were a model of simplicity by comparison :) Cheers, Nick. P.S. None of the above concerns apply to explicit inline scope declarations, as those are easy to explain by saying that the inline declarations work the same way as the scope declaration statements do, and can be applied universally to all name binding operations rather than being specific to ":= in comprehension scope". -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu May 10 12:49:49 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 10 May 2018 17:49:49 +0100 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: <5AF47579.1070203@stoneleaf.us> References: <17376e38-c17c-5249-b4cb-6b8e1d441403@egenix.com> <69ef44b1-d35f-d8f9-c329-655e060e4baf@egenix.com> <5AF47579.1070203@stoneleaf.us> Message-ID: On 10 May 2018 at 17:38, Ethan Furman wrote: > On 05/10/2018 09:29 AM, M.-A. Lemburg wrote: >> >> On 10.05.2018 15:57, Guido van Rossum wrote: >>> >>> On Thu, May 10, 2018 at 5:04 AM, M.-A. Lemburg wrote: > > >>>> To a (former Pascal) programmer, a := 1 doesn't read like an >>>> operator. It's an assignment expression. If embedded expressions >>>> is where Python is heading, it should be made very clear where >>>> the embedded expression starts and where it ends on a line. >>>> >>> >>> The rules we've arrived at are about as straightforward as it gets: the >>> RHS >>> of `:=` ends at the nearest comma or close parenthesis/bracket/brace. >> >> >> That may be easy for a computer to parse, but it's not for >> a programmer. It would be better to contain such expressions >> inside a safe container which is clearly visible to a human >> eye. >> >> ohoh = a := (1, 2, 3), 4, a * 2 > > > I have no problem reading that. > >> vs. >> >> aha = ((a := (1, 2, 3)), 4, a * 2) > > > The extra parens are unneeded line noise (at least for me). I found the version with extra parens *harder* to read. Although I will admit neither is particularly easy to read (not surprising as it's an artificial example intended to prove a point, not a real world use case) and in practice I'd write a = 1, 2, 3 ohoh = a, 4, a * 2 Look ma, no parens! Paul From alexander.belopolsky at gmail.com Thu May 10 13:18:21 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 10 May 2018 13:18:21 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On Thu, May 10, 2018 at 9:44 AM, Guido van Rossum wrote: > I'm sorry, but unless there's a sudden landslide of support for 'given' in > favor of ':=', I'm really not going to consider it. How much support was there for ":="? Are you serious about bringing back Pascal and Algol from their comfortable resting places? From rosuav at gmail.com Thu May 10 13:36:10 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 11 May 2018 03:36:10 +1000 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: <20180510012256.GA25502@kundert.designers-guide.com> Message-ID: On Fri, May 11, 2018 at 2:28 AM, Facundo Batista wrote: > 2018-05-10 10:34 GMT-03:00 Chris Angelico : >> A "!j" flag >> could take an iterable, format each element using the given format, >> and then join them. The letter "j" makes good sense then, as it > > Where would you indicate the separator if we use the bang notation? It's your proposal, you can decide what works best for you. :) I don't think there's anything quite like this, so go ahead and pick whatever you reckon makes the most sense. ChrisA From tim.peters at gmail.com Thu May 10 13:36:56 2018 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 10 May 2018 12:36:56 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: [Nick Coghlan ] > How would you expect this to work in cases where the generator expression > isn't immediately consumed? If "p" is nonlocal (or global) by default, then > that opens up the opportunity for it to be rebound between generator steps. > That gets especially confusing if you have multiple generator expressions in > the same scope iterating in parallel using the same binding target: I'm most interested in what sensible programmers can do easily that's of use, not really about pathologies that can be contrived. > # This is fine > gen1 = (p for p in range(10)) > gen2 = (p for p in gen1) > print(list(gen2)) Sure. > > # This is not (given the "let's reintroduce leaking from comprehensions" proposal) Be fair: it's not _re_introducing anything. It's brand new syntax for which "it's a very much intended feature" that a not-local name can be bound. You have to go out of your way to use it. Where it doesn't do what you want, don't use it. > p = 0 I'm not sure of the intent of that line. If `p` is otherwise unknown in this block, its appearance as a binding operator target in an immediately contained genexp establishes that `p` is local to this block. So `p = 0` here just establishes that directly. Best I can guess, the 0 value is never used below. > gen1 = (p := q for q in range(10)) I expect that's a compile time error, grouping as gen1 = (p := (q for q in range(10))) but without those explicit parentheses delimiting the "genexp part" it may not be _recognized_ as being a genexp. With the extra parens, it binds both `gen1` and `p` to the genexp, and `p` doesn't appear in the body of the genexp at all. Or did you intend gen1 = ((p := q) for q in range(10)) ? I'll assume that's so. > gen2 = (p, p := q for q in gen1) OK, I really have no guess about the intent there. Note that gen2 = (p, q for q in gen1) is a syntax error today, while gen2 = (p, (q for q in gen1)) builds a 2-tuple. Perhaps gen2 = ((p, p := q) for q in gen1) was intended? Summarizing: gen1 = ((p := q) for q in range(10)) gen2 = ((p, p := q) for q in gen1) is my best guess. > print(list(gen2)) [(0, 0), (1, 1), (2, 2), ..., (9, 9)] But let's not pretend it's impossible to do that today; e.g., this code produces the same: class Cell: def __init__(self, value=None): self.bind(value) def bind(self, value): self.value = value return value p = Cell() gen1 = (p.bind(q) for q in range(10)) gen2 = ((p.value, p.bind(q)) for q in gen1) print(list(gen2)) Someone using ":=" INTENDS to bind the name, just as much as someone deliberately using that `Cell` class. > It also reintroduces the original problem that comprehension scopes solved, > just in a slightly different form: > > # This is fine > for x in range(10): > for y in range(10): > transposed_related_coords = [y, x for x, y in related_coords(x, y)] I'm not clear on what "This is fine" means, other than that the code does whatever it does. That's part of why I so strongly prefer real-life use cases. In the code above, I can't imagine what the intent of the code might be _unless_ they're running tons of otherwise-useless code for _side effects_ performed by calling `related_coords()`. If "it's functional", they could do the same via x = y = 9 transposed_related_coords = [y, x for x, y in related_coords(x, y)] except that's a syntax error ;-) I assume transposed_related_coords = [(y, x) for x, y in related_coords(x, y)] was intended. BTW, I'd shoot anyone who tried to check in that code today ;-) It inherently relies on that the name `x` inside the listcomp refers to two entirely different scopes, and that's Poor Practice (the `x` in the `related_coords()` call refers to the `x` in `for x in range(10)`, but all other instances of `x` refer to the listcomp-local `x`). > # This is not (given the "let's reintroduce leaking from comprehensions" proposal) > for x in range(10): > for y in range(10): > related_interesting_coords = [x, y for x in related_x_coord(x, y) > if is_interesting(y := f(x))] Same syntax error there (you need parens around "x, y" at the start of the listcomp). Presumably they _intended_ to build (x, f(x)) pairs when and only when `f(x)` "is interesting". In what specific way does the code fail to do that? Yes, the outer `y` is rebound, but what of it? When the statement completes, `y` will be rebound to the next value from the inner range(10), and that's the value of `y` seen by `related_x_coord(x, y)` the next time the loop body runs. The binding done by `:=` is irrelevant to that. So I don't see your point in that specific example, although - sure! - of course it's possible to contrive examples where it really would matter. For example, change the above in some way to use `x` as the binding operator target inside the listcomp. Then that _could_ affect the value of `x` seen by `related_x_coord(x, y)` across inner loop iterations. > Deliberately reintroducing stateful side effects into a nominally functional > construct seems like a recipe for significant confusion, Side effects of any kind anywhere can create significant confusion. But Python is not a functional language, and it you don't want side effects due to ":=" in synthetic functions, you're not required to use ":=" in that context. That said, I agree "it would be nice" if advanced users had a way to explicitly say which scope they want. > even if there are some cases where it might arguably be useful to folks > that don't want to write a named function that returns multiple values instead. Sorry, I didn't follow that - functions returning multiple values? From ed.page at ni.com Thu May 10 13:30:20 2018 From: ed.page at ni.com (Ed Page) Date: Thu, 10 May 2018 17:30:20 +0000 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: , Message-ID: Greetings, ? Is there interest in a PEP for extending time, datetime / timedelta for arbitrary or extended precision fractional seconds? ? My company designs and manufactures scientific hardware that typically operate with nanoseconds -- sometimes even attoseconds -- levels of precision.? We?re in the process of providing Python APIs for some of these products and need to expose the full accuracy of the data to our customers.? Doing so would allow developers to do things like timestamp analog measurements for correlating with other events in their system, or precisely schedule a future time event for correctly interoperating with other high-speed devices.? ? The API we?ve been toying with is adding two new fields to time, datetime and timedelta - frac_seconds (int) - frac_seconds_exponent (int or new SITimeUnit enum) ? time.microseconds would be turned into a property that wraps frac_seconds for compatibility ? Challenges - Defining the new `max` or `resolution` - strftime / strptime.? I propose that we do nothing, just leave formatting / parsing to use `microseconds` at best.? On the other hand, __str__ could just specify the fractional seconds using scientific or engineering notation. ? Alternatives - My company create our own datetime library ? - Continued fracturing of time ... ecosystem (datetime, arrow, pendulum, delorean, datetime64, pandas.Timestamp ? all of which offer varying degrees of compatibility) - Add an `attosecond` field and have `microsecond` wrap this. ? - Effectively same except hard code `frac_seconds_exponent` to lowest value ? - The most common cases (milliseconds, microseconds) will always pay the cost of using a bigint as compared to the proposal which is a "pay for what you use" approach ? - How do we define what is "good enough" precision? - Continue to subdivide time by adding `nanosecond` that is "nanoseconds since last micosecond", `picosecond` that is "picoseconds since last micnanosecond", and? `attosecond` field that is "attoseconds since last picosecond" ? - Possibly surprising API; people might expect `picosecond` to be an offset since last second ? - Messy base 10 / base 2 conversions - Have `frac_seconds` be a float ? - This has precision issues. ? If anyone wants to have an impromptu BoF on the subject, I'm available at PyCon. Thanks Ed Page From tim.peters at gmail.com Thu May 10 14:05:53 2018 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 10 May 2018 13:05:53 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: Just a quickie - I'm out of time for now. [Guido] >> That's just one of several "don't do that" situations. *What will happen* >> is perhaps hard to see at a glance, but it's perfectly well specified. Not >> all legal code does something useful though, and in this case the obvious >> advice should be to use different variables. [Nick] > I can use that *exact same argument* to justify the Python 2 comprehension > variable leaking behaviour. We decided that was a bad idea based on ~18 > years of experience with it, Here's the practical difference: you can't write a listcomp or genexp AT ALL without a "for" clause, so whether "for" target names leak is an issue in virtually every listcomp or genexp ever written. Here's one where it isn't: [None for somelist[12] in range(10)] Which nobody has ever seen in real life ;-) But ":=" is never required to write one - you only use it when you go out of your way to use it. I expect that will be relatively rare in real life listcomps and genexps. > and there hasn't been a clear justification presented for going back on that > decision Nobody is suggesting going back on "all and only `for` target names are local to the genexp/listcomp". To the contrary, the proposal preserves that verbatim: It's not _adding_ "oh, ya, and binding operator targets are local too". Just about everything here follows from _not_ adding that. > presented beyond "Tim would like using it sometimes". So long as I'm the only one looking at real-life use cases, mine is the only evidence I care about ;-) I don't really care about contrived examples, unless they illustrate that a proposal is ill-defined, impossible to implement as intended, or likely to have malignant unintended consequences out-weighing their benefits. From brenbarn at brenbarn.net Thu May 10 14:23:00 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Thu, 10 May 2018 11:23:00 -0700 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: <5AF48E04.6060802@brenbarn.net> On 2018-05-10 11:05, Tim Peters wrote: > So long as I'm the only one looking at real-life use cases, mine is > the only evidence I care about ;-) I don't really care about > contrived examples, unless they illustrate that a proposal is > ill-defined, impossible to implement as intended, or likely to have > malignant unintended consequences out-weighing their benefits. You keep saying things like this with a smiley, and I realize you know what you're talking about (much more than I do), but I'd just like to push back a bit against that entire concept. Number one, I think many people have been bringing in real life uses cases. Number two, I disagree with the idea that looking at individual use cases and ignoring logical argumentation is the way to go. The problem with it is that a lot of the thorny issues arise in unanticipated interactions between constructs that were designed to handle separate use cases. I also do not think it's appropriate to say "if it turns out there's a weird interaction between two features, then just don't use those two things together". One of the great things about Python's design is that it doesn't just make it easy for us to write good code, but in many ways makes it difficult for us to write bad code. It is absolutely a good idea to think of the broad range of wacky things that COULD be done with a feature, not just the small range of things in the focal area of its intended use. We may indeed decide that some of the wacky cases are so unlikely that we're willing to accept them, but we can only decide that after we consider them. You seem to be suggesting that we shouldn't even bother thinking about such corner cases at all, which I think is a dangerous mistake. Taking the approach of "this individual use case justifies this individual feature", leads to things like JavaScript, a hellhole of special cases, unintended consequences, and incoherence between different corners of the language. There are real cognitive benefits to having language features make logical and conceptual sense IN ADDITION TO having practical utility, and fit together into a unified whole. Personally my feeling on this whole thread is that these changes, if implemented are likely to decrease the average readability of Python code, and I don't see the benefits as being worth the added complexity. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From storchaka at gmail.com Thu May 10 14:40:32 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 10 May 2018 21:40:32 +0300 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: Message-ID: 09.05.18 15:39, Facundo Batista ????: >>>> authors = ["John", "Mary", "Estela"] >>>> "Authors: {:, j}".format(authors) > 'Authors: John, Mary, Estela' In the case of the list of book or article authors it would be better to get "John, Mary and Estela" or "John, Mary, and Estela". In other cases "John, Mary & Estela" can be more appropriate. I mean that in real case you will need to use an external function that implements more complex algorithm for formatting a list. From mertz at gnosis.cx Thu May 10 15:15:03 2018 From: mertz at gnosis.cx (David Mertz) Date: Thu, 10 May 2018 19:15:03 +0000 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: This feels specialized enough to belong in a third party library. If that library can behave as transparently as possible interacting with Python datetime, so much the better. But the need is niche enough I don't think it belongs in standard library. ... this as someone who actually worked in a lab that measured MD simulations in attoseconds. I do understand the purpose. On Thu, May 10, 2018, 2:00 PM Ed Page wrote: > Greetings, > > Is there interest in a PEP for extending time, datetime / timedelta for > arbitrary or extended precision fractional seconds? > > My company designs and manufactures scientific hardware that typically > operate with nanoseconds -- sometimes even attoseconds -- levels of > precision. We?re in the process of providing Python APIs for some of these > products and need to expose the full accuracy of the data to our > customers. Doing so would allow developers to do things like timestamp > analog measurements for correlating with other events in their system, or > precisely schedule a future time event for correctly interoperating with > other high-speed devices. > > The API we?ve been toying with is adding two new fields to time, datetime > and timedelta > - frac_seconds (int) > - frac_seconds_exponent (int or new SITimeUnit enum) > > time.microseconds would be turned into a property that wraps frac_seconds > for compatibility > > Challenges > - Defining the new `max` or `resolution` > - strftime / strptime. I propose that we do nothing, just leave > formatting / parsing to use `microseconds` at best. On the other hand, > __str__ could just specify the fractional seconds using scientific or > engineering notation. > > Alternatives > - My company create our own datetime library > - Continued fracturing of time ... ecosystem (datetime, arrow, pendulum, > delorean, datetime64, pandas.Timestamp ? all of which offer varying degrees > of compatibility) > - Add an `attosecond` field and have `microsecond` wrap this. > - Effectively same except hard code `frac_seconds_exponent` to lowest > value > - The most common cases (milliseconds, microseconds) will always pay the > cost of using a bigint as compared to the proposal which is a "pay for what > you use" approach > - How do we define what is "good enough" precision? > - Continue to subdivide time by adding `nanosecond` that is "nanoseconds > since last micosecond", `picosecond` that is "picoseconds since last > micnanosecond", and `attosecond` field that is "attoseconds since last > picosecond" > - Possibly surprising API; people might expect `picosecond` to be an > offset since last second > - Messy base 10 / base 2 conversions > - Have `frac_seconds` be a float > - This has precision issues. > > If anyone wants to have an impromptu BoF on the subject, I'm available at > PyCon. > > Thanks > Ed Page > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gvanrossum at gmail.com Thu May 10 15:35:14 2018 From: gvanrossum at gmail.com (Guido van Rossum) Date: Thu, 10 May 2018 19:35:14 +0000 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: I have to agree with David that this seems too specialized to make room for in the stdlib. On Thu, May 10, 2018, 15:16 David Mertz wrote: > This feels specialized enough to belong in a third party library. If that > library can behave as transparently as possible interacting with Python > datetime, so much the better. But the need is niche enough I don't think it > belongs in standard library. > > ... this as someone who actually worked in a lab that measured MD > simulations in attoseconds. I do understand the purpose. > > On Thu, May 10, 2018, 2:00 PM Ed Page wrote: > >> Greetings, >> >> Is there interest in a PEP for extending time, datetime / timedelta for >> arbitrary or extended precision fractional seconds? >> >> My company designs and manufactures scientific hardware that typically >> operate with nanoseconds -- sometimes even attoseconds -- levels of >> precision. We?re in the process of providing Python APIs for some of these >> products and need to expose the full accuracy of the data to our >> customers. Doing so would allow developers to do things like timestamp >> analog measurements for correlating with other events in their system, or >> precisely schedule a future time event for correctly interoperating with >> other high-speed devices. >> >> The API we?ve been toying with is adding two new fields to time, datetime >> and timedelta >> - frac_seconds (int) >> - frac_seconds_exponent (int or new SITimeUnit enum) >> >> time.microseconds would be turned into a property that wraps frac_seconds >> for compatibility >> >> Challenges >> - Defining the new `max` or `resolution` >> - strftime / strptime. I propose that we do nothing, just leave >> formatting / parsing to use `microseconds` at best. On the other hand, >> __str__ could just specify the fractional seconds using scientific or >> engineering notation. >> >> Alternatives >> - My company create our own datetime library >> - Continued fracturing of time ... ecosystem (datetime, arrow, >> pendulum, delorean, datetime64, pandas.Timestamp ? all of which offer >> varying degrees of compatibility) >> - Add an `attosecond` field and have `microsecond` wrap this. >> - Effectively same except hard code `frac_seconds_exponent` to lowest >> value >> - The most common cases (milliseconds, microseconds) will always pay >> the cost of using a bigint as compared to the proposal which is a "pay for >> what you use" approach >> - How do we define what is "good enough" precision? >> - Continue to subdivide time by adding `nanosecond` that is "nanoseconds >> since last micosecond", `picosecond` that is "picoseconds since last >> micnanosecond", and `attosecond` field that is "attoseconds since last >> picosecond" >> - Possibly surprising API; people might expect `picosecond` to be an >> offset since last second >> - Messy base 10 / base 2 conversions >> - Have `frac_seconds` be a float >> - This has precision issues. >> >> If anyone wants to have an impromptu BoF on the subject, I'm available at >> PyCon. >> >> Thanks >> Ed Page >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu May 10 15:48:11 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 10 May 2018 19:48:11 +0000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: Yes. On Thu, May 10, 2018, 13:18 Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > On Thu, May 10, 2018 at 9:44 AM, Guido van Rossum > wrote: > > I'm sorry, but unless there's a sudden landslide of support for 'given' > in > > favor of ':=', I'm really not going to consider it. > > How much support was there for ":="? Are you serious about bringing > back Pascal and Algol from their comfortable resting places? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Thu May 10 15:56:31 2018 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 10 May 2018 15:56:31 -0400 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: <20180510012256.GA25502@kundert.designers-guide.com> Message-ID: On 5/10/18 12:28 PM, Facundo Batista wrote: > 2018-05-10 10:34 GMT-03:00 Chris Angelico : > >>> >>> Ideally, it will handle *any* iterable. >> >> If it's to handle arbitrary iterables, it can't be the normal style of >> "take this string, pass it to the object's __format__ method, and let >> it interpret it". That's why I suggested a bang notation instead. We >> have some already: > > Yes, I think it fits better, as it's not just a formatting, it's more > an "operation"... > >> A "!j" flag >> could take an iterable, format each element using the given format, >> and then join them. The letter "j" makes good sense then, as it > > Where would you indicate the separator if we use the bang notation? You would have to wedge it in before the colon. Something like: f'{lst!j, :20}' But then you couldn't have a colon in the separator. And this would be our first conversion character to be more than a single character. I'm opposed to this proposal. Just call str.join(), or write a helper function or class (depending on what you're trying to do). It's not worth complicating what's already a complicated topic. Eric From warste at gmail.com Thu May 10 15:58:12 2018 From: warste at gmail.com (stefano) Date: Thu, 10 May 2018 19:58:12 +0000 Subject: [Python-ideas] Sorry for yet another self discussion In-Reply-To: References: Message-ID: I know that "self" parameter have been discussed a lot, but still I didn't find this proposal. If it was instead take my sincere apologies and please forget this mail. The disturbing part of the "self parameter" is the asymmetry of the definition and the call. So I was thinking: why not do define the methods like: "def self.whatevermethod(par1, par2, etc)" instead of "def whatevermethod(self, par1, par2, etc)"? This will allow the call and the definition to be written exactly in the same way, still leaving the "clarity" of the additional input for the function. Moreover this can be made backward compatible (even still without making self a reserved word, to be considered anyway). I've been short by purpose but ready to elaborate if needed. Thank you for the attention. _Stefano From mertz at gnosis.cx Thu May 10 16:00:22 2018 From: mertz at gnosis.cx (David Mertz) Date: Thu, 10 May 2018 20:00:22 +0000 Subject: [Python-ideas] Have a "j" format option for lists In-Reply-To: References: <20180510012256.GA25502@kundert.designers-guide.com> Message-ID: -1 on idea. Far too many edge cases that can't be handled... Or absurd complexity added to format mini-language if it tries to handle them (and still fails). On Thu, May 10, 2018, 3:57 PM Eric V. Smith wrote: > On 5/10/18 12:28 PM, Facundo Batista wrote: > > 2018-05-10 10:34 GMT-03:00 Chris Angelico : > > > >>> > >>> Ideally, it will handle *any* iterable. > >> > >> If it's to handle arbitrary iterables, it can't be the normal style of > >> "take this string, pass it to the object's __format__ method, and let > >> it interpret it". That's why I suggested a bang notation instead. We > >> have some already: > > > > Yes, I think it fits better, as it's not just a formatting, it's more > > an "operation"... > > > >> A "!j" flag > >> could take an iterable, format each element using the given format, > >> and then join them. The letter "j" makes good sense then, as it > > > > Where would you indicate the separator if we use the bang notation? > > You would have to wedge it in before the colon. Something like: > > f'{lst!j, :20}' > > But then you couldn't have a colon in the separator. And this would be > our first conversion character to be more than a single character. > > I'm opposed to this proposal. Just call str.join(), or write a helper > function or class (depending on what you're trying to do). It's not > worth complicating what's already a complicated topic. > > Eric > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Thu May 10 16:00:26 2018 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 10 May 2018 20:00:26 +0000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: Probably going to completely lose this, but would it be possible to have a vote? +1 for either 'given' and/or ':='? On Thu, May 10, 2018 at 2:48 PM Guido van Rossum wrote: > Yes. > > On Thu, May 10, 2018, 13:18 Alexander Belopolsky < > alexander.belopolsky at gmail.com> wrote: > >> On Thu, May 10, 2018 at 9:44 AM, Guido van Rossum >> wrote: >> > I'm sorry, but unless there's a sudden landslide of support for 'given' >> in >> > favor of ':=', I'm really not going to consider it. >> >> How much support was there for ":="? Are you serious about bringing >> back Pascal and Algol from their comfortable resting places? >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan (????) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else https://refi64.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu May 10 16:05:41 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 10 May 2018 16:05:41 -0400 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: References: Message-ID: On 5/10/2018 9:53 AM, Guido van Rossum wrote: > On Thu, May 10, 2018 at 3:32 AM, Terry Reedy > > wrote: > > On 5/9/2018 11:33 PM, Guido van Rossum wrote: > > I now think that the best way out is to rule `:=` in the top > level expression of an expression statement completely > > > I would like to be able to interactively enter > > >>> a: = f(2,4) > > to have 'a' echoed as well as bound. > > > I hope that's a typo (the can be no space between `:` and `=`, since > `:=` is a single token, just like `<='). a := f(2,4) # corrected ;-) > We *could* make this work while still ruling out the ambiguous cases > (which involve top-level commas on either side of the assignment > expression). > > OTOH I worry that this particular feature would cause `:=` to become > part of many a teacher's bag of tricks to show off, until someone tries a := [0]*10000000 > exposing users to it way too early for any curriculum, > and it might then elicit complaints that > >>>> def f(): > ...???? a := 5 > ... >>>> f() >>>> > > doesn't print `5`. Although the reason the same as for any expression, I can believe that people will see it as different. A bare assignment expression *looks* like a statement in a way that other expressions do not. > So all in all I'm not sure I think this is important enough to support, > and the rule "Use `:=` in expressions, not as a top level assignment" > seems easier to explain and understand. -- Terry Jan Reedy From mertz at gnosis.cx Thu May 10 16:11:11 2018 From: mertz at gnosis.cx (David Mertz) Date: Thu, 10 May 2018 20:11:11 +0000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: Only the BDFL has a vote with non-zero weight. But I'll contribute my zero-weighted preference for :=. On Thu, May 10, 2018, 4:01 PM Ryan Gonzalez wrote: > Probably going to completely lose this, but would it be possible to have a > vote? +1 for either 'given' and/or ':='? > > On Thu, May 10, 2018 at 2:48 PM Guido van Rossum wrote: > >> Yes. >> >> On Thu, May 10, 2018, 13:18 Alexander Belopolsky < >> alexander.belopolsky at gmail.com> wrote: >> >>> On Thu, May 10, 2018 at 9:44 AM, Guido van Rossum >>> wrote: >>> > I'm sorry, but unless there's a sudden landslide of support for >>> 'given' in >>> > favor of ':=', I'm really not going to consider it. >>> >>> How much support was there for ":="? Are you serious about bringing >>> back Pascal and Algol from their comfortable resting places? >>> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > -- > Ryan (????) > Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else > https://refi64.com/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu May 10 16:21:29 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 10 May 2018 16:21:29 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On 5/10/2018 9:44 AM, Guido van Rossum wrote: > I'm sorry, but unless there's a sudden landslide of support for 'given' > in favor of ':=', I'm really not going to consider it. > > I'd pronounce "if (x := y) > 0" as either "if y (assigned to x) is > greater than zero" or "if x (assigned from y) is greater than zero". or "if x (bound to y) is greater than zero" ("do something with x"). -- Terry Jan Reedy From kirillbalunov at gmail.com Thu May 10 16:44:25 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Thu, 10 May 2018 23:44:25 +0300 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: 2018-05-10 16:44 GMT+03:00 Guido van Rossum : > I'm sorry, but unless there's a sudden landslide of support for 'given' in > favor of ':=', I'm really not going to consider it. > > I'd pronounce "if (x := y) > 0" as either "if y (assigned to x) is greater > than zero" or "if x (assigned from y) is greater than zero". > I think you do not quite objectively look at the current situation. Many just lost interest in attempts to move the topic at least a little bit in the other way, seeing how you and Tim so actively expresses support/protects this `:=` syntax, while ignoring or pushing out alternative opinions :-). Of course, the latter is partly due to the incredible number of different threads and messages on this topic. Briefly: Initially, the main argument in favor of `:=` was that this form is similar to the usual assignment statement, but can be used as an expression. Ok. Then everyone agreed with the idea that it's necessary to limit assignment target to name only. Although all this criticism was actually put forward in the first 50-100 messages on the topic. In the same first hundred it was actively discussed, that in fact, this idea gives a win only in `while` and `if` statemetns that probably will match 99%+ where it will be used for its intended purpose. At the same time, most of the criticism concerned exactly the use in generators and comprehenshions, they are already often overloaded for perception. And as you once said - "Language Design Is Not Just Solving Puzzles". There was also discussed the difference in perception between the `expr op name` and `name op expr`. Here the expression is something that is important, the name is only a convenient consequence. At the moment with all the constraints of `:=`, the discuscation is more like - trying to cram this syntax into the language. While for those who are familiar with Pascal, Icon and other languages that use this syntax, this - `:=` looks natural. For others and I believe such a majority among users, this syntax is, to put it mildly, not natural and ugly producing a line noise, the colon `:` symbol is already used in a lot of places. With all the changes, the limitations and magic with scopes. Is it now easier to explain all the differences between `=` and `:=`, than the difference between `if expr as name ...: ...` and `with expr as name:`? Therefore, I take Nick's insistent position as an attempt to at least somehow make an alternative look at this topic. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu May 10 17:22:59 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 10 May 2018 17:22:59 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On 10 May 2018 at 11:10, Guido van Rossum wrote: > Please no, it's not that easy. I can easily generate a stream of +1s or > -1s for any proposal. I'd need well-reasoned explanations and it would have > to come from people who are willing to spend significant time writing it up > eloquently. Nick has tried his best and failed to convince me. So the bar > is high. > > (Also note that most of the examples that have been brought up lately were > meant to illustrate the behavior in esoteric corner cases while I was > working out the fine details of the semantics. Users should use this > feature sparingly and stay very far away of those corner cases -- but they > have to be specified in order to be able to implement this thing.) > I raised this with some of the folks that were still here at the Education Summit (similar to what I did for data classes at the PyCon Australia education seminar last year), but whereas the reactions to data classes were "as easy or easier to teach as traditional classes", the reaction for this for the folks that I asked was almost entirely negative - the most positive reaction was "Yes, if it's as a wholesale replacement for the '=' spelling, since that sometimes gets confused with mathematical equality". As an *addition* to the existing spelling, and especially with the now proposed leaking semantics in comprehension scopes, it was "No, that would just confuse out students". It's one thing adding syntactic and semantic complexity for the sake of something that significantly increases the language's expressive power (which is what the original sublocal scopes proposal was aiming for: the ability to more readily express constrained scoping and name shadowing without explicit name aliasing and del statements), it's something else entirely to do it for the sake of purely cosmetic tweaks like flattening the occasional nested if-else chain or replacing a loop-and-a-half with an embedded assignment. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Thu May 10 17:32:02 2018 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 10 May 2018 16:32:02 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: <5AF48E04.6060802@brenbarn.net> References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> <5AF48E04.6060802@brenbarn.net> Message-ID: [Tim] >> ... >> So long as I'm the only one looking at real-life use cases, mine is >> the only evidence I care about ;-) I don't really care about >> contrived examples, unless they illustrate that a proposal is >> ill-defined, impossible to implement as intended, or likely to have >> malignant unintended consequences out-weighing their benefits. [Brendan Barnwell ] > You keep saying things like this with a smiley, and I realize you > know what you're talking about (much more than I do), but I'd just like to > push back a bit against that entire concept. I'm not so keen on meta-discussions either ;-) > Number one, I think many people have been bringing in real life uses > cases. Keep in mind the context here: _this_ thread is specifically about listcomps and genexps. I agree there have been tons of use cases presented for statement-oriented applications (some positive for the feature, some negative), but not so much for listcomps and genexps. It's worth noting again that "the" use case that started all this long ago was a listcomp that the current PEP points out still "won't work": total = 0 progressive_sums = [total := total + value for value in data] It's obvious what that's intended to do. It's not obvious why it blows up. It's a question of scope, and the scopes of names in synthesized functions is a thoroughly legitimate thing to question. The suggestion made in the first message of this thread was the obvious scope change needed to make that example work, although I was motivated by looking at _other_ listcomp/genexp use cases. They wanted the same scope decision as the example above. But I didn't realize that the example above was essentially the same thing until after I made the suggestion. > Number two, I disagree with the idea that looking at individual use > cases and ignoring logical argumentation is the way to go. Fine, then you argue, and I'll look at use cases ;-) Seriously, I don't at all ignore argument - but, yes, arguments are secondary to me. I don't give a rip about how elegant something is if it turns out to be unusable. Conversely, I don't _much_ care about how "usable" something is if the mental model for how it works is inexplicable. > The problem with it is that a lot of the thorny issues arise in > unanticipated interactions between constructs that were > designed to handle separate use cases. Sure. > I also do not think it's appropriate to say "if it turns out there's > a weird interaction between two features, then just don't use those two > things together". Sometimes it is, sometimes it isn't. For example, code using threads has to be aware of literal mountains of other features that may not work well (or at all) in a multi-threaded environment without major rewriting. Something as simple as "count += 1" may fail in mysterious ways otherwise. So it goes. But note that this is easily demonstrated by realistic code. > One of the great things about Python's design is that it doesn't just > make it easy for us to write good code, but in many ways makes > it difficult for us to write bad code. That one I disagree with. It's very easy to write bad code in every language I'm aware of. It's just that Python programmers are too enlightened to approve of doing so ;-) > It is absolutely a good idea to think of the broad range of wacky things that > COULD be done with a feature, So present some! > not just the small range of things in the focal area of its intended use. > We may indeed decide that some of the wacky cases are so unlikely that we're > willing to accept them, but we can only decide that after we consider them. > You seem to be suggesting that we shouldn't even bother thinking about such > corner cases at all, which I think is a dangerous mistake. To the contrary, bring 'em on. But there is no feature in Python you can't make "look bad" by contriving examples, from two-page regular expressions to `if` statements nested 16 deep. "But no sane person would do that" is usually - but not always - "refutation" enough for such stuff. > Taking the approach of "this individual use case justifies this > individual feature", leads to things like JavaScript, a hellhole of special > cases, unintended consequences, and incoherence between different corners of > the language. There are real cognitive benefits to having language features > make logical and conceptual sense IN ADDITION TO having practical utility, > and fit together into a unified whole. I haven't ignored that here. The scope rule for synthesized functions implementing regexps and listcomps _today_ is: The names local to that function are the names appearing as `for` targets. All other names resolve to the same scopes they resolve to in the block containing the synthesized function. The scope rule if the suggestion is adopted? The same, along with that a name appearing as a ":=" target establishes that the name is local to the containing block _if_ that name is otherwise unknown in the containing block. There's nothing incoherent or illogical about that, provided that you understand how Python scoping works at all. It's not, e.g., adding any _new_ concept of "scope" - just spelling out what the intended scopes are. Of course it's worth noting that the scope decision made for ";=" targets in listcomps/genexps differs from the decision made for `for` target names. It's use cases that decide, for me, whether that's "the tail" or "the dog". Look again at the `progressive_sums` example above, and tell me whether _you'll_ be astonished if it works. Then are you astonished that >>> x = 0 >>> ignore = [x := 1] >>> x 1 displays 1? Either way, are you astonished that >>> x = 0 >>> ignore = [x := 1 for i in range(1)] >>> x 1 also displays 1? If you want to argue about "logical and conceptual sense", I believe you'll get lost in abstractions unless you _apply_ your theories to realistic examples. > Personally my feeling on this whole thread is that these changes, if > implemented are likely to decrease the average readability of Python code, > and I don't see the benefits as being worth the added complexity. Of course consensus will never be reached. That's why Guido is paid riches beyond the dreams of avarice ;-) From alexander.belopolsky at gmail.com Thu May 10 18:13:24 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 10 May 2018 22:13:24 +0000 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: > Is there interest in a PEP for extending time, datetime / timedelta for arbitrary or extended precision fractional seconds? Having seen the utter disaster that similar ideas brought to numpy, I would say: no. On the other hand, nanoseconds are slowly making their way to the stdlib and to add nanoseconds to datetime we only need a fully backward compatible implementation, not even a PEP. See . From steve at pearwood.info Thu May 10 18:33:48 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 11 May 2018 08:33:48 +1000 Subject: [Python-ideas] Sorry for yet another self discussion In-Reply-To: References: Message-ID: <20180510223348.GQ9562@ando.pearwood.info> On Thu, May 10, 2018 at 07:58:12PM +0000, stefano wrote: > The disturbing part of the "self parameter" is the asymmetry of the > definition and the call. Why is that disturbing? There is always some asymmetry between *defining* a function and *calling* a function. Function definitions: def function(named parameter list): block Function calls: function(positional arguments) We don't repeat the keyword "def" or the colon in the call, and we obviously don't repeat the body of the block. We typically call the function with either positional arguments, or only a *subset* of the keyword arguments, relying on defaults to fill in the rest. The arguments we provide will often be expressions, sometimes complex expressions, not just simple names. Even if they are simple names, they rarely match the parameter name: Definition: function(spam, eggs, cheese=None, aardvark=True) Call: function((foo + bar) or baz, foobar) So whatever symmetry there is, it isn't much. > So I was thinking: why not do define the methods > like: "def self.whatevermethod(par1, par2, etc)" instead of "def > whatevermethod(self, par1, par2, etc)"? "Why not ..." is the wrong question. Adding new features to the language, or new syntax, is not Default Allow ("add the feature, unless there is a compelling reason not to"). There has to be a good, positive reason for adding the feature, not merely a lack of reason not to. What does this feature buy us? Not much. It doesn't add any new expressiveness to the language. It doesn't let the language do anything that couldn't be done before. What it does do is add complexity to the syntax, there now being two ways to specify the first parameter: def self.method(arg) def method(self, arg) both of which need to be supported, which means more complexity to the language, more documentation needed, more for people to learn, more for tutorials to have to cover, more tests needed, etc, and all those costs don't actually gain us any significant advantage for the code we write. And what happens with classmethods and staticmethods, or those people who for whatever reason refuse to use the name "self" for the first parameter? -- Steve From ethan at stoneleaf.us Thu May 10 19:14:02 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 10 May 2018 16:14:02 -0700 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: , Message-ID: <5AF4D23A.50709@stoneleaf.us> On 05/10/2018 10:30 AM, Ed Page wrote: > Alternatives > - My company create our own datetime library > - Continued fracturing of time ... ecosystem (datetime, arrow, pendulum, delorean, datetime64, pandas.Timestamp Or, team up with one of those (if you can). -- ~Ethan~ From greg.ewing at canterbury.ac.nz Thu May 10 19:43:17 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 11 May 2018 11:43:17 +1200 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: References: Message-ID: <5AF4D915.2020909@canterbury.ac.nz> Tim Peters wrote: > Umm ... that's the opposite of what the Reference Manual says "lower":means: > > """ > 6.16. Operator precedence > > The following table summarizes the operator precedence in Python, from > lowest precedence (least binding) to highest precedence (most > binding). > """ Which is also in accord with the usual English meaning of the word "precedence" -- things with higher precedence get done first. -- Greg From mertz at gnosis.cx Thu May 10 20:26:22 2018 From: mertz at gnosis.cx (David Mertz) Date: Thu, 10 May 2018 20:26:22 -0400 Subject: [Python-ideas] High Precision datetime In-Reply-To: <5AF4D23A.50709@stoneleaf.us> References: <5AF4D23A.50709@stoneleaf.us> Message-ID: In fairness, Pandas, datetime64, and Arrow are really the same thing. I don't know about Pendulum or Delorean. A common standard would be great, or at least strong interoperability. I'm sure the authors of those projects would want that... Arrow is entirely about interoperability, after all. On Thu, May 10, 2018, 7:11 PM Ethan Furman wrote: > On 05/10/2018 10:30 AM, Ed Page wrote: > > > Alternatives > > - My company create our own datetime library > > - Continued fracturing of time ... ecosystem (datetime, arrow, > pendulum, delorean, datetime64, pandas.Timestamp > > Or, team up with one of those (if you can). > > -- > ~Ethan~ > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu May 10 21:59:23 2018 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 10 May 2018 21:59:23 -0400 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: You don't mention the option of allowing time.microseconds to be a float, and I was curious about that since if it did work, then that might be a relatively smooth extension of the current API. The highest value you'd store in the microseconds field is 1e6, and at values around 1e6, double-precision floating point has precision of about 1e-10: In [8]: 1e6 - np.nextafter(1e6, 0) Out[8]: 1.1641532182693481e-10 So that could represent values to precision of ~0.116 femtoseconds, or 116 attoseconds. Too bad. Femtosecond precision would cover a lot of cases, if you really need attoseconds then it won't work. -n On Thu, May 10, 2018 at 1:30 PM, Ed Page wrote: > Greetings, > > Is there interest in a PEP for extending time, datetime / timedelta for arbitrary or extended precision fractional seconds? > > My company designs and manufactures scientific hardware that typically operate with nanoseconds -- sometimes even attoseconds -- levels of precision. We?re in the process of providing Python APIs for some of these products and need to expose the full accuracy of the data to our customers. Doing so would allow developers to do things like timestamp analog measurements for correlating with other events in their system, or precisely schedule a future time event for correctly interoperating with other high-speed devices. > > The API we?ve been toying with is adding two new fields to time, datetime and timedelta > - frac_seconds (int) > - frac_seconds_exponent (int or new SITimeUnit enum) > > time.microseconds would be turned into a property that wraps frac_seconds for compatibility > > Challenges > - Defining the new `max` or `resolution` > - strftime / strptime. I propose that we do nothing, just leave formatting / parsing to use `microseconds` at best. On the other hand, __str__ could just specify the fractional seconds using scientific or engineering notation. > > Alternatives > - My company create our own datetime library > - Continued fracturing of time ... ecosystem (datetime, arrow, pendulum, delorean, datetime64, pandas.Timestamp ? all of which offer varying degrees of compatibility) > - Add an `attosecond` field and have `microsecond` wrap this. > - Effectively same except hard code `frac_seconds_exponent` to lowest value > - The most common cases (milliseconds, microseconds) will always pay the cost of using a bigint as compared to the proposal which is a "pay for what you use" approach > - How do we define what is "good enough" precision? > - Continue to subdivide time by adding `nanosecond` that is "nanoseconds since last micosecond", `picosecond` that is "picoseconds since last micnanosecond", and `attosecond` field that is "attoseconds since last picosecond" > - Possibly surprising API; people might expect `picosecond` to be an offset since last second > - Messy base 10 / base 2 conversions > - Have `frac_seconds` be a float > - This has precision issues. > > If anyone wants to have an impromptu BoF on the subject, I'm available at PyCon. > > Thanks > Ed Page > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Nathaniel J. Smith -- https://vorpus.org From python at fwdaddr.fastmail.fm Thu May 10 22:31:00 2018 From: python at fwdaddr.fastmail.fm (Nick Malaguti) Date: Thu, 10 May 2018 22:31:00 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <1526005860.1311112.1368211880.63BB2460@webmail.messagingengine.com> Hi all. I've been lurking for a little while on this discussion and I thought I might contribute some thoughts. One of my hurdles for ":=" is understanding when I should use it rather than "=". Should I use it everywhere? Should I use it only where I can't use regular "="? Is it a personal choice? Will it become so common that I need to think harder because some people will use it really frequently or intermix them? I don't want to see "=" vs ":=" become like semicolons in JavaScript. When I work on a different codebase, am I going to have to follow an "always" or "never" for binding expressions? Maybe this is all overblown and PEP8 direction will keep everyone on the same page, but I guess I worry about there being 2 very similar, but not the same, ways to do it. What I really like about "given" is it makes it a lot clearer when I should use it. No one is going to want to write x given x = f() if they can just write x = f() If you need a binding expression in a comprehension or an if or while statement, you'll know the pattern of using "given" to save that loop and a half or to call a function and bind its result while iterating. Just like you know when to use a ternary if to save that extra temporary variable - there's little confusion about when to use a ternary, especially since a few if statements quickly prove clearer to read. 10 if x == 5 else 9 if x == 2 else 8 if x == 3 else 100 looks much better as: if x == 5: result = 10 elif x == 2: result = 9 elif x == 3: result = 8 else: result = 100 I feel the same way about given. If you feel tempted to go overboard with: x given x = y * 2 given y = z + 3 given z = f() Which should be equivalent to: x := (y := ((z := f()) + 3)) * 2 hopefully you'll think, "maybe I should just make 3 statements instead?" And also I have no trouble following what that statement actually does when using given. I didn't need any parenthesis to make sure I didn't bind the wrong expressions and I don't have to read it from the inside out. Each sub-expression is complete rather than being mixed together (even though I have to read it from right to left). I feel like the strongest argument for ":=" is for all the situations where someone will actually want a binding expression in real code, ":=" is more succinct. I'm just concerned that when given a new binding expression hammer, everything is going to look like a nail and all the places where someone could really benefit from a binding expression will be drowned out by the unnecessary usage of ":=" (and its side effects). -- Nick ----- Original message ----- From: Guido van Rossum To: "marky1991 ." Cc: "python-ideas" Subject: Re: [Python-ideas] Inline assignments using "given" clauses Date: Thu, 10 May 2018 11:10:50 -0400 Please no, it's not that easy. I can easily generate a stream of +1s or -1s for any proposal. I'd need well-reasoned explanations and it would have to come from people who are willing to spend significant time writing it up eloquently. Nick has tried his best and failed to convince me. So the bar is high. (Also note that most of the examples that have been brought up lately were meant to illustrate the behavior in esoteric corner cases while I was working out the fine details of the semantics. Users should use this feature sparingly and stay very far away of those corner cases -- but they have to be specified in order to be able to implement this thing.) On Thu, May 10, 2018 at 10:26 AM, marky1991 . wrote:> If it just needs a stream of +1s, I personally like the "given" > approach much more than the ":=" approach, for all of the many reasons > repeated many times in the various email chains. (I preferred it as > "as", but that's been struck down already) (and if it's between ":=" > and not having them at all, I would rather just not have them) -- --Guido van Rossum (python.org/~guido) _________________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Thu May 10 23:47:16 2018 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 10 May 2018 22:47:16 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: ... [Guido] >> You should really read Tim's initial post in this thread, where he >> explains his motivation. [Nick] > I did, and then I talked him out of it by pointing out how confusing it > would be to have the binding semantics of "x := y" be context dependent. Ya, that was an effective Jedi mind trick when I was overdue to go to sleep ;-) To a plain user, there's nothing about a listcomp or genexp that says "new function introduced here". It looks like, for all the world, that it's running _in_ the block that contains it. It's magical enough that `for` targets magically become local. But that's almost never harmful magic, and often helpful, so worth it. >... > It *is* different, because ":=" normally binds the same as any other name > binding operation including "for x in y:" (i.e. it creates a local > variable), while at comprehension scope, the proposal has now become for "x > := y" to create a local variable in the containing scope, while "for x in y" > doesn't. ":=" target names in a genexp/listcmp are treated exactly the same as any other non-for-target name: they resolve to the same scope as they resolve to in the block that contains them. The only twist is that if such a name `x` isn't otherwise known in the block, then `x` is established as being local to the block (which incidentally also covers the case when the genexp/listcomp is at module level, where "local to the block" and "global to the block" mean the same thing). Class scope may be an exception (I cheerfully never learned anything about how class scope works, because I don't write insane code ;-) ). > Comprehension scoping is already hard to explain when its just a > regular nested function that accepts a single argument, so I'm not looking > forward to having to explain that "x := y" implies "nonlocal x" at > comprehension scope It doesn't, necessarily. If `x` is already known as `global` in the block, then there's an implied `global x` at comprehension scope. > (except that unlike a regular nonlocal declaration, it also implicitly makes it a local > in the immediately surrounding scope). Only if `x` is otherwise _unknown_ in the block. If, e.g., `x` is already known in an enclosing scope E, then `x` also resolves to scope E in the comprehension. It is not made local to the enclosing scope in that case. I think it's more fruitful to explain the semantics than try to explain a concrete implementation. Python's has a "lumpy" scope system now, with hard breaks among global scopes, class scopes, and all other lexical scopes. That makes implementations artificially noisy to specify. "resolve to the same scope as they resolve to in the block that contains them, with a twist ..." avoids that noise (e.g., the words "global" and "nonlocal" don't even occur), and gets directly to the point: in which scope does a name live? If you think it's already clear enough which scope `y` resolves to in z = (x+y for x in range(10)) then it's exactly as clear which scope `y` resolves to in z = (x + (y := 7) for x in range(10)) with the twist that if `y` is otherwise unknown in the containing block, `y` becomes local to the block. > It isn't reasonable to wave this away as "It's only confusing to Nick > because he's intimately familiar with how comprehensions are implemented", As above, though, I'm gently suggesting that being so intimately familiar with implementation details may be interfering with seeing how all those details can _obscure_ rather than illuminate. Whenever you think you need to distinguish between, e.g., "nonlocal" and "global", you're too deep in the detail weeds. > as I also wrote some of the language reference docs for the current (already > complicated) comprehension scoping semantics, and I can't figure out how > we're going to document the proposed semantics in a way that will actually > be reasonably easy for readers to follow. Where are those docs? I expect to find such stuff in section 4 ("Execution model") of the Language Reference Manual, but listcomps and genexps are only mentioned in passing once in the 3.6.5 section 4 docs, just noting that they don't always play well at class scope. > ... > - for comprehensions at class scope, the class scope is ignored for purposes > of determining the target binding scope (and hence will implicitly create a > new global variable when used in a top level class definition, and new > function local when used in a class definition nested inside a function) Isn't all of that too covered by "resolve to the same scope as they resolve to in the block that contains them .."? For example, in class K: print(g) at module level, `g` obviously refers to the global `g`. Therefore any `g` appearing as a ";=" target in an immediately contained comprehension also refers to the global `g`, exactly the same as if `g` were any other non-for-target name in the comprehension. That's not a new rule: it's a consequence of how class scopes already work. Which remain inscrutable to me ;-) > ...; > P.S. None of the above concerns apply to explicit inline scope declarations, > as those are easy to explain by saying that the inline declarations work the > same way as the scope declaration statements do, and can be applied > universally to all name binding operations rather than being specific to ":= > in comprehension scope". You already know I'd be happy with being explicit too, but Guido didn't like it. Perhaps he'd like it better if it were even _more_ like regular declarations. Off the top of my head, say that a comprehension could start with a new optional declaration section, like def f(): g = 12 i = 8 genexp = ( g + (j := i*2) for i in range(2)) Of course that's contrived. When the genexp ran, the `g` would refer to the global `g` (and the f-local `g` would be ignored); the local-to-f `i` would end up bound to 1, and in this "all bindings are local by default" world the ":=" binding to `j` would simply vanish when the genexp ended. In practice, I'd be amazed to see anything much fancier than p = None # annoying but worth it ;-) that is, in this world the intended scope # for a nonlocal needs to be explicitly established while any(( n % p == 0 for p in small_primes)): n //= p Note too: a binding expression (":=") isn't even needed then for this class of use case. OTOH, it's inexplicable _unless_ someone learns something about how a synthetic function is being created to implement the genexp. From tim.peters at gmail.com Fri May 11 02:13:05 2018 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 11 May 2018 01:13:05 -0500 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: References: <17376e38-c17c-5249-b4cb-6b8e1d441403@egenix.com> Message-ID: [Guido] > ,,, > OT about the name: despite Tim's relentless pushing of "binding expressions" > in the end I think they should be called "assignment expressions" just like > in C. Ha! I already gave up on "binding expressions". For nearly a full day, I've been rigidly calling them "binding operators" instead. That's why everyone has warmed up to them so dramatically since yesterday. Oh. They haven't? Oh well - since verbal misdirection failed, I'll go back to "assignment expressions'. Chris always liked that better anyway too. From greg.ewing at canterbury.ac.nz Fri May 11 02:57:51 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 11 May 2018 18:57:51 +1200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <5AF53EEF.8020305@canterbury.ac.nz> +1 given, -1 := -- Greg From greg.ewing at canterbury.ac.nz Fri May 11 03:05:47 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 11 May 2018 19:05:47 +1200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <5AF540CB.3040708@canterbury.ac.nz> Guido van Rossum wrote: > I'd need well-reasoned explanations My reasoning is essentially the same as what I've already said about "where". To summarise, "given" sounds like something an English-speaking mathematician would write, whereas ":=" doesn't even have an obvious pronunciation. Some variation on "given" just seems greatly more pythonic to me. -- Greg From j.van.dorp at deonet.nl Fri May 11 03:33:06 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Fri, 11 May 2018 09:33:06 +0200 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: >[Tim] > Since this is all about scope, while I'm not 100% sure of what Guido > meant, I assumed he was saying "p can only have one scope in the > synthetic function: local or non-local, not both, and local is what I > propose". For example, let's flesh out his example a bit more: > > p = 42 > [p := p for p in range(10) if p == 3] > print(p) # 42? 3? 9? > > If `p` is local to the listcomp, it must print 42. If `p` is > not-local, it must print 9. If it's some weird mixture of both, 3 > makes most sense (the only time `p := p` is executed is when the `for` > target `p` is 3). With my limited experience, I'd consider 3 to make most sense, but 9 when thinking about it in the expanded form. If it's not 3 tho, then the following would make most sense: SyntaxError("Cannot re-bind for target name in a list comprehension") # Or something more clear. And the rest of that mail that convinces me even more that an error would be the correct solution here. Before I got on this mailinglist, i never even knew comprehensions introduced a new scope. I'm really that new. Two years ago I'd look up stackoverflow to check the difference between overriding and extending a method and to verify whether I made my super() calls the right way. If something goes to weird, I think just throwing exceptions is a sensible solution that keeps the language simple, rather than making that much of a headache of something so trivially avoided. Jacco From greg.ewing at canterbury.ac.nz Fri May 11 03:33:39 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 11 May 2018 19:33:39 +1200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <14d15390-fba5-023d-12f5-274d884b88a0@kellett.im> Message-ID: <5AF54753.2010604@canterbury.ac.nz> Gustavo Carneiro wrote: > IMHO, all these toy examples don't translate well to the real world > because they tend to use very short variable names while in real world > [good written code] tends to select longer more descriptive variable names. I don't believe that's always true. It depends on the context. Sometimes, using long variable names can make code *harder* to read. I don't think there's anything unrealistic about this example: if m given m = pattern.match(the_string): nugget = m.group(2) Most people's short-term memory is good enough to remember that "m" refers to the match object while they read the next couple of lines. IMO, using a longer name would serve no purpose and would just clutter things up. -- Greg From greg.ewing at canterbury.ac.nz Fri May 11 03:38:30 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 11 May 2018 19:38:30 +1200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <5AF54876.7010803@canterbury.ac.nz> Kirill Balunov wrote: > > While for those who > are familiar with Pascal, Icon and other languages that use this syntax, > this - `:=` looks natural. As someone familiar with Pascal, I think the similarity to the Pascal assignment operator is actually an argument *against* it. Knowing what it means in Pascal is confusing, because Pascal's ":=" is equivalent to Python's "=" (it's strictly a statement, and can't be used in expressions). -- Greg From tim.peters at gmail.com Fri May 11 03:59:00 2018 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 11 May 2018 02:59:00 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: [Tim] >> Since this is all about scope, while I'm not 100% sure of what Guido >> meant, I assumed he was saying "p can only have one scope in the >> synthetic function: local or non-local, not both, and local is what I >> propose". For example, let's flesh out his example a bit more: >> >> p = 42 >> [p := p for p in range(10) if p == 3] >> print(p) # 42? 3? 9? >> >> If `p` is local to the listcomp, it must print 42. If `p` is >> not-local, it must print 9. If it's some weird mixture of both, 3 >> makes most sense (the only time `p := p` is executed is when the `for` >> target `p` is 3). [Jacco van Dorp ] > With my limited experience, I'd consider 3 to make most sense, but 9 > when thinking about it in the expanded form. > > If it's not 3 tho, then the following would make most sense: > > SyntaxError("Cannot re-bind for target name in a list comprehension") > # Or something more clear. > > And the rest of that mail that convinces me even more that an error > would be the correct solution here. Good news, then: Nick & Guido recently agreed that it would be a compile-time error. Assuming it's added to the language at all, of course. > Before I got on this mailinglist, i never even knew comprehensions > introduced a new scope. I'm really that new. They didn't, at first. That changed over time. The real reason was so that `for` variables - which people typically give little thought to naming - didn't accidentally overwrite local variables that happened to share the same name. Like: >>> i = -42 >>> [i+1 for i in range(3)] [1, 2, 3] >>> i # unchanged! -42 But you can productively use list comprehensions without knowing anything about how they're implemented, and just think "ha! Python does some happy magic for me there :-)". > Two years ago I'd look up stackoverflow to check the difference between > overriding and extending a method and to verify whether I made my > super() calls the right way. > > If something goes to weird, I think just throwing exceptions is a > sensible solution that keeps the language simple, rather than making > that much of a headache of something so trivially avoided. Since Guido agreed with you in this case, that proves you're a true Pythonista - or maybe just that you're both Dutch ;-) From jmcs at jsantos.eu Fri May 11 04:49:06 2018 From: jmcs at jsantos.eu (=?UTF-8?B?Sm/Do28gU2FudG9z?=) Date: Fri, 11 May 2018 10:49:06 +0200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AF540CB.3040708@canterbury.ac.nz> References: <5AF540CB.3040708@canterbury.ac.nz> Message-ID: +1 to this reasoning. One of the main reason python is popular is because code is easy to read, while ":=" would clearly not be as readable as "given". For me the difference between "given" and ":=" is the same as between python and C for loops. On Fri, 11 May 2018 at 09:06 Greg Ewing wrote: > Guido van Rossum wrote: > > I'd need well-reasoned explanations > > My reasoning is essentially the same as what I've already > said about "where". To summarise, "given" sounds like > something an English-speaking mathematician would write, > whereas ":=" doesn't even have an obvious pronunciation. > Some variation on "given" just seems greatly more pythonic > to me. > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.van.dorp at deonet.nl Fri May 11 05:40:51 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Fri, 11 May 2018 11:40:51 +0200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> Message-ID: I dont really like "given". If we compare: if m given m = re.match(stuff): to if m := re.match(stuff) then I count 4+(name_length) more tokens and 2 more spaces. Since I believe := is perfectly clear, I don't see the reason for a far more verbose syntax. That all said, I would still prefer: if re.match(stuff) as m: which is exactly equal to the := in line length and parallels with. While that may -technically- be a different beast. For beginners the difference is really irrelevant, and you can just tell advanced people the full story(technically speaking the as in a with statement isn't an expression assignment, it's a part of the with statement, and it feeds in the value through the context manager machinery before binding it. Similar for the except statement.). But I've kind of given up on "as" (so no need to reply on that bit). From jmcs at jsantos.eu Fri May 11 05:56:34 2018 From: jmcs at jsantos.eu (=?UTF-8?B?Sm/Do28gU2FudG9z?=) Date: Fri, 11 May 2018 11:56:34 +0200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> Message-ID: Optimizing syntax for space makes sense for "mathematical" notation since it's commonly written by hand, but putting space above readability in a programming language design feels like a skewmorphism. On Fri, 11 May 2018 at 11:41 Jacco van Dorp wrote: > I dont really like "given". > > If we compare: > > if m given m = re.match(stuff): > > to > > if m := re.match(stuff) > > then I count 4+(name_length) more tokens and 2 more spaces. Since I > believe := is perfectly clear, I don't see the reason for a far more > verbose syntax. > > That all said, I would still prefer: > > if re.match(stuff) as m: > > which is exactly equal to the := in line length and parallels with. > While that may -technically- be a different beast. > For beginners the difference is really irrelevant, and you can just > tell advanced people the full story(technically speaking the as in a > with statement isn't an expression assignment, it's a part of the with > statement, and it feeds in the value through the context manager > machinery before binding it. Similar for the except statement.). > > But I've kind of given up on "as" (so no need to reply on that bit). > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.van.dorp at deonet.nl Fri May 11 06:14:50 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Fri, 11 May 2018 12:14:50 +0200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> Message-ID: 2018-05-11 11:56 GMT+02:00 Jo?o Santos : > Optimizing syntax for space makes sense for "mathematical" notation since > it's commonly written by hand, but putting space above readability in a > programming language design feels like a skewmorphism. You are assuming "given" to improve readability, where I stated ":= is perfectly clear ", at least in my opinion. Therefore, since clarity is already achieved, the rest is clutter that reduces readability. From srkunze at mail.de Fri May 11 06:51:32 2018 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 11 May 2018 12:51:32 +0200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AF54753.2010604@canterbury.ac.nz> References: <14d15390-fba5-023d-12f5-274d884b88a0@kellett.im> <5AF54753.2010604@canterbury.ac.nz> Message-ID: On 11.05.2018 09:33, Greg Ewing wrote: > Gustavo Carneiro wrote: >> IMHO, all these toy examples don't translate well to the real world >> because they tend to use very short variable names while in real >> world [good written code] tends to select longer more descriptive >> variable names. > > I don't believe that's always true. It depends on the context. > Sometimes, using long variable names can make code *harder* > to read. > > I don't think there's anything unrealistic about this > example: > > ?? if m given m = pattern.match(the_string): > ????? nugget = m.group(2) I gather we don't talk about interactive usage. So, I grepped through part of our code-base. Like it or not, it's almost always called "match" there. Like Gustavo, I also have the feeling that long-living, real-world code tends to have more descriptive names than all those toy examples/arguments. Cheers, Sven From e+python-ideas at kellett.im Fri May 11 07:00:21 2018 From: e+python-ideas at kellett.im (Ed Kellett) Date: Fri, 11 May 2018 12:00:21 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <14d15390-fba5-023d-12f5-274d884b88a0@kellett.im> Message-ID: On 2018-05-10 17:10, Gustavo Carneiro wrote: > IMHO, all these toy examples don't translate well to the real world because > they tend to use very short variable names while in real world [good > written code] tends to select longer more descriptive variable names. > > Try replacing "c" with a longer name, like input_command, then it becomes: > > while ((input_command = getchar()) != EOF) ... > > while (input_command = getchar(), input_command != EOF) ... This thread really isn't about variable naming. I'll simply say that I've always written the C idiom with `c` as the variable name, I've seen literally hundreds of others do the same, and I've never seen anyone spell it `input_command`. I do the same in Python in a few contexts, usually where the variable's meaning is very clear from its usage, or where the code in question doesn't know or care what the variable is used for: for i in range(10): def get_const(self, x): for k in self.defaults.keys(): etc., and if we had `given` clauses, I'd probably do this there too. I think one of the advantages of that syntax is that it makes it extremely clear what's going on: if m given m = re.match(...): I don't need to try to stuff an English description of m into its name, because the `given` clause already describes it perfectly. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From srkunze at mail.de Fri May 11 07:05:09 2018 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 11 May 2018 13:05:09 +0200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AF54876.7010803@canterbury.ac.nz> References: <5AF54876.7010803@canterbury.ac.nz> Message-ID: On 11.05.2018 09:38, Greg Ewing wrote: > Kirill Balunov wrote: >> >> While for those who are familiar with Pascal, Icon and other >> languages that use this syntax, this - `:=` looks natural. > > As someone familiar with Pascal, I think the similarity to > the Pascal assignment operator is actually an argument > *against* it. Knowing what it means in Pascal is confusing, > because Pascal's ":=" is equivalent to Python's "=" (it's > strictly a statement, and can't be used in expressions). Same here. It means something different. Also coding in Pascal was annoying from the beginning with its extremely verbose syntax like begin/end etc. So, ":=" also felt like "why the hell do we need a colon in front of the equal sign?" Absolutely unnecessary bloat, like almost everything in Pascal. Maybe that's also part, why I am -1 on the proposal. Who knows... From ncoghlan at gmail.com Fri May 11 07:15:19 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 11 May 2018 07:15:19 -0400 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: On 10 May 2018 at 23:47, Tim Peters wrote: > ... > > [Guido] > >> You should really read Tim's initial post in this thread, where he > >> explains his motivation. > > [Nick] > > I did, and then I talked him out of it by pointing out how confusing it > > would be to have the binding semantics of "x := y" be context dependent. > > Ya, that was an effective Jedi mind trick when I was overdue to go to > sleep ;-) > > To a plain user, there's nothing about a listcomp or genexp that says > "new function introduced here". It looks like, for all the world, > that it's running _in_ the block that contains it. It's magical > enough that `for` targets magically become local. But that's almost > never harmful magic, and often helpful, so worth it. > > > >... > > It *is* different, because ":=" normally binds the same as any other name > > binding operation including "for x in y:" (i.e. it creates a local > > variable), while at comprehension scope, the proposal has now become for > "x > > := y" to create a local variable in the containing scope, while "for x > in y" > > doesn't. > > ":=" target names in a genexp/listcmp are treated exactly the same as > any other non-for-target name: they resolve to the same scope as they > resolve to in the block that contains them. The only twist is that if > such a name `x` isn't otherwise known in the block, then `x` is > established as being local to the block (which incidentally also > covers the case when the genexp/listcomp is at module level, where > "local to the block" and "global to the block" mean the same thing). > Class scope may be an exception (I cheerfully never learned anything > about how class scope works, because I don't write insane code ;-) ). > That's all well and good, but it is *completely insufficient for the language specification*. For the language spec, we have to be able to tell implementation authors exactly how all of the "bizarre edge case" that you're attempting to hand wave away should behave by updating https://docs.python.org/dev/reference/expressions.html#displays-for-lists-sets-and-dictionaries appropriately. It isn't 1995 any more - while CPython is still the reference implementation for Python, we're far from being the only implementation, which means we have to be a lot more disciplined about how much we leave up to the implementation to define. The expected semantics for locals() are already sufficiently unclear that they're a source of software bugs (even in CPython) when attempting to run things under a debugger or line profiler (or anything else that sets a trace function). See https://www.python.org/dev/peps/pep-0558/ for details. "Comprehension scopes are already confusing, so it's OK to dial their weirdness all the way up to 11" is an *incredibly* strange argument to be attempting to make when the original better defined sublocal scoping proposal was knocked back as being overly confusing (even after it had been deliberately simplified by prohibiting nonlocal access to sublocals). Right now, the learning process for picking up the details of comprehension scopes goes something like this: * make the technically-incorrect-but-mostly-reliable-in-the-absence-of-name-shadowing assumption that "[x for x in data]" is semantically equivalent to a for loop (especially common for experienced Py2 devs where this really was the case!): _result = [] for x in data: _result.append(x) * discover that "[x for x in data]" is actually semantically equivalent to "list(x for x in data)" (albeit without the name lookup and optimised to avoid actually creating the generator-iterator) * make the still-technically-incorrect-but-even-more-reliable assumption that the generator expression "(x for x in data)" is equivalent to def _genexp(): for x in data: yield x _result = _genexp() * *maybe* discover that even the above expansion isn't quite accurate, and that the underlying semantic equivalent is actually this (one way to discover this by accident is to have a name error in the outermost iterable expression): def _genexp(_outermost_iter): for x in _outermost_iter: yield x _result = _genexp(_outermost_iter) * and then realise that the optimised list comprehension form is essentially this: def _listcomp(_outermost_iter): result = [] for x in _outermost_iter: result.append(x) return result _result = _listcomp(data) Now that "yield" in comprehensions has been prohibited, you've learned all the edge cases at that point - all of the runtime behaviour of things like name references, locals(), lambda expressions that close over the iteration variable, etc can be explained directly in terms of the equivalent functions and generators, so while comprehension iteration variable hiding may *seem* magical, it's really mostly explained by the deliberate semantic equivalence between the comprehension form and the constructor+genexp form. (That's exactly how PEP 3100 describes the change: "Have list comprehensions be syntactic sugar for passing an equivalent generator expression to list(); as a consequence the loop variable will no longer be exposed") As such, any proposal to have name bindings behave differently in comprehension and generator expression scope from the way they would behave in the equivalent nested function definitions *must be specified to an equivalent level of detail as the status quo*. All of the attempts at such a definition that have been made so far have been riddled with action and a distance and context-dependent compilation requirements: * whether to implicitly declare the binding target as nonlocal or global depends on whether or not you're at module scope or inside a function * the desired semantics at class scope have been left largely unclear * the desired semantics in the case of nested comprehensions and generator expressions has been left entirely unclear Now, there *are* ways to resolve these problems in a coherent way, and that would be to define "parent local scoping" as a new scope type, and introduce a corresponding "parentlocal NAME" compiler declaration to explicitly request those semantics for bound names (allowing the expansions of comprehensions and generator expressions as explicitly nested functions to be adjusted accordingly). But the PEP will need to state explicitly that that's what it is doing, and fully specify how those new semantics are expected to work in *all* of the existing scope types, not just the two where the desired behaviour is relatively easy to define in terms of nonlocal and global. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri May 11 07:21:40 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 11 May 2018 07:21:40 -0400 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: On 11 May 2018 at 07:15, Nick Coghlan wrote: > * *maybe* discover that even the above expansion isn't quite accurate, and > that the underlying semantic equivalent is actually this (one way to > discover this by accident is to have a name error in the outermost iterable > expression): > > def _genexp(_outermost_iter): > for x in _outermost_iter: > yield x > > _result = _genexp(_outermost_iter) > Typo here: the call argument should be "data", not a repeat of the parameter name, Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhodri at kynesim.co.uk Fri May 11 07:37:43 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Fri, 11 May 2018 12:37:43 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> Message-ID: <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> On 11/05/18 11:14, Jacco van Dorp wrote: > 2018-05-11 11:56 GMT+02:00 Jo?o Santos : >> Optimizing syntax for space makes sense for "mathematical" notation since >> it's commonly written by hand, but putting space above readability in a >> programming language design feels like a skewmorphism. > > You are assuming "given" to improve readability, where I stated ":= is > perfectly clear ", at least in my opinion. Therefore, since clarity is > already achieved, the rest is clutter that reduces readability. I respectfully disagree with your opinion (i.e. you're wrong :-) Consider: while (cmd := get_command()).token != CMD_QUIT: cmd.do_something() vs: while cmd.token != CMD_QUIT given cmd = get_command(): cmd.do_something() I find I write code like this[*] a fair bit, since my major use for Python is to write remote monitors for embedded kit, so it's pretty much a real world example. I don't find the first version using ":=" to be perfectly clear, in fact I think it's rather ugly. That may be partly the same reaction that many of us had to the asymmetry of assignment expressions in (over-)complicated comprehensions. The second version using "given" reads much more naturally to the mathematician in me, and not too badly to my English half either. [*] By "like this" I mean the clunky "while true:" spelling, obviously. -- Rhodri James *-* Kynesim Ltd From ncoghlan at gmail.com Fri May 11 07:43:01 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 11 May 2018 07:43:01 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AF54753.2010604@canterbury.ac.nz> References: <14d15390-fba5-023d-12f5-274d884b88a0@kellett.im> <5AF54753.2010604@canterbury.ac.nz> Message-ID: On 11 May 2018 at 03:33, Greg Ewing wrote: > Gustavo Carneiro wrote: > >> IMHO, all these toy examples don't translate well to the real world >> because they tend to use very short variable names while in real world >> [good written code] tends to select longer more descriptive variable names. >> > > I don't believe that's always true. It depends on the context. > Sometimes, using long variable names can make code *harder* > to read. > > I don't think there's anything unrealistic about this > example: > > if m given m = pattern.match(the_string): > nugget = m.group(2) > > Most people's short-term memory is good enough to remember > that "m" refers to the match object while they read the > next couple of lines. IMO, using a longer name would serve > no purpose and would just clutter things up. I've been thinking about this problem, and I think for the If/elif/while cases it's actually possible to allow the "binding is the same as the condition" case to be simplified to: if command = pattern.match(the_string): ... elif command = other_pattern.match(the_string): ... while data = read_data(): ... Allowing this would be part of the definition of the if/elif/while statement headers, rather than a general purpose assignment expression. The restriction of the LHS to a simple name target would need to be in the AST generator rather than in the grammar, but it's hardly the only case where we do that kind of thing. Switching to the given expression form would then only be necessary in cases where the condition *wasn't* the same as the binding target. A similar enhancement could be made to conditional expressions (adjusting their grammar to permit "EXPR if NAME = EXPR else EXPR") and filter clauses in comprehensions (allowing "EXPR for TARGET in EXPR if NAME = EXPR"). In essence, "if", "elif", and "while" would all allow for an "implied given" clause in order to simplify the 90% case where the desired condition and the bound expression are the same. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri May 11 07:46:27 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 11 May 2018 21:46:27 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AED70C4.1030008@canterbury.ac.nz> Message-ID: <20180511114627.GR9562@ando.pearwood.info> On Sat, May 05, 2018 at 06:24:07AM -0400, Juancarlo A?ez wrote: > I think that "expr as y" was discarded too quickly. This discussion started back in *February*. I don't think "too quickly" applies to ANYTHING about it. https://mail.python.org/pipermail/python-ideas/2018-February/048971.html And Chris' first draft of the PEP: https://mail.python.org/pipermail/python-ideas/2018-February/049041.html I have been one of the main proponents of "as". See, for example: https://mail.python.org/pipermail/python-ideas/2018-April/049880.html At least, I *was*. I'm now satisfied that "as" is the wrong solution, and I don't think it was discarded too quickly. Even though it makes me sad that "as" is not suitable, I'm satisfied that the problems with it would require too high a price to solve. The major problem is that it will clash with "except as" and "with as" statements. Of course we *could* introduce some sort of special treatment, possibly as simple as simply banning the use of binding- assignments inside except/with statements, but such special rules add complexity, make the feature less useful, harder to learn, and more surprising. Allowing or disallowing particular expressions after a certain keywork ought to be a last resort. Even though this was my preferred solution, I've now come to change my mind and think this would have been a mistake. (Thanks Chris for sticking to your guns and rejecting "as".) You say: > The special cases that may arise over "except" and "with" can be worked > out and documented. but there's no "may" about this. Using "as" does clash, it's not a matter of whether or not it will clash, we know it will. And it's easy to say that it "can" be worked out, but unless you have a concrete proposal to work it out, that's not really an argument in favour for "as", it is just a hope. Guido has also correctly pointed out that will "as" is used to bind names in other contexts, it doesn't *quite* work the same as regular = assignment. Again, the "with" statement is especially relevant: with expression as name does not bind the value of the expression to name, except by coincidence. It actually binds the value of expession.__enter__() to name. I still, and probably always will, like the look of result = (expression as spam) + spam**2 but I'm realistic to realise that it isn't practical. -- Steve From srkunze at mail.de Fri May 11 07:57:14 2018 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 11 May 2018 13:57:14 +0200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <14d15390-fba5-023d-12f5-274d884b88a0@kellett.im> <5AF54753.2010604@canterbury.ac.nz> Message-ID: <2d89e54b-2b30-b8e2-1305-a6c5ca097150@mail.de> On 11.05.2018 13:43, Nick Coghlan wrote: > I've been thinking about this problem, and I think for the > If/elif/while cases it's actually possible to allow the "binding is > the same as the condition" case to be simplified to: > > ? ? if command =? pattern.match(the_string): > ??????? ... > ? ? elif command =? other_pattern.match(the_string): > ??????? ... > > ??? while data = read_data(): > ??????? ... > > Allowing this would be part of the definition of the if/elif/while > statement headers, rather than a general purpose assignment expression. I can imagine that this will cover 80 to 90% of the usecases and it's readable as well. > A similar enhancement could be made to conditional expressions > (adjusting their grammar to permit "EXPR if NAME = EXPR else EXPR") > and filter clauses in comprehensions (allowing "EXPR for TARGET in > EXPR if NAME = EXPR"). Not sure if that is too much for now. List comprehensions tend to be longer than expected, the same goes for the ternary expression. Maybe, we could start with the 90% case and whether the need for more arises. > In essence, "if", "elif", and "while" would all allow for an "implied > given" clause in order to simplify the 90% case where the desired > condition and the bound expression are the same. Exactly. Maybe, even here: let's do just the 90% case. And if a lot of people need finer control, we can reconsider :=/given/as. Regards, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri May 11 08:24:14 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 11 May 2018 22:24:14 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <20180511122414.GS9562@ando.pearwood.info> On Fri, May 04, 2018 at 09:56:10PM -0400, Alexander Belopolsky wrote: > This proposal has finally made me realize why I did not > like PEP 572. The strong expression vs. statement dichotomy is one of > the key features that set Python apart from many other languages and > it makes Python programs much easier to understand. I'm not so sure that the "expression versus statement" dichotomy is as strong or as useful as Alexander says. If it were, we'd be writing much more imperative code, as if it were 1970 and we were using BASIC. Some of the greatest Python successes have been to add expression forms of what used to be purely imperative statements: - comprehensions (for-loops); - ternary if (if...else statement). In addition, we have more overlap between statements and expressions: - an expression form of def (lambda); - a functional form of the import statement (importlib.import_module, before that people used to use __import__(); - expression forms of augmented assignment (ordinary + etc operators), which is a rare case where the expression form came first and the imperative command came afterwards. I may have missed some. Also notable is that Python does not have "procedures" (pure statement callables). We use functions instead, and simply ignore the returned result. If the difference between statements and expressions was really a dichotomy, we would only need One Way to do these things, not two. I also have seen many people disappointed that Python doesn't treat "everything as an expression". The lack of assignment-expressions is a turn-off for some people: https://mail.python.org/pipermail/python-list/2018-May/732890.html See also: https://stackoverflow.com/questions/50090868/why-are-assignments-not-allowed-in-pythons-lambda-expressions -- Steve From rosuav at gmail.com Fri May 11 09:02:43 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 11 May 2018 23:02:43 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> Message-ID: On Fri, May 11, 2018 at 9:37 PM, Rhodri James wrote: > On 11/05/18 11:14, Jacco van Dorp wrote: >> >> 2018-05-11 11:56 GMT+02:00 Jo?o Santos : >>> >>> Optimizing syntax for space makes sense for "mathematical" notation since >>> it's commonly written by hand, but putting space above readability in a >>> programming language design feels like a skewmorphism. >> >> >> You are assuming "given" to improve readability, where I stated ":= is >> perfectly clear ", at least in my opinion. Therefore, since clarity is >> already achieved, the rest is clutter that reduces readability. > > > I respectfully disagree with your opinion (i.e. you're wrong :-) > > Consider: > > while (cmd := get_command()).token != CMD_QUIT: > cmd.do_something() > > vs: > > while cmd.token != CMD_QUIT given cmd = get_command(): > cmd.do_something() > Yes, I've considered it. And I don't like the fact that the evaluation is right-to-left. It isn't a problem when your condition is extremely simple, but I can guarantee you that people will use this with more complicated conditions. And when that happens, you have to be aware that the tail of the statement is actually evaluated before the primary expression. It's like with Perl: die("blah blah") unless some_condition Reverse-order evaluation is confusing and frequently annoying. It's not an instant failure of the proposal, but it's a serious cost, and I'd much rather avoid it by using := (which leaves the expression where it is). ChrisA From rosuav at gmail.com Fri May 11 09:08:57 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 11 May 2018 23:08:57 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: On Fri, May 11, 2018 at 9:15 PM, Nick Coghlan wrote: > * *maybe* discover that even the above expansion isn't quite accurate, and > that the underlying semantic equivalent is actually this (one way to > discover this by accident is to have a name error in the outermost iterable > expression): > > def _genexp(_outermost_iter): > for x in _outermost_iter: > yield x > > _result = _genexp(_outermost_iter) > > * and then realise that the optimised list comprehension form is essentially > this: > > def _listcomp(_outermost_iter): > result = [] > for x in _outermost_iter: > result.append(x) > return result > > _result = _listcomp(data) > > Now that "yield" in comprehensions has been prohibited, you've learned all > the edge cases at that point Not quite! You missed one, just because comprehensions aren't weird enough yet. AFAIK you can't tell with the list comp, but with the genexp you can (by not iterating over it). > def _genexp(_outermost_iter): > for x in _outermost_iter: > yield x > > _result = _genexp(data) It's actually this: def _genexp(_outermost_iter): for x in _outermost_iter: yield x _result = _genexp(iter(_outermost_iter)) I don't think there's anything in the main documentation that actually says this, although PEP 289 mentions it in the detaily bits. [1] ChrisA [1] https://www.python.org/dev/peps/pep-0289/#the-details From j.van.dorp at deonet.nl Fri May 11 09:25:40 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Fri, 11 May 2018 15:25:40 +0200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> Message-ID: >[Rhodri] > I respectfully disagree with your opinion (i.e. you're wrong :-) > > Consider: > > while (cmd := get_command()).token != CMD_QUIT: > cmd.do_something() > > vs: > > while cmd.token != CMD_QUIT given cmd = get_command(): > cmd.do_something() > Actually, the first version is more readable. It's got a lot to do with what Chris said about order of operations, but IMO, even more with grouping. There's a couple of things you might want to learn from this statement. First, will the while check succeed? Well.... while......get_command()).token != CMD_QUIT: yup, looks clear. I can ignore the cmd := part easily, and the bonus paren I see there doesn't matter that much. Another thing i might be curious about, is what is the value of cmd after ? while (cmd := get_command())........................: Looks like it has the value of getcommand(). Hey, that was both clear and readable. I can just ignore half the line and learn stuff. Great. Lets see with the other notation. What's the value of cmd ? while ................................................... cmd = get_command(): That works. Bit of line I had to skip. Will the check succeed ? while cmd.token != CMD_QUIT .....................................: Wait, what's the value of cmd ? Lets look in the code in the preceding lines....oh, ok, it's at the end of the line. I actually have to mentally parse the entire line to get what the check will work. This, along with what Chris said about order of operations, reduce the readability of the "given" version. From rhodri at kynesim.co.uk Fri May 11 09:34:56 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Fri, 11 May 2018 14:34:56 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> Message-ID: <9ba24601-d4a6-c208-3be3-2168cf7b617d@kynesim.co.uk> On 11/05/18 14:25, Jacco van Dorp wrote: > I actually have to mentally parse the entire line to get what the > check will work. This, along with what Chris said about order of > operations, reduce the readability of the "given" version. You say "I had to parse the entire line..." I hear "I had to read what I was making a snap decision on." Sounds like a win :-) -- Rhodri James *-* Kynesim Ltd From rosuav at gmail.com Fri May 11 09:42:50 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 11 May 2018 23:42:50 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <9ba24601-d4a6-c208-3be3-2168cf7b617d@kynesim.co.uk> References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <9ba24601-d4a6-c208-3be3-2168cf7b617d@kynesim.co.uk> Message-ID: On Fri, May 11, 2018 at 11:34 PM, Rhodri James wrote: > On 11/05/18 14:25, Jacco van Dorp wrote: >> >> I actually have to mentally parse the entire line to get what the >> check will work. This, along with what Chris said about order of >> operations, reduce the readability of the "given" version. > > > You say "I had to parse the entire line..." I hear "I had to read what I > was making a snap decision on." Sounds like a win :-) > No, not a win. Do you read the entire source code for an entire project before trying to comprehend one part of it? I doubt it. Do you read an entire file before trying to comprehend a single function in that file? No. Do you even read an entire function before processing one line in that function? Unlikely. It's normal and correct to seek to understand one part of some code while ignoring other parts. That's why we have proper variable names, even inside functions - we could just use "slot0" and "slot1" and so on, since that's how they work to the interpreter. But we use good names, so that you can understand some code without having to first read the thing that created that variable. ChrisA From jmcs at jsantos.eu Fri May 11 09:47:05 2018 From: jmcs at jsantos.eu (=?UTF-8?B?Sm/Do28gU2FudG9z?=) Date: Fri, 11 May 2018 15:47:05 +0200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> Message-ID: How do you read something like " while (cmd := get_command()).token != CMD_QUIT:" in plain english? On Fri, 11 May 2018 at 12:15 Jacco van Dorp wrote: > 2018-05-11 11:56 GMT+02:00 Jo?o Santos : > > Optimizing syntax for space makes sense for "mathematical" notation since > > it's commonly written by hand, but putting space above readability in a > > programming language design feels like a skewmorphism. > > You are assuming "given" to improve readability, where I stated ":= is > perfectly clear ", at least in my opinion. Therefore, since clarity is > already achieved, the rest is clutter that reduces readability. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri May 11 09:59:13 2018 From: guido at python.org (Guido van Rossum) Date: Fri, 11 May 2018 09:59:13 -0400 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: (Note: this is an off-topic side thread, unrelated to assignment expressions. Inline comment below.) On Fri, May 11, 2018 at 9:08 AM, Chris Angelico wrote: > On Fri, May 11, 2018 at 9:15 PM, Nick Coghlan wrote: > > * *maybe* discover that even the above expansion isn't quite accurate, > and > > that the underlying semantic equivalent is actually this (one way to > > discover this by accident is to have a name error in the outermost > iterable > > expression): > > > > def _genexp(_outermost_iter): > > for x in _outermost_iter: > > yield x > > > > _result = _genexp(_outermost_iter) > > > > * and then realise that the optimised list comprehension form is > essentially > > this: > > > > def _listcomp(_outermost_iter): > > result = [] > > for x in _outermost_iter: > > result.append(x) > > return result > > > > _result = _listcomp(data) > > > > Now that "yield" in comprehensions has been prohibited, you've learned > all > > the edge cases at that point > > Not quite! You missed one, just because comprehensions aren't weird > enough yet. AFAIK you can't tell with the list comp, but with the > genexp you can (by not iterating over it). > > > def _genexp(_outermost_iter): > > for x in _outermost_iter: > > yield x > > > > _result = _genexp(data) > > It's actually this: > > def _genexp(_outermost_iter): > for x in _outermost_iter: > yield x > > _result = _genexp(iter(_outermost_iter)) > > I don't think there's anything in the main documentation that actually > says this, although PEP 289 mentions it in the detaily bits. [1] > > ChrisA > > [1] https://www.python.org/dev/peps/pep-0289/#the-details > I'm not sure this is the whole story. I tried to figure out how often __iter__ is called in a genexpr. I found that indeed I see iter() is called as soon as the generator is brought to life, but it is *not* called a second time the first time you call next(). However the translation you show has a 'for' loop which is supposed to call iter() again. So how is this done? It seems the generated bytecode isn't equivalent to a for-loop, it's equivalent to s while loop that just calls next(). Disassembly of a regular generator: def foo(a): for x in a: yield x * 2 0 SETUP_LOOP 18 (to 20)* 2 LOAD_FAST 0 (a) * 4 GET_ITER* >> 6 FOR_ITER 10 (to 18) 8 STORE_FAST 1 (x) 10 LOAD_FAST 1 (x) 12 YIELD_VALUE 14 POP_TOP 16 JUMP_ABSOLUTE 6 >> 18 POP_BLOCK >> 20 LOAD_CONST 0 (None) 22 RETURN_VALUE But for a generator: g = (x for x in C()) 1 0 LOAD_FAST 0 (.0) >> 2 FOR_ITER 10 (to 14) 4 STORE_FAST 1 (x) 6 LOAD_FAST 1 (x) 8 YIELD_VALUE 10 POP_TOP 12 JUMP_ABSOLUTE 2 >> 14 LOAD_CONST 0 (None) 16 RETURN_VALUE Note the lack of SETUP_LOOP and GET_ITER (but otherwise they are identical). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.van.dorp at deonet.nl Fri May 11 10:04:44 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Fri, 11 May 2018 16:04:44 +0200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <9ba24601-d4a6-c208-3be3-2168cf7b617d@kynesim.co.uk> Message-ID: A while ago, we had this gem: 2018-04-06 8:19 GMT+02:00 Serhiy Storchaka : > Using currently supported syntax: > > smooth_signal = [average for average in [0] for x in signal for average in [(1-decay)*average + decay*x]] Go ahead and understand that line in 1 go. It's currently legal syntax for a running average for a smoothing signal, which remembers something about it. (Subject: Proposal: A Reduce-Map Comprehension and a "last" builtin) You're not allowed to work it out bit by bit, just understand the entire line or nothing. Any failure of yours proves my point. > [Jo?o] > How do you read something like " while (cmd := get_command()).token != CMD_QUIT:" in plain english? while open-paren cee em dee colon is call get-underscore-command close-paren dot token doesn't equal all-caps cee em dee underscore quit colon. Might be some dutch in there. But far more importantly, I can hold the concept into my head, or just the parts of it that I need. How we call it in english is actually not a good argument - whether we can easily mentally parse it is, since I tend not to code by voice command, but with a keyboard. Your mileage may vary, but I think we should optimize for keyboard coding over voice chat coding. And when I need to refer to it, I say "this bit here" or I copy paste it. From clint.hepner at gmail.com Fri May 11 10:06:19 2018 From: clint.hepner at gmail.com (Clint Hepner) Date: Fri, 11 May 2018 10:06:19 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> Message-ID: <9A3709D9-4FB3-4D83-ACEA-25E445ACC5F0@gmail.com> > On 2018 May 11 , at 7:37 a, Rhodri James wrote: > > On 11/05/18 11:14, Jacco van Dorp wrote: >> 2018-05-11 11:56 GMT+02:00 Jo?o Santos : >>> Optimizing syntax for space makes sense for "mathematical" notation since >>> it's commonly written by hand, but putting space above readability in a >>> programming language design feels like a skewmorphism. >> You are assuming "given" to improve readability, where I stated ":= is >> perfectly clear ", at least in my opinion. Therefore, since clarity is >> already achieved, the rest is clutter that reduces readability. > > I respectfully disagree with your opinion (i.e. you're wrong :-) > > Consider: > > while (cmd := get_command()).token != CMD_QUIT: > cmd.do_something() > > vs: > > while cmd.token != CMD_QUIT given cmd = get_command(): > cmd.do_something() > > > I find I write code like this[*] a fair bit, since my major use for Python is to write remote monitors for embedded kit, so it's pretty much a real world example. I don't find the first version using ":=" to be perfectly clear, in fact I think it's rather ugly. That may be partly the same reaction that many of us had to the asymmetry of assignment expressions in (over-)complicated comprehensions. The second version using "given" reads much more naturally to the mathematician in me, and not too badly to my English half either. I would write this using a for loop and the two-argument form of iter: for cmd in iter(get_command, ''): if cmd.token == CMD_QUIT: break cmd.do_something() or from itertools import take while for cmd in takewhile(lambda x: x.token != CMD_QUIT, iter(get_command, '')): cmd.do_something() Depending on what get_command actually returns, you might be able to construct a valid sentinel that doesn't require an explicit test of cmd.token. (This reminds that I wish ``iter`` could take a predicate instead of a sentinel as its second argument. Then you could just write for cmd in iter(get_command, lambda x: x.token == CMD_QUIT): cmd.do_something() ) -- Clint From kirillbalunov at gmail.com Fri May 11 11:00:25 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Fri, 11 May 2018 18:00:25 +0300 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <9A3709D9-4FB3-4D83-ACEA-25E445ACC5F0@gmail.com> References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <9A3709D9-4FB3-4D83-ACEA-25E445ACC5F0@gmail.com> Message-ID: 2018-05-11 17:06 GMT+03:00 Clint Hepner : > > > (This reminds that I wish ``iter`` could take a predicate instead of a > sentinel as its second argument. Then > you could just write > > for cmd in iter(get_command, lambda x: x.token == CMD_QUIT): > cmd.do_something() > ) > > But you can do it right now: class P: def __init__(self, key): self.key = key def __eq__(self, other): return self.key(other) for cmd in iter(get_command, P(lambda x: x.token == CMD_QUIT)): cmd.do_something() With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri May 11 11:10:45 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 12 May 2018 01:10:45 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <1526005860.1311112.1368211880.63BB2460@webmail.messagingengine.com> References: <1526005860.1311112.1368211880.63BB2460@webmail.messagingengine.com> Message-ID: <20180511151045.GV9562@ando.pearwood.info> On Thu, May 10, 2018 at 10:31:00PM -0400, Nick Malaguti wrote: > One of my hurdles for ":=" is understanding when I should use it rather > than "=". Should I use it everywhere? Should I use it only where I can't > use regular "="? Is it a personal choice? Will it become so common that > I need to think harder because some people will use it really frequently > or intermix them? When should I use "x = x + 1" and when should I use "x += 1"? When should I write alist.sort() and when sorted(alist)? When should I use a list comp and when should I use a for-loop? I could give a dozen more examples. We always have a choice in writing code. There is almost never *only* one way to do it. Nevertheless, we manage, and I believe that most of the things which perplex us today will be so trivially easy to understand tomorrow that we'll wonder what the fuss was all about. It is sometimes humbling to remember back at the things that we swore were terrible, terrible mistakes. I hated augmented assignment, until I gave in and started using them. I hated != instead of <> and I actually seriously considered using "from __future__ import barry_as_FLUFL" for a while. Depending on whether you are a "glass half full" or "glass half empty" kind of guy, you could say either that: - we're irrationally risk adverse, and would rather miss out on something fantastic than risk something slightly not so good; - or like Stockholm Syndrome victims, we can get used to, and even learn to enjoy, the most awful crap if it goes on long enough. (Or possibly both at the same time.) Personally, I think that in hindsight my dislike of != was irrational and rather silly, rather than a prophetic realisation that the dropping of <> began the ruination of Python. YMMV. > I don't want to see "=" vs ":=" become like semicolons in JavaScript. The difference between = and := is nothing like automatic semicolon insertion in Javascript. The Python interpreter is never going to (only sometimes) insert a colon to make your "name = expression" work and it is never going to turn two statements into a single broken expression because you used = instead of := or vice versa. Of course people can still screw up the precedence and get the wrong results, but the same applies to all operators and the same solution applies: when in doubt, add parentheses to make it clear. Some arguments against := are better than others: will it encourage people to write unreadable one-liners instead of using multiple statements? Maybe, but I think most people will show more sense and restraint. You haven't seen many quadruply-nested list comprehensions, or ternary if operators nested ten deep, have you? I haven't. > When I work on a different codebase, am I going to have to follow an > "always" or "never" for binding expressions? Yes, of course you are. Some places will be conservative and say Never, and some will be gung ho and say Always, but the majority will say "Use them when they make the code better". Just like when you work on some code bases you will have to always follow PEP 8, and when you work on other code bases, you will have to follow different rules. > Maybe this is all overblown > and PEP8 direction will keep everyone on the same page, but I guess I > worry about there being 2 very similar, but not the same, ways to do it. > What I really like about "given" is it makes it a lot clearer when I > should use it. You seem to have two contradictory opinions here. Paraphrasing: "I worry that the same people who never abuse ternary if by writing obfuscated one-liners will suddenly turn around and abuse := binding expressions by writing obfuscated one-liners." and "clearly nobody will abuse 'given' binding expressions by writing obfuscated one-liners, because they don't abuse ternary if to write obfuscated one-liners." I understand being pessimistic about the common-sense of my fellow programmers. I understand being optimistic about the common-sense of my fellow programmers. I don't understand doing both at the same time. If people will abuse := they will abuse "given". If they won't abuse "given", they surely won't abuse := either. Since we have over a quarter of a century of experience showing that the Python community, as a whole, tends not to abuse syntax to write unreadable one-liners, I think that fears about people abusing := are overblown. -- Steve From steve at pearwood.info Fri May 11 11:37:44 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 12 May 2018 01:37:44 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> Message-ID: <20180511153744.GW9562@ando.pearwood.info> On Fri, May 11, 2018 at 11:40:51AM +0200, Jacco van Dorp wrote: > I dont really like "given". > > If we compare: > > if m given m = re.match(stuff): > > to > > if m := re.match(stuff) > > then I count 4+(name_length) more tokens and 2 more spaces. Since I > believe := is perfectly clear, I don't see the reason for a far more > verbose syntax. I agree with Jacco here. We have to name the variable twice, even if it is only used once, and we have a relatively long keyword, five characters, longer than average for all keywords, and only one char short of the maximum. I know the aim isn't to absolutely minimise the number of keystrokes, but it does seem strange to use such a long symbol which requires duplicating the target, unless the intent is to discourage people from using it. -- Steve From rosuav at gmail.com Fri May 11 11:45:20 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 12 May 2018 01:45:20 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <9ba24601-d4a6-c208-3be3-2168cf7b617d@kynesim.co.uk> Message-ID: On Sat, May 12, 2018 at 12:04 AM, Jacco van Dorp wrote: >> [Jo?o] >> How do you read something like " while (cmd := get_command()).token != CMD_QUIT:" in plain english? > > while open-paren cee em dee colon is call get-underscore-command > close-paren dot token doesn't equal all-caps cee em dee underscore > quit colon. > > Might be some dutch in there. While cee-em-dee (from get-command) dot token isn't CMD_Quit ChrisA From steve at pearwood.info Fri May 11 12:17:24 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 12 May 2018 02:17:24 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> Message-ID: <20180511161724.GX9562@ando.pearwood.info> On Fri, May 11, 2018 at 12:37:43PM +0100, Rhodri James wrote: > Consider: > > while (cmd := get_command()).token != CMD_QUIT: > cmd.do_something() > > vs: > > while cmd.token != CMD_QUIT given cmd = get_command(): > cmd.do_something() Okay, considered. I think the first is preferable. Much earlier in the PEP 572 discussion, I strongly argued in favour of the expr as name syntax on the basis that the most important part of the overall expression is "expr", not the assignment target, and therefore that should come first. Even though I have accepted that "as" is not viable, I still believe that it is preferable to have the expression first, or if not first, at least as close to the left as we can get it. This "given" syntax puts the expr part all the way to the far right of the line. A line which is made all the longer for needing to use "given" and redundantly state the target name. It's like we're trying to maximize the distance the eye has to travel back and forth when reading. I have to read to the end of the line before I have any idea where cmd has come from or what it is. The fact that it comes from a "given" expression comes as a surprise at the end of the line. Now obviously this doesn't matter if I'm reading lines of code in careful detail, but I don't do that all the time. I skim code far more than I read it in careful detail, and the closer things are to the left, the more likely I am to see them while skimming. The further out they are, the easier they are to miss. I think that "given" will *literally* make reading harder, in that the eye has to travel further to spot the relevant expression while skimming over code. As I said, I don't think it makes any difference when reading closely in detail. But most of my reading of code is skimming to find the relevant line or section, and then read closely. I would probably skim a hundred lines for every one I read closely. We read more code than we write, but writing is important too. I think the verbosity of "given" (six chars versus two) and the redundancy of needing to repeat the name of the target even if you only use it once will soon make using this syntax seem like a chore. -- Steve From guido at python.org Fri May 11 12:25:46 2018 From: guido at python.org (Guido van Rossum) Date: Fri, 11 May 2018 12:25:46 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AF540CB.3040708@canterbury.ac.nz> References: <5AF540CB.3040708@canterbury.ac.nz> Message-ID: Maybe you could propose some kind of syntax using "whereas"? (It can be used as a preamble.) On Fri, May 11, 2018 at 3:05 AM, Greg Ewing wrote: > Guido van Rossum wrote: > >> I'd need well-reasoned explanations >> > > My reasoning is essentially the same as what I've already > said about "where". To summarise, "given" sounds like > something an English-speaking mathematician would write, > whereas ":=" doesn't even have an obvious pronunciation. > Some variation on "given" just seems greatly more pythonic > to me. > > -- > Greg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri May 11 12:33:10 2018 From: guido at python.org (Guido van Rossum) Date: Fri, 11 May 2018 12:33:10 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AF54753.2010604@canterbury.ac.nz> References: <14d15390-fba5-023d-12f5-274d884b88a0@kellett.im> <5AF54753.2010604@canterbury.ac.nz> Message-ID: On Fri, May 11, 2018 at 3:33 AM, Greg Ewing wrote: > Most people's short-term memory is good enough to remember > that "m" refers to the match object while they read the > next couple of lines. IMO, using a longer name would serve > no purpose and would just clutter things up. Indeed. A thought just occurred to me. Maybe we need to instigate a cultural shift where people think about style guides as less dictated by hard-coded rules that were "passed down from the mountain" and more as derived from research that we can all understand about usability. A lot more is known about how human perception and various types of memory and learning work than it was when the "7 things plus/minus 2" rule was invented ( https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two). It would be fascinating to imagine a future where language designers could talk about such topic with as much confidence as they talk about the efficiency of hash tables. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri May 11 12:43:14 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 12 May 2018 02:43:14 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> Message-ID: <20180511164314.GY9562@ando.pearwood.info> On Fri, May 11, 2018 at 03:47:05PM +0200, Jo?o Santos wrote: > How do you read something like " while (cmd := get_command()).token != > CMD_QUIT:" in plain english? I wouldn't if I could avoid it. I hardly ever program by talking about code in plain English. Often the lines are gobblydegook: zreplace = '%c%02d%02d' % (sign, h, m) # datetime.py and even when they are technically pronouncable English: # subprocess.py (p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite) = self._get_handles(stdin, stdout, stderr) my brain would glaze over by the second "p2c". I prefer to read and write code than speak it, and if I need to discuss it, I prefer to use a whiteboard so I can write things down. But if I really needed to, I'd probably start by reading it as: while open bracket command defined as get-command close bracket dot token is not equal to command-quit and then I'd probably drop the "defined" and swap the order around. Actually, that's not true. I probably wouldn't say that, not in a real conversation. What I'd probably say is, So, like, I've got this command object, see, which I get from calling get-command, right, and so I get the, um, token attribute, okay, and if that's not equal to the quit value, I loop until it is. Right? (And this is why I prefer *writing* code than *saying* code.) -- Steve From tim.peters at gmail.com Fri May 11 12:45:13 2018 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 11 May 2018 11:45:13 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <14d15390-fba5-023d-12f5-274d884b88a0@kellett.im> <5AF54753.2010604@canterbury.ac.nz> Message-ID: [Gustavo Carneiro] >>> IMHO, all these toy examples don't translate well to the real world >>> because they tend to use very short variable names while in real world [good >>> written code] tends to select longer more descriptive variable names. [Greg Ewing] >> I don't believe that's always true. It depends on the context. >> Sometimes, using long variable names can make code *harder* >> to read. >> >> I don't think there's anything unrealistic about this >> example: >> >> if m given m = pattern.match(the_string): >> nugget = m.group(2) >> >> Most people's short-term memory is good enough to remember >> that "m" refers to the match object while they read the >> next couple of lines. IMO, using a longer name would serve >> no purpose and would just clutter things up. [Nick Coghlan] > I've been thinking about this problem, and I think for the If/elif/while > cases it's actually possible to allow the "binding is the same as the > condition" case to be simplified to: > > if command = pattern.match(the_string): > ... > elif command = other_pattern.match(the_string): > ... > > while data = read_data(): Unless there's some weird font problem on my machine, that looks like a single "equals sign". In which case we'd be reproducing C's miserable confusion about whether: if (i = 1) was a too-hastily-typed spelling of the intended: if (i == 1) or whether they were thinking "equals" and typed "=" by mistake. If so, that would get an instant -1 from any number of core devs, who have vivid painful memories of being burned by that in C. That's not just speculation - it came up a number of times in the PEP 572 threads. > Allowing this would be part of the definition of the if/elif/while statement > headers, rather than a general purpose assignment expression. > > The restriction of the LHS to a simple name target would need to be in the > AST generator rather than in the grammar, but it's hardly the only case > where we do that kind of thing. Switching to the given expression form would > then only be necessary in cases where the condition *wasn't* the same as the > binding target. > > A similar enhancement could be made to conditional expressions (adjusting > their grammar to permit "EXPR if NAME = EXPR else EXPR") and filter clauses > in comprehensions (allowing "EXPR for TARGET in EXPR if NAME = EXPR"). In > essence, "if", "elif", and "while" would all allow for an "implied given" > clause in order to simplify the 90% case where the desired condition and the > bound expression are the same. Spell it ":=" (colon equals) instead, and a few core devs would stop objecting ;-) From steven at rigetti.com Fri May 11 12:36:31 2018 From: steven at rigetti.com (steven at rigetti.com) Date: Fri, 11 May 2018 09:36:31 -0700 (PDT) Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> Message-ID: <772c2592-681e-4d7c-9527-27ea52c78ca5@googlegroups.com> Hi everyone, I?m also a first time poster to python-ideas so I apologize if reviving a week old thread is bad form. I emailed Guido out of the blue to share some thoughts on the JavaScript pattern matching proposal?s applicability to Python and he encouraged me to post those thoughts here. The best argument for pattern matching is to support what Daniel F Mossat above calls ?structural patterns?. These go beyond simple value matching or boolean conditions that are better served with other constructs like if statements. Structural pattern matching allows for reasoning about the shape of data. As a practical example, in my day job I work as a software engineer at a startup that builds quantum computers. Python has been a great language for writing physics experiments and doing numerical simulations. However, our codebase contains a lot of `isinstance` calls due to the need to write converters from the physics experiment definition language to quantum assembly instructions. There?s some examples in our open source code as well, for instance here: https://github.com/rigetticomputing/pyquil/blob/master/pyquil/quil.py#L641 . This could be written more succinctly as: match instr: case Gate(name, params, qubits): result.append(Gate(name, params, [ qubit_mapping[q] for q in qubits]) case Measurement(qubit, classical_reg): result.append(Measurement( qubit_mapping[qubit], classical_reg) else: result.append(instr) Something that I also haven?t seen discussed yet is how well this would work with the new Python 3.7 dataclasses. Dataclasses allow us to create more structured data than using dicts alone. For instance, each instruction in our internal assembly language has its own dataclass. Pattern matching would make it easy to create readable code which loops over a list of these instructions and performs some sort of optimizations/transformations. Finally I want to talk about matching on the structure of built-in data structures like lists and dicts. The javascript proposal does a great job of supporting these data types and I think this would also be natural for Python which also has some destructuring bind support for these types on assignment. Consider the example of accessing nested dictionaries. This comes up a lot when working with JSON. If you had a dictionary like this: target = {'galaxy': {'system': {'planet': 'jupiter'}}} Then trying to access the value ?jupiter? would mean one of these alternatives: # Risks getting a ValueError my_planet = target[?galaxy?][?system?][?planet?] print(my_planet) # Awkward to read my_planet = target.get(?galaxy?, {}).get(?system?, {}).get(?planet?) if my_planet: print(my_planet) With pattern matching this could become more simply: match target: case {?galaxy?: {?system?: {?planet?: my_planet}}}: print(my_planet) This example was stolen from the tutorial for the glom library https://sedimental.org/glom_restructured_data.html which works on nested data. Structural pattern matching is a more universal concept and could eliminate the need for these kinds of helper functions. >From reading through this thread (as well as other background reading like https://groups.google.com/forum/#!msg/python-ideas/aninkpPpEAw/wCQ1IH5mAQAJ and the Javascript proposal) a couple things seem clear to me for pattern matching in the case of Python: - it should be statement based - it should have great support for built-in data types like lists, dicts, namedtuples, and dataclasses - it should form a coherent story with other similar Python concepts like unpacking on assignment There are a ton of details to be worked out obviously and we should go slow as Guido suggested. However, I believe that it would be worth doing the work. To that end: if there?s anyone else who?d like to collaborate and come up with a first draft of a more well-defined proposal I would love to commit my time to this, my email is below. Also, I sent this email from PyCon in Cleveland if anyone would like to brainstorm in person :). Steven Heidel Rigetti Quantum Computing steven at rigetti.com On Friday, May 4, 2018 at 4:37:43 PM UTC, Guido van Rossum wrote: > > Can I recommend going slow here? This is a very interesting topic where > many languages have gone before. I liked Daniel F Moisset's analysis about > the choices of a language designer and his conclusion that match should be > a statement. > > I just noticed the very similar proposal for JavaScript linked to by the > OP: https://github.com/tc39/proposal-pattern-matching -- this is more > relevant than what's done in e.g. F# or Swift because Python and JS are > much closer. (Possibly Elixir is also relevant, it seems the JS proposal > borrows from that.) > > A larger topic may be how to reach decisions. If I've learned one thing > from PEP 572 it's that we need to adjust how we discuss and evaluate > proposals. I'll think about this and start a discussion at the Language > Summit about this. > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri May 11 13:11:01 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 12 May 2018 03:11:01 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <14d15390-fba5-023d-12f5-274d884b88a0@kellett.im> <5AF54753.2010604@canterbury.ac.nz> Message-ID: <20180511171101.GZ9562@ando.pearwood.info> On Fri, May 11, 2018 at 12:33:10PM -0400, Guido van Rossum wrote: > A thought just occurred to me. Maybe we need to instigate a cultural shift > where people think about style guides as less dictated by hard-coded rules > that were "passed down from the mountain" and more as derived from research > that we can all understand about usability. That would be fantastic, but it's a real uphill battle. How long have we known about the effects of low-contrast on readability? https://www.wired.com/2016/10/how-the-web-became-unreadable/ http://contrastrebellion.com/ And don't get me started on the awful, awful choices made for chart design and data visualisation. We know how to make good charts: https://en.wikipedia.org/wiki/Edward_Tufte and yet: https://i.redd.it/0w0y1r1ba8x01.jpg -- Steve From rhodri at kynesim.co.uk Fri May 11 13:14:28 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Fri, 11 May 2018 18:14:28 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <9ba24601-d4a6-c208-3be3-2168cf7b617d@kynesim.co.uk> Message-ID: On 11/05/18 15:04, Jacco van Dorp wrote: > A while ago, we had this gem: > > 2018-04-06 8:19 GMT+02:00 Serhiy Storchaka: >> Using currently supported syntax: >> >> smooth_signal = [average for average in [0] for x in signal for average in [(1-decay)*average + decay*x]] > Go ahead and understand that line in 1 go. It's currently legal syntax > for a running average for a smoothing signal, which remembers > something about it. (Subject: Proposal: A Reduce-Map Comprehension and > a "last" builtin) > > You're not allowed to work it out bit by bit, just understand the > entire line or nothing. Any failure of yours proves my point. Personally I thought it proved the point that you shouldn't be trying to squash things like that into a list comprehension in the first place, because average = 0 smooth_signal = [] for x in signal: average = (1 - decay) * average + decay * x smooth_signal.append(average) is quite a bit more comprehensible. -- Rhodri James *-* Kynesim Ltd From steve at pearwood.info Fri May 11 13:27:25 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 12 May 2018 03:27:25 +1000 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: References: Message-ID: <20180511172725.GB9562@ando.pearwood.info> On Thu, May 10, 2018 at 09:53:40AM -0400, Guido van Rossum wrote: [...] > So all in all I'm not sure I think this is important enough to support, and > the rule "Use `:=` in expressions, not as a top level assignment" seems > easier to explain and understand. Like Terry, I too would find it useful to be able to use x := expression in the interactive interpreter to see the value immediately. But I'd rather start off by prohibiting := as a stand-alone statement, and relax the restriction later, than by allowing it and then regretting it later. I think the rule "binding expressions are prohibited as a stand-alone statement" might reduce some of the anxiety about having two ways to do assignment, even if the cost is that they are treated as a special case. (All other expressions are allowed as statements, even if they're pointless.) -- Steve From alexander.belopolsky at gmail.com Fri May 11 13:42:55 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 11 May 2018 13:42:55 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180511153744.GW9562@ando.pearwood.info> References: <5AF540CB.3040708@canterbury.ac.nz> <20180511153744.GW9562@ando.pearwood.info> Message-ID: On Fri, May 11, 2018 at 11:43 AM Steven D'Aprano wrote: > ... > I agree with Jacco here. We have to name the variable twice, even if it > is only used once, and we have a relatively long keyword, five > characters, longer than average for all keywords, and only one char > short of the maximum. To be fair when counting the keystrokes, you should take into account that the colon and the parentheses that appear in the := syntax are the upper register keys that counting the shift require two keystrokes each. From brenbarn at brenbarn.net Fri May 11 14:08:41 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Fri, 11 May 2018 11:08:41 -0700 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180511161724.GX9562@ando.pearwood.info> References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <20180511161724.GX9562@ando.pearwood.info> Message-ID: <5AF5DC29.9030000@brenbarn.net> On 2018-05-11 09:17, Steven D'Aprano wrote: > Much earlier in the PEP 572 discussion, I strongly argued in favour > of the > > expr as name > > syntax on the basis that the most important part of the overall > expression is "expr", not the assignment target, and therefore that > should come first. Even though I have accepted that "as" is not viable, > I still believe that it is preferable to have the expression first, or > if not first, at least as close to the left as we can get it. > > This "given" syntax puts the expr part all the way to the far right of > the line. A line which is made all the longer for needing to use "given" > and redundantly state the target name. > > It's like we're trying to maximize the distance the eye has to travel > back and forth when reading. > > I have to read to the end of the line before I have any idea where cmd > has come from or what it is. The fact that it comes from a "given" > expression comes as a surprise at the end of the line. > > Now obviously this doesn't matter if I'm reading lines of code in > careful detail, but I don't do that all the time. I skim code far more > than I read it in careful detail, and the closer things are to the left, > the more likely I am to see them while skimming. The further out they > are, the easier they are to miss. > > I think that "given" will*literally* make reading harder, in that the > eye has to travel further to spot the relevant expression while skimming > over code. As I said, I don't think it makes any difference when reading > closely in detail. But most of my reading of code is skimming to find > the relevant line or section, and then read closely. I would probably > skim a hundred lines for every one I read closely. That is an interesting argument --- interesting to me because I agree with a lot of it, but it leads me to the opposite conclusion, and also because it highlights some of the relevant factors for me. The main part I disagree with is that the most important thing is the definition of expr. Rather, what I think is most important is the role of expr within the surrounding expression. For simple cases, it doesn't much matter which comes first: if x := do_some_stuff(): if x where x = do_some_stuff(): . . . and it's true the latter is a bit more verbose in that case for little extra benefit. But when the locally-defined value is used within a more complicated expression (like the quadratic formula example), I think readability goes down significantly. To appease Tim, instead of using the quadratic formula, though, I will use a more realistic example that comes up fairly often for me: wanting to do some kind of normalization on a piece of data for a comparison, while keeping the unnormalized data for use within the block: if some_condition and (stuff:= get_user_input()).lower().strip().replace('-', ''): versus if some_condition and stuff.lower().strip().replace('-', '') given stuff = get_user_input(): Now, the latter is still more verbose. But to me it is now more readable, because the assignment does not disrupt the flow of reading the surrounding expression. This benefit increases the more complicated the surrounding expression is. Your point about reading ease is well taken, but also relevant to me is that we only read a piece of code *for the first time* once. The advantage of the given-style assignment is that on multiple readings, it foregrounds how the assigned value is USED, not how it is DEFINED. This encourages a "top-down" understanding of the expression in which you first understand the overall picture, and then later "drill down" into definition of what the components are. I wonder if some of the disagreement about the relative merits of the two cases comes from people focusing on different kinds of examples. As I said, I think the advantages of "cleft assignment" (i.e., where the assignment is shunted to the end of the line) become more pronounced as the surrounding expression becomes more complex. They also become somewhat greater as the definition of the expression becomes more complex, because there is more to skip over when finding out how the value is used. But many of the examples we're seeing are very simple ones, and in particular have a trivial surrounding expression. (That is, in something like "m := re.match()" the assigned value is not being used in any larger expression; the assigned value constitutes the whole of the expression.) I'm also finding it useful to think of parallel situations in prose writing, particularly journalistic-style prose writing. The current Python behavior, in which assignments must be done before the block, I think of as akin to something like this: "The IAAEA (International Association of Assignment Expression Advocates) is an organization dedicated to the promotion of assignment expressions. Its president is Eddie McEquals. In a statement yesterday, McEquals called for a new syntax to bring assignment expressions to the masses." The inline-assignment syntax is akin to this: In a statement yesterday, Eddie McEquals, president of the International Association of Assignment Expression Advocates (IAAEA), an organization dedicated to the promotion of assignment expressions, called for a new syntax to bring assignment expressions to the masses. The cleft-assignment syntax is akin to this: In a statement yesterday, Eddie McEquals called for a new syntax to bring assignment expressions to the masses. McEquals is the president of the International Association of Assignment Expression Advocates (IAAEA), which is an organization dedicated to the promotion of assignment expressions. Now of course I'm fudging a bit on the details (like whether you have IAAEA in parentheses or its full expansion), but the point is that the last version foregrounds the "bottom line" or "predicate" --- what actually happened. The first one foregrounds the "participants", or who/what was involved, and saves what actually happened for the end. But the middle one, to my eye, foregrounds nothing. It stuffs everything into one big clause where the descriptions of the participants occur as asides that disrupt the flow of reading. By the time we get to the predicate, we have to jump back to the beginning to remember who it is we're talking about, Again, the difference in readability varies depending on the complexity of the different components. If we just have "Eddie McEquals, president of the International Association of Assignment Expression advocates, delivered a speech yesterday at the organization's convention", the inline appositive is not so disruptive. But the more complex the inline definitions become, and especially the more complex the expression in which they are embedded, the lower readability goes, for me. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From brenbarn at brenbarn.net Fri May 11 14:13:53 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Fri, 11 May 2018 11:13:53 -0700 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AF5DC29.9030000@brenbarn.net> References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <20180511161724.GX9562@ando.pearwood.info> <5AF5DC29.9030000@brenbarn.net> Message-ID: <5AF5DD61.4050809@brenbarn.net> On 2018-05-11 11:08, Brendan Barnwell wrote: > . . . and it's true the latter is a bit more verbose in that case for > little extra benefit. But when the locally-defined value is used within > a more complicated expression (like the quadratic formula example), I > think readability goes down significantly. To appease Tim, instead of > using the quadratic formula, though, I will use a more realistic example > that comes up fairly often for me: wanting to do some kind of > normalization on a piece of data for a comparison, while keeping the > unnormalized data for use within the block: > > if some_condition and (stuff:= > get_user_input()).lower().strip().replace('-', ''): > > versus > > if some_condition and stuff.lower().strip().replace('-', '') given > stuff = get_user_input(): Ironically I weakened my argument by forgetting to finish my expression there. I intended that chain of method calls to be used in a comparison to make the surrounding expression more complex. So revise the above to if some_condition and (stuff := get_user_input()).lower().strip().replace('-', '') == existing_value: versus if some_condition and stuff.lower().strip().replace('-', '') == existing_value given stuff = get_user_input(): -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From tim.peters at gmail.com Fri May 11 14:44:41 2018 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 11 May 2018 13:44:41 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AF5DD61.4050809@brenbarn.net> References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <20180511161724.GX9562@ando.pearwood.info> <5AF5DC29.9030000@brenbarn.net> <5AF5DD61.4050809@brenbarn.net> Message-ID: [Brendan Barnwell] >> >> . . . and it's true the latter is a bit more verbose in that case for >> little extra benefit. But when the locally-defined value is used within >> a more complicated expression (like the quadratic formula example), I >> think readability goes down significantly. To appease Tim, instead of >> using the quadratic formula, though, I will use a more realistic example >> that comes up fairly often for me: wanting to do some kind of >> normalization on a piece of data for a comparison, while keeping the >> unnormalized data for use within the block: >> >> if some_condition and (stuff:= >> get_user_input()).lower().strip().replace('-', ''): >> >> versus >> >> if some_condition and stuff.lower().strip().replace('-', '') given >> stuff = get_user_input(): [also Brendan] > Ironically I weakened my argument by forgetting to finish my > expression there. I intended that chain of method calls to be used in a > comparison to make the surrounding expression more complex. So revise the > above to > > if some_condition and (stuff := > get_user_input()).lower().strip().replace('-', '') == existing_value: > > versus > > if some_condition and stuff.lower().strip().replace('-', '') == > existing_value given stuff = get_user_input(): Even more ironically, to my eyes the original more strongly supported your view than the rewrite ;-) "given stuff =" stuck out in the original because it was preceded by punctuation (a right parenthesis). I had to read the rewrite 3 times before i realized you were even _using_ "given", because there it's buried between two other names - "existing_value given stuff" - and visually looks more like it's actually the 3rd of 4 words (because of the underscore in "existing_value"). Of course that would have been obvious in a Python-aware editor that colored "given" differently, but as-is I found the original easy to read but the rewrite a puzzle to decode. Similarly, in the rewritten assignment expression spelling, it's obvious at a glance that the test is of the form some_condition and some_messy_expression == existing_value but in the rewritten "given" sample that's obscured because "existing_value" not only doesn't end the statement, it's not even followed by punctuation. Of course coloring "given" differently would remove that visual uncertainty too. For a dumb display, I'd write it if some_condition and ( stuff.lower().strip().replace('-', '') == existing_value) given stuff = get_user_input(): instead (added parens so that "existing_value" and "given" are separated by punctuation). From marcidy at gmail.com Fri May 11 14:52:02 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Fri, 11 May 2018 11:52:02 -0700 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AF5DD61.4050809@brenbarn.net> References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <20180511161724.GX9562@ando.pearwood.info> <5AF5DC29.9030000@brenbarn.net> <5AF5DD61.4050809@brenbarn.net> Message-ID: Apology for top post, but this is a general statement about Readability and not a response to an individual. it would be nice to list the objective parts separate from the "argument" (i.e. debate, not fight), perhaps list them then make a case for which metric is a more important, and which values produce better results. Key strokes, keyboard location, location of information vs where it is used, cognitive load (objective barring neuroscience changes) are all objective points (along with many other points raised). "Better" will be decided by Guido I guess, but listing objective points with explanatory examples gives a basis for that discussion. Legibility, for example, is not objective at all, it has nothing to do with syntax. This covers fonts, colors, monitors, lighting, chairs, lunch, etc. None of this is relevent to the symbols or their ordering in a file we all must read. Teachability likewise. My opinion here is learnability is far more important anyways, I am 90% self taught going back 25 years, but this is equally unquantifiable. Perhaps just trust students to learn as an author must trust a reader. Of course, let it not be lost that determining teachability and learnability for something which doesn't even exist yet is quite challenging. Any quantification will give more information than only naked impassioned pleas to Readability. Note Tim came up with a real metric: 2 * count(":=")/len(statement). It's objective. it's just unclear if a higher score is better or worse. However, one could say "a Tim of .3 is considered too high" as a guideline. Perhaps coders find these opportunities to express feelings and opinion cathartic after speaking to the most pedantic creature on Earth (compilier/computer) but I think exercising the high skill level available here to dissect and find objective statements is a worthy puzzle. On Fri, May 11, 2018, 11:14 Brendan Barnwell wrote: > On 2018-05-11 11:08, Brendan Barnwell wrote: > > . . . and it's true the latter is a bit more verbose in that case for > > little extra benefit. But when the locally-defined value is used within > > a more complicated expression (like the quadratic formula example), I > > think readability goes down significantly. To appease Tim, instead of > > using the quadratic formula, though, I will use a more realistic example > > that comes up fairly often for me: wanting to do some kind of > > normalization on a piece of data for a comparison, while keeping the > > unnormalized data for use within the block: > > > > if some_condition and (stuff:= > > get_user_input()).lower().strip().replace('-', ''): > > > > versus > > > > if some_condition and stuff.lower().strip().replace('-', '') given > > stuff = get_user_input(): > > Ironically I weakened my argument by forgetting to finish my > expression > there. I intended that chain of method calls to be used in a comparison > to make the surrounding expression more complex. So revise the above to > > if some_condition and (stuff := > get_user_input()).lower().strip().replace('-', '') == existing_value: > > versus > > if some_condition and stuff.lower().strip().replace('-', '') == > existing_value given stuff = get_user_input(): > > -- > Brendan Barnwell > "Do not follow where the path may lead. Go, instead, where there is no > path, and leave a trail." > --author unknown > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Fri May 11 15:51:25 2018 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 11 May 2018 14:51:25 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <20180511161724.GX9562@ando.pearwood.info> <5AF5DC29.9030000@brenbarn.net> <5AF5DD61.4050809@brenbarn.net> Message-ID: <16350c20a60.27a3.db5b03704c129196a4e9415e55413ce6@gmail.com> On May 11, 2018 1:45:27 PM Tim Peters wrote: > [Brendan Barnwell] >>> >>> . . . and it's true the latter is a bit more verbose in that case for >>> little extra benefit. But when the locally-defined value is used within >>> a more complicated expression (like the quadratic formula example), I >>> think readability goes down significantly. To appease Tim, instead of >>> using the quadratic formula, though, I will use a more realistic example >>> that comes up fairly often for me: wanting to do some kind of >>> normalization on a piece of data for a comparison, while keeping the >>> unnormalized data for use within the block: >>> >>> if some_condition and (stuff:= >>> get_user_input()).lower().strip().replace('-', ''): >>> >>> versus >>> >>> if some_condition and stuff.lower().strip().replace('-', '') given >>> stuff = get_user_input(): > > [also Brendan] >> Ironically I weakened my argument by forgetting to finish my >> expression there. I intended that chain of method calls to be used in a >> comparison to make the surrounding expression more complex. So revise the >> above to >> >> if some_condition and (stuff := >> get_user_input()).lower().strip().replace('-', '') == existing_value: >> >> versus >> >> if some_condition and stuff.lower().strip().replace('-', '') == >> existing_value given stuff = get_user_input(): > > Even more ironically, to my eyes the original more strongly supported > your view than the rewrite ;-) > > "given stuff =" stuck out in the original because it was preceded by > punctuation (a right parenthesis). > > I had to read the rewrite 3 times before i realized you were even > _using_ "given", because there it's buried between two other names - > "existing_value given stuff" - and visually looks more like it's > actually the 3rd of 4 words (because of the underscore in > "existing_value"). There are some variants of tanks like 'if let' where the bindings come *first*, unlike 'given' where they come last (like Haskell's 'where'). > > Of course that would have been obvious in a Python-aware editor that > colored "given" differently, but as-is I found the original easy to > read but the rewrite a puzzle to decode. Well, you've partly explained the reason: our eyes are drawn to what sticks out. In this case, the := stuck out in a section heavy on black letters. In a proper editor, it may even be the other way around: they tend to highlight keywords in a stronger manner (like bolding) than operators. > > Similarly, in the rewritten assignment expression spelling, it's > obvious at a glance that the test is of the form > > some_condition and some_messy_expression == existing_value > > but in the rewritten "given" sample that's obscured because > "existing_value" not only doesn't end the statement, it's not even > followed by punctuation. Of course coloring "given" differently would > remove that visual uncertainty too. For a dumb display, I'd write it > > if some_condition and ( > stuff.lower().strip().replace('-', '') == existing_value) given > stuff = get_user_input(): > > instead (added parens so that "existing_value" and "given" are > separated by punctuation). > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From goosey15 at gmail.com Fri May 11 17:12:05 2018 From: goosey15 at gmail.com (Angus Hollands) Date: Fri, 11 May 2018 22:12:05 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: I'd like to address Steven D'Aprano's reply earlier in the list concerning "given" vs ":=" clauses. My stance on the PEP is that the general ability to assign locally within an expression is undesirable. I gave several reasons, but in general, it means processing lines becomes more complex, and it makes refactoring more difficult. I don't think that there are enough cases where it would be useful to justify its negatives. I can recall but several occasions (writing regexes, mainly) that I've wanted something like this. However, if we were to assert that PEP 572 was mandatory, and then dispute its implementation, then I would argue as follows: *Cost:* A lot of people seem quite fixed upon the character cost of "given" vs ":=". I think this is a straw man. Most people spend longer reading than writing code, so if you agree with that observation, then it's the line length that you are concerned with. While it is true that overly long lines are not desirable, it's also true that incredibly short lines are equally difficult to understand. I don't think that keyword length can be independently taken too seriously as support for any proposal. Once you have to include grouping parentheses, this length difference quickly diminishes anyway. *Repetition of variable:* In addition, several people seem unhappy that the "given" keyword approach sees a repetition of the state variable in question. I think this is actually a benefit for two reasons. Firstly, you can save the expression in question to the intermediate variable, and then access a sub expression in a condition e.g while value.in_use given value = get_next_pool_item(): print(value.refcount()) Instead of while (value:=get_next_pool_item()) and value.in_use: print(value.refcount()) Secondly, it reads in the order that matters. When reading the first line, one encounters what the condition is evaluating *first*, and then the implementation details (m=p.match) second. It reads as one would describe a mathematical equation in a paper, and clearly separates *what you're interested in* from *what it depends upon*. This is what I particularly dislike about the ":=" operator approach, the value, and name it is bound to, are unrelated at the point of evaluation, yet are right next to each other. It's visual noise. In the example above, the reader has to read the entire line when trying to find the loop condition. Someone earlier suggested this was a bad thing, that the expression was far to the right of the line. I disagree. In cases where you might want to assign an expression to a variable, it is going to be used at least twice (otherwise just use the expression directly). At this point, the target name should be explicit in what it represents. This depends upon context slightly, such as what is happening in the local region of code. An example; when matching regexes, the match conditions (if x.group(2) == ...) are more important than what you matched on, usually. In addition, the two cases are not identical - if the API is such that get_next_pool_item should not return None values, this wouldn't be caught in the ":=" approach, unless you add more conditions. Yes, you could argue that I've cherry picked an example here. Actually, I haven't; I observed this after writing the example. What am I getting at here? In effect, the "given" keyword provides a superset of use cases to that of ":=". Dare I say it, but *explicit is better than implicit*. *Readability:* A smaller point is that I don't feel that ":=" is very readable. If we had to use an operator, I think $= is better, but me reasoning for this is weak. I think it derives from my observation that ":=" is slow to distinguish from "=". Regards, Angus Hollands On Fri, 11 May 2018 at 17:45 wrote: > Send Python-ideas mailing list submissions to > python-ideas at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/python-ideas > or, via email, send a message with subject or body 'help' to > python-ideas-request at python.org > > You can reach the person managing the list at > python-ideas-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-ideas digest..." > > > Today's Topics: > > 1. Re: Inline assignments using "given" clauses (Steven D'Aprano) > 2. Re: Inline assignments using "given" clauses (Guido van Rossum) > 3. Re: Inline assignments using "given" clauses (Guido van Rossum) > 4. Re: Inline assignments using "given" clauses (Steven D'Aprano) > 5. Re: Inline assignments using "given" clauses (Tim Peters) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 12 May 2018 02:17:24 +1000 > From: Steven D'Aprano > To: python-ideas at python.org > Subject: Re: [Python-ideas] Inline assignments using "given" clauses > Message-ID: <20180511161724.GX9562 at ando.pearwood.info> > Content-Type: text/plain; charset=us-ascii > > On Fri, May 11, 2018 at 12:37:43PM +0100, Rhodri James wrote: > > > Consider: > > > > while (cmd := get_command()).token != CMD_QUIT: > > cmd.do_something() > > > > vs: > > > > while cmd.token != CMD_QUIT given cmd = get_command(): > > cmd.do_something() > > Okay, considered. I think the first is preferable. > > Much earlier in the PEP 572 discussion, I strongly argued in favour > of the > > expr as name > > syntax on the basis that the most important part of the overall > expression is "expr", not the assignment target, and therefore that > should come first. Even though I have accepted that "as" is not viable, > I still believe that it is preferable to have the expression first, or > if not first, at least as close to the left as we can get it. > > This "given" syntax puts the expr part all the way to the far right of > the line. A line which is made all the longer for needing to use "given" > and redundantly state the target name. > > It's like we're trying to maximize the distance the eye has to travel > back and forth when reading. > > I have to read to the end of the line before I have any idea where cmd > has come from or what it is. The fact that it comes from a "given" > expression comes as a surprise at the end of the line. > > Now obviously this doesn't matter if I'm reading lines of code in > careful detail, but I don't do that all the time. I skim code far more > than I read it in careful detail, and the closer things are to the left, > the more likely I am to see them while skimming. The further out they > are, the easier they are to miss. > > I think that "given" will *literally* make reading harder, in that the > eye has to travel further to spot the relevant expression while skimming > over code. As I said, I don't think it makes any difference when reading > closely in detail. But most of my reading of code is skimming to find > the relevant line or section, and then read closely. I would probably > skim a hundred lines for every one I read closely. > > We read more code than we write, but writing is important too. I think > the verbosity of "given" (six chars versus two) and the redundancy of > needing to repeat the name of the target even if you only use it once > will soon make using this syntax seem like a chore. > > > > -- > Steve > > > ------------------------------ > > Message: 2 > Date: Fri, 11 May 2018 12:25:46 -0400 > From: Guido van Rossum > To: Greg Ewing > Cc: python-ideas > Subject: Re: [Python-ideas] Inline assignments using "given" clauses > Message-ID: > inDi7dFiF7s5R7DrAQXw at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Maybe you could propose some kind of syntax using "whereas"? (It can be > used as a preamble.) > > On Fri, May 11, 2018 at 3:05 AM, Greg Ewing > wrote: > > > Guido van Rossum wrote: > > > >> I'd need well-reasoned explanations > >> > > > > My reasoning is essentially the same as what I've already > > said about "where". To summarise, "given" sounds like > > something an English-speaking mathematician would write, > > whereas ":=" doesn't even have an obvious pronunciation. > > Some variation on "given" just seems greatly more pythonic > > to me. > > > > -- > > Greg > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/python-ideas/attachments/20180511/d6443de1/attachment-0001.html > > > > ------------------------------ > > Message: 3 > Date: Fri, 11 May 2018 12:33:10 -0400 > From: Guido van Rossum > To: Greg Ewing > Cc: Python-Ideas > Subject: Re: [Python-ideas] Inline assignments using "given" clauses > Message-ID: > 7SU3GT3TWjZkRTqpg at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On Fri, May 11, 2018 at 3:33 AM, Greg Ewing > wrote: > > > Most people's short-term memory is good enough to remember > > that "m" refers to the match object while they read the > > next couple of lines. IMO, using a longer name would serve > > no purpose and would just clutter things up. > > > Indeed. > > A thought just occurred to me. Maybe we need to instigate a cultural shift > where people think about style guides as less dictated by hard-coded rules > that were "passed down from the mountain" and more as derived from research > that we can all understand about usability. A lot more is known about how > human perception and various types of memory and learning work than it was > when the "7 things plus/minus 2" rule was invented ( > https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two > ). > > It would be fascinating to imagine a future where language designers could > talk about such topic with as much confidence as they talk about the > efficiency of hash tables. > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/python-ideas/attachments/20180511/808769ad/attachment-0001.html > > > > ------------------------------ > > Message: 4 > Date: Sat, 12 May 2018 02:43:14 +1000 > From: Steven D'Aprano > To: python-ideas at python.org > Subject: Re: [Python-ideas] Inline assignments using "given" clauses > Message-ID: <20180511164314.GY9562 at ando.pearwood.info> > Content-Type: text/plain; charset=iso-8859-1 > > On Fri, May 11, 2018 at 03:47:05PM +0200, Jo?o Santos wrote: > > > How do you read something like " while (cmd := get_command()).token != > > CMD_QUIT:" in plain english? > > I wouldn't if I could avoid it. I hardly ever program by talking about > code in plain English. Often the lines are gobblydegook: > > zreplace = '%c%02d%02d' % (sign, h, m) # datetime.py > > and even when they are technically pronouncable English: > > # subprocess.py > (p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite) = > self._get_handles(stdin, stdout, stderr) > > my brain would glaze over by the second "p2c". I prefer to read and > write code than speak it, and if I need to discuss it, I prefer to use a > whiteboard so I can write things down. > > But if I really needed to, I'd probably start by reading it as: > > while open bracket command defined as get-command close bracket dot > token is not equal to command-quit > > and then I'd probably drop the "defined" and swap the order around. > > Actually, that's not true. I probably wouldn't say that, not in a > real conversation. What I'd probably say is, > > So, like, I've got this command object, see, which I get > from calling get-command, right, and so I get the, um, > token attribute, okay, and if that's not equal to the > quit value, I loop until it is. Right? > > > (And this is why I prefer *writing* code than *saying* code.) > > > -- > Steve > > > ------------------------------ > > Message: 5 > Date: Fri, 11 May 2018 11:45:13 -0500 > From: Tim Peters > To: Nick Coghlan > Cc: Greg Ewing , Python-Ideas > > Subject: Re: [Python-ideas] Inline assignments using "given" clauses > Message-ID: > qEw at mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > [Gustavo Carneiro] > >>> IMHO, all these toy examples don't translate well to the real world > >>> because they tend to use very short variable names while in real world > [good > >>> written code] tends to select longer more descriptive variable names. > > [Greg Ewing] > >> I don't believe that's always true. It depends on the context. > >> Sometimes, using long variable names can make code *harder* > >> to read. > >> > >> I don't think there's anything unrealistic about this > >> example: > >> > >> if m given m = pattern.match(the_string): > >> nugget = m.group(2) > >> > >> Most people's short-term memory is good enough to remember > >> that "m" refers to the match object while they read the > >> next couple of lines. IMO, using a longer name would serve > >> no purpose and would just clutter things up. > > [Nick Coghlan] > > I've been thinking about this problem, and I think for the If/elif/while > > cases it's actually possible to allow the "binding is the same as the > > condition" case to be simplified to: > > > > if command = pattern.match(the_string): > > ... > > elif command = other_pattern.match(the_string): > > ... > > > > while data = read_data(): > > Unless there's some weird font problem on my machine, that looks like > a single "equals sign". In which case we'd be reproducing C's > miserable confusion about whether: > > if (i = 1) > > was a too-hastily-typed spelling of the intended: > > if (i == 1) > > or whether they were thinking "equals" and typed "=" by mistake. > > If so, that would get an instant -1 from any number of core devs, who > have vivid painful memories of being burned by that in C. That's not > just speculation - it came up a number of times in the PEP 572 > threads. > > > > Allowing this would be part of the definition of the if/elif/while > statement > > headers, rather than a general purpose assignment expression. > > > > The restriction of the LHS to a simple name target would need to be in > the > > AST generator rather than in the grammar, but it's hardly the only case > > where we do that kind of thing. Switching to the given expression form > would > > then only be necessary in cases where the condition *wasn't* the same as > the > > binding target. > > > > A similar enhancement could be made to conditional expressions (adjusting > > their grammar to permit "EXPR if NAME = EXPR else EXPR") and filter > clauses > > in comprehensions (allowing "EXPR for TARGET in EXPR if NAME = EXPR"). In > > essence, "if", "elif", and "while" would all allow for an "implied given" > > clause in order to simplify the 90% case where the desired condition and > the > > bound expression are the same. > > Spell it ":=" (colon equals) instead, and a few core devs would stop > objecting ;-) > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > > ------------------------------ > > End of Python-ideas Digest, Vol 138, Issue 90 > ********************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri May 11 17:44:24 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 12 May 2018 07:44:24 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: On Sat, May 12, 2018 at 7:12 AM, Angus Hollands wrote: > e.g > > while value.in_use given value = get_next_pool_item(): > print(value.refcount()) > > Instead of > > while (value:=get_next_pool_item()) and value.in_use: > print(value.refcount()) > > In addition, the two cases are not identical - if the API is such that > get_next_pool_item should not return None values, this wouldn't be caught in > the ":=" approach, unless you add more conditions. Yes, you could argue that > I've cherry picked an example here. Actually, I haven't; I observed this > after writing the example. > > What am I getting at here? In effect, the "given" keyword provides a > superset of use cases to that of ":=". Dare I say it, but explicit is better > than implicit. I'm not sure what you're getting at here. To make your two versions equivalent, you'd need to write them like this: while value and value.in_use given value = get_next_pool_item(): print(value.refcount()) while (value:=get_next_pool_item()) and value.in_use: print(value.refcount()) or like this: while value.in_use given value = get_next_pool_item(): print(value.refcount()) while (value:=get_next_pool_item()).in_use: print(value.refcount()) There, now they're identical. There's no superset or subset of use cases. And I have no idea how that connects with the oft-misused "explicit is better than implicit". (I'm still fairly sure that "explicit" and "strongly typed" are both synonyms for "stuff I like", with their antonyms "implicit" and "weakly typed" both being synonyms for "stuff I don't like". Years of discussion have not disproven this theory yet.) ChrisA From e+python-ideas at kellett.im Fri May 11 17:56:58 2018 From: e+python-ideas at kellett.im (Ed Kellett) Date: Fri, 11 May 2018 22:56:58 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <05471209-9e91-b087-d260-60f76db44065@kellett.im> On 2018-05-11 22:12, Angus Hollands wrote: > while (value:=get_next_pool_item()) and value.in_use: > print(value.refcount()) Just as a heads-up, I believe the prescribed way of doing that is: while (value := get_next_pool_item()).in_use: Of course you'd need additional mess to do something else with value. I don't like the asymmetry here: while (value := get_next_pool_item()).in_use and value is not blah: > Secondly, it reads in the order that matters. When reading the first line, > one encounters what the condition is evaluating *first*, and then the > implementation details (m=p.match) second. It reads as one would describe a > mathematical equation in a paper, and clearly separates *what you're > interested in* from *what it depends upon*. This is what I particularly > dislike about the ":=" operator approach, the value, and name it is bound > to, are unrelated at the point of evaluation, yet are right next to each > other. It's visual noise. In the example above, the reader has to read the > entire line when trying to find the loop condition. I'm inclined to agree. But several people have argued that this is more readable than the alternative. I don't buy the reasoning, but they still like it better, and there's probably no point in going any further into this aspect. I doubt people are going to be convinced. > What am I getting at here? In effect, the "given" keyword provides a > superset of use cases to that of ":=". Dare I say it, but *explicit is > better than implicit*. I'm not sure that it's strictly a superset. It's arguably the reverse, since it's restricted to statements with a condition rather than arbitrary expressions. I think the more important thing is that it's--subjectively--better at the subset of use cases that people seem to actually have (as listed in the OP). > *Readability:* > A smaller point is that I don't feel that ":=" is very readable. If we had > to use an operator, I think $= is better, but me reasoning for this is > weak. I think it derives from my observation that ":=" is slow to > distinguish from "=". Clearly the objectively best choice is "<-". -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Fri May 11 18:04:17 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 12 May 2018 08:04:17 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <05471209-9e91-b087-d260-60f76db44065@kellett.im> References: <05471209-9e91-b087-d260-60f76db44065@kellett.im> Message-ID: On Sat, May 12, 2018 at 7:56 AM, Ed Kellett wrote: >> *Readability:* >> A smaller point is that I don't feel that ":=" is very readable. If we had >> to use an operator, I think $= is better, but me reasoning for this is >> weak. I think it derives from my observation that ":=" is slow to >> distinguish from "=". > > Clearly the objectively best choice is "<-". It's already legal syntax. That's an advantage, right? ChrisA From steve at pearwood.info Fri May 11 19:50:06 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 12 May 2018 09:50:06 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <20180511161724.GX9562@ando.pearwood.info> <5AF5DC29.9030000@brenbarn.net> <5AF5DD61.4050809@brenbarn.net> Message-ID: <20180511235006.GC9562@ando.pearwood.info> On Fri, May 11, 2018 at 11:52:02AM -0700, Matt Arcidy wrote: > Note Tim came up with a real metric: > 2 * count(":=")/len(statement). > It's objective. it's just unclear if a higher score is better or worse. > However, one could say "a Tim of .3 is considered too high" as a guideline. I think Tim was making a joke about demanding objective measurements of subjective things. Certainly he hasn't done any research or study to justify that metric. He just plucked the formula out of thin air. Or at least no peer reviewed research. -- Steve From steve at pearwood.info Fri May 11 19:51:19 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 12 May 2018 09:51:19 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> <20180511153744.GW9562@ando.pearwood.info> Message-ID: <20180511235119.GD9562@ando.pearwood.info> On Fri, May 11, 2018 at 01:42:55PM -0400, Alexander Belopolsky wrote: > On Fri, May 11, 2018 at 11:43 AM Steven D'Aprano > wrote: > > ... > > I agree with Jacco here. We have to name the variable twice, even if it > > is only used once, and we have a relatively long keyword, five > > characters, longer than average for all keywords, and only one char > > short of the maximum. > > To be fair when counting the keystrokes, you should take into account that > the colon and the parentheses that appear in the := syntax are the upper > register keys that counting the shift require two keystrokes each. That's a good point, but also to be fair, I didn't mention keystrokes, only characters :-) -- Steve From steve at pearwood.info Fri May 11 19:57:24 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 12 May 2018 09:57:24 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <20180511235724.GE9562@ando.pearwood.info> On Sat, May 12, 2018 at 07:44:24AM +1000, Chris Angelico wrote: > (I'm still fairly sure that "explicit" and "strongly typed" are both > synonyms for "stuff I like", with their antonyms "implicit" and > "weakly typed" both being synonyms for "stuff I don't like". Years of > discussion have not disproven this theory yet.) That's certainly how they are used the great majority of the time. A bit like how "strawman argument" is mostly used to mean "dammit, you just spotted an unwelcome consequence and/or flaw in my position which I have no counter too". :-) -- Steve From tim.peters at gmail.com Fri May 11 20:04:22 2018 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 11 May 2018 19:04:22 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180511235006.GC9562@ando.pearwood.info> References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <20180511161724.GX9562@ando.pearwood.info> <5AF5DC29.9030000@brenbarn.net> <5AF5DD61.4050809@brenbarn.net> <20180511235006.GC9562@ando.pearwood.info> Message-ID: [Matt Arcidy] >> Note Tim came up with a real metric: >> 2 * count(":=")/len(statement). >> It's objective. it's just unclear if a higher score is better or worse. >> However, one could say "a Tim of .3 is considered too high" as a guideline. [Steven D'Aprano] > I think Tim was making a joke about demanding objective measurements of > subjective things. > > Certainly he hasn't done any research or study to justify that metric. > He just plucked the formula out of thin air. It was the outcome of an intense 17-year research project. > Or at least no peer reviewed research. Au contraire! My peers are here, and that message was reviewed by at least 3 people on this list. That said, I am a fan of objectively measuring subjective things, just not of taking the measurements seriously ;-) If people do want to take it seriously, check out prior Python art first: http://radon.readthedocs.io/en/latest/intro.html From apalala at gmail.com Fri May 11 20:41:07 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Fri, 11 May 2018 20:41:07 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180511161724.GX9562@ando.pearwood.info> References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <20180511161724.GX9562@ando.pearwood.info> Message-ID: > > while (cmd := get_command()).token != CMD_QUIT: > > cmd.do_something() > > while get_command() as cmd: if cmd.token == CMD_QUIT: break cmd.do_something() -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Fri May 11 21:39:50 2018 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 11 May 2018 21:39:50 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <369b6cef-c979-0daf-1f86-3017093d62e9@trueblade.com> On 5/11/18 5:12 PM, Angus Hollands wrote: > *Readability:* > A smaller point is that I don't feel that ":=" is very readable. If we > had to use an operator, I think $= is better, but me reasoning for this > is weak. I think it derives from my observation that ":=" is slow to > distinguish from "=". := would prevent you from using assignment expressions inside f-strings, which could be argued is a good thing. To demonstrate, and just for giggles, this works in 3.6, and appears to have the desired behavior: -------------------------- class X: def __init__(self, value): self.value = value def __str__(self): return str(self.value) def __format__(self, fmt): assert fmt[0] == '=' self.value = eval(fmt[1:]) return '' x = X(3) print(x) f'{x:=4}' # Behold! print(x) -------------------------- Produces: 3 4 I kid, of course. Eric From rob.cliffe at btinternet.com Fri May 11 21:37:20 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sat, 12 May 2018 02:37:20 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <9ba24601-d4a6-c208-3be3-2168cf7b617d@kynesim.co.uk> Message-ID: Yeah, well, I'm totally lost.? Even after trying out this code, and refactoring it once if not twice, I didn't understand it.? I don't know what point you're trying to prove, but you seem to have comprehensively proved it. On 11/05/2018 15:04, Jacco van Dorp wrote: > A while ago, we had this gem: > > 2018-04-06 8:19 GMT+02:00 Serhiy Storchaka : >> Using currently supported syntax: >> >> smooth_signal = [average for average in [0] for x in signal for average in [(1-decay)*average + decay*x]] > Go ahead and understand that line in 1 go. It's currently legal syntax > for a running average for a smoothing signal, which remembers > something about it. (Subject: Proposal: A Reduce-Map Comprehension and > a "last" builtin) > > You're not allowed to work it out bit by bit, just understand the > entire line or nothing. Any failure of yours proves my point. > >> [Jo?o] >> How do you read something like " while (cmd := get_command()).token != CMD_QUIT:" in plain english? > while open-paren cee em dee colon is call get-underscore-command > close-paren dot token doesn't equal all-caps cee em dee underscore > quit colon. > > Might be some dutch in there. > > But far more importantly, I can hold the concept into my head, or just > the parts of it that I need. How we call it in english is actually not > a good argument - whether we can easily mentally parse it is, since I > tend not to code by voice command, but with a keyboard. Your mileage > may vary, but I think we should optimize for keyboard coding over > voice chat coding. And when I need to refer to it, I say "this bit > here" or I copy paste it. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > --- > This email has been checked for viruses by AVG. > http://www.avg.com From raiderrobert at gmail.com Fri May 11 22:01:01 2018 From: raiderrobert at gmail.com (Robert Roskam) Date: Fri, 11 May 2018 19:01:01 -0700 (PDT) Subject: [Python-ideas] Pattern Matching Syntax In-Reply-To: <772c2592-681e-4d7c-9527-27ea52c78ca5@googlegroups.com> References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> <772c2592-681e-4d7c-9527-27ea52c78ca5@googlegroups.com> Message-ID: <2b64fcb0-b8af-41b6-af84-58f2f5189fbd@googlegroups.com> Hey Steven, I'm also at PyCon. Shall we take this off list and attempt to meet up and discuss? On Friday, May 11, 2018 at 12:36:32 PM UTC-4, ste... at rigetti.com wrote: > > Hi everyone, I?m also a first time poster to python-ideas so I apologize > if reviving a week old thread is bad form. I emailed Guido out of the blue > to share some thoughts on the JavaScript pattern matching proposal?s > applicability to Python and he encouraged me to post those thoughts here. > > The best argument for pattern matching is to support what Daniel F Mossat > above calls ?structural patterns?. These go beyond simple value matching or > boolean conditions that are better served with other constructs like if > statements. Structural pattern matching allows for reasoning about the > shape of data. > > As a practical example, in my day job I work as a software engineer at a > startup that builds quantum computers. Python has been a great language for > writing physics experiments and doing numerical simulations. However, our > codebase contains a lot of `isinstance` calls due to the need to write > converters from the physics experiment definition language to quantum > assembly instructions. There?s some examples in our open source code as > well, for instance here: > https://github.com/rigetticomputing/pyquil/blob/master/pyquil/quil.py#L641 > . This could be written more succinctly as: > > match instr: > case Gate(name, params, qubits): result.append(Gate(name, params, [ > qubit_mapping[q] for q in qubits]) > case Measurement(qubit, classical_reg): result.append(Measurement( > qubit_mapping[qubit], classical_reg) > else: result.append(instr) > > Something that I also haven?t seen discussed yet is how well this would > work with the new Python 3.7 dataclasses. Dataclasses allow us to create > more structured data than using dicts alone. For instance, each instruction > in our internal assembly language has its own dataclass. Pattern matching > would make it easy to create readable code which loops over a list of these > instructions and performs some sort of optimizations/transformations. > > Finally I want to talk about matching on the structure of built-in data > structures like lists and dicts. The javascript proposal does a great job > of supporting these data types and I think this would also be natural for > Python which also has some destructuring bind support for these types on > assignment. > > Consider the example of accessing nested dictionaries. This comes up a lot > when working with JSON. If you had a dictionary like this: > > target = {'galaxy': {'system': {'planet': 'jupiter'}}} > > Then trying to access the value ?jupiter? would mean one of these > alternatives: > > # Risks getting a ValueError > my_planet = target[?galaxy?][?system?][?planet?] > print(my_planet) > > # Awkward to read > my_planet = target.get(?galaxy?, {}).get(?system?, {}).get(?planet?) > if my_planet: > print(my_planet) > > With pattern matching this could become more simply: > > match target: > case {?galaxy?: {?system?: {?planet?: my_planet}}}: print(my_planet) > > This example was stolen from the tutorial for the glom library > https://sedimental.org/glom_restructured_data.html which works on nested > data. Structural pattern matching is a more universal concept and could > eliminate the need for these kinds of helper functions. > > From reading through this thread (as well as other background reading like > https://groups.google.com/forum/#!msg/python-ideas/aninkpPpEAw/wCQ1IH5mAQAJ > and the Javascript proposal) a couple things seem clear to me for pattern > matching in the case of Python: > - it should be statement based > - it should have great support for built-in data types like lists, dicts, > namedtuples, and dataclasses > - it should form a coherent story with other similar Python concepts like > unpacking on assignment > > There are a ton of details to be worked out obviously and we should go > slow as Guido suggested. However, I believe that it would be worth doing > the work. To that end: if there?s anyone else who?d like to collaborate and > come up with a first draft of a more well-defined proposal I would love to > commit my time to this, my email is below. > > Also, I sent this email from PyCon in Cleveland if anyone would like to > brainstorm in person :). > > Steven Heidel > Rigetti Quantum Computing > ste... at rigetti.com > > On Friday, May 4, 2018 at 4:37:43 PM UTC, Guido van Rossum wrote: >> >> Can I recommend going slow here? This is a very interesting topic where >> many languages have gone before. I liked Daniel F Moisset's analysis about >> the choices of a language designer and his conclusion that match should be >> a statement. >> >> I just noticed the very similar proposal for JavaScript linked to by the >> OP: https://github.com/tc39/proposal-pattern-matching -- this is more >> relevant than what's done in e.g. F# or Swift because Python and JS are >> much closer. (Possibly Elixir is also relevant, it seems the JS proposal >> borrows from that.) >> >> A larger topic may be how to reach decisions. If I've learned one thing >> from PEP 572 it's that we need to adjust how we discuss and evaluate >> proposals. I'll think about this and start a discussion at the Language >> Summit about this. >> >> -- >> --Guido van Rossum (python.org/~guido) >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Sat May 12 00:38:23 2018 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 11 May 2018 23:38:23 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: [Tim[ >> ... >> ":=" target names in a genexp/listcmp are treated exactly the same as >> any other non-for-target name: they resolve to the same scope as they >> resolve to in the block that contains them. The only twist is that if >> such a name `x` isn't otherwise known in the block, then `x` is >> established as being local to the block (which incidentally also >> covers the case when the genexp/listcomp is at module level, where >> "local to the block" and "global to the block" mean the same thing). >> Class scope may be an exception (I cheerfully never learned anything >> about how class scope works, because I don't write insane code ;-) ). [Nick] > That's all well and good, but it is *completely insufficient for the > language specification*. I haven't been trying to write reference docs here, but so far as supplying a rigorous specification goes, I maintain the above gets "pretty close". It needs more words, and certainly isn't in the _style_ of Python's current reference docs, but that's all repairable. Don't dismiss it just because it's brief. Comprehensions already exist in the language, and so do nested scopes, so it's not necessary for this PEP to repeat any of the stuff that goes into those. Mostly it needs to specify the scopes of assignment expression target names - and the _intent_ here is really quite simple. Here with more words, restricted to the case of assignment expressions in comprehensions (the only case with any subtleties): Consider a name `y` appearing in the top level of a comprehension as an assignment expression target, where the comprehension is immediately contained in scope C, and the names belonging to scopes containing C have already been determined: ... (y := expression) ... We can ignore that `y` also appears as a `for` target at the comprehension's top level, because it was already decided that's a compile-time error. Consider what the scope of `y` would be if `(y := expression)` were textually replaced by `(y)`. Then what would the scope of `y` be? The answer relies solely on what the docs _already_ specify. There are three possible answers: 1. The docs say `y` belongs to scope S (which may be C itself, or a scope containing C). Then y's scope in the original comprehension is S. 2. The docs say name `y` is unknown. Then y's scope in the original comprehension is C. 3. The docs are unclear about whether #1 or #2 applies. Then the language is _already_ ill-defined. It doesn't matter to this whether the assignment expression is, or is not, in the expression that defines the iterable for the outermost `for`. What about that is hand-wavy? Defining semantics clearly and unambiguously doesn't require specifying a concrete implementation (the latter is one possible way to achieve the goal - but _here_ it's a convoluted PITA because Python has no way to explicitly declare intended scopes). Since all questions about scope are reduced by the above to questions about Python's _current_ scope rules, it's as clear and unambiguous as Python's current scope rules. Now those may not be the _intended_ rules in all cases. That deserves deep scrutiny. But claiming it's too vague to scrutinize doesn't fly with me. If there's a scope question you suspect can't be answered by the above, or that the above gives an unintended answer to, by all means bring that up! If your question isn't about scope, then I'd probably view it as being irrelevant to the current PEP (e.g., what `locals()` returns depends on how the relevant code object attributes are set, which are in turn determined by which scopes names belong to relative to the code block's local scope, and it's certainly not _this_ PEP's job to redefine what `locals()` does with that info). Something to note: for-target names appearing in the outermost `for` _may_ have different scopes in different parts of the comprehension. y = 12 [y for y in range(y)] There the first two `y`'s have scope local to the comprehension, but the last `y` is local to the containing block. But an assignment expression target name always has the same scope within a comprehension. In that specific sense, their scope rules are "more elegant" than for-target names. This isn't a new rule, but a logical consequence of the scope-determining algorithm given above. It's a _conceptual_ consequence of that assignment statement targets are "intended to act like" the bindings are performed _in_ scope C rather than in the comprehension's scope. And that's no conceptually weirder than that it's _already_ the case that the expression defining the iterable of the outermost `for` _is_ evaluated in scope C (which I'm not a fan of, but which is rhetorically convenient to mention here ;-) ). As I've said more than once already, I don't know whether this should apply to comprehensions at class scope too - I've never used a comprehension in class scope, and doubt I ever will. Without use cases I'm familiar with, I have no idea what might be most useful there. Best uninformed guess is that the above makes decent sense at class scope too, especially given that I've picked up on that people are already baffled by some comprehension behavior at class scope. I suspect that you already know, but find it rhetorically convenient to pretend this is all so incredibly unclear you can't possibly guess ;-) > For the language spec, we have to be able to tell implementation authors > exactly how all of the "bizarre edge case" Which are? > that you're attempting to hand wave away Not attempting to wave them way - don't know what you're referring to. The proposed scope rules are defined entirely by straightforward reference to existing scope rules - and stripped of all the excess verbiage amount to no more than "same scope in the comprehension as in the containing scope". > should behave by updating > https://docs.python.org/dev/reference/expressions.html#displays-for-lists-sets-and-dictionaries Thanks for the link! I hadn't seen that before. If the PEP gets that far, I'd think harder about how it really "ought to be" documented. I think, e.g., that scope issues should be more rigorously handled in section 4.2 (which is about binding and name resolution). > appropriately. It isn't 1995 any more - while CPython is still the reference > implementation for Python, we're far from being the only implementation, > which means we have to be a lot more disciplined about how much we leave up > to the implementation to define. What in the "more words" above was left to the implementation's discretion? I can already guess you don't _like_ the way it's worded, but that's not what I'm asking about. > The expected semantics for locals() are already sufficiently unclear that > they're a source of software bugs (even in CPython) when attempting to run > things under a debugger or line profiler (or anything else that sets a trace > function). See https://www.python.org/dev/peps/pep-0558/ for details. As above, what does that have to do with PEP 572? The docs you referenced as a model don't even mention `locals()` - but PEP 572 must? Well, fine: from the explanation above, it's trivially deduced that all names appearing as assignment expression targets in comprehensions will appear as free variables in their code blocks, except for when they resolve to the global scope. In the former case, looks like `locals()` will return them, despite that they're _not_ local to the block. But that's the same thing `locals()` does for free variables created via any means whatsoever - it appears to add all the names in code_object.co_freevars to the returned dict. I have no idea why it acts that way, and wouldn't have done it that way myself. But if that's "a bug", it would be repaired for the PEP 572 cases at the same time and in the same way as for all other freevars cases. Again, the only thing at issue here is specifying intended scopes. There's nothing inherently unique about that.. > "Comprehension scopes are already confusing, so it's OK to dial their > weirdness all the way up to 11" is an *incredibly* strange argument to be > attempting That's an extreme characterization of what, in reality, is merely specifying scopes. That total = 0 sums = [total := total + value for value in data] blows up without the change is at least as confusing - and is more confusing to me. > to make when the original better defined sublocal scoping > proposal was knocked back as being overly confusing (even after it had been > deliberately simplified by prohibiting nonlocal access to sublocals). I'm done arguing about this part ;-) > Right now, the learning process for picking up the details of comprehension > scopes goes something like this: Who needs to do this? I'm not denying that many people do, but is that a significant percentage of those who merely want to _use_ comprehensions? We already did lots of heroic stuff apparently attempting to cater to those who _don't_ want to learn about their implementation, like evaluating the outer iterable "at once" outside the comprehension scope, and - indeed - bothering to create a new scope for them at all. Look at the "total := total + value" example again and really try to pretend you don't know anything about the implementation. "It works!" is a happy experience :-) For the rest of this message, it's an entertaining and educational development. I'm not clear on what it has to do with the PEP, though. > * make the > technically-incorrect-but-mostly-reliable-in-the-absence-of-name-shadowing > assumption that "[x for x in data]" is semantically equivalent to a for loop > (especially common for experienced Py2 devs where this really was the > case!): > > _result = [] > for x in data: > _result.append(x) > > * discover that "[x for x in data]" is actually semantically equivalent to > "list(x for x in data)" (albeit without the name lookup and optimised to > avoid actually creating the generator-iterator) > * make the still-technically-incorrect-but-even-more-reliable assumption > that the generator expression "(x for x in data)" is equivalent to > > def _genexp(): > for x in data: > yield x > > _result = _genexp() > > * *maybe* discover that even the above expansion isn't quite accurate, and > that the underlying semantic equivalent is actually this (one way to > discover this by accident is to have a name error in the outermost iterable > expression): > > def _genexp(_outermost_iter): > for x in _outermost_iter: > yield x > > _result = _genexp(_outermost_iter) > > * and then realise that the optimised list comprehension form is essentially > this: > > def _listcomp(_outermost_iter): > result = [] > for x in _outermost_iter: > result.append(x) > return result > > _result = _listcomp(data) > > Now that "yield" in comprehensions has been prohibited, you've learned all > the edge cases at that point - all of the runtime behaviour of things like > name references, locals(), lambda expressions that close over the iteration > variable, etc can be explained directly in terms of the equivalent functions > and generators, so while comprehension iteration variable hiding may *seem* > magical, it's really mostly explained by the deliberate semantic equivalence > between the comprehension form and the constructor+genexp form. (That's > exactly how PEP 3100 describes the change: "Have list comprehensions be > syntactic sugar for passing an equivalent generator expression to list(); as > a consequence the loop variable will no longer be exposed") > > As such, any proposal to have name bindings behave differently in > comprehension and generator expression scope from the way they would behave > in the equivalent nested function definitions *must be specified to an > equivalent level of detail as the status quo*. I don't see any of those Python workalike examples in the docs. So which "status quo" are you referring to? You already know it's possible, and indeed straightforward, to write functions that model the proposed scope rules in any given case, so what;s your real point? They're "just like" the stuff above, possibly adding a sprinkling of "nonlocal" and/or "global" declarations. They don't require changing anything fundamental about the workalike examples you've already given - just adding cruft to specify scopes. I don't want to bother doing it here, because it's just tedious, and you _already know_ it. Most tediously, because there's no explicit way to declare a non-global scope in Python, in the """ 2. The docs say name `y` is unknown. Then y's scope in the original comprehension is C. """ case it's necessary to do something like: if 0: y = None in the scope containing the synthetic function so that the contained "nonlocal y" declaration knows which scope `y` is intended to live in. (The "if 0:" block is optimized out of existence, but after the compiler has noticed the local assignment to `y` and so records that `y` is containing-scope-local.) Crap like that isn't really illuminating. > All of the attempts at such a definition that have been made so far have > been riddled with action and a distance and context-dependent compilation > requirements: > > * whether to implicitly declare the binding target as nonlocal or global > depends on whether or not you're at module scope or inside a function That's artificial silliness, though. Already suggested that Python repair one of its historical scope distinctions by teaching `nonlocal` that nonlocal x in a top-level function is a synonym for global x in a top-level function. In every relevant conceptual sense, the module scope _is_ the top-level lexical scope. It seems pointlessly pedantic to me to insist that `nonlocal` _only_ refer to a non-global enclosing lexical scope. Who cares? The user-level semantically important part is "containing scope", not "is implemented by a cell object". In the meantime, BFD. So long as the language keyword insists on making that distinction, ya, it's a distinction that needs to be made by users too (and by the compiler regardless). This isn't some inherently new burden for the compiler either. When it sees a not-local name in a function, it already has to figure out whether to reference a cell or pump out a LOAD_GLOBAL opcode. > * the desired semantics at class scope have been left largely unclear Covered before. Someone who knows something about _desired_ class scope behavior needs to look at that. That's not me. > * the desired semantics in the case of nested comprehensions and generator > expressions has been left entirely unclear See the "more words" version above. It implies that scopes need to be resolved "outside in" for nesting of any kind. Which they need to be anyway, e.g., to make the "is this not-local name a cell or a global?" distinction in any kind of function code. > Now, there *are* ways to resolve these problems in a coherent way, and that > would be to define "parent local scoping" as a new scope type, and introduce > a corresponding "parentlocal NAME" compiler declaration to explicitly > request those semantics for bound names (allowing the expansions of > comprehensions and generator expressions as explicitly nested functions to > be adjusted accordingly). Sorry, I don't know what that means. I don't even know what "compiler declaration" alone means. Regardless, there's nothing here that can't be explained easily enough by utterly vanilla lexically nested scopes. All the apparent difficulties stem from the inability to explicitly declare a name's intended scope, and that the "nonlocal" keyword in a top-level function currently refuses to acknowledge that the global scope _is_ the containing not-local scope. If you mean adding a new statement to Python parentlocal NAME ... sure, that could work. But it obscures that the problem just isn't hard enough to require such excessive novelty in Python's scope gimmicks. The correct place to declare NAME's scope is _in_ NAME's intended scope, the same as in every other language with lexical scoping. There's also that the plain English meaning of "parent local' only applies to rule #2 at the top, and to the proper subset of cases in rule #1 where it turns out that S is C. In the other rule #1 cases, "parentlocal" would be a misleading name for the less specific "nonlocal" or the more specific "global". Writing workalike functions by hand isn't difficult regardless, just tedious (even without the current proposal!), and I don't view it as a significant use case regardless. I expect the minority who do it have real fun with it for a day or two, and then quite possibly never again. Which is a fair summary of my own life ;-) > But the PEP will need to state explicitly that that's what it is doing, and > fully specify how those new semantics are expected to work in *all* of the > existing scope types, not just the two where the desired behaviour is > relatively easy to define in terms of nonlocal and global. So you finally admit they _are_ relatively easy to define ;-) What, specifically, _are_ "*all" of the existing scope types"? There are only module, class, and function scopes in my view of the world. (and "comprehension scope" is just a name given at obvious times to function scope in my view of the world). If you also want piles of words about, e.g., how PEP 572 acts in all cases in smaller blocks, like code typed at a shell, or strings passed to eval() or exec(), you'll first have to explain why this was never necessary for any previous feature. PS: I hope you appreciate that I didn't whine about microscopic differences in the workalike examples' generated byte code ;-) From jelle.zijlstra at gmail.com Sat May 12 01:42:03 2018 From: jelle.zijlstra at gmail.com (Jelle Zijlstra) Date: Fri, 11 May 2018 22:42:03 -0700 Subject: [Python-ideas] Fwd: Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> <772c2592-681e-4d7c-9527-27ea52c78ca5@googlegroups.com> <2b64fcb0-b8af-41b6-af84-58f2f5189fbd@googlegroups.com> Message-ID: 2018-05-11 22:01 GMT-04:00 Robert Roskam : > Hey Steven, > > I'm also at PyCon. Shall we take this off list and attempt to meet up and > discuss? > > I'm also at PyCon and interested in meeting about this. I just wrote up a basic and incomplete implementation for pattern-matching yesterday between and after: talks: https://github.com/JelleZijlstra/cpython/blob/ matchcase/Lib/test/test_matching.py. It's nowhere near complete, but an implementation like this can help inform what the syntax should look like. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat May 12 03:37:28 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 12 May 2018 17:37:28 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <9ba24601-d4a6-c208-3be3-2168cf7b617d@kynesim.co.uk> Message-ID: <20180512073728.GB12683@ando.pearwood.info> On Sat, May 12, 2018 at 02:37:20AM +0100, Rob Cliffe via Python-ideas wrote: > Yeah, well, I'm totally lost.? Even after trying out this code, and > refactoring it once if not twice, I didn't understand it.? I don't know > what point you're trying to prove, but you seem to have comprehensively > proved it. Do you mean Serhiy's example of currently supported syntax? smooth_signal = [average for average in [0] for x in signal for average in [(1-decay)*average + decay*x]] It helps if you know the algorithm for exponential smoothing: for each value x (aside from the first), the average is equal to a mix of the current value x and the previous average A, split by some proportion P: A = (1-P)*A + P*x If P is 0.5, that is equivalent to taking the ordinary average between the current value and the previous average: A = (A+x)/2 # when P == 0.5 In the comprehension, P is called "decay" and A is called "average": average = (1-decay)*average + decay*x Writing the comprehension as a single line is hard to read. Let's give it some structure: smooth_signal = [average # append average to the results for average in [0] for x in signal for average in [(1-decay)*average + decay*x] ] Horrible as it is, it is perfectly legal Python right now. It uses for name in SINGLE_ITEM_LIST to perform an assignment. So that's equivalent to: average = 0 for x in signal average = (1-decay)*average + decay*x append average to the results Pull the initial value of average out of the comprehension, and use the PEP 572 syntax: average = 0 smooth_signal = [(average := (1-decay)*average + decay*x) for x in signal] which is a huge improvement in my opinion. It would be more obvious if the expression being calculated came first: smooth_signal = [(1-decay)*average + decay*x as average for x in signal] but there are good reasons why the "as" syntax won't work. So it looks like we're stuck with needing to look ahead past the := to see the actual value being appended to the list. A minor inconvenience, equivalent to that in ternary if, where we have to look ahead to see the condition: [target := COMPREHENSION_VALUE for x in sequence] true_value if CONDITION else false_value So I expect that it will take me a little while to learn to look ahead and read binding-expressions fluently. (Like comprehensions themselves, really. It took me a few months to stop needing to pull them apart to understand them.) He's Nick's version, as best as I am able to tell: average = 0 smooth_signal = [(average given average = (1-decay)*average + decay*x) for x in signal] So we have the same look-ahead needed to see the expression we care about, but instead of merely having two characters := needed to do the binding, we need "given average =". -- Steve From marcidy at gmail.com Sat May 12 04:10:31 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Sat, 12 May 2018 01:10:31 -0700 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <20180511161724.GX9562@ando.pearwood.info> <5AF5DC29.9030000@brenbarn.net> <5AF5DD61.4050809@brenbarn.net> <20180511235006.GC9562@ando.pearwood.info> Message-ID: On Fri, May 11, 2018, 17:04 Tim Peters wrote: > [Matt Arcidy] > >> Note Tim came up with a real metric: > >> 2 * count(":=")/len(statement). > >> It's objective. it's just unclear if a higher score is better or worse. > >> However, one could say "a Tim of .3 is considered too high" as a > guideline. > > [Steven D'Aprano] > > I think Tim was making a joke about demanding objective measurements of > > subjective things. > > > > Certainly he hasn't done any research or study to justify that metric. > > He just plucked the formula out of thin air. > > It was the outcome of an intense 17-year research project. > > > > Or at least no peer reviewed research. > > Au contraire! My peers are here, and that message was reviewed by at > least 3 people on this list. > > That said, I am a fan of objectively measuring subjective things, just > not of taking the measurements seriously ;-) > apparently my joke was objectively not funny :-) I thought calling it a "Tim" was sufficient. Im not serious about actually ranking for the purposes of a PEP. I brought it up when I felt the subjectivity was making the debate worse. Reiterating my point, n long sub-threads about fonts, screens, etc are ridiculous when those exist outside the pyfile. I don't know why personal preference for a font would stop a useful tool. Hopefully those arguments are ignored. Likewise for googlability, teachability, cross-language similarity and familiarity. If the tool is useful, that's all that will matter with respect to these points, they solve themselves. I happen to be working on a ranking tool for code (not quality, just an ordering to find entry points for new devs), so i tossed the idea in. it seemed appropriate to remind people that the fact that not everyone uses green to highlight "+" doesn't make "+" somehow more or less _useful_ (people -1'd just for legibility alone because of their personal feelings) I'm not sure where people stand on usefulness, but it's clear this tool is a pattern. No counter example of "but I can already do this" is related to other counter examples in the way that named expressions solves all of them, and does it succinctly regardless of chosen syntax. Some required imports! Obviously making these decisions with the future unknown is nearly impossible and requires careful consideration of all points, but I don't think Hypothetical Bill's perscription glasses should determine syntax decisions. Best of luck with the hard parts, clearly I hope the PEP makes it. > > > If people do want to take it seriously, check out prior Python art first: > Awesome, thanks! > > http://radon.readthedocs.io/en/latest/intro.html > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From goosey15 at gmail.com Sat May 12 04:36:11 2018 From: goosey15 at gmail.com (Angus Hollands) Date: Sat, 12 May 2018 09:36:11 +0100 Subject: [Python-ideas] Python-ideas Digest, Vol 138, Issue 95 In-Reply-To: References: Message-ID: Concerning my previous email, Yes, my mistake. I'd forgotten (ironically) that the whole point is that it's an expression itself. So > while (value:=get_next_pool_item()).in_use: > print(value.refcount()) would be the appropriate analogue. Consequently, my example is invalid. A better example would have been where one needs to access more than one attribute of the expression while (node.x, node.y) > (5.0, 5.0) given node = get_neighbours(node): pass This wouldn't be so easy to do without writing a lambda, or conditional break. Again, the example is completely arbitrary, simply there to indicate that moving the assignment out of the expression itself gives more scope for use. Hence, by superset, I meant that all cases using ":=" can be expressed with given, but not visa-versa. --- > Clearly the objectively best choice is "<-". I'm convinced ;) > while (cmd := get_command()).token != > CMD_QUIT: I think this example is concise enough that it doesn't consider the readability when get_command is nontrivial. Examples: https://gist.github.com/agoose77/7ac98e74c3b7f1892789575c8cd50536 If i'm refactoring this kind of code, or new to the module and attempting to unpick what is happening, I would argue that the latter example presents me with what I most care about up-front (the condition) and maybe care about (the implementation) second. Furthermore, the assignment is very obvious, which I think is quite important. I don't think that the ':=' approach is entirely cryptic, of course, I can read it. But I find the clarity in putting the expression before the assignment to be worthwhile. --- > A bit like how "strawman argument" is mostly used to mean "dammit, you > just spotted an unwelcome consequence and/or flaw in my position which I > have no counter too". Not sure if this is directed at me (will assume so). I'm sure people misuse the term, but here I think it's appropriate. It doesn't quite fit the exact definition, I grant you, but arguing about character cost seems premature in the question of whether one should care about character cost (within reason). --- >:= would prevent you from using assignment expressions inside f-strings, which could be argued is a good thing. Oh wow, that's rather interesting! I'll put it up there with "py2 f-strings as codecs". :) Cheers, Angus -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat May 12 04:56:27 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 12 May 2018 18:56:27 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: <5AF48E04.6060802@brenbarn.net> References: <5AF48E04.6060802@brenbarn.net> Message-ID: <20180512085626.GC12683@ando.pearwood.info> There's a lot of things in Brendan's email which I disagree with but will skip to avoid dragging this out even further. But there's one point in particular which I think is important to comment on. On Thu, May 10, 2018 at 11:23:00AM -0700, Brendan Barnwell wrote: > One of the great things about Python's design is that > it doesn't just make it easy for us to write good code, but in many ways > makes it difficult for us to write bad code. I don't think this concept survives even a cursory look at the language. Make it difficult to write bad code? Let's see now: Anyone who have been caught by the "mutual default" gotcha will surely disagree: def func(arg, x=[]): ... And the closures-are-shared gotcha: py> addone, addtwo, addthree = [lambda x: x + i for i in (1, 2, 3)] py> addone(100) 103 py> addtwo(100) 103 We have no enforced encapsulation, no "private" or "protected" state for classes. Every single pure-Python class is 100% open for modification, both by subclasses and by direct monkey-patching of the class. The term "monkey-patch" was, if Wikipedia is to be believed, invented by the Python community, long before Ruby took to it as a life-style. We have no compile-time type checks to tell us off if we use the same variable as a string, a list, an int, a float and a dict all in the one function. The compiler won't warn us if we assign to something which ought to be constant. We can reach into other modules' namespaces and mess with their variables, even replacing builtins. Far from making it *hard* to do bad things, Python makes it *easy*. And that's how we love it! Consenting adults applies. We trust that code is not going to abuse these features, we trust that people aren't generally going to write list comps nested six levels deep, or dig deep into our module and monkey-patch our functions: import some_module some_module.function.__defaults__ = (123,) # Yes, this works. As a community, we use these powers wisely. We don't make a habit of shooting ourselves in the foot. We don't write impenetrable forests of nested comprehensions inside lambdas or stack ternary-if expressions six deep, or write meta-metaclasses. Binding expressions can be abused. But they have good uses too. I trust the Python community will use this for the good uses, and not change the character of the language. Just as the character of the language was not ruined by comprehensions, ternary-if or decorators. -- Steve From tim.peters at gmail.com Sat May 12 04:56:53 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 12 May 2018 03:56:53 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: [Tim] > ... > But an assignment expression target name always has the > same scope within a comprehension. [which is a consequence > of the rules - not a rule of its own] Something related to ponder: what's the meaning of the following _without_ the proposed scope change? So the Golden Binding Rule (GBR) applies then: GBR: binding a name by any means always makes the name local to the block the binding appears in, unless the name is declared "global" or "nonlocal" in the block. def f(): ys = [y for _ in range(y := 5)] The second instance of `y` is local - but local to what? Since the range is evaluated _in_ f's scope, presumably that instance of `y` is local to `f`. What about the first instance of `y`? Is that _not_ local to the comprehension despite that the GBR insists it must be local to the comprehension? Or does it raise UnboundLocalError for consistency with the GBR, and "well, so just don't use any name in a comprehension that appears as an assignment expression target in the expression defining the iterable for the outermost `for` ". Or is it that despite that `range(y := 5)` is executed in f's scope, the _binding_ is actually performed in the comprehension's scope to a comprehension-local `y`, to both preserve GBR and avoid the UnboundLocalError? . But then what if `print(y)` is added after? If `range(y := 5)` really was executed in f's scope, surely that must print 5. Then what about [y for y in range(y := 5)] ? Now that there's another binding inside the comprehension establishing that `y` is local to the comprehension "for real", does that work fine and the rule changes to well, so just don't use any name in a comprehension that appears as an assignment expression target in the expression E defining the iterable for the outermost `for` - unless the name is _also_ used in a binding context in the comprehension outside of E too ? Or is that a compile-time error despite that the first 2 y's are now obviously comprehension-local and the final y obviously f-local? Or are assignment expressions disallowed in the expression defining the iterable for the outermost `for`, and both examples are compile-time errors? Talk about incoherent ;-) Under the proposed change, all instances of `y` are local to `f` in the first example, and the second example is a compile-time error for a _coherent_ reason (the ":=" binding implies "not local" for `y` - which has nothing to do with that it's in the outermost `for` -, but the "for y in" binding implies "local" for `y`). From kirillbalunov at gmail.com Sat May 12 06:07:50 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Sat, 12 May 2018 13:07:50 +0300 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: 2018-05-12 11:36 GMT+03:00 Angus Hollands : > Concerning my previous email, > > Yes, my mistake. I'd forgotten (ironically) that the whole point is that > it's an expression itself. > > So > > while (value:=get_next_pool_item()).in_use: > > print(value.refcount()) > would be the appropriate analogue. > > Consequently, my example is invalid. A better example would have been > where one needs to access more than one attribute of the expression > > while (node.x, node.y) > (5.0, 5.0) given node = get_neighbours(node): > pass > In addition, this form gives you all the advantages of tuple unpacking: while (x, y) > (5.0, 5.0) given x, y = get_neighbours(node): pass There was some criticism about the length of the `given`, maybe it is possible to _repurpose_ `with` keyword: while (x, y) > (5.0, 5.0) with x, y = get_neighbours(node): pass In this context, the with statement and with expression are clearly distinguishable both for the parser and for the person. But maybe many will find this as a bad style ... since the semantics are too different. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From steven at rigetti.com Sat May 12 07:51:44 2018 From: steven at rigetti.com (Steven Heidel) Date: Sat, 12 May 2018 07:51:44 -0400 Subject: [Python-ideas] Fwd: Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> <772c2592-681e-4d7c-9527-27ea52c78ca5@googlegroups.com> <2b64fcb0-b8af-41b6-af84-58f2f5189fbd@googlegroups.com> Message-ID: Great! Also emailed you with logistics. On Sat, May 12, 2018, 01:43 Jelle Zijlstra wrote: > 2018-05-11 22:01 GMT-04:00 Robert Roskam : > >> Hey Steven, >> >> I'm also at PyCon. Shall we take this off list and attempt to meet up and >> discuss? >> >> I'm also at PyCon and interested in meeting about this. I just wrote up a > basic and incomplete implementation for pattern-matching yesterday between > and after: talks: > https://github.com/JelleZijlstra/cpython/blob/matchcase/Lib/test/test_matching.py. > It's nowhere near complete, but an implementation like this can help inform > what the syntax should look like. > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/nqW2_-kKrNg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/nqW2_-kKrNg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adelfino at gmail.com Sat May 12 10:49:45 2018 From: adelfino at gmail.com (=?UTF-8?Q?Andr=C3=A9s_Delfino?=) Date: Sat, 12 May 2018 11:49:45 -0300 Subject: [Python-ideas] Fwd: Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> <772c2592-681e-4d7c-9527-27ea52c78ca5@googlegroups.com> <2b64fcb0-b8af-41b6-af84-58f2f5189fbd@googlegroups.com> Message-ID: I find it weird for case statements to be "inside" match statements. There's isn't a statement "group" that works this way right now, AFAIK. This would also be weird: match X: case Y: ... I thought a form based on try would make more coherent: match: suite case x: suite1 else: suite2 suite would be executed, and the last expression would be checked against each case. If no matching case is found, suite2 would be executed. Does it make sense? On Sat, May 12, 2018 at 8:51 AM, Steven Heidel wrote: > Great! Also emailed you with logistics. > > On Sat, May 12, 2018, 01:43 Jelle Zijlstra > wrote: > >> 2018-05-11 22:01 GMT-04:00 Robert Roskam : >> >>> Hey Steven, >>> >>> I'm also at PyCon. Shall we take this off list and attempt to meet up >>> and discuss? >>> >>> I'm also at PyCon and interested in meeting about this. I just wrote up >> a basic and incomplete implementation for pattern-matching yesterday >> between and after: talks: https://github.com/JelleZijlstra/cpython/blob/ >> matchcase/Lib/test/test_matching.py. It's nowhere near complete, but an >> implementation like this can help inform what the syntax should look like. >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/ >> topic/python-ideas/nqW2_-kKrNg/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/ >> topic/python-ideas/nqW2_-kKrNg/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Sat May 12 11:16:07 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 12 May 2018 08:16:07 -0700 (PDT) Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: I love given compared with := mainly because Simpler is better than complex: * given breaks a complex statement into two simpler ones, which is putting people off in the simple examples shown here (some people are annoyed by the extra characters). However, when given is used in a list comprehension to prevent having to re-express it as a for loop, then two simple statements are easier to understand than one complex statement. This is a common difference between code written at programming contests versus code written by those same software engineers years later at big companies. Code that you write for yourself can be compact because you already understand it, but code you write professionally is read many many more times than it is written. Accessibility is much more important than concision. * Python has a reputation for being working pseudocode, and given reads like pseudocode. := needs to be deciphered by comparison especially in the complicated cases where multiple := operators are used on one line. * there are no difficult mental questions about evaluation order, e.g., in a bracketed expression having multiple assignments. Similarly, instead of (a.b(a) given a = c.d()) do I write (a.b(a := c.d())) or ((a := c.d()).b(a)) ? * it avoids the question of what happens when := is used in a switch: (a if (b := c) else d) Sometimes you want the assignment to happen unconditionally (a if (b:=c) else d) + b; sometimes you don't. How do you force one case or the other? given makes it obvious by separating assignment from the usage of its assignment target. Style: * it avoids the big style question of when to use and when not to use :=. (Even if you ask people not to, people are going to write the expression-statement a := b as a synonym for the statement a = b.) * it looks a lot like the existing Python "for" and "if" clauses, which also do in-expression assignments. This makes formatting the code obvious too: (expression given a = b) compared with expresion ( a := b ) rest of expression which quickly gets ugly. Best, Neil On Thursday, May 10, 2018 at 9:46:01 AM UTC-4, Guido van Rossum wrote: > > I'm sorry, but unless there's a sudden landslide of support for 'given' in > favor of ':=', I'm really not going to consider it. > > I'd pronounce "if (x := y) > 0" as either "if y (assigned to x) is greater > than zero" or "if x (assigned from y) is greater than zero". > > On Thu, May 10, 2018 at 6:39 AM, Nick Coghlan > wrote: > >> On 8 May 2018 at 04:19, Brett Cannon > >> wrote: >> >>> My brain wants to drop the variable name in front of 'given': >>> >>> if given m = pattern.search(data): >>> >>> while given m = pattern.search(remaining_data): >>> >>> Maybe it's because the examples use such a short variable name? >>> >> >> Does that change if the condition isn't just "bool(name)"? For example: >> >> if y > 0 given y = f(x): >> ... >> >> That's the situation where I strongly prefer the postfix operator >> spelling, since it's pretty clear how I should pronounce it (i.e. "if y is >> greater than zero, given y is set to f-of-x, then ..."). By contrast, while >> a variety of plausible suggestions have been made, I still don't really >> know how to pronounce "if (y := f(x)) > 0:)" in a way that's going to be >> clear to an English-speaking listener (aside from pronouncing it the same >> way as I'd pronounce the version using "given", but that then raises the >> question of "Why isn't it written the way it is pronounced?"). >> >> I do agree with Tim that the name repetition would strongly encourage the >> use of short names rather than long ones (since you're always typing them >> at least twice), such that we'd probably see code like: >> >> while not probable_prime(n) given (n = >> highbit | randrange(1, highbit, 2)): >> pass >> >> Rather than the more explicit: >> >> while not probable_prime(candidate) given (candidate = >> highbit | randrange(1, highbit, 2)): >> pass >> >> However, I'd still consider both of those easier to follow than: >> >> while not probable_prime(candidate := highbit | randrange(1, highbit, >> 2)): >> pass >> >> since it's really unclear to me that "candidate" in the latter form is a >> positional argument being bound to a name in the local environment, and >> *not* a keyword argument being passed to "probable_prime". >> >> I've also been pondering what the given variant might look like as a >> generally available postfix operator, rather than being restricted to >> if/elif/while clauses, and I think that would have interesting implications >> for the flexibility of its usage in comprehensions, since there would now >> be *three* places where "given" could appear (as is already the case for >> the inline binding operator spelling): >> >> - in the result expression >> - in the iterable expression >> - in the filter expression >> >> That is: >> >> [(x, y, x - y) given y = f(x) for x in data] >> [(x, data) for x in data given data = get_data()] >> [(x, y, x/y) for x in data if y given y = f(x)] >> >> Rather than: >> >> [(x, y := f(x), x - y) for x in data] >> [(x, data) for x in data := get_data()] >> [(x, y, x/y) for x in data if y := f(x)] >> >> Opening it up that way would allow for some odd usages that might need to >> be discouraged in PEP 8 (like explicitly preferring "probable_prime(n) >> given n = highbit | randrange(1, highbit, 2)" to "probable_prime(n given n >> = highbit | randrange(1, highbit, 2))"), but it would probably still be >> simpler overall than attempting to restrict the construct purely to >> if/elif/while. >> >> Even as a generally available postfix keyword, "given" should still be >> amenable to the treatment where it could be allowed as a variable name in a >> non-operator context (since we don't allow two adjacent expressions to >> imply a function call, it's only prefix keywords that have to be disallowed >> as names to avoid ambiguity in the parser). >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncog... at gmail.com | Brisbane, >> Australia >> >> _______________________________________________ >> Python-ideas mailing list >> Python... at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat May 12 12:26:52 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 13 May 2018 02:26:52 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <20180512162652.GE12683@ando.pearwood.info> On Sun, May 06, 2018 at 02:00:35AM +1000, Nick Coghlan wrote: > 3. You can stick in an explicit "if True" if you don't need the given > variable in the filter condition > > [(fx**2, fx**3) for x in xs if True given fx = f(x)] o_O I'm replying to an email which is a week old. I haven't seen anyone other than Tim comment on that "if True" boilerplate. Are you still proposing that? If not, you can ignore the following. > And then once you've had an entire release where the filter condition was > mandatory for the comprehension form, allowing the "if True" in "[(fx**2, > fx**3) for x in xs given fx = f(x)]" to be implicit would be less ambiguous. I'm sorry, perhaps I'm having another slow day, but I don't see the ambiguity in the first place. And I *certainly* do not see how adding in a logically superfluorous "if True" would make it unambiguous. For comparison sake, here is the PEP 572 proposal compared to yours: [((fx := f(x))**2, fx**3) for x in xs] I'd read that as for each x in xs, let fx be f of x, return fx squared and fx cube [(fx**2, fx**3) for x in xs if True given fx = f(x)] which I'd read as for each x in xs, if True, return fx squared and fx cube, given fx is f of x and then wonder why on earth the test is there. Explain to me again why this boilerplate is necessary, please. Especially since you're already suggesting that in a future release it could be dropped without changing the meaning. [Tim Peters] > > In > > > > if match given match = pattern.search(data): > > > > the annoying visual redundancy (& typing) persists. > > Right, but that's specific to the case where the desired condition really > is just "bool(target)". That really isn't the case. if result.method() given result = compare(data, arg): versus the PEP 572 syntax: if (result := compare(data, arg)).method(): So it certainly isn't just the bool(target) case that is redundant. -- Steve From steve at pearwood.info Sat May 12 13:24:27 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 13 May 2018 03:24:27 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <20180512172427.GF12683@ando.pearwood.info> On Sat, May 12, 2018 at 08:16:07AM -0700, Neil Girdhar wrote: > I love given compared with := mainly because > > Simpler is better than complex: > * given breaks a complex statement into two simpler ones, (Foreshadowing: this argument applies to augmented assignment. See below.) I don't see how you justify that statement about "given". I think that it is "given" which is more complex. Significantly so. Let's compare the syntax: target := expr That is a single, simple expression with a single side-effect: it assigns the value to . That's it. Like all expressions, it returns a value, namely the result of "expr". Like all expressions, you can embed it in other expressions (possibly wrapping it in parens to avoid precedence issues), or not, as required. (That surrounding expression can be as simple or complex as you like.) Now here's Nick's syntax: target given target = expr Exactly like the := version above, we can say that this is a single expression with a single side-effect. Like all expressions, it returns a value, namely the result of "expr", and like all expressions, you can embed it in other expressions. So far the two are precisely the same. There is no difference in the complexity, because they are exactly the same except for the redundant and verbose "given" spelling. But actually, I lied. Nick's syntax is *much more complicated* than the := syntax. Any arbitrary expression can appear on the left of "given". It need not even involve the binding target! So to make a fair comparison, I ought to compare: target := expr which evaluates a single expression, binds it, and returns it, to: another_expr given target := expr which evaluates "expr", binds it to "target", evaluates a SECOND unrelated expression, and returns that. If you want to argue that this is more useful, then fine, say so. But to say that it is *simpler* makes no sense to me. Option 1: evaluate and bind a single expression Option 2: exactly the same as Option 1, and then evaluate a second expression How do you justify that Option 2 "given", which does everything := does PLUS MORE, is simpler than Option 1? That's not a rhetorical question. > which is putting people off in the simple examples shown here (some > people are annoyed by the extra characters). However, when given is > used in a list comprehension to prevent having to re-express it as a > for loop, then two simple statements are easier to understand than one > complex statement. I'd like to see an example of one of these list comprehensions that is simpler written with given. Here's an earlier example, an exponentially weighted running average: average = 0 smooth_signal = [(average := (1-decay)*average + decay*x) for x in signal] assert average == smooth_signal[-1] I'm not even sure if "given" will support this. Nick is arguing strongly that bound targets should be local to the comprehension, and so I think you can't even write this example at all with Nick's scoping rule. But let's assume that the scoping rule is the same as the above. In that case, I make it: average = 0 smooth_signal = [average given average = (1-decay)*average + decay*x) for x in signal] Is it longer, requiring more typing? Absolutely. Does it contain a redundantly repetitious duplication of the repeated target name? Certainly. But is it simpler? I don't think so. If you don't like the exponential running average, here's a simple running total: total = 0 running_totals = [total := total + x for x in xs] versus total = 0 running_totals = [total given total = total + x for x in xs] If you don't like this example either, please show me an example of an actual list comp that given makes simpler. > This is a > common difference between code written at programming contests versus code > written by those same software engineers years later at big companies. > Code that you write for yourself can be compact because you already > understand it, but code you write professionally is read many many more > times than it is written. Accessibility is much more important than > concision. Ah, nice rhetorical argument: "given" is for professionals, := is a hack for amateurs and programming contests. Seriously? Do you use augmented assignment? Your simple versus complex argument for "given" applies well to augmented assignment. Augmented assignment combines two conceptual operations: x = x + 1 - addition (for example) - assignment into a single operator: x += 1 By your argument, augmented assignment is more complex, and we ought to prefer splitting it into two separate operations x = x + 1 because that's simpler. I think I agree that x = x + 1 *is* simpler. We can understand it by understanding the two parts separately: x+1, followed by assignment. Whereas += requires us to understand that the syntax not only calls a dunder method __iadd__ (or __add__ if that doesn't exist), which potentially can operate in place, but it also does an assignment, all in one conceptual operation. There's a whole lot of extra complexity there. That doesn't mean I agree with your conclusion that we ought to prefer the simpler version, let alone that the complex case (augmented assignment) is fit only for programming contests and that professionals ought to choose the simpler one. -- Steve From steven at rigetti.com Sat May 12 14:03:06 2018 From: steven at rigetti.com (steven at rigetti.com) Date: Sat, 12 May 2018 11:03:06 -0700 (PDT) Subject: [Python-ideas] Fwd: Pattern Matching Syntax In-Reply-To: References: <878t90oo7j.fsf@ender.lizardnet> <864bb060-1963-49ca-468b-6817f37a292e@kellett.im> <5e74130e-7506-4fc0-a59d-1c50457a0a97@googlegroups.com> <6f504af8-080f-4184-981e-2e0c3e914796@googlegroups.com> <772c2592-681e-4d7c-9527-27ea52c78ca5@googlegroups.com> <2b64fcb0-b8af-41b6-af84-58f2f5189fbd@googlegroups.com> Message-ID: <2db15c18-0fbe-446a-a197-f3e6fc9c183e@googlegroups.com> In some sense async and await are like this group you describe, the keyword "await" doesn't have meaning outside a function annotated with "async". The issue I'd have with your proposal is that it requires the "suite" to be an expression-based multi-line statement, ie. where the last statement becomes the value that is matched against. Similar to the original proposal that started this thread: these sorts of expression-based constructs don't feel that natural for Python. On Saturday, May 12, 2018 at 2:50:49 PM UTC, Andr?s Delfino wrote: > > I find it weird for case statements to be "inside" match statements. > There's isn't a statement "group" that works this way right now, AFAIK. > > This would also be weird: > > match X: > case Y: > ... > > I thought a form based on try would make more coherent: > > match: > suite > case x: > suite1 > else: > suite2 > > suite would be executed, and the last expression would be checked against > each case. If no matching case is found, suite2 would be executed. > > Does it make sense? > > On Sat, May 12, 2018 at 8:51 AM, Steven Heidel > wrote: > >> Great! Also emailed you with logistics. >> >> On Sat, May 12, 2018, 01:43 Jelle Zijlstra > > wrote: >> >>> 2018-05-11 22:01 GMT-04:00 Robert Roskam >> >: >>> >>>> Hey Steven, >>>> >>>> I'm also at PyCon. Shall we take this off list and attempt to meet up >>>> and discuss? >>>> >>>> I'm also at PyCon and interested in meeting about this. I just wrote up >>> a basic and incomplete implementation for pattern-matching yesterday >>> between and after: talks: >>> https://github.com/JelleZijlstra/cpython/blob/matchcase/Lib/test/test_matching.py. >>> It's nowhere near complete, but an implementation like this can help inform >>> what the syntax should look like. >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "python-ideas" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/python-ideas/nqW2_-kKrNg/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> python-ideas... at googlegroups.com . >>> For more options, visit https://groups.google.com/d/optout. >>> _______________________________________________ >>> Python-ideas mailing list >>> Python... at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "python-ideas" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/python-ideas/nqW2_-kKrNg/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> python-ideas... at googlegroups.com . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> _______________________________________________ >> Python-ideas mailing list >> Python... at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Sat May 12 14:13:05 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 12 May 2018 13:13:05 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180512172427.GF12683@ando.pearwood.info> References: <20180512172427.GF12683@ando.pearwood.info> Message-ID: Just clarifying a fine point here: [Steven D'Aprano ] > ... > average = 0 > smooth_signal = [(average := (1-decay)*average + decay*x) for x in signal] > assert average == smooth_signal[-1] > > I'm not even sure if "given" will support this. Nick is arguing strongly > that bound targets should be local to the comprehension, and so I think > you can't even write this example at all with Nick's scoping rule. You can't under Nick's proposal(s), at least not directly (there are always "tricks"). But it also blows up with UnboundLocalError (for the "average" in "(1-decay)*average") under the current PEP 572 (the ":=" PEP). I've proposed to change 572's scoping rules for targets of assignment expressions appearing in comprehensions so that "it would just work" instead, but that's getting strong opposition too. My favorite so far was Nick's (Coghlan's) entertainingly hyperbolic: "Comprehension scopes are already confusing, so it's OK to dial their weirdness all the way up to 11" is an *incredibly* strange argument to be attempting :-) The scope issues are logically independent of assignment-expression spelling, but it's a pretty safe guess Nick is opposed to that example ever "just working" regardless of spelling, while PEP 572 doesn't currently support it anyway. Last I heard, Chris (Angelico - the PEP's author) didn't seem keen on changing it either. From carl.input at gmail.com Sat May 12 14:14:39 2018 From: carl.input at gmail.com (Carl Smith) Date: Sat, 12 May 2018 19:14:39 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180512172427.GF12683@ando.pearwood.info> References: <20180512172427.GF12683@ando.pearwood.info> Message-ID: Minor gripe: The ability to *articulate* Python is not the same as the ability to *type Python verbally*. Nobody articulates `def area(width, height): return width * height` as *def area, open paren, width, comma, space, height, closed paren..*. They would say something like *def area as a function of width and height, equal to width by height*. -- Carl Smith carl.input at gmail.com On 12 May 2018 at 18:24, Steven D'Aprano wrote: > On Sat, May 12, 2018 at 08:16:07AM -0700, Neil Girdhar wrote: > > > I love given compared with := mainly because > > > > Simpler is better than complex: > > * given breaks a complex statement into two simpler ones, > > (Foreshadowing: this argument applies to augmented assignment. See > below.) > > I don't see how you justify that statement about "given". I think that > it is "given" which is more complex. Significantly so. > > Let's compare the syntax: > > target := expr > > > That is a single, simple expression with a single side-effect: it > assigns the value to . That's it. > > Like all expressions, it returns a value, namely the result of "expr". > Like all expressions, you can embed it in other expressions (possibly > wrapping it in parens to avoid precedence issues), or not, as required. > > (That surrounding expression can be as simple or complex as you like.) > > > Now here's Nick's syntax: > > target given target = expr > > Exactly like the := version above, we can say that this is a single > expression with a single side-effect. Like all expressions, it returns a > value, namely the result of "expr", and like all expressions, you can > embed it in other expressions. > > So far the two are precisely the same. There is no difference in the > complexity, because they are exactly the same except for the redundant > and verbose "given" spelling. > > But actually, I lied. > > Nick's syntax is *much more complicated* than the := syntax. Any > arbitrary expression can appear on the left of "given". It need not > even involve the binding target! So to make a fair comparison, I ought > to compare: > > target := expr > > which evaluates a single expression, binds it, and returns it, to: > > another_expr given target := expr > > which evaluates "expr", binds it to "target", evaluates a SECOND > unrelated expression, and returns that. > > If you want to argue that this is more useful, then fine, say so. But to > say that it is *simpler* makes no sense to me. > > Option 1: evaluate and bind a single expression > > Option 2: exactly the same as Option 1, and then evaluate a second > expression > > How do you justify that Option 2 "given", which does everything := does > PLUS MORE, is simpler than Option 1? > > That's not a rhetorical question. > > > > > which is putting people off in the simple examples shown here (some > > people are annoyed by the extra characters). However, when given is > > used in a list comprehension to prevent having to re-express it as a > > for loop, then two simple statements are easier to understand than one > > complex statement. > > I'd like to see an example of one of these list comprehensions that is > simpler written with given. Here's an earlier example, an exponentially > weighted running average: > > > average = 0 > smooth_signal = [(average := (1-decay)*average + decay*x) for x in signal] > assert average == smooth_signal[-1] > > > I'm not even sure if "given" will support this. Nick is arguing strongly > that bound targets should be local to the comprehension, and so I think > you can't even write this example at all with Nick's scoping rule. > > But let's assume that the scoping rule is the same as the above. In that > case, I make it: > > average = 0 > smooth_signal = [average given average = (1-decay)*average + decay*x) for > x in signal] > > Is it longer, requiring more typing? Absolutely. > > Does it contain a redundantly repetitious duplication of the repeated > target name? Certainly. > > But is it simpler? I don't think so. > > > If you don't like the exponential running average, here's a simple > running total: > > > total = 0 > running_totals = [total := total + x for x in xs] > > > versus > > > total = 0 > running_totals = [total given total = total + x for x in xs] > > > If you don't like this example either, please show me an example of an > actual list comp that given makes simpler. > > > > This is a > > common difference between code written at programming contests versus > code > > written by those same software engineers years later at big companies. > > Code that you write for yourself can be compact because you already > > understand it, but code you write professionally is read many many more > > times than it is written. Accessibility is much more important than > > concision. > > Ah, nice rhetorical argument: "given" is for professionals, := is a hack > for amateurs and programming contests. Seriously? > > Do you use augmented assignment? Your simple versus complex argument for > "given" applies well to augmented assignment. > > Augmented assignment combines two conceptual operations: > > x = x + 1 > - addition (for example) > - assignment > > into a single operator: > > x += 1 > > > By your argument, augmented assignment is more complex, and we ought to > prefer splitting it into two separate operations x = x + 1 because > that's simpler. > > I think I agree that x = x + 1 *is* simpler. We can understand it by > understanding the two parts separately: x+1, followed by assignment. > > Whereas += requires us to understand that the syntax not only calls a > dunder method __iadd__ (or __add__ if that doesn't exist), which > potentially can operate in place, but it also does an assignment, all in > one conceptual operation. There's a whole lot of extra complexity there. > > That doesn't mean I agree with your conclusion that we ought to prefer > the simpler version, let alone that the complex case (augmented > assignment) is fit only for programming contests and that professionals > ought to choose the simpler one. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat May 12 14:27:35 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 13 May 2018 04:27:35 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <20180512182735.GG12683@ando.pearwood.info> Part 2. On Sat, May 12, 2018 at 08:16:07AM -0700, Neil Girdhar wrote: > I love given compared with := mainly because [...] > * Python has a reputation for being working pseudocode, and given reads > like pseudocode. := needs to be deciphered by comparison especially in the > complicated cases where multiple := operators are used on one line. Until you learn and become familiar with a new syntax, there is generally going to be a period you have to decipher it. I spent a long time mentally translating list comprehensions into mathematical set builder notation before it became second nature to me. Even now, I know people who find decorators and comprehensions indecipherable. Or at least, so they claim, and they aren't motivated to bother learning them. Oh well, that's their loss. Binding-expressions aren't like asynchronous programming, where the entire coding paradigm is different, and you literally have to think about your algorithms in another way. Whether you spell the binding operation target := expr expr as target expr -> target target given target = expr let target = expr : target expr ; (that last one is stolen from Forth, and should not be taken as a serious suggestion) is just a matter of spelling. We'll get used to whatever spelling it is. Some may be more convenient than others, or more error-prone, or may be harder to parse, but they're secondary issues. (Important, but still secondary.) Fundamentally, the operation is the same regardless of the spelling: - evaluate an expression - bind that value to a name - return the value (Except of course Nick's "given" suggestion is more complex, since the returned value is not necessarily the same as the bound value.) > * there are no difficult mental questions about evaluation order, e.g., in > a bracketed expression having multiple assignments. Aren't there just? x = 1 print( x + x given x = 50 ) Will that print 100, or 51? Brackets ought to make it clearer: (x + x given x = 50) # this ought to return 100 x + (x given x = 50) # this ought to return 51 but if I leave the brackets out, I have no idea what I'll get. > Similarly, instead of > (a.b(a) given a = c.d()) do I write (a.b(a := c.d())) or ((a := > c.d()).b(a)) ? I would expect that your first example is a NameError: a.b(a := c.d()) since Python evaluates arguments to a method *after* looking up the method. So that corresponds to: - look up "a" # NameError, unless you've already got an "a" - look up "a.b" - evaluate c.d() - assign that value to a - pass that to the a.b method we found earlier What you probably want is the second version: (a := c.d()).b(a) which of course looks like utter crap. It might look better if we use at least half-way sensible variable names and a more realistic looking example, instead of obfuscated one-letter names. (widget := widget_builder.new(*args)).method(widget) > * it avoids the question of what happens when := is used in a switch: (a > if (b := c) else d) Sometimes you want the assignment to happen > unconditionally (a if (b:=c) else d) + b; sometimes you don't. How do you > force one case or the other? I think that this argument is really weak. The obvious answer is, if you don't want the assignment to happen unconditionally, then don't do the assignment unconditionally. Where you do it depends on when you want the assignment to take place. There's no mystery here. # unconditionally pop from a list (spam if mylist.pop(idx) else eggs) + mylist # conditionally pop from a list (mylist.pop(idx) if condition else eggs) + mylist (spam if condition else mylist.pop(idx)) + mylist Take your choice of which you want: (spam if (mylist := expr) else eggs) + mylist ((mylist := expr) if condition else eggs) + mylist (spam if condition else (mylist := expr)) + mylist Of course, in the last two cases, you're going to get a NameError when you try to add mylist if the ternary operator took the wrong branch. Unless you already defined mylist earlier. If you want something else, you have to explain what you want. > given makes it obvious by separating > assignment from the usage of its assignment target. This is just a purely mechanical source transformation from one to the other. Just replace ":" with "mylist given mylist " and you are done. # unconditional version (spam if (mylist given mylist = expr) else eggs) + mylist # conditional versions ((mylist given mylist = expr) if condition else eggs) + mylist (spam if condition else (mylist given mylist = expr)) + mylist If you can write the "given" versions, then just do the reverse transformation and replace the redundant verbosity with a colon. > Style: > * it avoids the big style question of when to use and when not to use :=. > (Even if you ask people not to, people are going to write the > expression-statement a := b as a synonym for the statement a = b.) What if they do? Is it really the end of the world if some ex-Pascal coders or people with an over-developed desire for (foolish) consistency add a colon to their statement level assignments? Some people add semi-colons to their statements too. Do we care? No. But if it aggitates people so much, then I'm perfectly happy with Guido's suggestion that we simply ban top level binding expressions and require them to leave the colons out. > * it looks a lot like the existing Python "for" and "if" clauses, which > also do in-expression assignments. "if" clauses do an assignment? Have you borrowed the keys to Guido's time machine, and are writing from 2025 and Python 4.2? *wink* I don't think "given" expressions look even remotely similar to either. for target in iterable: if condition: another_expr given target = expr Aside from "all three use a keyword". -- Steve From turnbull.stephen.fw at u.tsukuba.ac.jp Sat May 12 16:37:25 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 13 May 2018 05:37:25 +0900 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <23287.20613.651267.265792@turnbull.sk.tsukuba.ac.jp> David Mertz writes: > Only the BDFL has a vote with non-zero weight. "Infinitesimal" != "zero". Pedantically yours, From rosuav at gmail.com Sat May 12 16:52:49 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 May 2018 06:52:49 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <23287.20613.651267.265792@turnbull.sk.tsukuba.ac.jp> References: <23287.20613.651267.265792@turnbull.sk.tsukuba.ac.jp> Message-ID: On Sun, May 13, 2018 at 6:37 AM, Stephen J. Turnbull wrote: > David Mertz writes: > > > Only the BDFL has a vote with non-zero weight. > > "Infinitesimal" != "zero". > > Pedantically yours, Correct, infinitesimal is not zero. This does not contradict David's statement. ChrisA From e+python-ideas at kellett.im Sat May 12 17:33:48 2018 From: e+python-ideas at kellett.im (Ed Kellett) Date: Sat, 12 May 2018 22:33:48 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180512172427.GF12683@ando.pearwood.info> References: <20180512172427.GF12683@ando.pearwood.info> Message-ID: <5703c5ad-7f5b-7788-be3e-4442144805e8@kellett.im> On 2018-05-12 18:24, Steven D'Aprano wrote: > On Sat, May 12, 2018 at 08:16:07AM -0700, Neil Girdhar wrote: >> I love given compared with := mainly because >> >> Simpler is better than complex: * given breaks a complex statement >> into two simpler ones, > > [snip] > > Nick's syntax is *much more complicated* than the := syntax. Any > arbitrary expression can appear on the left of "given". It need not > even involve the binding target! So to make a fair comparison, I > ought to compare: > > target := expr > > which evaluates a single expression, binds it, and returns it, to: > > another_expr given target := expr > > which evaluates "expr", binds it to "target", evaluates a SECOND > unrelated expression, and returns that. > > If you want to argue that this is more useful, then fine, say so. > But to say that it is *simpler* makes no sense to me. > > Option 1: evaluate and bind a single expression > > Option 2: exactly the same as Option 1, and then evaluate a second > expression > > How do you justify that Option 2 "given", which does everything := > does PLUS MORE, is simpler than Option 1? Your interpretation of the argument you quote, as I understand it, is that "`given` is simpler than `:=`". I interpreted it as, and would argue myself, more like "`given` makes *the code that uses it* simpler". The reason the `given` syntax might be simpler has nothing to do with the number of syntactic elements (brainfuck is down the hall...). Rather, it is that `given` separates things: if m.group(2) in words given m = word_re.match(s): versus: if (m := word_re.match(s)).group(2) in words: In the `:=` version, the assignment is embedded in the expression. It's different. Most of the time it will save at least a few characters. But it's not obviously--and certainly not objectively--simpler. One somewhat concrete difference I can think of is that in expressions that refer to the result multiple times, the `:=` assignment must always be positioned such that it is the first to be evaluated, moving around should that ever change. In cases where evaluation depends on runtime information (if-else, and, or), I'm not actually sure what you would do. In general, I feel like you're focusing on the simplicity or otherwise of the operator itself, rather than whether it has a simplifying effect on code that uses it. As you say in part 2 (which arrived while I was still taking forever to write this; sorry if it reads a bit confused), we'll learn the syntax, eventually. What's not so explicitly spelled out is that we'll be reading new code that uses it forever. > average = 0 smooth_signal = [(average := (1-decay)*average + > decay*x) for x in signal] assert average == smooth_signal[-1] Like, holy smoke, man. Sure, `:=` is probably better for cramming side-effects into list comprehensions. Please don't cram side-effects into list comprehensions. > total = 0 running_totals = [total := total + x for x in xs] > > versus > > total = 0 running_totals = [total given total = total + x for x in xs] What is it with the cramming side-effects into list comprehensions? Do you realise itertools.accumulate exists? See the OP: On 2018-05-04 13:06, Nick Coghlan wrote: > 3. Sharing values between filtering clauses and result expressions in > comprehensions: > > result = [(x, y, x/y) for x in data if (y := f(x))] I don't have a more concrete example of this to hand, mostly because I can't do this today, and faking it is hard enough (I didn't even know about the `for value in [stuff]` hack until this thread) that I just write for loops. On 2018-05-12 18:24, Steven D'Aprano wrote: > By your argument, augmented assignment is more complex, and we ought > to prefer splitting it into two separate operations x = x + 1 because > that's simpler. > > I think I agree that x = x + 1 *is* simpler. We can understand it by > understanding the two parts separately: x+1, followed by assignment. > > Whereas += requires us to understand that the syntax not only calls > a dunder method __iadd__ (or __add__ if that doesn't exist), which > potentially can operate in place, but it also does an assignment, > all in one conceptual operation. There's a whole lot of extra > complexity there. While I'm not sorry that I can do `x += 1`, it does have a substantial cost--one that is perhaps justified by the fact that it's *not* equivalent to `x = x + 1`, and that the possibilities it opens up are both useful and easy to understand. It's not simpler in and of itself, but that's not really the issue. On 2018-05-12 19:27, Steven D'Aprano wrote: > What you probably want is the second version: > > (a := c.d()).b(a) > > which of course looks like utter crap. It might look better if we > use at least half-way sensible variable names and a more realistic > looking example, instead of obfuscated one-letter names. > > (widget := widget_builder.new(*args)).method(widget) I believe that that's missing the point: to wit, in x.method(y) is `x` or `y` evaluated first? I didn't know. I don't think the order is documented anywhere or guaranteed not to change. I don't know what other languages do in general, though I know in C it's explicitly unspecified. So, sure, you can do this fine with `:=`. But it forces your code to depend on what I'd regard as a subtlety of the implementation. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 484 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Sat May 12 17:42:19 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 May 2018 07:42:19 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5703c5ad-7f5b-7788-be3e-4442144805e8@kellett.im> References: <20180512172427.GF12683@ando.pearwood.info> <5703c5ad-7f5b-7788-be3e-4442144805e8@kellett.im> Message-ID: On Sun, May 13, 2018 at 7:33 AM, Ed Kellett wrote: > On 2018-05-12 19:27, Steven D'Aprano wrote: >> What you probably want is the second version: >> >> (a := c.d()).b(a) >> >> which of course looks like utter crap. It might look better if we >> use at least half-way sensible variable names and a more realistic >> looking example, instead of obfuscated one-letter names. >> >> (widget := widget_builder.new(*args)).method(widget) > > I believe that that's missing the point: to wit, in > > x.method(y) > > is `x` or `y` evaluated first? I didn't know. I don't think the order is > documented anywhere or guaranteed not to change. I don't know what other > languages do in general, though I know in C it's explicitly unspecified. It's documented. It's guaranteed not to change. https://docs.python.org/3/reference/expressions.html#evaluation-order ChrisA From e+python-ideas at kellett.im Sat May 12 17:49:39 2018 From: e+python-ideas at kellett.im (Ed Kellett) Date: Sat, 12 May 2018 22:49:39 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512172427.GF12683@ando.pearwood.info> <5703c5ad-7f5b-7788-be3e-4442144805e8@kellett.im> Message-ID: <86910926-8732-8afa-ccd0-f178887a5649@kellett.im> On 2018-05-12 22:42, Chris Angelico wrote: > It's documented. It's guaranteed not to change. > > https://docs.python.org/3/reference/expressions.html#evaluation-order Thanks. That's good to know, though I continue to hope nobody makes a habit of writing code that depends on it. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From cs at cskk.id.au Sat May 12 17:07:16 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 13 May 2018 07:07:16 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <20180512210716.GA34320@cskk.homeip.net> On 06May2018 02:00, Nick Coghlan wrote: >On 5 May 2018 at 13:36, Tim Peters wrote: >> If only one trailing "given" clause can be given per `if` test >> expression, presumably I couldn't do that without trickery. > >I was actually thinking that if we did want to allow multiple assignments, >and we limited targets to single names, we could just use a comma as a >separator: > > if diff and g > 1 given diff = x - x_base, g = gcd(diff, n): > return g > >Similar to import statements, optional parentheses could be included in the >grammar, allowing the name bindings to be split across multiple lines: > > if diff and g > 1 given ( > diff = x - x_base, > g = gcd(diff, n), > ): > return g I'm well behind, but... this! This turns "given" into a +0.8 for me. That's really nice. It reads clearly too. I was hitherto in the "expression as name" camp, which I gather is already rejected. Cheers, Cameron Simpson From cs at cskk.id.au Sat May 12 17:24:20 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 13 May 2018 07:24:20 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180512210716.GA34320@cskk.homeip.net> References: <20180512210716.GA34320@cskk.homeip.net> Message-ID: <20180512212420.GA77042@cskk.homeip.net> On 13May2018 07:07, Cameron Simpson wrote: >>Similar to import statements, optional parentheses could be included in the >>grammar, allowing the name bindings to be split across multiple lines: >> >> if diff and g > 1 given ( >> diff = x - x_base, >> g = gcd(diff, n), >> ): >> return g > >I'm well behind, but... this! This turns "given" into a +0.8 for me. >That's really nice. It reads clearly too. And if we're still worried about new keywords or reserved words, this: if diff and g > 1 with ( diff = x - x_base, g = gcd(diff, n), ): return g or: if diff and g > 1 with ( x - x_base as diff, gcd(diff, n) as g ): return g or even: if diff and g > 1 with ( x - x_base, gcd(diff, n), ) as diff, g: return g read nearly as well to my eye. My main point here is that "with" works as well as "given" in this form from an English prose point of view. Cheers, Cameron Simpson From tim.peters at gmail.com Sat May 12 18:11:00 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 12 May 2018 17:11:00 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: Just showing an example of "by hand" code emulating nesting of comprehensions, with a highly dubious rebinding, in the inner comprehension, of an outer comprehension's local for-target. list(i + sum((i := i+1) + i for j in range(i)) for i in range(5)) I don't believe I have compelling use cases for nesting listcomps/genexps, so that's just made up to be an example of atrocious feature abuse :-) In the outer genexp, `i` is obviously local, as is `j` in the inner genexp. But the assignment expression in the inner genexp demands that `i` _there_ be not-local. To which scope does the inner `i` belong? To the same scope it would belong if `i := i+1` were replaced by `i`, which the docs today say is the outer genexp's scope. So that's what it is. Here's code to emulate all that, with a bit more to demonstrate that `i` and `j` in the scope containing that statement remain unchanged: The only "novelty" is that a `nonlocal` declaration is needed to establish an intended scope. def f(): i = 42 j = 53 def outer(it): def inner(it): nonlocal i for j in it: i = i+1 yield i for i in it: yield i + sum(inner(range(i))) + i print(list(outer(range(5)))) print(i, j) f() The output: [0, 5, 13, 24, 38] 42 53 Since the code is senseless, so is the list it generates ;-) Showing it this way may make it clearer: [0+(0)+0, 1+(2)+2, 2+(3+4)+4, 3+(4+5+6)+6, 4+(5+6+7+8)+8] From tim.peters at gmail.com Sat May 12 18:28:14 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 12 May 2018 17:28:14 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: Ah, fudge - I pasted in the wrong "high-level" code. Sorry! The code that's actually being emulated is not > list(i + sum((i := i+1) + i for j in range(i)) > for i in range(5)) but list(i + sum((i := i+1) for j in range(i)) + i for i in range(5)) > ... I have piles of these, but they're all equally tedious so I'll stop with this one ;-) From apalala at gmail.com Sat May 12 18:47:11 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Sat, 12 May 2018 18:47:11 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180512212420.GA77042@cskk.homeip.net> References: <20180512210716.GA34320@cskk.homeip.net> <20180512212420.GA77042@cskk.homeip.net> Message-ID: > My main point here is that "with" works as well as "given" in this form > from an English prose point of view. > +1 for "with...as", -1 for ":=" About affecting existing contexts, it seems that "with..as" would create a new context just for the expression, and the control statement it is embedded in, similar to what the current "with" statement does. These are semantics that are really easy to explain. Cheers! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Sat May 12 18:52:22 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 12 May 2018 18:52:22 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180512210716.GA34320@cskk.homeip.net> References: <20180512210716.GA34320@cskk.homeip.net> Message-ID: On Sat, May 12, 2018 at 5:54 PM Cameron Simpson wrote: > On 06May2018 02:00, Nick Coghlan wrote: > >On 5 May 2018 at 13:36, Tim Peters wrote: > >> If only one trailing "given" clause can be given per `if` test > >> expression, presumably I couldn't do that without trickery. > > > >I was actually thinking that if we did want to allow multiple assignments, > >and we limited targets to single names, we could just use a comma as a > >separator: > > > > if diff and g > 1 given diff = x - x_base, g = gcd(diff, n): > > return g > > > >Similar to import statements, optional parentheses could be included in > the > >grammar, allowing the name bindings to be split across multiple lines: > > > > if diff and g > 1 given ( > > diff = x - x_base, > > g = gcd(diff, n), > > ): > > return g > > I'm well behind, but... this! This turns "given" into a +0.8 for me. > > That's really nice. It reads clearly too. > > I was hitherto in the "expression as name" camp, which I gather is already > rejected. > I love given, but that's the one thing I don't like. I prefer this: if (diff and g > 1 given diff = x - x_base given g = gcd(diff, n)): return g ?just like for and if subexpressions. Doing this can also open up weirdness if someone tries to roll something like: a = f(), # Make a tuple of length 1 into a given statement. Now, where do you up the parentheses? given ( a = (f(),), b = whatever? ) Seems weird. > Cheers, > Cameron Simpson > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/CFuqwmE8s-E/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat May 12 18:55:07 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 May 2018 08:55:07 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512210716.GA34320@cskk.homeip.net> <20180512212420.GA77042@cskk.homeip.net> Message-ID: On Sun, May 13, 2018 at 8:47 AM, Juancarlo A?ez wrote: > >> My main point here is that "with" works as well as "given" in this form >> from an English prose point of view. > > > +1 for "with...as", -1 for ":=" > > About affecting existing contexts, it seems that "with..as" would create a > new context just for the expression, and the control statement it is > embedded in, similar to what the current "with" statement does. These are > semantics that are really easy to explain. The trouble with every variant involving 'with' is that the semantics LOOK similar, but are subtly different. The current 'with' statement doesn't create a subscope; the only "context" it creates is regarding the resource represented by the context manager. For instance, opening a file in a 'with' block will close the file at the end of the block - but you still have a (closed) file object. Using "with... as" for name bindings wouldn't call __enter__ or __exit__, so it won't create that kind of context; and whether it creates a subscope for the variable or not, it's not going to match the 'with' statement. ChrisA From apalala at gmail.com Sat May 12 19:36:33 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Sat, 12 May 2018 19:36:33 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512210716.GA34320@cskk.homeip.net> <20180512212420.GA77042@cskk.homeip.net> Message-ID: > LOOK similar, but are subtly different. The current 'with' statement > doesn't create a subscope; the only "context" it creates is regarding > the resource represented by the context manager. For instance, opening > a file in a 'with' block will close the file at the end of the block - > but you still have a (closed) file object. Using "with... as" for name > bindings wouldn't call __enter__ or __exit__, so it won't create that > kind of context; and whether it creates a subscope for the variable or > not, it's not going to match the 'with' statement. > That is all valid, but it still would be familiar, and easier to explain. Python already uses "in", which is used in other languages to introduce context. The statement structure of "with...as" seems desirable, just asking for a word that is not "with" or "given". I don't remember if "when" was already rejected. http://www.thesaurus.com/browse/with?s=t http://www.thesaurus.com/browse/given?s=t http://www.thesaurus.com/browse/considering?s=t http://www.thesaurus.com/browse/assume?s=t http://www.thesaurus.com/browse/when?s=t Cheers! -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Sat May 12 19:37:33 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 12 May 2018 19:37:33 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180512172427.GF12683@ando.pearwood.info> References: <20180512172427.GF12683@ando.pearwood.info> Message-ID: On Sat, May 12, 2018 at 1:25 PM Steven D'Aprano wrote: > On Sat, May 12, 2018 at 08:16:07AM -0700, Neil Girdhar wrote: > > > I love given compared with := mainly because > > > > Simpler is better than complex: > > * given breaks a complex statement into two simpler ones, > > (Foreshadowing: this argument applies to augmented assignment. See > below.) > > I don't see how you justify that statement about "given". I think that > it is "given" which is more complex. Significantly so. > > Let's compare the syntax: > > target := expr > > > That is a single, simple expression with a single side-effect: it > assigns the value to . That's it. > > Like all expressions, it returns a value, namely the result of "expr". > Like all expressions, you can embed it in other expressions (possibly > wrapping it in parens to avoid precedence issues), or not, as required. > > (That surrounding expression can be as simple or complex as you like.) > > > Now here's Nick's syntax: > > target given target = expr > > Exactly like the := version above, we can say that this is a single > expression with a single side-effect. Like all expressions, it returns a > value, namely the result of "expr", and like all expressions, you can > embed it in other expressions. > > So far the two are precisely the same. There is no difference in the > complexity, because they are exactly the same except for the redundant > and verbose "given" spelling. > > But actually, I lied. > > Nick's syntax is *much more complicated* than the := syntax. Any > arbitrary expression can appear on the left of "given". It need not > even involve the binding target! So to make a fair comparison, I ought > to compare: > > target := expr > > which evaluates a single expression, binds it, and returns it, to: > > another_expr given target := expr > > which evaluates "expr", binds it to "target", evaluates a SECOND > unrelated expression, and returns that. > > If you want to argue that this is more useful, then fine, say so. But to > say that it is *simpler* makes no sense to me. > It's simpler in the sense that each of the components is simpler, and that the components exhibit *separation of concerns*. The bindings of the expressions are separate from calculation of the returned expression. The separation is both conceptual since the bindings are all distinct pieces of the whole expression, and temporal since the bindings all happen in some order, and must happen first. It can be very hard to get the ordering right. What if you are implementing something using the visitor pattern and you have something like: (visitor.access(database, key=database.key) given database = kwargs.pop("database") given visitor = database.default_visitor()) The beauty of given is that in this pattern: EXPRESSION given A = B given C = D you can completely forget about * B and D when thinking about EXPRESSION, * D and EXPRESSION when thinking about B, and * B and EXPRESSION when thinking about C. That's the separation of concerns. Now with :=, it's tricky because you can't set the visitor without first setting the database, but the visitor shows up first in the expression! Even if you could do something like: (visitor := database.default_visitor()).access((database := kwargs.pop("database")), key=database.key) I think this is much more complicated because all of the pieces are in the same giant expression, and so the concerns aren't separate. It's not obvious how to temporally order the pieces. And it takes a couple minutes to work out which parts of the expression fit with which other parts of the expression. If anyone ever writes code like that, you can guarantee that the Google style guide will preclude := except for maybe its the simplest uses. > > Option 1: evaluate and bind a single expression > > Option 2: exactly the same as Option 1, and then evaluate a second > expression > > How do you justify that Option 2 "given", which does everything := does > PLUS MORE, is simpler than Option 1? > > That's not a rhetorical question. > > > > > which is putting people off in the simple examples shown here (some > > people are annoyed by the extra characters). However, when given is > > used in a list comprehension to prevent having to re-express it as a > > for loop, then two simple statements are easier to understand than one > > complex statement. > > I'd like to see an example of one of these list comprehensions that is > simpler written with given. Here's an earlier example, an exponentially > weighted running average: > > > average = 0 > smooth_signal = [(average := (1-decay)*average + decay*x) for x in signal] > assert average == smooth_signal[-1] > > Back when I was on the ACM team, people wrote code like that. It's fine if you're a brilliant contestant who has a few hours to solve a bunch of problems on one computer. It makes sense to optimize for number of characters, and anyway, ideally you were the only one reading your code unless your team had to help you debug it. However, and I'm sorry to be so bold, but this code will never pass a code review at a big company. It may have been easy for you to write, but it is not easy to understand at all. You're using the binding of := to update a a value that's used in subsequent iterations of the loop. That's way too complicated for a one-liner. You can write code like this for yourself, but no one else should have to decipher that. This is what I meant when I mentioned the difference between code at programming contests versus code written by those same software engineers years later at big companies. In my professional career (most recently at Google), the main goal was writing clear, accessible code so that the next person looking at it (or even me months later) would not have to think about it at all. It is immediately obvious that simple code is right. Even if you're brilliant enough to understand the code above, it needs to be immediately obviously correct to next person looking at it. Compare it with the for loop: def leaky_integral(signals, decay): value = 0.0 for x in signals: value = value * decay + x * (1 - decay) yield value > I'm not even sure if "given" will support this. Nick is arguing strongly > that bound targets should be local to the comprehension, and so I think > you can't even write this example at all with Nick's scoping rule. > Nick is right in my opinion to argue this. If you want the bound target to leak out, just write the for loop. Otherwise you're asking too much of the reader. > But let's assume that the scoping rule is the same as the above. In that > case, I make it: > > average = 0 > smooth_signal = [average given average = (1-decay)*average + decay*x) for > x in signal] > > Is it longer, requiring more typing? Absolutely. > > Does it contain a redundantly repetitious duplication of the repeated > target name? Certainly. > > But is it simpler? I don't think so. > > > If you don't like the exponential running average, here's a simple > running total: > > > total = 0 > running_totals = [total := total + x for x in xs] > > > versus > > > total = 0 > running_totals = [total given total = total + x for x in xs] > > > If you don't like this example either, please show me an example of an > actual list comp that given makes simpler. > I showed one above where := isn't even possible. > > > This is a > > common difference between code written at programming contests versus > code > > written by those same software engineers years later at big companies. > > Code that you write for yourself can be compact because you already > > understand it, but code you write professionally is read many many more > > times than it is written. Accessibility is much more important than > > concision. > > Ah, nice rhetorical argument: "given" is for professionals, := is a hack > for amateurs and programming contests. Seriously? > Sorry, I didn't mean to offend you. I fleshed out what I meant in this email. Two separate parts of my life as a programmer were my contest life and my professional career, and each had different ideals. > > Do you use augmented assignment? Your simple versus complex argument for > "given" applies well to augmented assignment. > > Augmented assignment combines two conceptual operations: > > x = x + 1 > - addition (for example) > - assignment > > into a single operator: > > x += 1 > > > By your argument, augmented assignment is more complex, and we ought to > prefer splitting it into two separate operations x = x + 1 because > that's simpler. > > I think I agree that x = x + 1 *is* simpler. We can understand it by > understanding the two parts separately: x+1, followed by assignment. > I don't agree with that. += is simpler since in that case the number of elements is fewer. The reason I'm saying that := is not simpler is that it tangles together unrelated subelements, which I explained in my separation of concerns paragraphs above. > Whereas += requires us to understand that the syntax not only calls a > dunder method __iadd__ (or __add__ if that doesn't exist), which > potentially can operate in place, but it also does an assignment, all in > one conceptual operation. There's a whole lot of extra complexity there. > > That doesn't mean I agree with your conclusion that we ought to prefer > the simpler version, let alone that the complex case (augmented > assignment) is fit only for programming contests and that professionals > ought to choose the simpler one. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/CFuqwmE8s-E/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Sat May 12 19:51:56 2018 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 13 May 2018 00:51:56 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180512210716.GA34320@cskk.homeip.net> References: <20180512210716.GA34320@cskk.homeip.net> Message-ID: <274b99a3-e8c2-a412-3870-8b0daed2bbe0@mrabarnett.plus.com> On 2018-05-12 22:07, Cameron Simpson wrote: > On 06May2018 02:00, Nick Coghlan wrote: >>On 5 May 2018 at 13:36, Tim Peters wrote: >>> If only one trailing "given" clause can be given per `if` test >>> expression, presumably I couldn't do that without trickery. >> >>I was actually thinking that if we did want to allow multiple assignments, >>and we limited targets to single names, we could just use a comma as a >>separator: >> >> if diff and g > 1 given diff = x - x_base, g = gcd(diff, n): >> return g >> >>Similar to import statements, optional parentheses could be included in the >>grammar, allowing the name bindings to be split across multiple lines: >> >> if diff and g > 1 given ( >> diff = x - x_base, >> g = gcd(diff, n), >> ): >> return g > > I'm well behind, but... this! This turns "given" into a +0.8 for me. > > That's really nice. It reads clearly too. > That's longer than this: diff = x - x_base g = gcd(diff, n) if diff and g > 1: return g which is already valid. [snip] From mistersheik at gmail.com Sat May 12 20:06:02 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 12 May 2018 20:06:02 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180512182735.GG12683@ando.pearwood.info> References: <20180512182735.GG12683@ando.pearwood.info> Message-ID: On Sat, May 12, 2018 at 2:28 PM Steven D'Aprano wrote: > Part 2. > > On Sat, May 12, 2018 at 08:16:07AM -0700, Neil Girdhar wrote: > > I love given compared with := mainly because > > [...] > > * Python has a reputation for being working pseudocode, and given reads > > like pseudocode. := needs to be deciphered by comparison especially in > the > > complicated cases where multiple := operators are used on one line. > > Until you learn and become familiar with a new syntax, there is > generally going to be a period you have to decipher it. I spent a long > time mentally translating list comprehensions into mathematical set > builder notation before it became second nature to me. > > Even now, I know people who find decorators and comprehensions > indecipherable. Or at least, so they claim, and they aren't motivated to > bother learning them. Oh well, that's their loss. > > Binding-expressions aren't like asynchronous programming, where the > entire coding paradigm is different, and you literally have to think > about your algorithms in another way. Whether you spell the binding > operation > > target := expr > expr as target > expr -> target > target given target = expr > let target = expr > : target expr ; > > (that last one is stolen from Forth, and should not be taken as a > serious suggestion) > > is just a matter of spelling. We'll get used to whatever spelling it is. > Some may be more convenient than others, or more error-prone, or may be > harder to parse, but they're secondary issues. (Important, but still > secondary.) Fundamentally, the operation is the same regardless of the > spelling: > > - evaluate an expression > - bind that value to a name > - return the value > > > (Except of course Nick's "given" suggestion is more complex, since the > returned value is not necessarily the same as the bound value.) > > > > * there are no difficult mental questions about evaluation order, e.g., > in > > a bracketed expression having multiple assignments. > > Aren't there just? > > x = 1 > print( x + x given x = 50 ) > > > Will that print 100, or 51? Brackets ought to make it clearer: > > (x + x given x = 50) # this ought to return 100 > x + (x given x = 50) # this ought to return 51 > > but if I leave the brackets out, I have no idea what I'll get. > It has to be 100, or else outer parentheses change how an expression is evaluated. > > > > > Similarly, instead of > > (a.b(a) given a = c.d()) do I write (a.b(a := c.d())) or ((a := > > c.d()).b(a)) ? > > > I would expect that your first example is a NameError: > > a.b(a := c.d()) > > since Python evaluates arguments to a method *after* looking up the > method. So that corresponds to: > > - look up "a" # NameError, unless you've already got an "a" > - look up "a.b" > - evaluate c.d() > - assign that value to a > - pass that to the a.b method we found earlier > > > What you probably want is the second version: > > (a := c.d()).b(a) > > which of course looks like utter crap. It might look better if we use at > least half-way sensible variable names and a more realistic looking > example, instead of obfuscated one-letter names. > > (widget := widget_builder.new(*args)).method(widget) > > > > * it avoids the question of what happens when := is used in a switch: > (a > > if (b := c) else d) Sometimes you want the assignment to happen > > unconditionally (a if (b:=c) else d) + b; sometimes you don't. How do > you > > force one case or the other? > > I think that this argument is really weak. The obvious answer is, if you > don't want the assignment to happen unconditionally, then don't do the > assignment unconditionally. Where you do it depends on when you want the > assignment to take place. There's no mystery here. > > # unconditionally pop from a list > (spam if mylist.pop(idx) else eggs) + mylist > > # conditionally pop from a list > (mylist.pop(idx) if condition else eggs) + mylist > (spam if condition else mylist.pop(idx)) + mylist > > > Take your choice of which you want: > > (spam if (mylist := expr) else eggs) + mylist > ((mylist := expr) if condition else eggs) + mylist > (spam if condition else (mylist := expr)) + mylist > > Of course, in the last two cases, you're going to get a NameError when > you try to add mylist if the ternary operator took the wrong branch. > Unless you already defined mylist earlier. > That's the problem I'm showing. This is impossible: (spam if (mylist := expr) else eggs) + mylist but just fine with given: ((spam if mylist else eggs) + mylist) given mylist = expr) > If you want something else, you have to explain what you want. > > > > given makes it obvious by separating > > assignment from the usage of its assignment target. > > This is just a purely mechanical source transformation from one to the > other. Just replace ":" with "mylist given mylist " and you are done. > First of all, you cannot convert all := expressions to given expressions. Even if you could, the point is that they are separated. I went over the separation of concerns in my other mail. > # unconditional version > (spam if (mylist given mylist = expr) else eggs) + mylist > > # conditional versions > ((mylist given mylist = expr) if condition else eggs) + mylist > (spam if condition else (mylist given mylist = expr)) + mylist > > If you can write the "given" versions, then just do the reverse > transformation and replace the redundant verbosity with a colon. > You can't always do that. I've given you two examples now where you cannot go backwards. > > > Style: > > * it avoids the big style question of when to use and when not to use > :=. > > (Even if you ask people not to, people are going to write the > > expression-statement a := b as a synonym for the statement a = b.) > > What if they do? Is it really the end of the world if some ex-Pascal > coders or people with an over-developed desire for (foolish) consistency > add a colon to their statement level assignments? > Some people add semi-colons to their statements too. Do we care? No. > > But if it aggitates people so much, then I'm perfectly happy with > Guido's suggestion that we simply ban top level binding expressions and > require them to leave the colons out. > > > > * it looks a lot like the existing Python "for" and "if" clauses, which > > also do in-expression assignments. > > "if" clauses do an assignment? Have you borrowed the keys to Guido's > time machine, and are writing from 2025 and Python 4.2? *wink* > > I don't think "given" expressions look even remotely similar to either. > > for target in iterable: > > if condition: > > another_expr given target = expr > I meant the clauses not the statements: (expression for x in it if x.y given z = x.z) All three clauses have the same format, and obvious temporal ordering. for and given both bind a name that is visible to the expression and to clauses below. > Aside from "all three use a keyword". > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/CFuqwmE8s-E/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Sat May 12 20:12:41 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 12 May 2018 19:12:41 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <274b99a3-e8c2-a412-3870-8b0daed2bbe0@mrabarnett.plus.com> References: <20180512210716.GA34320@cskk.homeip.net> <274b99a3-e8c2-a412-3870-8b0daed2bbe0@mrabarnett.plus.com> Message-ID: [attributions lost - sorry, but I can't get 'em back] ... >>> Similar to import statements, optional parentheses could be included in >>> the >>> grammar, allowing the name bindings to be split across multiple lines: >>> >>> if diff and g > 1 given ( >>> diff = x - x_base, >>> g = gcd(diff, n), >>> ): >>> return g >> I'm well behind, but... this! This turns "given" into a +0.8 for me. >> >> That's really nice. It reads clearly too. >> > That's longer than this: > > diff = x - x_base > g = gcd(diff, n) > if diff and g > 1: > return g > > which is already valid. Since that was my example to begin with, I think it's fair to point out that they all miss a key part of the original example: this code is working with multi-thousand bit integers, and calling gcd() is expensive. It was a key point that if (diff := x - x_base) and (g := gcd(diff, n)) > 1: return g didn't call gcd() _at all_ unless `diff` was non-zero. The original real-life code was: diff = x - x_base if diff: g = gcd(diff, n) if g > 1: return g From apalala at gmail.com Sat May 12 20:13:01 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Sat, 12 May 2018 20:13:01 -0400 Subject: [Python-ideas] Inline changes to context, or PEP572 Message-ID: A new thread just to suggest taking the discussion about PEP572 well beyond python-ideas (PyConn is good for that). The least anyone should want is a language change that immediately gets tagged on the networks as "don't use", or "use only for...", etc. To be honest, I'll likely be on the "don't use :=, unless" band of pundits (already a filibuster). ":=" is like going back to "reduce()", which is almost defunct thanks to.. us! Cheers! -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Sat May 12 20:27:53 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Sat, 12 May 2018 20:27:53 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512210716.GA34320@cskk.homeip.net> <274b99a3-e8c2-a412-3870-8b0daed2bbe0@mrabarnett.plus.com> Message-ID: > if (diff := x - x_base) and (g := gcd(diff, n)) > 1: > return g > > I don't see the advantage in that succinctness: g = special_gcd(x - x_base, n) if g: return g The code bases I work on constantly move towards having the next guy grok what's going on just by reading the code. It could also be: if special_gcd(x - x_base, n) as g: return g Cheers! Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From andre.roberge at gmail.com Sat May 12 20:34:33 2018 From: andre.roberge at gmail.com (Andre Roberge) Date: Sat, 12 May 2018 21:34:33 -0300 Subject: [Python-ideas] "given" vs ":=" in list comprehensions Message-ID: Sorry for chiming in so late; I was lurking using google groups and had to subscribe to post - hence this new thread. I gather that *where* has been discarded as a possible new keywords given its use as a function in numpy ( https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.where.html) ... Still, I will include it below for completeness (and as I think it reads better than the other choices) Here are two sets of two examples illustrating a situation that I have not seen before, and which read (to me) much better when using a keyword (given or where) than a symbol (:=). Furthermore, the position of the temporary assignment can, in my opinion, be done differently and help in making the code clearer. First example: single temporary assignment, done four different ways. 1) using := real_roots = [ (-b/(2*a) + (D:= sqrt( (b/(2*a))**2 - c/a), -b/(2*a) - D) for a in range(10) for b in range(10) for c in range(10) if D >= 0] 2) using *given* at the very end real_roots = [ (-b/(2*a) + D, -b/(2*a) - D) for a in range(10) for b in range(10) for c in range(10) if D >= 0 given D= sqrt( (b/(2*a))**2 - c/a)] 3) using *given* before the iterations real_roots = [ (-b/(2*a) + D, -b/(2*a) - D) given D= sqrt( (b/(2*a))**2 - c/a) for a in range(10) for b in range(10) for c in range(10) if D >= 0] 4) using *where* before the iterations (which would be my preferred choice if it were available) real_roots = [ (-b/(2*a) + D, -b/(2*a) - D) where D= sqrt( (b/(2*a))**2 - c/a) for a in range(10) for b in range(10) for c in range(10) if D >= 0] Second example: multiple assignments. When we have multiple temporary assignments, the situation can be more complicated. In the following series of examples, I will start in reverse order compared to above. 5) using *where* before the iterations real_roots2 = [ (-b/(2*a) + D, -b/(2*a) - D) where D= sqrt( (b/(2*a))**2 - c/a) where c = c_values/100 for c_values in range(1000) if D >= 0] 6) using *given* before the iterations real_roots2 = [ (-b/(2*a) + D, -b/(2*a) - D) given D= sqrt( (b/(2*a))**2 - c/a) given c = c_values/100 for c_values in range(1000) if D >= 0] 7) using *given* at the very end real_roots2 = [ (-b/(2*a) + D, -b/(2*a) - D) for c_values in range(1000) if D >= 0 given D= sqrt( (b/(2*a))**2 - c/a) given c = c_values/100] 8) Using := real_roots2 = [ ( -b/(2*a) + (D:= sqrt( (b/(2*a))**2 - (c:=c_values/100)/a), -b/(2*a) - D) for c_values in range(1000) if D >= 0] I find this last version extremely difficult to understand compared with the others where a keyword is used. Perhaps it is because I do not fully understand how := should be used... Finally ... if "where" cannot be used, given the very special status of such temporary assignments, could "where_" (with a trailing underscore) be considered? I would argue that any word followed by an underscore would be more readable than a compound symbol such as ":=". Andr? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat May 12 20:36:32 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 May 2018 10:36:32 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512182735.GG12683@ando.pearwood.info> Message-ID: On Sun, May 13, 2018 at 10:06 AM, Neil Girdhar wrote: > >> Take your choice of which you want: >> >> (spam if (mylist := expr) else eggs) + mylist >> ((mylist := expr) if condition else eggs) + mylist >> (spam if condition else (mylist := expr)) + mylist >> >> Of course, in the last two cases, you're going to get a NameError when >> you try to add mylist if the ternary operator took the wrong branch. >> Unless you already defined mylist earlier. > > > That's the problem I'm showing. This is impossible: > > (spam if (mylist := expr) else eggs) + mylist > > but just fine with given: > > ((spam > if mylist > else eggs) + mylist) > given mylist = expr) I don't understand. How is that impossible with the colon-equals form? The 'if' expression is always going to be evaluated, followed by exactly one of 'spam' and 'eggs'. So mylist will be properly assigned before you get to adding it onto the end. ChrisA From rosuav at gmail.com Sat May 12 20:41:21 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 May 2018 10:41:21 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512210716.GA34320@cskk.homeip.net> <274b99a3-e8c2-a412-3870-8b0daed2bbe0@mrabarnett.plus.com> Message-ID: On Sun, May 13, 2018 at 10:27 AM, Juancarlo A?ez wrote: > >> if (diff := x - x_base) and (g := gcd(diff, n)) > 1: >> return g >> > > I don't see the advantage in that succinctness: > > g = special_gcd(x - x_base, n) > > if g: > > return g > > > The code bases I work on constantly move towards having the next guy grok > what's going on just by reading the code. > > It could also be: > > if special_gcd(x - x_base, n) as g: > > return g > Now I have to go read elsewhere to figure out what "special_gcd" does. With Tim's original code, I could see the effect right there. It might not seem significant with a single function, but if you had this situation come up a dozen times, now you have a dozen functions, each one used only once. You have to go a LONG way out of line to find the meaning of this name. Remember: Giving a function a useful name means figuring out a name that means you do not need to read the function's body to understand what it does. It's easy to say "just make it a function", but that's utterly useless if the next reader has to grok the function's implementation to understand its usage. ChrisA From rosuav at gmail.com Sat May 12 20:55:12 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 May 2018 10:55:12 +1000 Subject: [Python-ideas] "given" vs ":=" in list comprehensions In-Reply-To: References: Message-ID: On Sun, May 13, 2018 at 10:34 AM, Andre Roberge wrote: > Second example: multiple assignments. > > When we have multiple temporary assignments, the situation can be more > complicated. In the following series of examples, I will start in reverse > order compared to above. > > 5) using *where* before the iterations > > real_roots2 = [ (-b/(2*a) + D, -b/(2*a) - D) > where D= sqrt( (b/(2*a))**2 - c/a) > where c = c_values/100 > for c_values in range(1000) > if D >= 0] > > 6) using *given* before the iterations > > real_roots2 = [ (-b/(2*a) + D, -b/(2*a) - D) > given D= sqrt( (b/(2*a))**2 - c/a) > given c = c_values/100 > for c_values in range(1000) > if D >= 0] > > 7) using *given* at the very end > > real_roots2 = [ (-b/(2*a) + D, -b/(2*a) - D) > for c_values in range(1000) > if D >= 0 > given D= sqrt( (b/(2*a))**2 - c/a) > given c = c_values/100] In what order are multiple assignments performed? I guarantee you that whichever you pick, people will wonder why you didn't pick the other. For instance, you've used the word "given" twice, and they appear to nest; but can you use 'c' in the main body? The 'for' loops all execute left to right. Now you're introducing 'given' statements that execute from right to left. I think. > 8) Using := > > real_roots2 = [ ( -b/(2*a) + (D:= sqrt( (b/(2*a))**2 - > (c:=c_values/100)/a), > -b/(2*a) - D) > for c_values in range(1000) > if D >= 0] > > I find this last version extremely difficult to understand compared with the > others where a keyword is used. Perhaps it is because I do not fully > understand how := should be used... It's used where you want to have an expression that you then capture the value of. > Finally ... if "where" cannot be used, given the very special status of such > temporary assignments, could "where_" (with a trailing underscore) be > considered? I would argue that any word followed by an underscore would be > more readable than a compound symbol such as ":=". Definitely not. There is a strong convention around Python that goes the opposite way: if you want to use a keyword like "class" in your API, you use "class_" to avoid the clash. Creating a keyword that inverts this is going to cause nothing but pain. For the situation you're looking for, 'given' is going to be at least as good. But I am still far from convinced that it's better than ':='. Pushing the expression out-of-line creates confusing order of evaluation, whereas prepending 'NAME :=' to an expression doesn't change when anything's evaluated. ChrisA From cs at cskk.id.au Sat May 12 20:53:50 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 13 May 2018 10:53:50 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <20180513005350.GA67965@cskk.homeip.net> On 13May2018 08:55, Chris Angelico wrote: >On Sun, May 13, 2018 at 8:47 AM, Juancarlo A?ez wrote: >> >>> My main point here is that "with" works as well as "given" in this form >>> from an English prose point of view. >> >> >> +1 for "with...as", -1 for ":=" >> >> About affecting existing contexts, it seems that "with..as" would create a >> new context just for the expression, and the control statement it is >> embedded in, similar to what the current "with" statement does. These are >> semantics that are really easy to explain. > >The trouble with every variant involving 'with' is that the semantics >LOOK similar, but are subtly different. The current 'with' statement >doesn't create a subscope; the only "context" it creates is regarding >the resource represented by the context manager. For instance, opening >a file in a 'with' block will close the file at the end of the block - >but you still have a (closed) file object. Using "with... as" for name >bindings wouldn't call __enter__ or __exit__, so it won't create that >kind of context; and whether it creates a subscope for the variable or >not, it's not going to match the 'with' statement. For myself, I'm not a fan of a narrow scope. I'd be happy with an inline assignment landing in the function scope (or whatever the current innermost scope is). Like normal assignments. In how many other places does Python adopt a narrower scope than the function/method/class/module? Consider: r = re.compile("bah(baz)") if m given m = r.match("foo barbaz"): thing = m.group(1) I _want_ to access "m" after the expression. My other dislike of narrow scopes is shadowing. I had an unpleasant experience last year working in Go, which has its own ":=" assignment. While Go's := semantics are a little different to the assignment expressions being considered here, it has an IMO dangerous shadowing effect. I'm going to digress into an explaination of this issue in Go here because I want to highlight my dislike of easy accidental shadowing, which is a problem in many areas, and I think that conflating inline assignment with an implied narrow scope in Python would make this worse. In Go you can declare variables 2 ways: var ok = True and: ok := True Also, because Go doesn't use exceptions a common idiom is to return a success/fail value and the task's result: ok, result := do_something(...) That handily declares "ok" and "result" and gives them values. Because the ":=" is so convenient, you might often declare variables as they get used: ok, result1 := do_something() ok, result2 := do_something_else() and so on. You're allowed to mix existing names with new (undeclared) names provided there's at least one new name left of the ":=". So far this is all just ordinary assignments, with convenient declaration included. But consider the function that calls other functions: func thing_outer(x) { ok, result1 := do_something() if ok { ok, result2 := do_something_else() if ok { fmt.Println("ok!") } } return ok, result } That's how it might go in Pythonic form. But Go lets you preceed the test condition with a statement, somewhat like the C "for (setup values; test; advance)" form: for (i=0; i<10; i++) So you may wish to write the function like this: func thing_outer(x) { if ok, result1 := do_something(); ok { if ok, result2 := do_something_else(); ok { fmt.Println("ok!") } } return ok, result } and here is where the scoping causes a disaster. In Go, the declarations _within_ the "if" statement have the "if" statement as their scope. In particular, the inner "ok" shadows the outer "ok", such that an inner failure (setting "ok" to false) doesn't affect the outer "ok" which was true. And so the overall function returns apparent success. And that is entirely enabled by the implied scoping in the "if" statement. The above example is subtly wrong because I'm doing this from memory, but I had a very simple real world example bite me this way and it was a PITA to debug because the effect was so surprising. So much so that from then on I pretty much eschewed the ":=" declaration as a dangerous construct and went with "var" all the time, effectively giving me _one_ scope within any function. Cheers, Cameron Simpson From cs at cskk.id.au Sat May 12 20:59:24 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 13 May 2018 10:59:24 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <274b99a3-e8c2-a412-3870-8b0daed2bbe0@mrabarnett.plus.com> References: <274b99a3-e8c2-a412-3870-8b0daed2bbe0@mrabarnett.plus.com> Message-ID: <20180513005924.GA12937@cskk.homeip.net> On 13May2018 00:51, MRAB wrote: >On 2018-05-12 22:07, Cameron Simpson wrote: >>On 06May2018 02:00, Nick Coghlan wrote: >>>Similar to import statements, optional parentheses could be included in the >>>grammar, allowing the name bindings to be split across multiple lines: >>> >>> if diff and g > 1 given ( >>> diff = x - x_base, >>> g = gcd(diff, n), >>> ): >>> return g >> >>I'm well behind, but... this! This turns "given" into a +0.8 for me. >> >> That's really nice. It reads clearly too. >> >That's longer than this: > >diff = x - x_base >g = gcd(diff, n) >if diff and g > 1: > return g > >which is already valid. Yes, but in a slightly larger scope such as a cascaded if/elif/.../else sequence embedding the "given" into the expression has 2 advantages: it stops the if-else cascading across the terminal purely to embed assignments before the next test, and it lets one clearly associate these particular assignments as relevant to the test itself. Obviously there are circumstances for each, and I expect overuse of "given" degrades readability. However, I still find Nick's "if this given that" formulation quite compelling as a logical form. Cheers, Cameron Simpson From tim.peters at gmail.com Sat May 12 22:17:54 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 12 May 2018 21:17:54 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: [Nick] >> That's all well and good, but it is *completely insufficient for the >> language specification*. > I haven't been trying to write reference docs here, but so far as > supplying a rigorous specification goes, I maintain the above gets > "pretty close". It needs more words, and certainly isn't in the > _style_ of Python's current reference docs, but that's all repairable. > Don't dismiss it just because it's brief. Comprehensions already > exist in the language, and so do nested scopes, so it's not necessary > for this PEP to repeat any of the stuff that goes into those. Mostly > it needs to specify the scopes of assignment expression target names - > and the _intent_ here is really quite simple. > > Here with more words, restricted to the case of assignment expressions > in comprehensions (the only case with any subtleties): > ... And if you didn't like those words, you're _really_ gonna hate this ;-) I don't believe more than just the following is actually necessary, although much more than this would be helpful. I spent the first 15-plus years of my career writing compilers for a living, so am sadly resigned to the "say the minimum necessary for a long argument to conclude that it really was the minimum necessary" style of language specs. That's why I exult in giving explanations and examples that might actually be illuminating - it's not because I _can't_ be cryptically terse ;-) Section 4.2.1 (Binding of names) of the Language Reference Manual has a paragraph starting with "The following constructs bind names:". It really only needs another two-sentence paragraph after that to capture all of the PEP's intended scope semantics (including my suggestion): """ An assignment expression binds the target, except in a function F synthesized to implement a list comprehension or generator expression (see XXX). In the latter case, if the target is not in F's environment (see section 4.2.2) , the target is bound in the block containing F. """ That explicitly restates my earlier "rule #2" in the language already used by the manual. My "rule #1" essentially vanishes as such, because it's subsumed by what the manual already means by "F's environment". This may also be the best place to add another new sentence.: """ Regardless, if the target also appears as an identifier target of a `for` loop header in F, a `SyntaxError` exception is raised. """ Earlier, for now-necessary disambiguation, I expect that in ... targets that are identifiers if occurring in an assignment, ... " statement" should be inserted before the comma. From steve at pearwood.info Sat May 12 22:27:26 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 13 May 2018 12:27:26 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512172427.GF12683@ando.pearwood.info> Message-ID: <20180513022725.GH12683@ando.pearwood.info> On Sat, May 12, 2018 at 01:13:05PM -0500, Tim Peters wrote: > Just clarifying a fine point here: > > [Steven D'Aprano ] > > ... > > average = 0 > > smooth_signal = [(average := (1-decay)*average + decay*x) for x in signal] > > assert average == smooth_signal[-1] > > > > I'm not even sure if "given" will support this. Nick is arguing strongly > > that bound targets should be local to the comprehension, and so I think > > you can't even write this example at all with Nick's scoping rule. > > You can't under Nick's proposal(s), at least not directly (there are > always "tricks"). But it also blows up with UnboundLocalError (for > the "average" in "(1-decay)*average") under the current PEP 572 (the > ":=" PEP). Yes, but I've sort of assumed that if PEP 572 has even a microscopic chance of being accepted, it will have to be changed, given that Guido has already stated that the only behaviour that makes sense is what you and I have been suggesting. Unless Guido has changed his mind, the relevant links are: https://mail.python.org/pipermail/python-ideas/2018-May/050411.html https://mail.python.org/pipermail/python-ideas/2018-May/050456.html So maybe this is a tiny bit naughty (or a lot...) but I've just been ignoring what the PEP currently says and going by what it ought to say :-) -- Steve From mertz at gnosis.cx Sat May 12 22:55:41 2018 From: mertz at gnosis.cx (David Mertz) Date: Sat, 12 May 2018 22:55:41 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <23287.20613.651267.265792@turnbull.sk.tsukuba.ac.jp> References: <23287.20613.651267.265792@turnbull.sk.tsukuba.ac.jp> Message-ID: Are votes cast in Edwin Hewitt's Hyperreals? I had thought python-ideas voted in the domain R rather than *R. :-) On Sat, May 12, 2018, 5:20 PM Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > David Mertz writes: > > > Only the BDFL has a vote with non-zero weight. > > "Infinitesimal" != "zero". > > Pedantically yours, > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Sat May 12 23:04:45 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 12 May 2018 23:04:45 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <23287.20613.651267.265792@turnbull.sk.tsukuba.ac.jp> Message-ID: Another benefit of given compared with := that I just thought of is this. Suppose you have a generator like (expression(f(x), y, z) for x in xs for y in ys(x) for z in zs(y)) With given notation you can optimize: (expression(f_x, y, z) for x in xs given f_x = f(x) for y in ys(x) for z in zs(y)) whereas with :=, you can't. Best, Neil On Sat, May 12, 2018 at 10:56 PM David Mertz wrote: > Are votes cast in Edwin Hewitt's Hyperreals? I had thought python-ideas > voted in the domain R rather than *R. :-) > > On Sat, May 12, 2018, 5:20 PM Stephen J. Turnbull < > turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > >> David Mertz writes: >> >> > Only the BDFL has a vote with non-zero weight. >> >> "Infinitesimal" != "zero". >> >> Pedantically yours, >> >> -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/CFuqwmE8s-E/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/CFuqwmE8s-E/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat May 12 23:47:41 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 13 May 2018 13:47:41 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <23287.20613.651267.265792@turnbull.sk.tsukuba.ac.jp> Message-ID: <20180513034741.GI12683@ando.pearwood.info> On Sat, May 12, 2018 at 11:04:45PM -0400, Neil Girdhar wrote: > Another benefit of given compared with := that I just thought of is this. > Suppose you have a generator like > > (expression(f(x), y, z) > for x in xs > for y in ys(x) > for z in zs(y)) > > With given notation you can optimize: > > (expression(f_x, y, z) > for x in xs > given f_x = f(x) > for y in ys(x) > for z in zs(y)) > > whereas with :=, you can't. Is that legal syntax? You're splitting the "given" expression by sticking the for clause in the middle of it. I don't think that will be legal. It would be trying to split an ternary if: (true_expr for x in xs if condition else false_expr for y in ys) (true_expr if condition for x in xs else false_expr for y in ys) But whether legal or not, Neil, you went on at length about how professionals don't write code like this and such overly dense comprehensions are only fit for competitions. Now you want your cake and to eat it too: "given is better, because it doesn't allow the awful unreadable code that := gives; oh, and it's also better, because it allows *this* awful unreadable code that := doesn't allow" In any case, of course you can write this with := binding expression. You just shouldn't do it: (expression(f_x, y, z) for x in xs for y in ys(x) for z in zs(y) if (f_x := f(x)) or True) ) That's fine for mucking about, but I wouldn't do it for serious code. Replacing the colon with "given f_x" doesn't change that. -- Steve From mistersheik at gmail.com Sun May 13 00:10:08 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Sun, 13 May 2018 00:10:08 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180513034741.GI12683@ando.pearwood.info> References: <23287.20613.651267.265792@turnbull.sk.tsukuba.ac.jp> <20180513034741.GI12683@ando.pearwood.info> Message-ID: On Sat, May 12, 2018 at 11:48 PM Steven D'Aprano wrote: > On Sat, May 12, 2018 at 11:04:45PM -0400, Neil Girdhar wrote: > > Another benefit of given compared with := that I just thought of is this. > > Suppose you have a generator like > > > > (expression(f(x), y, z) > > for x in xs > > for y in ys(x) > > for z in zs(y)) > > > > With given notation you can optimize: > > > > (expression(f_x, y, z) > > for x in xs > > given f_x = f(x) > > for y in ys(x) > > for z in zs(y)) > > > > whereas with :=, you can't. > > Is that legal syntax? You're splitting the "given" expression by > sticking the for clause in the middle of it. I don't think that will be > legal. It would be trying to split an ternary if: > > (true_expr for x in xs if condition else false_expr for y in ys) > > (true_expr if condition for x in xs else false_expr for y in ys) > > > You're right that it's a different proposal, but I thought it would be a natural extension to this proposal since at the start of this discussion someone mentioned how this could be accomplished with something like (expression for x in xs for f_x in [f(x)]) The regular given proposal is an extension of (I think) "expr" into something like expr ["given" test annassign] I think a natural extension of that is to add it to "testlist_comp" and "dictorsetmaker" so that given name = value can be used a cleaner synonym of something like for name in [value] But your'e right, it's a different proposal and I'm getting ahead of things. But whether legal or not, Neil, you went on at length about how > professionals don't write code like this and such overly dense > comprehensions are only fit for competitions. > > Now you want your cake and to eat it too: > > "given is better, because it doesn't allow the awful > unreadable code that := gives; oh, and it's also better, > because it allows *this* awful unreadable code that := > doesn't allow" > > > In any case, of course you can write this with := binding expression. > You just shouldn't do it: > > (expression(f_x, y, z) for x in xs > for y in ys(x) > for z in zs(y) > if (f_x := f(x)) or True) > ) > > > That's fine for mucking about, but I wouldn't do it for serious code. > Replacing the colon with "given f_x" doesn't change that. > > My thought is that if each component is simple enough *and* if each component can be reasoned about independently of the others, then it's fine. The issue I had with the := code you presented was that it was impossible to reason about the components independently from the whole. Ultimately, this is one of feelings. I think every one of us has a unique set of experiences, and those experiences led us to having different programming aesthetics. I used to write code one way, and then after lots of code reviews, my experience has made me write code a different way. Best, Neil > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/CFuqwmE8s-E/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at cskk.id.au Sun May 13 00:05:32 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 13 May 2018 14:05:32 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <20180513040532.GA38182@cskk.homeip.net> On 11May2018 11:40, Jacco van Dorp wrote: >[...] That all said, I would still prefer: > >if re.match(stuff) as m: > >which is exactly equal to the := in line length and parallels with. >While that may -technically- be a different beast. Yeah, this is my favorite also. Has the same feel and ordering as with, import and except. Could someone point me to a post which nicely describes the rationale behind its rejection? I'm sure there's one in the many in this discussion but I've not found it yet. Cheers, Cameron Simpson From cs at cskk.id.au Sun May 13 00:01:19 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 13 May 2018 14:01:19 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <20180513040119.GA64546@cskk.homeip.net> On 11May2018 11:45, Tim Peters wrote: >[Gustavo Carneiro] >>>> IMHO, all these toy examples don't translate well to the real world >>>> because they tend to use very short variable names while in real world [good >>>> written code] tends to select longer more descriptive variable names. > >[Greg Ewing] >>> I don't believe that's always true. It depends on the context. >>> Sometimes, using long variable names can make code *harder* >>> to read. >>> >>> I don't think there's anything unrealistic about this >>> example: >>> >>> if m given m = pattern.match(the_string): >>> nugget = m.group(2) >>> >>> Most people's short-term memory is good enough to remember >>> that "m" refers to the match object while they read the >>> next couple of lines. IMO, using a longer name would serve >>> no purpose and would just clutter things up. > >[Nick Coghlan] >> I've been thinking about this problem, and I think for the If/elif/while >> cases it's actually possible to allow the "binding is the same as the >> condition" case to be simplified to: >> >> if command = pattern.match(the_string): >> ... >> elif command = other_pattern.match(the_string): >> ... >> >> while data = read_data(): > >Unless there's some weird font problem on my machine, that looks like >a single "equals sign". In which case we'd be reproducing C's >miserable confusion about whether: > > if (i = 1) > >was a too-hastily-typed spelling of the intended: > > if (i == 1) > >or whether they were thinking "equals" and typed "=" by mistake. Sorry for the huge quote, but I'm also -1 on this. It reads beautifully, but we're straight into the = vs == area and that outweighs things IMO. >If so, that would get an instant -1 from any number of core devs, who >have vivid painful memories of being burned by that in C. That's not >just speculation - it came up a number of times in the PEP 572 >threads. Disclaimer: I'm not a core dev! Cheers, Cameron Simpson From rosuav at gmail.com Sun May 13 00:23:08 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 May 2018 14:23:08 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180513040532.GA38182@cskk.homeip.net> References: <20180513040532.GA38182@cskk.homeip.net> Message-ID: On Sun, May 13, 2018 at 2:05 PM, Cameron Simpson wrote: > On 11May2018 11:40, Jacco van Dorp wrote: >> >> [...] That all said, I would still prefer: >> >> if re.match(stuff) as m: >> >> which is exactly equal to the := in line length and parallels with. >> While that may -technically- be a different beast. > > > Yeah, this is my favorite also. Has the same feel and ordering as with, > import and except. > > Could someone point me to a post which nicely describes the rationale behind > its rejection? I'm sure there's one in the many in this discussion but I've > not found it yet. https://www.python.org/dev/peps/pep-0572/#special-casing-conditional-statements https://www.python.org/dev/peps/pep-0572/#alternative-spellings I'm not sure which version you're looking at, so there's the rejections of both. ChrisA From cs at cskk.id.au Sun May 13 00:58:20 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 13 May 2018 14:58:20 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: Message-ID: <20180513045820.GA81072@cskk.homeip.net> On 13May2018 14:23, Chris Angelico wrote: >On Sun, May 13, 2018 at 2:05 PM, Cameron Simpson wrote: >> Could someone point me to a post which nicely describes the rationale behind >> its rejection? I'm sure there's one in the many in this discussion but I've >> not found it yet. > >https://www.python.org/dev/peps/pep-0572/#special-casing-conditional-statements >https://www.python.org/dev/peps/pep-0572/#alternative-spellings > >I'm not sure which version you're looking at, so there's the rejections of >both. I meant the latter, but I'd already looked at that part of the PEP and found its explaination... unfulfilling. It says: EXPR as NAME: stuff = [[f(x) as y, x/y] for x in range(5)] Since EXPR as NAME already has meaning in except and with statements (with different semantics), this would create unnecessary confusion or require special-casing (eg to forbid assignment within the headers of these statements). All you need to disambiguate, say: with expr as expr_as as with_as: is to require parentheses for (expr as exp_as) if someone wanted that complications (assuming that is even necessary - it seems unambiguous to my eye already, unless the token lookahead requirement in Python's grammar prevents that. So I'd hoped for a post to the list discussing this aspect and outlining why it was considered unsatisfactory. Cheers, Cameron Simpson From rosuav at gmail.com Sun May 13 01:05:28 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 May 2018 15:05:28 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180513045820.GA81072@cskk.homeip.net> References: <20180513045820.GA81072@cskk.homeip.net> Message-ID: On Sun, May 13, 2018 at 2:58 PM, Cameron Simpson wrote: > On 13May2018 14:23, Chris Angelico wrote: >> >> On Sun, May 13, 2018 at 2:05 PM, Cameron Simpson wrote: >>> >>> Could someone point me to a post which nicely describes the rationale >>> behind >>> its rejection? I'm sure there's one in the many in this discussion but >>> I've >>> not found it yet. >> >> >> >> https://www.python.org/dev/peps/pep-0572/#special-casing-conditional-statements >> https://www.python.org/dev/peps/pep-0572/#alternative-spellings >> >> I'm not sure which version you're looking at, so there's the rejections of >> both. > > > I meant the latter, but I'd already looked at that part of the PEP and found > its explaination... unfulfilling. It says: > > EXPR as NAME: > > stuff = [[f(x) as y, x/y] for x in range(5)] > > Since EXPR as NAME already has meaning in except and with statements (with > different semantics), this would create unnecessary confusion or require > special-casing (eg to forbid assignment within the headers of these > statements). > > All you need to disambiguate, say: > > with expr as expr_as as with_as: > > is to require parentheses for (expr as exp_as) if someone wanted that > complications (assuming that is even necessary - it seems unambiguous to my > eye already, unless the token lookahead requirement in Python's grammar > prevents that. > > So I'd hoped for a post to the list discussing this aspect and outlining why > it was considered unsatisfactory. There were a large number of posts, so I can't really point to one of them. The problem isn't the double-as case that you describe; it's that these two are ALMOST identical: with expr as target: with (expr as target): In fact, they are functionally identical for many situations - "with (open(...) as target):" is exactly the same as the form without the parens. It'd make for data-dependent bugs, which are a really REALLY bad idea. ChrisA From tim.peters at gmail.com Sun May 13 02:41:08 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 13 May 2018 01:41:08 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: [Tim, suggests changes to the Reference Manual's 4.2.1] > """ > An assignment expression binds the target, except in a function F > synthesized to implement a list comprehension or generator expression > (see XXX). In the latter case, if the target is not in F's > environment (see section 4.2.2) , the target is bound in the block > containing F. > """ Let me try that again ;-) The notion of "environment" includes the global scope, but that's not really wanted here. "Environment" has more of a runtime flavor anyway. And since nobody will tell me anything about class scope, I read the docs myself ;-) And that's a problem, I think! If a comprehension C is in class scope S, apparently the class locals are _not_ in C's environment. Since C doesn't even have read access to S's locals, it seems to me bizarre that ":=" could _create_ a local in S. Since I personally couldn't care less about running comprehensions of any kind at class scope, I propose to make `:=` a SyntaxError if someone tries to use a comprehension with ':=' at class scope (of course they may be able to use ":=" in nested comprehensions anyway - not that anyone would). If someone objects to that, fine, you figure it out ;-) So here's another stab. """ An assignment expression binds the target, except in a function F synthesized to implement a list comprehension or generator expression (see XXX). In the latter case[1]: - If the target also appears as an identifier target of a `for` loop header in F, a `SyntaxError` exception is raised. - If the block containing F is a class block, a `SyntaxError` exception is raised. - If the target is not local to any function enclosing F, and is not declared `global` in the block containing F, then the target is bound in the block containing F. Footnote: [1] The intent is that runtime binding of the target occurs as if the binding were performed in the block containing F. Because that necessarily makes the target not local in F, it's an error if the target also appears in a `for` loop header, which is a local binding for the same target. If the containing block is a class block, F has no access to that block's scope, so it doesn't make sense to consider the containing block. If the target is already known to the containing block, the target inherits its scope resolution from the containing block. Else the target is established as local to the containing block. """ I realize the docs don't generally use bullet lists. Convert to WallOfText if you must. The material in the footnote would usually go in a "Rationale" doc instead, but we don't have one of those, and I think the intent is too hard to deduce without that info. And repeating the other point, to keep a self-contained account: > Earlier, for now-necessary disambiguation, I expect that in > > ... targets that are identifiers if occurring in an assignment, ... > > " statement" should be inserted before the comma. From ethan at stoneleaf.us Sun May 13 03:17:48 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 13 May 2018 00:17:48 -0700 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: <5AF7E69C.4040408@stoneleaf.us> On 05/12/2018 11:41 PM, Tim Peters wrote: > [Tim, suggests changes to the Reference Manual's 4.2.1] >> """ >> An assignment expression binds the target, except in a function F >> synthesized to implement a list comprehension or generator expression >> (see XXX). In the latter case, if the target is not in F's >> environment (see section 4.2.2) , the target is bound in the block >> containing F. >> """ > > Let me try that again ;-) The notion of "environment" includes the > global scope, but that's not really wanted here. "Environment" has > more of a runtime flavor anyway. And since nobody will tell me > anything about class scope, I read the docs myself ;-) > > And that's a problem, I think! If a comprehension C is in class scope > S, apparently the class locals are _not_ in C's environment. Since C > doesn't even have read access to S's locals, it seems to me bizarre > that ":=" could _create_ a local in S. Python 3.7.0b3+ (heads/bpo-33217-dirty:28c1790, Apr 5 2018, 13:10:10) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. --> class C: ... huh = 7 ... hah = [i for i in range(huh)] ... --> C.hah [0, 1, 2, 3, 4, 5, 6] Same results clear back to 3.3 (the oldest version of 3 I have). Are the docs wrong? Or maybe they just refer to functions: --> class C: ... huh = 7 ... hah = [i for i in range(huh)] ... heh = lambda: [i for i in range(huh)] ... --> C.hah [0, 1, 2, 3, 4, 5, 6] --> C.heh() Traceback (most recent call last): File "test_class_comp.py", line 7, in print(C.heh()) File "test_class_comp.py", line 4, in heh = lambda: [i for i in range(huh)] NameError: global name 'huh' is not defined So a class-scope comprehension assignment expression should behave as you originally specified. -- ~Ethan~ From rosuav at gmail.com Sun May 13 03:19:31 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 May 2018 17:19:31 +1000 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: <5AF7E69C.4040408@stoneleaf.us> References: <5AF7E69C.4040408@stoneleaf.us> Message-ID: On Sun, May 13, 2018 at 5:17 PM, Ethan Furman wrote: > On 05/12/2018 11:41 PM, Tim Peters wrote: >> >> [Tim, suggests changes to the Reference Manual's 4.2.1] >>> >>> """ >>> An assignment expression binds the target, except in a function F >>> synthesized to implement a list comprehension or generator expression >>> (see XXX). In the latter case, if the target is not in F's >>> environment (see section 4.2.2) , the target is bound in the block >>> containing F. >>> """ >> >> >> Let me try that again ;-) The notion of "environment" includes the >> global scope, but that's not really wanted here. "Environment" has >> more of a runtime flavor anyway. And since nobody will tell me >> anything about class scope, I read the docs myself ;-) >> >> And that's a problem, I think! If a comprehension C is in class scope >> S, apparently the class locals are _not_ in C's environment. Since C >> doesn't even have read access to S's locals, it seems to me bizarre >> that ":=" could _create_ a local in S. > > > Python 3.7.0b3+ (heads/bpo-33217-dirty:28c1790, Apr 5 2018, 13:10:10) > [GCC 4.8.2] on linux > Type "help", "copyright", "credits" or "license" for more information. > --> class C: > ... huh = 7 > ... hah = [i for i in range(huh)] > ... > --> C.hah > [0, 1, 2, 3, 4, 5, 6] > > Same results clear back to 3.3 (the oldest version of 3 I have). Are the > docs wrong? >>> class C: ... huh = 7 ... hah = [i for _ in [0] for i in range(huh)] ... Traceback (most recent call last): File "", line 1, in File "", line 3, in C File "", line 3, in NameError: name 'huh' is not defined The outermost iterable is a special case. Otherwise, you can't reference anything. ChrisA From peter.ed.oconnor at gmail.com Sun May 13 05:21:24 2018 From: peter.ed.oconnor at gmail.com (Peter O'Connor) Date: Sun, 13 May 2018 11:21:24 +0200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <23287.20613.651267.265792@turnbull.sk.tsukuba.ac.jp> References: <23287.20613.651267.265792@turnbull.sk.tsukuba.ac.jp> Message-ID: target := expr expr as target expr -> target target given target = expr let target = expr : target expr ; Although in general "target:=exp" seems the most palatable of these to me, there is one nice benefit to the "given" syntax: Suppose you have a comprehension wherein you want to pass forward an internal "state" between iterations, but not return it as the output: In today's python, you'd to: outputs = [] state = initial_state for inp in inputs: out, state = my_update_func(state) outputs.append(state) This could not be neatly compacted into: state = initial_state outputs = [out given out, state = my_update_func(inp, state) for inp in inputs] Or maybe: outputs = [out given out, state = my_update_func(inp, state) for inp in inputs given state=initial_state] Though I agree for the much more common case of assigning a value inline "x given x=y" seems messily redundant. On Sat, May 12, 2018 at 10:37 PM, Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > David Mertz writes: > > > Only the BDFL has a vote with non-zero weight. > > "Infinitesimal" != "zero". > > Pedantically yours, > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.ed.oconnor at gmail.com Sun May 13 05:23:07 2018 From: peter.ed.oconnor at gmail.com (Peter O'Connor) Date: Sun, 13 May 2018 11:23:07 +0200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <23287.20613.651267.265792@turnbull.sk.tsukuba.ac.jp> Message-ID: *Correction: Above code should read: outputs = [] state = initial_state for inp in inputs: out, state = my_update_func(inp, state) outputs.append(out) On Sun, May 13, 2018 at 11:21 AM, Peter O'Connor wrote: > target := expr > expr as target > expr -> target > target given target = expr > let target = expr > : target expr ; > > > Although in general "target:=exp" seems the most palatable of these to me, > there is one nice benefit to the "given" syntax: > > Suppose you have a comprehension wherein you want to pass forward an > internal "state" between iterations, but not return it as the output: > > In today's python, you'd to: > > outputs = [] > state = initial_state > for inp in inputs: > out, state = my_update_func(state) > outputs.append(state) > > This could not be neatly compacted into: > > state = initial_state > outputs = [out given out, state = my_update_func(inp, state) for inp > in inputs] > > Or maybe: > > outputs = [out given out, state = my_update_func(inp, state) for inp > in inputs given state=initial_state] > > Though I agree for the much more common case of assigning a value inline > "x given x=y" seems messily redundant. > > > > On Sat, May 12, 2018 at 10:37 PM, Stephen J. Turnbull < > turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > >> David Mertz writes: >> >> > Only the BDFL has a vote with non-zero weight. >> >> "Infinitesimal" != "zero". >> >> Pedantically yours, >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dickinsm at gmail.com Sun May 13 05:51:18 2018 From: dickinsm at gmail.com (Mark Dickinson) Date: Sun, 13 May 2018 10:51:18 +0100 Subject: [Python-ideas] "given" vs ":=" in list comprehensions In-Reply-To: References: Message-ID: On Sun, May 13, 2018 at 1:34 AM, Andre Roberge wrote: > First example: single temporary assignment, done four different ways. > > 1) using := > > real_roots = [ (-b/(2*a) + (D:= sqrt( (b/(2*a))**2 - c/a), -b/(2*a) - D) > for a in range(10) > for b in range(10) > for c in range(10) > if D >= 0] > Unless PEP 572 is doing something horribly magical, this doesn't look as though it should work at all, so it may not be a good target for comparisons with other syntax possibilities. I'd expect that the `D` name lookup in the `if D >= 0` clause would occur before the (D := ...) assignment in the target expression, resulting in an UnboundLocalError. (I tried to check out Chris's reference implementation branch to verify, but the compilation failed with a traceback.) And a random mathematical nitpick: as a non-NaN result of a square root operation, D will always satisfy D >= 0; for this use-case we want to check the *argument* to the `sqrt` call for nonnegativity, rather than the *result*. So I think the target statement for the comparison with other syntaxes should look something like this instead: real_roots = [ (-b/(2*a) + sqrt(D), -b/(2*a) - sqrt(D)) for a in range(1, 10) # start at 1 to avoid division by zero for b in range(10) for c in range(10) if (D := (b/(2*a))**2 - c/a) >= 0 ] Or possibly like this, using an extra assignment expression to avoid the repeated computation of the square root: real_roots = [ (-b/(2*a) + (s := sqrt(D)), -b/(2*a) - s) for a in range(1, 10) for b in range(10) for c in range(10) if (D := (b/(2*a))**2 - c/a) >= 0 ] Similar order-of-evaluation issues apply to example (8), and to the other examples based on hypothetical syntax, depending on exactly how that syntax is hypothesised to work. -- Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Sun May 13 05:55:33 2018 From: srkunze at mail.de (Sven R. Kunze) Date: Sun, 13 May 2018 11:55:33 +0200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <23287.20613.651267.265792@turnbull.sk.tsukuba.ac.jp> Message-ID: <7755d68f-8484-1608-b9eb-5d93ccee4a94@mail.de> On 13.05.2018 11:23, Peter O'Connor wrote: > ?*Correction: Above code should read: > > ? ? outputs = [] > state = initial_state > for inp in inputs: > out, state = my_update_func(inp, state) > outputs.append(out) > Question still stands if this type of code needs compaction in the first place? List comprehensions usually have some sort of declarative touch (set builder notation). Even though, striving for a more compacted version, I tend to think that using a declarative version of it doesn't serve it well in the long term. We recently came across the following code snippet in our source base (1st answer of https://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-whilst-preserving-order). It was absolutely not comprehensible. Your example is inherently imperative because the internal state changes from iteration to iteration; something unusual for set builder notation. Regards, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Sun May 13 06:45:29 2018 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 13 May 2018 06:45:29 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180513045820.GA81072@cskk.homeip.net> Message-ID: <54daa379-bebc-23df-3cf7-dbd52c6627b6@trueblade.com> On 5/13/18 1:05 AM, Chris Angelico wrote: > On Sun, May 13, 2018 at 2:58 PM, Cameron Simpson wrote: >> On 13May2018 14:23, Chris Angelico wrote: >>> >>> On Sun, May 13, 2018 at 2:05 PM, Cameron Simpson wrote: >>>> >>>> Could someone point me to a post which nicely describes the rationale >>>> behind >>>> its rejection? I'm sure there's one in the many in this discussion but >>>> I've >>>> not found it yet. >>> >>> >>> >>> https://www.python.org/dev/peps/pep-0572/#special-casing-conditional-statements >>> https://www.python.org/dev/peps/pep-0572/#alternative-spellings >>> >>> I'm not sure which version you're looking at, so there's the rejections of >>> both. >> >> >> I meant the latter, but I'd already looked at that part of the PEP and found >> its explaination... unfulfilling. It says: >> >> EXPR as NAME: >> >> stuff = [[f(x) as y, x/y] for x in range(5)] >> >> Since EXPR as NAME already has meaning in except and with statements (with >> different semantics), this would create unnecessary confusion or require >> special-casing (eg to forbid assignment within the headers of these >> statements). >> >> All you need to disambiguate, say: >> >> with expr as expr_as as with_as: >> >> is to require parentheses for (expr as exp_as) if someone wanted that >> complications (assuming that is even necessary - it seems unambiguous to my >> eye already, unless the token lookahead requirement in Python's grammar >> prevents that. >> >> So I'd hoped for a post to the list discussing this aspect and outlining why >> it was considered unsatisfactory. > > There were a large number of posts, so I can't really point to one of > them. The problem isn't the double-as case that you describe; it's > that these two are ALMOST identical: > > with expr as target: > with (expr as target): > > In fact, they are functionally identical for many situations - "with > (open(...) as target):" is exactly the same as the form without the > parens. It'd make for data-dependent bugs, which are a really REALLY > bad idea. A little more detail: The first one is partially: target = expr.__enter__() The second one is partially: target = expr For many objects, obj.__enter__() just returns obj, it often looks like these two statements do the same thing, but they do not. Chris's concern, which I share, is that users wouldn't know why these are different, and why the second on works for some objects but not others. I agree the PEP could use more detail in explaining this particular issue. Eric From steve at pearwood.info Sun May 13 07:23:48 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 13 May 2018 21:23:48 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512210716.GA34320@cskk.homeip.net> <20180512212420.GA77042@cskk.homeip.net> Message-ID: <20180513112336.GJ12683@ando.pearwood.info> On Sat, May 12, 2018 at 07:36:33PM -0400, Juancarlo A?ez wrote: > Python already uses "in", which is used in other languages to introduce > context. Fortunately, we don't have to come up with syntax that works with other languages, only Python. > The statement structure of "with...as" seems desirable But it's not a statement, its an expression. > just asking for a word that is not "with" or "given". How about "antidisestablishmentarianism"? That's unlikely to be used in many programs, so we could make it a keyword. *wink* I jest, of course. But I don't think "with" reads well, and given doesn't really work for me either *as prose*. In my experience mathematicians put the given *before* the statement: Given a, b, c three sides of a triangle, then Area = sqrt(s*(s-a)*(s-b)*(s-c)) where s = (a + b + c)/2 is the semi-perimeter of the triangle. For the record, that is almost exactly what I wrote for a student earlier today, and its not just me, it is very similar to the wording used on both Wolfram Mathworld and Wikipedia's pages on Heron's Formula. http://mathworld.wolfram.com/HeronsFormula.html https://en.wikipedia.org/wiki/Heron%27s_formula Putting "given" after the expression is backwards. -- Steve From steve at pearwood.info Sun May 13 07:32:55 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 13 May 2018 21:32:55 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512210716.GA34320@cskk.homeip.net> <274b99a3-e8c2-a412-3870-8b0daed2bbe0@mrabarnett.plus.com> Message-ID: <20180513113255.GL12683@ando.pearwood.info> On Sat, May 12, 2018 at 08:27:53PM -0400, Juancarlo A?ez wrote: > > if (diff := x - x_base) and (g := gcd(diff, n)) > 1: > > return g > > > > > I don't see the advantage in that succinctness: > > g = special_gcd(x - x_base, n) > > if g: > > return g > > > The code bases I work on constantly move towards having the next guy grok > what's going on just by reading the code. That's an excellent point. What's "special_gcd" and how does it differ from normal gcd? How am I supposed to grok that just from reading the code above? Do I have to dig into the source of special_gcd to understand it? What happens if the normal gcd would return zero? Is that going to lead to a bug? > It could also be: > > if special_gcd(x - x_base, n) as g: > return g No it can't, because "as" is not an option. -- Steven From steve at pearwood.info Sun May 13 07:28:16 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 13 May 2018 21:28:16 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180513005350.GA67965@cskk.homeip.net> References: <20180513005350.GA67965@cskk.homeip.net> Message-ID: <20180513112815.GK12683@ando.pearwood.info> On Sun, May 13, 2018 at 10:53:50AM +1000, Cameron Simpson wrote: > For myself, I'm not a fan of a narrow scope. I'd be happy with an inline > assignment landing in the function scope (or whatever the current innermost > scope is). Like normal assignments. In how many other places does Python > adopt a narrower scope than the function/method/class/module? None except for comprehensions and generators. And may it stay that way :-) > I'm going to digress into an explaination of this issue in Go here because > I want to highlight my dislike of easy accidental shadowing, which is a > problem in many areas, and I think that conflating inline assignment with > an implied narrow scope in Python would make this worse. [...] Thanks Cameron! This is a really good practical example of why narrow scopes should be avoided. -- Steve From greg.ewing at canterbury.ac.nz Sun May 13 08:31:24 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 14 May 2018 00:31:24 +1200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180513112336.GJ12683@ando.pearwood.info> References: <20180512210716.GA34320@cskk.homeip.net> <20180512212420.GA77042@cskk.homeip.net> <20180513112336.GJ12683@ando.pearwood.info> Message-ID: <5AF8301C.2090600@canterbury.ac.nz> Steven D'Aprano wrote: > In my experience mathematicians put the given *before* the statement: > > Given a, b, c three sides of a triangle, then > > Area = sqrt(s*(s-a)*(s-b)*(s-c)) > > where s = (a + b + c)/2 is the semi-perimeter of the triangle. I agree, and "where" would be my first choice, but... given... that "where" is not available, "given" seems like the next best choice. At least it doesn't sound blatantly wrong in that position. -- Greg From rob.cliffe at btinternet.com Sun May 13 08:55:48 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sun, 13 May 2018 13:55:48 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512210716.GA34320@cskk.homeip.net> Message-ID: <24fe2ef7-1ede-7212-c2e0-914e69ca4d90@btinternet.com> On 12/05/2018 23:52, Neil Girdhar wrote: > > > On Sat, May 12, 2018 at 5:54 PM Cameron Simpson > wrote: > > On 06May2018 02:00, Nick Coghlan > wrote: > >On 5 May 2018 at 13:36, Tim Peters > wrote: > >> If only one trailing "given" clause can be given per `if` test > >> expression, presumably I couldn't do that without trickery. > > > >I was actually thinking that if we did want to allow multiple > assignments, > >and we limited targets to single names, we could just use a comma > as a > >separator: > > > >? ? if diff and g > 1 given diff = x - x_base, g = gcd(diff, n): > >? ? ? ? return g > > > >Similar to import statements, optional parentheses could be > included in the > >grammar, allowing the name bindings to be split across multiple > lines: > > > >? ? if diff and g > 1 given ( > >? ? ? ? diff = x - x_base, > >? ? ? ? g = gcd(diff, n), > >? ? ): > >? ? ? ? return g > > I'm well behind, but... this! This turns "given" into a +0.8 for me. > > That's really nice. It reads clearly too. > > I was hitherto in the "expression as name" camp, which I gather is > already > rejected. > > > I love given, but that's the one thing I don't like.? I prefer this: > ? ? if (diff and g > 1 > ? ? ? ? ? ? given diff = x - x_base > ? ? ? ? ? ? given g = gcd(diff, n)): > ? ? ? ? return g > I don't like the consecutive "given"s.? Reading it aloud in English suggests to me that the second "given" should be evaluated before the first, which I'm sure is not the intention. (Just as I think that multiple for-loops inside a comprehension sound the wrong way round. :-( But that ship has sailed.) Rob Cliffe > ?just like for and if subexpressions.? ?Doing this can also open up > weirdness if someone tries to roll something like: > > a = f(),? # Make a tuple of length 1 > > into a given statement.? Now, where do you up the parentheses? > > given ( > ? ? a = (f(),), > ? ? b = whatever? > ) > > Seems weird. > > > Cheers, > Cameron Simpson > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in > the Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/CFuqwmE8s-E/unsubscribe. > To unsubscribe from this group and all its topics, send an email > to python-ideas+unsubscribe at googlegroups.com > . > For more options, visit https://groups.google.com/d/optout. > > > > Virus-free. www.avg.com > > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Sun May 13 10:20:14 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Sun, 13 May 2018 10:20:14 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180513113255.GL12683@ando.pearwood.info> References: <20180512210716.GA34320@cskk.homeip.net> <274b99a3-e8c2-a412-3870-8b0daed2bbe0@mrabarnett.plus.com> <20180513113255.GL12683@ando.pearwood.info> Message-ID: > That's an excellent point. What's "special_gcd" and how does it differ > from normal gcd? How am I supposed to grok that just from reading the > code above? > > Do I have to dig into the source of special_gcd to understand it? > > One would usually define functions like special_gcd() inline, right before it is first used. -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun May 13 10:55:03 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 14 May 2018 00:55:03 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <23287.20613.651267.265792@turnbull.sk.tsukuba.ac.jp> <20180513034741.GI12683@ando.pearwood.info> Message-ID: <20180513145459.GM12683@ando.pearwood.info> On Sun, May 13, 2018 at 12:10:08AM -0400, Neil Girdhar wrote: > > > Suppose you have a generator like > > > > > > (expression(f(x), y, z) > > > for x in xs > > > for y in ys(x) > > > for z in zs(y)) > > > > > > With given notation you can optimize: Actually, as you admitted, you can't, since this isn't part of the proposed syntax or semantics. But let's put that aside. > > > (expression(f_x, y, z) > > > for x in xs > > > given f_x = f(x) > > > for y in ys(x) > > > for z in zs(y)) > > > > > > whereas with :=, you can't. Except of course you can: > > (expression(f_x, y, z) for x in xs > > for y in ys(x) > > for z in zs(y) > > if (f_x := f(x)) or True) > > ) [Neil] > My thought is that if each component is simple enough *and* if each > component can be reasoned about independently of the others, then it's > fine. The issue I had with the := code you presented was that it was > impossible to reason about the components independently from the whole. I said it was rubbish code which I wouldn't use for serious work. But "impossible" to reason about? That's pretty strong, and definite, words for something which is actually pretty easy to reason about. (expression(f_x, y, z) for x in xs for y in ys(x) for z in zs(y) if (f_x := f(x)) or True) ) I'm pretty sure you can reason about "expression(f_x, y, z)" on its own. After all, the above code was (apart from the := line) your own example. If you can't even reason about your own examples, you're in a bad place. I think you can reason about the three for-loops "for x in xs" etc on their own. Again, they were your idea in the first place. I expect you can reason about "if or True" on its own. It's a pretty basic Python technique, to call for its side-effect without caring about its return value. Not something I would use for serious work, but this is YOUR example, not mine, so if you hate this generator expression, you were the one who suggested it. Only the is new or different from ordinary Python code that works now: f_x := f(x) The semantics of := are pretty damn simple (at least from the high-level overview without worrying about precedence or possible scoping issues): - evaluate f(x) - assign that value to f_x - return the value of f_x which is EXACTLY THE SAME SEMANTICS OF "f_x given f_x = f(x)": - evaluate f(x) - assign that value to f_x - return the value of f_x And you would reason about them the same way. -- Steve From mistersheik at gmail.com Sun May 13 11:08:31 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Sun, 13 May 2018 11:08:31 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180513145459.GM12683@ando.pearwood.info> References: <23287.20613.651267.265792@turnbull.sk.tsukuba.ac.jp> <20180513034741.GI12683@ando.pearwood.info> <20180513145459.GM12683@ando.pearwood.info> Message-ID: On Sun, May 13, 2018 at 10:56 AM Steven D'Aprano wrote: > On Sun, May 13, 2018 at 12:10:08AM -0400, Neil Girdhar wrote: > > > > > Suppose you have a generator like > > > > > > > > (expression(f(x), y, z) > > > > for x in xs > > > > for y in ys(x) > > > > for z in zs(y)) > > > > > > > > With given notation you can optimize: > > Actually, as you admitted, you can't, since this isn't part of > the proposed syntax or semantics. But let's put that aside. > > > > > (expression(f_x, y, z) > > > > for x in xs > > > > given f_x = f(x) > > > > for y in ys(x) > > > > for z in zs(y)) > > > > > > > > whereas with :=, you can't. > > Except of course you can: > > > > (expression(f_x, y, z) for x in xs > > > for y in ys(x) > > > for z in zs(y) > > > if (f_x := f(x)) or True) > > > ) > > [Neil] > > My thought is that if each component is simple enough *and* if each > > component can be reasoned about independently of the others, then it's > > fine. The issue I had with the := code you presented was that it was > > impossible to reason about the components independently from the whole. > > I said it was rubbish code which I wouldn't use for serious work. But > "impossible" to reason about? That's pretty strong, and definite, words > for something which is actually pretty easy to reason about. > I was talking about this snippet you wrote: smooth_signal = [(average := (1-decay)*average + decay*x) for x in signal] Not impossible to reason about, but not pretty either. > > (expression(f_x, y, z) for x in xs > for y in ys(x) > for z in zs(y) > if (f_x := f(x)) or True) > ) > > Not that it matters, but this doesn't work in general since this relies on f_x being evaluable to bool. You can always just write "for f_x in [f(x)]". This use of given was more of an aside. > I'm pretty sure you can reason about "expression(f_x, y, z)" on its own. > After all, the above code was (apart from the := line) your own example. > If you can't even reason about your own examples, you're in a bad place. > > I think you can reason about the three for-loops "for x in xs" etc on > their own. Again, they were your idea in the first place. > > I expect you can reason about "if or True" on its own. It's a > pretty basic Python technique, to call for its side-effect > without caring about its return value. > > Not something I would use for serious work, but this is YOUR example, > not mine, so if you hate this generator expression, you were the one who > suggested it. > > Only the is new or different from ordinary Python code that > works now: > > f_x := f(x) > > The semantics of := are pretty damn simple (at least from the high-level > overview without worrying about precedence or possible scoping issues): > > - evaluate f(x) > - assign that value to f_x > - return the value of f_x > > which is EXACTLY THE SAME SEMANTICS OF "f_x given f_x = f(x)": > > - evaluate f(x) > - assign that value to f_x > - return the value of f_x > > And you would reason about them the same way. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/CFuqwmE8s-E/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adelfino at gmail.com Sun May 13 11:17:17 2018 From: adelfino at gmail.com (=?UTF-8?Q?Andr=C3=A9s_Delfino?=) Date: Sun, 13 May 2018 12:17:17 -0300 Subject: [Python-ideas] Fwd: Pattern Matching Syntax Message-ID: Is some sense, yes, but await isn't part of the "async def" structure. await is just an independent statement, related to async def, yes, but independent. My proposal links match with case/else statements as a single flow control structure, and it writes them as such, like if/for/while/try. My proposal didn't *require* a suite to be multi-line, it just allowed it. I do see that it doesn't make sense for it to be multi-line, though. Perhaps multi-line can be a SyntaxError? I too think, like you, that this doesn't feel 100% pythonic either, but I felt that it was... less unpythonic? that the proposals seen until now. On Sat, May 12, 2018 at 3:03 PM, wrote: > In some sense async and await are like this group you describe, the > keyword "await" doesn't have meaning outside a function annotated with > "async". > > The issue I'd have with your proposal is that it requires the "suite" to > be an expression-based multi-line statement, ie. where the last statement > becomes the value that is matched against. Similar to the original proposal > that started this thread: these sorts of expression-based constructs don't > feel that natural for Python. > > On Saturday, May 12, 2018 at 2:50:49 PM UTC, Andr?s Delfino wrote: >> >> I find it weird for case statements to be "inside" match statements. >> There's isn't a statement "group" that works this way right now, AFAIK. >> >> This would also be weird: >> >> match X: >> case Y: >> ... >> >> I thought a form based on try would make more coherent: >> >> match: >> suite >> case x: >> suite1 >> else: >> suite2 >> >> suite would be executed, and the last expression would be checked against >> each case. If no matching case is found, suite2 would be executed. >> >> Does it make sense? >> >> On Sat, May 12, 2018 at 8:51 AM, Steven Heidel >> wrote: >> >>> Great! Also emailed you with logistics. >>> >>> On Sat, May 12, 2018, 01:43 Jelle Zijlstra wrote: >>> >>>> 2018-05-11 22:01 GMT-04:00 Robert Roskam : >>>> >>>>> Hey Steven, >>>>> >>>>> I'm also at PyCon. Shall we take this off list and attempt to meet up >>>>> and discuss? >>>>> >>>>> I'm also at PyCon and interested in meeting about this. I just wrote >>>> up a basic and incomplete implementation for pattern-matching yesterday >>>> between and after: talks: https://github.com/Jell >>>> eZijlstra/cpython/blob/matchcase/Lib/test/test_matching.py. It's >>>> nowhere near complete, but an implementation like this can help inform what >>>> the syntax should look like. >>>> >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "python-ideas" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>> pic/python-ideas/nqW2_-kKrNg/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> python-ideas... at googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python... at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "python-ideas" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>> pic/python-ideas/nqW2_-kKrNg/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> python-ideas... at googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python... at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun May 13 12:04:31 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 14 May 2018 02:04:31 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512182735.GG12683@ando.pearwood.info> Message-ID: <20180513160431.GN12683@ando.pearwood.info> On Sat, May 12, 2018 at 08:06:02PM -0400, Neil Girdhar wrote: > On Sat, May 12, 2018 at 2:28 PM Steven D'Aprano wrote: > > > * there are no difficult mental questions about evaluation order, e.g., > > > in a bracketed expression having multiple assignments. > > > > Aren't there just? > > > > x = 1 > > print( x + x given x = 50 ) > > > > > > Will that print 100, or 51? Brackets ought to make it clearer: > > > > (x + x given x = 50) # this ought to return 100 > > x + (x given x = 50) # this ought to return 51 > > > > but if I leave the brackets out, I have no idea what I'll get. > > > > It has to be 100, or else outer parentheses change how an expression is > evaluated. Whether that it right or wrong, you're missing the point. You said that we don't have to care about evaluation order. But we do: even if your analysis is correct, not everyone will realise it. I didn't. And I still don't, because I think your analyse is wrong. "Outer parentheses" is a red herring. I should have written: (x + x) given x = 50 # this ought to return 100 x + (x given x = 50) # this ought to return 51 and now there are no outer parentheses to worry about. The question is now whether "given" binds more tightly than + or not. We get to choose the precedence, and whatever we choose, it will surprise (or annoy) some people, and more importantly, some people simply won't know, and will have to ponder the difficult question of what the evaluation order is. This is already true today with code that has side-effects. I can simulate "given" using exec. It only works in the module scope: # another_expr given target = expr exec("target = expr") or another_expr but now we can see how precedence makes a real difference: x = 1 x + (exec("x = 50") or x) versus: x = 1 exec("x = 50) or (x + x) Regardless of which behaviour we choose, some people won't know it and will have to deal with "difficult mental questions about evaluation order". That's unavoidable, regardless of how we spell it, := or "given". -- Steve From barry at barrys-emacs.org Sun May 13 12:08:05 2018 From: barry at barrys-emacs.org (Barry Scott) Date: Sun, 13 May 2018 17:08:05 +0100 Subject: [Python-ideas] keywords and backward compatibility Message-ID: <455E64D6-6BF7-4E77-8CDB-F2FB384F0E2D@barrys-emacs.org> There seems to be a recurring theme of ideas wanting a new keyword but not being able to have one because of backwards compatibility. What if I have a way to ask for the new keyword? Would that make it easier to accept and implement some of the new ideas? The "given" keyword for example could be allow with a statement like: from __future__ import keyword_given If this method of extending the python language was used for a while I can see that the downside would be that an author of a new module will end up potentially putting a lot of from __future__ statements as a preamble. Barry From rosuav at gmail.com Sun May 13 12:58:46 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 14 May 2018 02:58:46 +1000 Subject: [Python-ideas] keywords and backward compatibility In-Reply-To: <455E64D6-6BF7-4E77-8CDB-F2FB384F0E2D@barrys-emacs.org> References: <455E64D6-6BF7-4E77-8CDB-F2FB384F0E2D@barrys-emacs.org> Message-ID: On Mon, May 14, 2018 at 2:08 AM, Barry Scott wrote: > There seems to be a recurring theme of ideas wanting a new keyword > but not being able to have one because of backwards compatibility. > > What if I have a way to ask for the new keyword? Would that make it > easier to accept and implement some of the new ideas? > > The "given" keyword for example could be allow with a statement like: > > from __future__ import keyword_given > > If this method of extending the python language was used for a while > I can see that the downside would be that an author of a new module > will end up potentially putting a lot of from __future__ statements as a > preamble. A future import always [1] has a target version beyond which its effect becomes automatic. For instance, "from __future__ import generator_stop" means that StopIteration leaking out of a generator function is an error, but starting with Python 3.7, the altered behaviour happens whether you have the directive or not. So a future import does not replace backward compatibility considerations; it simply allows the feature to be brought in progressively across several versions. The main problem with creating keywords is that APIs get broken. Let's suppose that I have a module which has "def given(...)" in it. If you import my module and try to call that function, you have to go to some crazy shenanigans to avoid writing "mymodule.given()", which isn't going to work. The point of a future directive is to make changes on a module-by-module basis; that's fine for some ("division" changes the byte code created for the slash operator, "print_function" changes the way the word 'print' is compiled, "generator_stop" flags the generator function), but not everything can be implemented that way. Creating a new keyword is still a high cost. ChrisA [1] Except for "barry_as_FLUFL", the April Fools one, although even it has one, technically. Kinda. From tim.peters at gmail.com Sun May 13 13:04:41 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 13 May 2018 12:04:41 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: <5AF7E69C.4040408@stoneleaf.us> References: <5AF7E69C.4040408@stoneleaf.us> Message-ID: [Tim[ >> ... If a comprehension C is in class scope S, apparently the >> class locals are _not_ in C's environment. Since C doesn't >>> even have read access to S's locals, it seems to me bizarre >> that ":=" could _create_ a local in S. [Ethan Furman ] > Python 3.7.0b3+ (heads/bpo-33217-dirty:28c1790, Apr 5 2018, 13:10:10) > [GCC 4.8.2] on linux > Type "help", "copyright", "credits" or "license" for more information. > --> class C: > ... huh = 7 > ... hah = [i for i in range(huh)] > ... > --> C.hah > [0, 1, 2, 3, 4, 5, 6] > > Same results clear back to 3.3 (the oldest version of 3 I have). Are the > docs wrong? > > Or maybe they just refer to functions: > > --> class C: > ... huh = 7 > ... hah = [i for i in range(huh)] > ... heh = lambda: [i for i in range(huh)] > ... > --> C.hah > [0, 1, 2, 3, 4, 5, 6] > --> C.heh() > Traceback (most recent call last): > File "test_class_comp.py", line 7, in > print(C.heh()) > File "test_class_comp.py", line 4, in > heh = lambda: [i for i in range(huh)] > NameError: global name 'huh' is not defined As Chris already explained (thanks!), the expression defining the iterable for the outermost `for` (which, perhaps confusingly, is the _leftmost_ `for`) is treated specially in a comprehension (or genexp), evaluated at once _in_ the scope containing the comprehension, not in the comprehension's own scope. Everything else in the comprehension is evaluated in the comprehension's scope. I just want to add that it's really the same thing as your lambda example. Comprehensions are also implemented as lambdas (functions), but invisible functions created by magic. The synthesized function takes one argument, which is the expression defining the iterable for the outermost `for`. So, skipping irrelevant-to-the-point details, your original example is more like: class C: huh = 7 def _magic(it): return [i for i in it] hah = _magic(range(huh)) Since the `range(huh)` part is evaluated _in_ C's scope, no problem. For a case that blows up, as Chris did you can add another `for` as "outermost", or just try to reference a class local in the body of the comprehension: class C2: huh = 7 hah = [huh for i in range(5)] That blows up (NameError on `huh`) for the same reason your lambda example blows up, because it's implemented like: class C: huh = 7 def _magic(it): return [huh for i in it] hah = _magic(range(5)) and C's locals are not in the environment seen by any function called from C's scope. A primary intent of the proposed ":= in comprehensions" change is that you _don't_ have to learn this much about implementation cruft to guess what a comprehension will do when it contains an assignment expression. The intent of total = 0 sums = [total := total + value for value in data] is obvious - until you think too much about it ;-) Because there's no function in sight, there's no reason to guess that the `total` in `total = 0` has nothing to do with the instances of `total` inside the comprehension. The point of the change is to make them all refer to the same thing, as they already do in (the syntactically similar, but useless): total = 0 sums = [total == total + value for value in data] Except even _that_ doesn't work "as visually expected" in class scope today. The `total` inside the comprehension refers to the closest (if any) scope (_containing_ the `class` statement) in which `total` is local (usually the module scope, but may be a function scope if the `class` is inside nested functions). In function and module scopes, the second `total` example does work in "the obvious" way, so in those scopes I'd like to see the first `total` example do so too. From rosuav at gmail.com Sun May 13 13:21:18 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 14 May 2018 03:21:18 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512210716.GA34320@cskk.homeip.net> <274b99a3-e8c2-a412-3870-8b0daed2bbe0@mrabarnett.plus.com> <20180513113255.GL12683@ando.pearwood.info> Message-ID: On Mon, May 14, 2018 at 12:20 AM, Juancarlo A?ez wrote: > >> That's an excellent point. What's "special_gcd" and how does it differ >> from normal gcd? How am I supposed to grok that just from reading the >> code above? >> >> Do I have to dig into the source of special_gcd to understand it? >> > > > One would usually define functions like special_gcd() inline, right before > it is first used. Okay, then. What happens to the succinctness? On Sun, May 13, 2018 at 10:27 AM, Juancarlo A?ez wrote: > >> if (diff := x - x_base) and (g := gcd(diff, n)) > 1: >> return g > > I don't see the advantage in that succinctness: > > g = special_gcd(x - x_base, n) > if g: > return g It's actually this: def special_gcd(diff, n): return diff and gcd(diff, n) g = special_gcd(x - x_base, n) if g: return g Yes, very succinct. You can't have it both ways; either you need a name that completely defines the function, such that it can be placed out-of-line without needing to be read; or you're just creating an inline helper function, which doesn't shorten your code at all, and just adds another layer of wrapping around everything. Remember, you're contrasting with one of two options: either a nested if, or an inline 'and' with the ability to capture values. What you've created here is a multi-line way of capturing a value. That's all. ChrisA From mistersheik at gmail.com Sun May 13 13:39:44 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Sun, 13 May 2018 13:39:44 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180513160431.GN12683@ando.pearwood.info> References: <20180512182735.GG12683@ando.pearwood.info> <20180513160431.GN12683@ando.pearwood.info> Message-ID: On Sun, May 13, 2018 at 12:06 PM Steven D'Aprano wrote: > On Sat, May 12, 2018 at 08:06:02PM -0400, Neil Girdhar wrote: > > On Sat, May 12, 2018 at 2:28 PM Steven D'Aprano > wrote: > > > > > * there are no difficult mental questions about evaluation order, > e.g., > > > > in a bracketed expression having multiple assignments. > > > > > > Aren't there just? > > > > > > x = 1 > > > print( x + x given x = 50 ) > > > > > > > > > Will that print 100, or 51? Brackets ought to make it clearer: > > > > > > (x + x given x = 50) # this ought to return 100 > > > x + (x given x = 50) # this ought to return 51 > > > > > > but if I leave the brackets out, I have no idea what I'll get. > > > > > > > It has to be 100, or else outer parentheses change how an expression is > > evaluated. > > Whether that it right or wrong, you're missing the point. You said that > we don't have to care about evaluation order. But we do: even if your > analysis is correct, not everyone will realise it. I didn't. > I think the way to think about "given" is the same way you think about "for" and "if" clauses: (x + x for x in it) (x + x if condition else y) (x + x where x = f(y)) > > And I still don't, because I think your analyse is wrong. > > "Outer parentheses" is a red herring. I should have written: > > (x + x) given x = 50 # this ought to return 100 > x + (x given x = 50) # this ought to return 51 > > and now there are no outer parentheses to worry about. The question is > now whether "given" binds more tightly than + or not. > > We get to choose the precedence, and whatever we choose, it will > surprise (or annoy) some people, and more importantly, some people > simply won't know, and will have to ponder the difficult question of > what the evaluation order is. > This is already true today with code that has side-effects. I can > simulate "given" using exec. It only works in the module scope: > > # another_expr given target = expr > exec("target = expr") or another_expr > > but now we can see how precedence makes a real difference: > > x = 1 > x + (exec("x = 50") or x) > > versus: > > x = 1 > exec("x = 50) or (x + x) > > > Regardless of which behaviour we choose, some people won't know it and > will have to deal with "difficult mental questions about evaluation > order". > > That's unavoidable, regardless of how we spell it, := or "given". > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/CFuqwmE8s-E/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun May 13 14:19:59 2018 From: guido at python.org (Guido van Rossum) Date: Sun, 13 May 2018 14:19:59 -0400 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions Message-ID: As anyone still following the inline assignment discussion knows, a problem with designing new syntax is that it's hard to introduce new keywords into the language, since all the nice words seem to be used as method names in popular packages. (E.g. we can't use 'where' because there's numpy.where , and we can't use 'given' because it's used in Hypothesis .) The idea I had (not for the first time :-) is that in many syntactic positions we could just treat keywords as names, and that would free up these keywords. For example, we could allow keywords after 'def' and after a period, and then the following would become legal: class C: def and(self, other): return ... a = C() b = C() print(a.and(b)) This does not create syntactic ambiguities because after 'def' and after a period the grammar *always* requires a NAME. There are other positions where we could perhaps allow this, e.g. in a decorator, immediately after '@' (the only keyword that's *syntactically* legal here is 'not', though I'm not sure it would ever be useful). Of course this would still not help for names of functions that might be imported directly (do people write 'from numpy import where'?). And it would probably cause certain typos be harder to diagnose. I should also mention that this was inspired from some messages where Tim Peters berated the fashion of using "reserved words", waxing nostalgically about the old days of Fortran (sorry, FORTRAN), which doesn't (didn't?) have reserved words at all (nor significant whitespace, apart from the "start in column 7" rule). Anyway, just throwing this out. Please tear it apart! -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Sun May 13 14:27:38 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Sun, 13 May 2018 11:27:38 -0700 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180513112336.GJ12683@ando.pearwood.info> References: <20180512210716.GA34320@cskk.homeip.net> <20180512212420.GA77042@cskk.homeip.net> <20180513112336.GJ12683@ando.pearwood.info> Message-ID: <5AF8839A.2060408@brenbarn.net> On 2018-05-13 04:23, Steven D'Aprano wrote: > In my experience mathematicians put the given *before* the statement: > > Given a, b, c three sides of a triangle, then > > Area = sqrt(s*(s-a)*(s-b)*(s-c)) > > where s = (a + b + c)/2 is the semi-perimeter of the triangle. > > For the record, that is almost exactly what I wrote for a student > earlier today, and its not just me, it is very similar to the wording > used on both Wolfram Mathworld and Wikipedia's pages on Heron's Formula. > > http://mathworld.wolfram.com/HeronsFormula.html > > https://en.wikipedia.org/wiki/Heron%27s_formula > > > Putting "given" after the expression is backwards. Yes, but that's because we're ruling out the use of "where". At this point I would be fine with "snicklefritz" as the keyword. The point is that I want to put SOMETHING after the expression, and this is not at all unusual. See for instance Wikipedia pages on the Reimann zeta function (https://en.wikipedia.org/wiki/Riemann_zeta_function#Definition), gravitation equation (https://en.wikipedia.org/wiki/Gravity#Newton%27s_theory_of_gravitation), and compound interest (https://en.wikipedia.org/wiki/Compound_interest#Mathematics_of_interest_rate_on_loans). If we have to use the word "given" even though the word mathematicians would use in that position is "where", that's not such a big deal. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From eltrhn at gmail.com Sun May 13 14:42:03 2018 From: eltrhn at gmail.com (Elias Tarhini) Date: Sun, 13 May 2018 11:42:03 -0700 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: I like it! The obvious question, though: How would "*from package import keyword*" be handled, if not simply by SyntaxError? Would *from package import keyword as keyword_* be allowed? In a similar vein, what would happen with stdlib functions like operator.not_? The thought of writing "operator.not" is appealing, but being forced to use *that* (with *from operator import not* being non-allowable) may not be. On Sun, May 13, 2018, 11:20 AM Guido van Rossum wrote: > As anyone still following the inline assignment discussion knows, a > problem with designing new syntax is that it's hard to introduce new > keywords into the language, since all the nice words seem to be used as > method names in popular packages. (E.g. we can't use 'where' because > there's numpy.where > , > and we can't use 'given' because it's used in Hypothesis > .) > > The idea I had (not for the first time :-) is that in many syntactic > positions we could just treat keywords as names, and that would free up > these keywords. > > For example, we could allow keywords after 'def' and after a period, and > then the following would become legal: > > class C: > def and(self, other): > return ... > > a = C() > b = C() > print(a.and(b)) > > This does not create syntactic ambiguities because after 'def' and after a > period the grammar *always* requires a NAME. > > There are other positions where we could perhaps allow this, e.g. in a > decorator, immediately after '@' (the only keyword that's *syntactically* > legal here is 'not', though I'm not sure it would ever be useful). > > Of course this would still not help for names of functions that might be > imported directly (do people write 'from numpy import where'?). And it > would probably cause certain typos be harder to diagnose. > > I should also mention that this was inspired from some messages where Tim > Peters berated the fashion of using "reserved words", waxing nostalgically > about the old days of Fortran (sorry, FORTRAN), which doesn't (didn't?) > have reserved words at all (nor significant whitespace, apart from the > "start in column 7" rule). > > Anyway, just throwing this out. Please tear it apart! > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericfahlgren at gmail.com Sun May 13 14:42:28 2018 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Sun, 13 May 2018 11:42:28 -0700 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: On Sun, May 13, 2018 at 11:20 AM Guido van Rossum wrote: > For example, we could allow keywords after 'def' and after a period, and > then the following would become legal: > ?Our modeling database overloads getattr/setattr (think SQLAlchemy) to allow us to access to database fields as if they were Python data members. Nothing new here, but we do have problems with keyword collisions on some of the objects, as we are wrapping an already-existing modeling language (MSC Adams Solver dataset) with our objects. We were pleased with 'print' became a function, because it removed the restriction from that one, but one of the remaining ones is 'return', like this: class Sensor: def __init__(self): setattr(self, "print", 0) setattr(self, "return", 0) s = Sensor() s.print # Works now, didn't in Python 2. s.return # Bork. I have a decades old +1 on this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun May 13 14:45:11 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 14 May 2018 04:45:11 +1000 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: On Mon, May 14, 2018 at 4:19 AM, Guido van Rossum wrote: > The idea I had (not for the first time :-) is that in many syntactic > positions we could just treat keywords as names, and that would free up > these keywords. > ... > I should also mention that this was inspired from some messages where Tim > Peters berated the fashion of using "reserved words", waxing nostalgically > about the old days of Fortran (sorry, FORTRAN), which doesn't (didn't?) have > reserved words at all (nor significant whitespace, apart from the "start in > column 7" rule). I spent most of the 1990s coding in REXX, which has exactly zero reserved words. You can write code like this: if = 1 then = "spam" else = "ham" if if then then else else do = 5 do do print("Doobee doobee doo" end The problem is that you can go a long way down the road of using a particular name, only to find that suddenly you can't use it in some particular context. Common words like "if" and "do" are basically never going to get reused (so there's no benefit over having actual keywords), but with less common words (which would include the proposed "where" for binding expressions), it's entirely possible to get badly bitten. So the question is: Is it better to be able to use a keyword as an identifier for a while, and then run into trouble later, or would you prefer to be told straight away "no, sorry, pick a different name"? ChrisA From stephanh42 at gmail.com Sun May 13 14:48:17 2018 From: stephanh42 at gmail.com (Stephan Houben) Date: Sun, 13 May 2018 20:48:17 +0200 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: Note that Javascript does something similar, in that reserved keywords are allowed in member access. var x = {if : 42}; console.log(x.if); Stephan 2018-05-13 20:42 GMT+02:00 Eric Fahlgren : > On Sun, May 13, 2018 at 11:20 AM Guido van Rossum > wrote: > >> For example, we could allow keywords after 'def' and after a period, and >> then the following would become legal: >> > > ?Our modeling database overloads getattr/setattr (think SQLAlchemy) to > allow us to access to database fields as if they were Python data members. > Nothing new here, but we do have problems with keyword collisions on some > of the objects, as we are wrapping an already-existing modeling language > (MSC Adams Solver dataset) with our objects. We were pleased with 'print' > became a function, because it removed the restriction from that one, but > one of the remaining ones is 'return', like this: > > class Sensor: > def __init__(self): > setattr(self, "print", 0) > setattr(self, "return", 0) > > s = Sensor() > s.print # Works now, didn't in Python 2. > s.return # Bork. > > I have a decades old +1 on this. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Sun May 13 14:49:11 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Sun, 13 May 2018 11:49:11 -0700 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: <5AF888A7.6090807@brenbarn.net> On 2018-05-13 11:19, Guido van Rossum wrote: > As anyone still following the inline assignment discussion knows, a > problem with designing new syntax is that it's hard to introduce new > keywords into the language, since all the nice words seem to be used as > method names in popular packages. (E.g. we can't use 'where' because > there's numpy.where > , > and we can't use 'given' because it's used in Hypothesis > .) > > The idea I had (not for the first time :-) is that in many syntactic > positions we could just treat keywords as names, and that would free up > these keywords. > > For example, we could allow keywords after 'def' and after a period, and > then the following would become legal: > > class C: > def and(self, other): > return ... > > a = C() > b = C() > print(a.and(b)) > > This does not create syntactic ambiguities because after 'def' and after > a period the grammar *always* requires a NAME. > > There are other positions where we could perhaps allow this, e.g. in a > decorator, immediately after '@' (the only keyword that's > *syntactically* legal here is 'not', though I'm not sure it would ever > be useful). > > Of course this would still not help for names of functions that might be > imported directly (do people write 'from numpy import where'?). And it > would probably cause certain typos be harder to diagnose. Spooky! :-) People definitely do write "from numpy import where". You can see examples just by googling for that string. And that's only finding the cases where "where" is the first thing in the from clause; there are probably many more where it's something like "from numpy import array, sin, cos, where". I think this kind of special casing would be really confusing though. It would mean that these two would work (and do the same thing): import np np.array(...) from np import array array(...) And this would work: import np np.where(...) But then this would fail: from np import where where(...) That would be really puzzling. Plus, we'd still have the problem of backwards compatibility. Before the new "where" keyword was introduced, the last example WOULD work, but then afterwards you'd have to change it to look like the third example. This might induce wary programmers to hide all names behind a period (that is, find a way to do "foo.bar" instead of just "bar" whenever possible) in order to guard against future keywordization of those names. The other thing is that I suspect use of such a privilege would explode due to the attractiveness of the reserved words. That is, many libraries would start defining things named "and", "or", "with", "is", "in", etc., because the names are so nice and short and are useful in so many situations. So there'd be nowhere to hide from the deluge of names shadowing keywords. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From eltrhn at gmail.com Sun May 13 14:50:57 2018 From: eltrhn at gmail.com (Elias Tarhini) Date: Sun, 13 May 2018 11:50:57 -0700 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: Apologies for my initial response. Looks like I failed to expand the initial email fully, which would have shown me the following :) > Of course this would still not help for names of functions that might be imported directly (do people write 'from numpy import where'?). -- I do think the *import keyword as keyword_* concept has merit, however. (This would also be a simple retroactive solution to the asyncio.async problem, wouldn't it?) On Sun, May 13, 2018, 11:45 AM Chris Angelico wrote: > On Mon, May 14, 2018 at 4:19 AM, Guido van Rossum > wrote: > > The idea I had (not for the first time :-) is that in many syntactic > > positions we could just treat keywords as names, and that would free up > > these keywords. > > ... > > I should also mention that this was inspired from some messages where Tim > > Peters berated the fashion of using "reserved words", waxing > nostalgically > > about the old days of Fortran (sorry, FORTRAN), which doesn't (didn't?) > have > > reserved words at all (nor significant whitespace, apart from the "start > in > > column 7" rule). > > I spent most of the 1990s coding in REXX, which has exactly zero > reserved words. You can write code like this: > > if = 1 > then = "spam" > else = "ham" > > if if then then > else else > > do = 5 > do do > print("Doobee doobee doo" > end > > The problem is that you can go a long way down the road of using a > particular name, only to find that suddenly you can't use it in some > particular context. Common words like "if" and "do" are basically > never going to get reused (so there's no benefit over having actual > keywords), but with less common words (which would include the > proposed "where" for binding expressions), it's entirely possible to > get badly bitten. > > So the question is: Is it better to be able to use a keyword as an > identifier for a while, and then run into trouble later, or would you > prefer to be told straight away "no, sorry, pick a different name"? > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcidy at gmail.com Sun May 13 14:53:20 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Sun, 13 May 2018 11:53:20 -0700 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AF8839A.2060408@brenbarn.net> References: <20180512210716.GA34320@cskk.homeip.net> <20180512212420.GA77042@cskk.homeip.net> <20180513112336.GJ12683@ando.pearwood.info> <5AF8839A.2060408@brenbarn.net> Message-ID: On Sun, May 13, 2018, 11:28 Brendan Barnwell wrote: > On 2018-05-13 04:23, Steven D'Aprano wrote: > > In my experience mathematicians put the given *before* the statement: > > > > Given a, b, c three sides of a triangle, then > > > > Area = sqrt(s*(s-a)*(s-b)*(s-c)) > > > > where s = (a + b + c)/2 is the semi-perimeter of the triangle. > > > > For the record, that is almost exactly what I wrote for a student > > earlier today, and its not just me, it is very similar to the wording > > used on both Wolfram Mathworld and Wikipedia's pages on Heron's Formula. > > > > http://mathworld.wolfram.com/HeronsFormula.html > > > > https://en.wikipedia.org/wiki/Heron%27s_formula > > > > > > Putting "given" after the expression is backwards. > > Yes, but that's because we're ruling out the use of "where". At > this > point I would be fine with "snicklefritz" as the keyword. The point is > that I want to put SOMETHING after the expression, and this is not at > all unusual. See for instance Wikipedia pages on the Reimann zeta > function > (https://en.wikipedia.org/wiki/Riemann_zeta_function#Definition), > gravitation equation > (https://en.wikipedia.org/wiki/Gravity#Newton%27s_theory_of_gravitation), > and > compound interest > ( > https://en.wikipedia.org/wiki/Compound_interest#Mathematics_of_interest_rate_on_loans). > > If we have to use the word "given" even though the word mathematicians > would use in that position is "where", that's not such a big deal. > it is a big deal. postfix requires more cognitive load, we will have no idea up front what's going on except for trivial exames. more givens, more cognitive load. if you think spending that is fine for you, I can't argue, but to say it doesn't matter isn't correct. 2.exames which get far worse for complex cases. left for the for can be as complex.as.you wish. 1: [ x + y for t in range(10) ... ] 2: x = 10 y = 20 [ x + y for t in range(10) ...] up till you read ... you have no idea there even will be a substitution. The lower is even worse, you think you know, but then have to redo the whole problem with new information. also : mathematicians don't just put the _word_ "given", they put givens, things that are known or assumed to be true. Axioms and definitions, where definitions assign names to values. This is for formal arguements. reassigning values is handled in post fix occasionally once it is clear what x and y are. but that's not what we are talking about if the name doesn't exist already. again, you want to use given, that's fine, but the math argument is wrong, as is the "it doesn't matter" argument, assuming the current neurological model for working memory continues to hold. Maybe the difference is small, especially after familiarity sets in, but that doesn't mean the difference in load isn't there. it will only increase for more complex statements with more givens. > > -- > Brendan Barnwell > "Do not follow where the path may lead. Go, instead, where there is no > path, and leave a trail." > --author unknown > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Sun May 13 14:59:49 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sun, 13 May 2018 19:59:49 +0100 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: <78e8c1d4-0597-a3ae-1f52-b7e0d192df4e@btinternet.com> On 13/05/2018 19:19, Guido van Rossum wrote: > As anyone still following the inline assignment discussion knows, a > problem with designing new syntax is that it's hard to introduce new > keywords into the language, since all the nice words seem to be used > as method names in popular packages. (E.g. we can't use 'where' > because there's numpy.where > , > and we can't use 'given' because it's used in Hypothesis > .) > > The idea I had (not for the first time :-) is that in many syntactic > positions we could just treat keywords as names, and that would free > up these keywords. > > For example, we could allow keywords after 'def' and after a period, > and then the following would become legal: > > class C: > ??? def and(self, other): > ??????? return ... > > a = C() > b = C() > print(a.and(b)) > > This does not create syntactic ambiguities because after 'def' and > after a period the grammar *always* requires a NAME. > > There are other positions where we could perhaps allow this, e.g. in a > decorator, immediately after '@' (the only keyword that's > *syntactically* legal here is 'not', though I'm not sure it would ever > be useful). So you would be allowing "second class" identifiers - legal in some positions where an identifier is allowed, not legal in others. With respect, that seems like a terrible idea; neither those who want to use such identifiers, nor those who don't, would be happy.? Especially if it encourages work-arounds such as ??? def and(x, y): ??????? return ... # ? and(1,2)?????????? #? Oops, SyntaxError.? Oh, I know: ??? globals()['and'](1,2) ?? # Works! and a zillion others. > > [snip] And it would probably cause certain typos be harder to diagnose. No doubt. > > I should also mention that this was inspired from some messages where > Tim Peters berated the fashion of using "reserved words", waxing > nostalgically about the old days of Fortran (sorry, FORTRAN), which > doesn't (didn't?) have reserved words at all (nor significant > whitespace, apart from the "start in column 7" rule). > > Anyway, just throwing this out. Please tear it apart! Thanks. :-) > > -- > --Guido van Rossum (python.org/~guido ) > Best wishes Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From neatnate at gmail.com Sun May 13 15:01:54 2018 From: neatnate at gmail.com (Nathan Schneider) Date: Sun, 13 May 2018 15:01:54 -0400 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: Care would have to be taken in the grammar to avoid syntactic ambiguity. For example: x = 1 def not(x): ... if not - x: # Is this 'not' the keyword or the identifier? not (-x), or not minus x? ... Nathan On Sun, May 13, 2018 at 2:20 PM Guido van Rossum wrote: > As anyone still following the inline assignment discussion knows, a > problem with designing new syntax is that it's hard to introduce new > keywords into the language, since all the nice words seem to be used as > method names in popular packages. (E.g. we can't use 'where' because > there's numpy.where > , > and we can't use 'given' because it's used in Hypothesis > .) > > The idea I had (not for the first time :-) is that in many syntactic > positions we could just treat keywords as names, and that would free up > these keywords. > > For example, we could allow keywords after 'def' and after a period, and > then the following would become legal: > > class C: > def and(self, other): > return ... > > a = C() > b = C() > print(a.and(b)) > > This does not create syntactic ambiguities because after 'def' and after a > period the grammar *always* requires a NAME. > > There are other positions where we could perhaps allow this, e.g. in a > decorator, immediately after '@' (the only keyword that's *syntactically* > legal here is 'not', though I'm not sure it would ever be useful). > > Of course this would still not help for names of functions that might be > imported directly (do people write 'from numpy import where'?). And it > would probably cause certain typos be harder to diagnose. > > I should also mention that this was inspired from some messages where Tim > Peters berated the fashion of using "reserved words", waxing nostalgically > about the old days of Fortran (sorry, FORTRAN), which doesn't (didn't?) > have reserved words at all (nor significant whitespace, apart from the > "start in column 7" rule). > > Anyway, just throwing this out. Please tear it apart! > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Sun May 13 15:04:37 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Sun, 13 May 2018 12:04:37 -0700 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512210716.GA34320@cskk.homeip.net> <20180512212420.GA77042@cskk.homeip.net> <20180513112336.GJ12683@ando.pearwood.info> <5AF8839A.2060408@brenbarn.net> Message-ID: <5AF88C45.3050204@brenbarn.net> On 2018-05-13 11:53, Matt Arcidy wrote: > again, you want to use given, that's fine, but the math argument is > wrong, as is the "it doesn't matter" argument, assuming the current > neurological model for working memory continues to hold. Sorry, but that's nonsense. In the message you're replying to, what I'm arguing "doesn't matter" is the WORD used to do the postfixing. Are you seriously saying that the "neurological model for working memory" means that "x + y where x = foo" is easier to understand, on a fundamental neurological basis, than "x + y given x = foo"? Since you're the one who was advocating for objective measures, I'm sure you'll understand if I want some solid objective evidence for that! If you're saying that the entire concept of postfixing the givens (rather than putting them before the expression) creates greater cognitive load, then how do you explain the use of this format in the Wikipedia articles I linked, not to mention various math papers, texts, etc. throughout the world? Finally, even if we allow that postfixed-givens does increase cognitive load on a single, initial reading, that's not sufficient. Part of what I'm saying is that on LATER readings it's faster to see the overall expression first, because you don't have to plow through the definitions of the givens. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From mistersheik at gmail.com Sun May 13 15:13:24 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Sun, 13 May 2018 15:13:24 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512210716.GA34320@cskk.homeip.net> <20180512212420.GA77042@cskk.homeip.net> <20180513112336.GJ12683@ando.pearwood.info> <5AF8839A.2060408@brenbarn.net> Message-ID: On Sun, May 13, 2018 at 2:54 PM Matt Arcidy wrote: > > > On Sun, May 13, 2018, 11:28 Brendan Barnwell > wrote: > >> On 2018-05-13 04:23, Steven D'Aprano wrote: >> > In my experience mathematicians put the given *before* the statement: >> > >> > Given a, b, c three sides of a triangle, then >> > >> > Area = sqrt(s*(s-a)*(s-b)*(s-c)) >> > >> > where s = (a + b + c)/2 is the semi-perimeter of the triangle. >> > >> > For the record, that is almost exactly what I wrote for a student >> > earlier today, and its not just me, it is very similar to the wording >> > used on both Wolfram Mathworld and Wikipedia's pages on Heron's Formula. >> > >> > http://mathworld.wolfram.com/HeronsFormula.html >> > >> > https://en.wikipedia.org/wiki/Heron%27s_formula >> > >> > >> > Putting "given" after the expression is backwards. >> > I usually read the vertical bar "given" in set-builder notation: {x | 10 < x < 100, x is prime} Conditional probabilities, expectations, entropy, etc.: P(X | Y), E(X| Y), H(X | Y), And, the so-called "evaluation bar": https://math.stackexchange.com/questions/52651/what-is-the-name-of-the-vertical-bar-in-x21-vert-x-4-or-left-left In words, I agree that where is probably better in text because writing "given the gravitational constant G" puts the subject last, unlike "where G is the gravitational constant". However, the given proposal for Python is putting the subject first: "given x = y". >> Yes, but that's because we're ruling out the use of "where". At >> this >> point I would be fine with "snicklefritz" as the keyword. The point is >> that I want to put SOMETHING after the expression, and this is not at >> all unusual. See for instance Wikipedia pages on the Reimann zeta >> function >> (https://en.wikipedia.org/wiki/Riemann_zeta_function#Definition), >> gravitation equation >> (https://en.wikipedia.org/wiki/Gravity#Newton%27s_theory_of_gravitation), >> and >> compound interest >> ( >> https://en.wikipedia.org/wiki/Compound_interest#Mathematics_of_interest_rate_on_loans). >> >> If we have to use the word "given" even though the word mathematicians >> would use in that position is "where", that's not such a big deal. >> > > it is a big deal. postfix requires more cognitive load, we will have no > idea up front what's going on except for trivial exames. more givens, more > cognitive load. > > if you think spending that is fine for you, I can't argue, but to say it > doesn't matter isn't correct. > > 2.exames which get far worse for complex cases. left for the for can be > as complex.as.you wish. > 1: > [ x + y for t in range(10) ... ] > > 2: > x = 10 > y = 20 > [ x + y for t in range(10) ...] > > up till you read ... you have no idea there even will be a substitution. > The lower is even worse, you think you know, but then have to redo the > whole problem with new information. > > also : > mathematicians don't just put the _word_ "given", they put givens, things > that are known or assumed to be true. Axioms and definitions, where > definitions assign names to values. This is for formal arguements. > reassigning values is handled in post fix occasionally once it is clear > what x and y are. but that's not what we are talking about if the name > doesn't exist already. > Most (if not all) uses of the vertical are read as "given" and they all put the givens to the right of it. > > again, you want to use given, that's fine, but the math argument is wrong, > as is the "it doesn't matter" argument, assuming the current neurological > model for working memory continues to hold. > > Maybe the difference is small, especially after familiarity sets in, but > that doesn't mean the difference in load isn't there. it will only > increase for more complex statements with more givens. > >> >> -- >> Brendan Barnwell >> "Do not follow where the path may lead. Go, instead, where there is no >> path, and leave a trail." >> --author unknown >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/CFuqwmE8s-E/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/CFuqwmE8s-E/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Sun May 13 16:01:09 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 13 May 2018 15:01:09 -0500 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: [Guido] > ... > I should also mention that this was inspired from some messages where Tim > Peters berated the fashion of using "reserved words", waxing nostalgically > about the old days of Fortran (sorry, FORTRAN), which doesn't (didn't?) have > reserved words at all (nor significant whitespace, apart from the "start in > column 7" rule). The Fortran standards continued evolving, and still are, but despite that in many ways it's become a much more "modern" language, while it continues to add ever more keywords it _still_ has no reserved words. They have a strong reason for that: there are numeric libraries coded decades ago in Fortran still in use, which nobody understands anymore, so anything that breaks old code is fiercely opposed. If you think people whine today about an incompatible Python change, wait another 30+ years to get a sense of what it's like in the Fortran community ;-) I'm not sure people realize that Algol had no reserved words either. But that was "a trick" :-) Algol was originally designed as a formal notation for publishing algorithms, not really for executing them. Published Algol "code" always showed the language's keywords in boldface.. Similarly, the nearly incomprehensible (to me) Algol 68's 60 "reserved words" are essentially defined to be in boldface in the "representation" (for publication) language, but can also be used as identifiers etc. The intimately related but distinct "reference language" needs to be used for computer input. In that the reserved words are decorated in some way, to distinguish them from user-defined names of the same spelling. There's more than one way to do that, and - indeed - it's required that Algol 68 compilers support several specific ways. Anyone looking for more pain can get it here: https://en.wikipedia.org/wiki/ALGOL_68#Program_representation Note that Perl largely (but not entirely) managed to dodge the problem by requiring a punctuation character (like $ for scalar and @ for array) before variable names. In addition to REXX (already mentioned), Prolog and PL/I had no reserved words. End of brain dump ;-) But I didn't suggest that for Python. It's a bit too late for that anyway ;-) It was more in line with what you suggested here: think more about ways in which reserved words _could_ be allowed as identifiers in specific contexts where implementing that doesn't require heroic effort. From python at fwdaddr.fastmail.fm Sun May 13 16:29:06 2018 From: python at fwdaddr.fastmail.fm (Nick Malaguti) Date: Sun, 13 May 2018 16:29:06 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512210716.GA34320@cskk.homeip.net> <20180512212420.GA77042@cskk.homeip.net> <20180513112336.GJ12683@ando.pearwood.info> <5AF8839A.2060408@brenbarn.net> Message-ID: <1526243346.2855696.1370586496.57E2BEDF@webmail.messagingengine.com> I think Peter tried to outline this earlier, but what he was laying out wasn't clear to me at first. There seem to be 4 variations when it comes to assignment expressions. I'm going to try to ignore exact keywords here since we can sort those out once we have settled on which variation we prefer. 1. infix: TARGET := EXPR 2. infix: EXPR as TARGET 3. prefix: let TARGET = EXPR in ANOTHER_EXPR 4. postfix: ANOTHER_EXPR given TARGET = EXPR Both 1 and 2 may appear in the context of a larger expression where TARGET may or may not be used: 1. 99 + (TARGET := EXPR) ** 2 + TARGET 2. 99 + (EXPR as TARGET) ** 2 + TARGET 3 and 4 require that TARGET appear in ANOTHER_EXPR, even if TARGET is the only thing contained in that expression, whereas with 1 and 2, TARGET need not be used again. Example I: 1. x := 10 2. 10 as x 3. let x = 10 in x 4. x given x = 10 In the simple case where the goal of the assignment expression is to bind the EXPR to the TARGET so that TARGET can be used in a future statement, 1 and 2 are clearly the most straightforward because they do not require ANOTHER_EXPR. # Please ignore that m.group(2) doesn't do anything useful here Example II: 1. if m := re.match(...): m.group(2) 2. if re.match(...) as m: res = m.group(2) 3. if let m = re.match(...) in m: m.group(2) 4. if m given m = re.match(...): m.group(2) I also think expressions that use "or" or "and" to make a compound expression benefit from the infix style, mostly because each sub-expression stands on its own and is only made longer with the repetition of TARGET: Example III: 1. if (diff := x - x_base) and (g := gcd(diff, n)) > 1: ... 2. if (x - x_base as diff) and (gcd(diff, n) as g) > 1: ... 3. if (let diff = x - x_base in diff) and (let g = gcd(diff, n) in g > 1): ... 4. if (diff given diff = x - x_base) and (g > 1 given g = gcd(diff, n)): ... In the more complex case where TARGET is reused in the expression, I find 3 and 4 to benefit as there is a separation of the binding from its usage. I can consider each expression separately and I don't have to deal with the assignment side effects at the same time. I believe this is what Neil is mostly arguing for. # Borrowing from Andre, please forgive any mathematical problems like division by 0 Example IV: 1: [(-b/(2*a) + (D := sqrt( (b/(2*a))**2 - c/a), -b/(2*a) - D) for a in range(10) for b in range(10) for c in range(10) if D >= 0] 2: [(-b/(2*a) + (sqrt( (b/(2*a))**2 - c/a as D), -b/(2*a) - D) for a in range(10) for b in range(10) for c in range(10) if D >= 0] 3. [let D = sqrt( (b/(2*a))**2 - c/a) in (-b/(2*a) + D, -b/(2*a) - D) for a in range(10) for b in range(10) for c in range(10) if D >= 0] 4. [(-b/(2*a) + D, -b/(2*a) - D) for a in range(10) for b in range(10) for c in range(10) if D >= 0 given D = sqrt( (b/(2*a))**2 - c/a)] Also in the case with multiple bindings I find that 3 and 4 benefit over 1 and 2: Example V: 1. [(x := f(y := (z := f(i) ** 2) + 1)) for i in range(10)] 2. [(f((f(i) ** 2 as z) + 1 as y) as x) for i in range(10)] 3. [let x = f(y), y = z + 1, z = f(i) ** 2 in x for i in range(10)] # maybe the order of the let expressions should be reversed? 4. [x given x = f(y) given y = z + 1 given z = f(i) ** 2 for i in range(10)] No matter which variation we prefer, there are plenty of arguments to be made that multiple assignment expressions in a single expression or usage of the TARGET later in the expression is harder to work with in most cases,. And since 1 and 2 (at least to me) are more difficult to parse in those situations, I'm more likely to push back on whoever writes that code to do it another way or split it into multiple statements. I feel that Steven prefers 1, mostly for the reason that it makes Examples I, II, and III easier to write and easier to read. Neil prefers 4 because Examples I, II, and II still aren't that bad with 4, and are easier to work with in Examples IV and V. If you feel that Examples IV and V should be written differently in the first place, you probably prefer infix (1 or 2). If you feel that Examples IV and V are going to be written anyway and you want them to be as readable as possible, you probably prefer prefix (3) or postfix (4). If you want to know what all the TARGETs are assigned to up front, you probably prefer 1 or 3 (for reading from left to right). If you want to see how the TARGET is used in the larger expression up front and are willing to read to the end to find out if or where the TARGET has been defined, you probably prefer 4. In my mind, all 4 variations have merit. I think I prefer prefix or postfix (postfix feels very natural to me) because I believe more complex expressions should be separateable (Neil argues better than I can for this). But Steven has gone a long way to convince me that the sky won't fall if we choose an infix variation because in practice our better angels will push us away from using expressions that are too complex. Prefix vs postfix is a discussion worth having if we decide that infix isn't the right choice. I would love to see us reach consensus (too optimistic?) or at least an acknowledgment of the explicit tradeoffs for whichever variation we ultimately choose. -- Nick ----- Original message ----- From: Matt Arcidy To: Brendan Barnwell Cc: "python-ideas" Subject: Re: [Python-ideas] Inline assignments using "given" clauses Date: Sun, 13 May 2018 11:53:20 -0700 On Sun, May 13, 2018, 11:28 Brendan Barnwell wrote: > On 2018-05-13 04:23, Steven D'Aprano wrote: > > In my experience mathematicians put the given *before* the statement: > > > >? ? ?Given a, b, c three sides of a triangle, then > > > >? ? ? ? ?Area = sqrt(s*(s-a)*(s-b)*(s-c)) > > > >? ? ?where s = (a + b + c)/2 is the semi-perimeter of the triangle. > > > > For the record, that is almost exactly what I wrote for a student > > earlier today, and its not just me, it is very similar to the wording > > used on both Wolfram Mathworld and Wikipedia's pages on Heron's Formula. > > > > http://mathworld.wolfram.com/HeronsFormula.html > > > > https://en.wikipedia.org/wiki/Heron%27s_formula > > > > > > Putting "given" after the expression is backwards. > > ? ? ? ? Yes, but that's because we're ruling out the use of "where".? At this > point I would be fine with "snicklefritz" as the keyword.? The point is > that I want to put SOMETHING after the expression, and this is not at > all unusual.? See for instance Wikipedia pages on the Reimann zeta > function > (https://en.wikipedia.org/wiki/Riemann_zeta_function#Definition), > gravitation equation > (https://en.wikipedia.org/wiki/Gravity#Newton%27s_theory_of_gravitation), and > compound interest > (https://en.wikipedia.org/wiki/Compound_interest#Mathematics_of_interest_rate_on_loans). > ? If we have to use the word "given" even though the word mathematicians > would use in that position is "where", that's not such a big deal. it is a big deal.? postfix requires more cognitive load, we will have no idea up front what's going on except for trivial exames.? more givens, more cognitive load. if you think spending that is fine for you, I can't argue, but to say it doesn't matter isn't correct. 2.exames which get far worse for complex cases.? left for the for can be as complex.as.you wish. 1: [ x + y for t in range(10)? ... ] 2: x = 10 y = 20 [ x + y for t in range(10) ...] up till you read ... you have no idea there even will be a substitution.? The lower is even worse, you think you know, but then have to redo the whole problem with new information. also : mathematicians don't just put the _word_ "given", they put givens, things that are known or assumed to be true.? Axioms and definitions, where definitions assign names to values.? This is for formal arguements.? reassigning values is handled in post fix occasionally once it is clear what x and y are.? but that's not what we are talking about if the name doesn't exist already. again, you want to use given, that's fine, but the math argument is wrong, as is the "it doesn't matter" argument, assuming the current neurological model for working memory continues to hold. ?Maybe the difference is small, especially after familiarity sets in, but that doesn't mean the difference in load isn't there.? it will only increase for more complex statements with more givens. > > -- > Brendan Barnwell > "Do not follow where the path may lead.? Go, instead, where there is no > path, and leave a trail." > ? ? --author unknown > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ _________________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ From mistersheik at gmail.com Sun May 13 16:48:51 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Sun, 13 May 2018 16:48:51 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <1526243346.2855696.1370586496.57E2BEDF@webmail.messagingengine.com> References: <20180512210716.GA34320@cskk.homeip.net> <20180512212420.GA77042@cskk.homeip.net> <20180513112336.GJ12683@ando.pearwood.info> <5AF8839A.2060408@brenbarn.net> <1526243346.2855696.1370586496.57E2BEDF@webmail.messagingengine.com> Message-ID: On Sun, May 13, 2018 at 4:30 PM Nick Malaguti wrote: > I think Peter tried to outline this earlier, but what he was laying out > wasn't clear to me at first. > > There seem to be 4 variations when it comes to assignment expressions. I'm > going to try to ignore exact keywords here since we can sort those out once > we have settled on which variation we prefer. > > 1. infix: TARGET := EXPR > 2. infix: EXPR as TARGET > 3. prefix: let TARGET = EXPR in ANOTHER_EXPR > 4. postfix: ANOTHER_EXPR given TARGET = EXPR > > Both 1 and 2 may appear in the context of a larger expression where TARGET > may or may not be used: > > 1. 99 + (TARGET := EXPR) ** 2 + TARGET > 2. 99 + (EXPR as TARGET) ** 2 + TARGET > > 3 and 4 require that TARGET appear in ANOTHER_EXPR, even if TARGET is the > only thing contained in that expression, whereas with 1 and 2, TARGET need > not be used again. > > Example I: > > 1. x := 10 > 2. 10 as x > 3. let x = 10 in x > 4. x given x = 10 > > In the simple case where the goal of the assignment expression is to bind > the EXPR to the TARGET so that TARGET can be used in a future statement, 1 > and 2 are clearly the most straightforward because they do not require > ANOTHER_EXPR. > > # Please ignore that m.group(2) doesn't do anything useful here > > Example II: > > 1. if m := re.match(...): m.group(2) > 2. if re.match(...) as m: res = m.group(2) > 3. if let m = re.match(...) in m: m.group(2) > 4. if m given m = re.match(...): m.group(2) > > I also think expressions that use "or" or "and" to make a compound > expression benefit from the infix style, mostly because each sub-expression > stands on its own and is only made longer with the repetition of TARGET: > > Example III: > > 1. if (diff := x - x_base) and (g := gcd(diff, n)) > 1: ... > 2. if (x - x_base as diff) and (gcd(diff, n) as g) > 1: ... > 3. if (let diff = x - x_base in diff) and (let g = gcd(diff, n) in g > 1): > ... > 4. if (diff given diff = x - x_base) and (g > 1 given g = gcd(diff, n)): > ... > > In the more complex case where TARGET is reused in the expression, I find > 3 and 4 to benefit as there is a separation of the binding from its usage. > I can consider each expression separately and I don't have to deal with the > assignment side effects at the same time. I believe this is what Neil is > mostly arguing for. > Yes. > > # Borrowing from Andre, please forgive any mathematical problems like > division by 0 > > Example IV: > > 1: [(-b/(2*a) + (D := sqrt( (b/(2*a))**2 - c/a), -b/(2*a) - D) > for a in range(10) > for b in range(10) > for c in range(10) > if D >= 0] > 2: [(-b/(2*a) + (sqrt( (b/(2*a))**2 - c/a as D), -b/(2*a) - D) > for a in range(10) > for b in range(10) > for c in range(10) > if D >= 0] > 3. [let D = sqrt( (b/(2*a))**2 - c/a) in > (-b/(2*a) + D, -b/(2*a) - D) > for a in range(10) > for b in range(10) > for c in range(10) > if D >= 0] > 4. [(-b/(2*a) + D, -b/(2*a) - D) > for a in range(10) > for b in range(10) > for c in range(10) > if D >= 0 > given D = sqrt( (b/(2*a))**2 - c/a)] > > Also in the case with multiple bindings I find that 3 and 4 benefit over 1 > and 2: > > Example V: > > 1. [(x := f(y := (z := f(i) ** 2) + 1)) for i in range(10)] > 2. [(f((f(i) ** 2 as z) + 1 as y) as x) for i in range(10)] > 3. [let x = f(y), y = z + 1, z = f(i) ** 2 in x for i in range(10)] # > maybe the order of the let expressions should be reversed? > 4. [x given x = f(y) given y = z + 1 given z = f(i) ** 2 for i in > range(10)] > > No matter which variation we prefer, there are plenty of arguments to be > made that multiple assignment expressions in a single expression or usage > of the TARGET later in the expression is harder to work with in most > cases,. And since 1 and 2 (at least to me) are more difficult to parse in > those situations, I'm more likely to push back on whoever writes that code > to do it another way or split it into multiple statements. > > I feel that Steven prefers 1, mostly for the reason that it makes Examples > I, II, and III easier to write and easier to read. Neil prefers 4 because > Examples I, II, and II still aren't that bad with 4, and are easier to work > with in Examples IV and V. > > If you feel that Examples IV and V should be written differently in the > first place, you probably prefer infix (1 or 2). > > If you feel that Examples IV and V are going to be written anyway and you > want them to be as readable as possible, you probably prefer prefix (3) or > postfix (4). > > If you want to know what all the TARGETs are assigned to up front, you > probably prefer 1 or 3 (for reading from left to right). > > If you want to see how the TARGET is used in the larger expression up > front and are willing to read to the end to find out if or where the TARGET > has been defined, you probably prefer 4. > > In my mind, all 4 variations have merit. I think I prefer prefix or > postfix (postfix feels very natural to me) because I believe more complex > expressions should be separateable (Neil argues better than I can for this). > > But Steven has gone a long way to convince me that the sky won't fall if > we choose an infix variation because in practice our better angels will > push us away from using expressions that are too complex. > > Prefix vs postfix is a discussion worth having if we decide that infix > isn't the right choice. > > I would love to see us reach consensus (too optimistic?) or at least an > acknowledgment of the explicit tradeoffs for whichever variation we > ultimately choose. > In my mind, your given clauses are upside-down. The way I see it, code like this: for a in range(10): if a != 5: for b in range(10): for c in range(10): D = sqrt((b/(2*a))**2 - c/a) if D >= 0: yield (-b/(2*a) + D, -b/(2*a) - D) should be cast into a generator like this: ((-b/(2*a) + D, -b/(2*a) - D) for a in range(10) if a != 5 for b in range(10) for c in range(10) given D = sqrt( (b/(2*a))**2 - c/a) if D >= 0) Just leave everything in the order you wrote it except the yield statement turns into a bare expression in front. Regarding prefix versus postfix, we already do postfix binding using "for" clauses and I want to be able to interleave given clauses with for clauses as above to prevent a bracketing mess that prefix would require. > -- > Nick > > ----- Original message ----- > From: Matt Arcidy > To: Brendan Barnwell > Cc: "python-ideas" > Subject: Re: [Python-ideas] Inline assignments using "given" clauses > Date: Sun, 13 May 2018 11:53:20 -0700 > > > > On Sun, May 13, 2018, 11:28 Brendan Barnwell > wrote: > > On 2018-05-13 04:23, Steven D'Aprano wrote: > > > In my experience mathematicians put the given *before* the statement: > > > > > > Given a, b, c three sides of a triangle, then > > > > > > Area = sqrt(s*(s-a)*(s-b)*(s-c)) > > > > > > where s = (a + b + c)/2 is the semi-perimeter of the triangle. > > > > > > For the record, that is almost exactly what I wrote for a student > > > earlier today, and its not just me, it is very similar to the wording > > > used on both Wolfram Mathworld and Wikipedia's pages on Heron's > Formula. > > > > > > http://mathworld.wolfram.com/HeronsFormula.html > > > > > > https://en.wikipedia.org/wiki/Heron%27s_formula > > > > > > > > > Putting "given" after the expression is backwards. > > > > Yes, but that's because we're ruling out the use of "where". > At this > > point I would be fine with "snicklefritz" as the keyword. The point is > > that I want to put SOMETHING after the expression, and this is not at > > all unusual. See for instance Wikipedia pages on the Reimann zeta > > function > > (https://en.wikipedia.org/wiki/Riemann_zeta_function#Definition), > > gravitation equation > > (https://en.wikipedia.org/wiki/Gravity#Newton%27s_theory_of_gravitation), > and > > compound interest > > ( > https://en.wikipedia.org/wiki/Compound_interest#Mathematics_of_interest_rate_on_loans). > > > If we have to use the word "given" even though the word > mathematicians > > would use in that position is "where", that's not such a big deal. > > it is a big deal. postfix requires more cognitive load, we will have no > idea up front what's going on except for trivial exames. more givens, more > cognitive load. > > if you think spending that is fine for you, I can't argue, but to say it > doesn't matter isn't correct. > > 2.exames which get far worse for complex cases. left for the for can be > as complex.as.you wish. > 1: > [ x + y for t in range(10) ... ] > > 2: > x = 10 > y = 20 > [ x + y for t in range(10) ...] > > up till you read ... you have no idea there even will be a substitution. > The lower is even worse, you think you know, but then have to redo the > whole problem with new information. > > also : > mathematicians don't just put the _word_ "given", they put givens, things > that are known or assumed to be true. Axioms and definitions, where > definitions assign names to values. This is for formal arguements. > reassigning values is handled in post fix occasionally once it is clear > what x and y are. but that's not what we are talking about if the name > doesn't exist already. > > again, you want to use given, that's fine, but the math argument is wrong, > as is the "it doesn't matter" argument, assuming the current neurological > model for working memory continues to hold. > > Maybe the difference is small, especially after familiarity sets in, but > that doesn't mean the difference in load isn't there. it will only > increase for more complex statements with more givens. > > > > -- > > Brendan Barnwell > > "Do not follow where the path may lead. Go, instead, where there is no > > path, and leave a trail." > > --author unknown > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _________________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/CFuqwmE8s-E/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at cskk.id.au Sun May 13 17:13:02 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 14 May 2018 07:13:02 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180513160431.GN12683@ando.pearwood.info> References: <20180513160431.GN12683@ando.pearwood.info> Message-ID: <20180513211302.GA79555@cskk.homeip.net> On 14May2018 02:04, Steven D'Aprano wrote: >"Outer parentheses" is a red herring. I should have written: > > (x + x) given x = 50 # this ought to return 100 > x + (x given x = 50) # this ought to return 51 > >and now there are no outer parentheses to worry about. The question is >now whether "given" binds more tightly than + or not. Just to this: as a general principle I think "words" should bind less tightly than "punctuationlike operators". Certainly for myself I read most code as "words surrounding expressions". While "given" and "for (comprehension)" and so forth are all words within expressions as far as the language grammar goes, to my intuition and _visually_ they are bulky separators between terse "mathy" things. Even the Boolean words "and", "or", "not"...) fit this role. So as a matter of egonomic design "given" ought also to bind less tightly than "+". Arithmetic before logic before declaration before control, if you like. Cheers, Cameron Simpson From cs at cskk.id.au Sun May 13 17:00:27 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 14 May 2018 07:00:27 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <54daa379-bebc-23df-3cf7-dbd52c6627b6@trueblade.com> References: <54daa379-bebc-23df-3cf7-dbd52c6627b6@trueblade.com> Message-ID: <20180513210027.GA34554@cskk.homeip.net> On 13May2018 06:45, Eric V. Smith wrote: >On 5/13/18 1:05 AM, Chris Angelico wrote: >>On Sun, May 13, 2018 at 2:58 PM, Cameron Simpson wrote: >>>On 13May2018 14:23, Chris Angelico wrote: >>>>https://www.python.org/dev/peps/pep-0572/#alternative-spellings >>> >>>[...] I'd already looked at that part of the PEP and found >>>its explaination... unfulfilling. It says: >>> >>> EXPR as NAME: >>> >>> stuff = [[f(x) as y, x/y] for x in range(5)] >>> >>> Since EXPR as NAME already has meaning in except and with statements (with >>> different semantics), this would create unnecessary confusion or require >>> special-casing (eg to forbid assignment within the headers of these >>> statements). [...] >>There were a large number of posts, so I can't really point to one of >>them. The problem isn't the double-as case that you describe; it's >>that these two are ALMOST identical: >> >>with expr as target: >>with (expr as target): >> >>In fact, they are functionally identical for many situations - "with >>(open(...) as target):" is exactly the same as the form without the >>parens. It'd make for data-dependent bugs, which are a really REALLY >>bad idea. > >A little more detail: > >The first one is partially: > >target = expr.__enter__() > >The second one is partially: > >target = expr > >For many objects, obj.__enter__() just returns obj, it often looks >like these two statements do the same thing, but they do not. Ah, thank you! Technically I knew this; in practice I'd forgotten and was thus vulnerable in exactly the same way as you anticipate. >Chris's concern, which I share, is that users wouldn't know why these are >different, and why the second on works for some objects but not others. While I can see the issue, on a personal basis I'd accept it: import and except don't do this and they happily use "as". In my mind the problem lies with "with" (and is a perfectly acceptable inconsistency there, given what with does for us). I agree the change in semantics is not obvious. But we've got a similar thing with tuples as well: x = (1) x = (1,) with a different spelling. >I agree the PEP could use more detail in explaining this particular issue. Yes. This issue certainly isn't clear to me from the PEP's wording. I'm no longer trying to push the "as" agenda here, BTW. I still like it, but that ship seems sailed. Cheers, Cameron Simpson From tim.peters at gmail.com Sun May 13 17:30:21 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 13 May 2018 16:30:21 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: <20180507114208.GV9562@ando.pearwood.info> <20180507142636.GW9562@ando.pearwood.info> Message-ID: [Tim] > ... > - If the target is not local to any function enclosing F, and is not > declared `global` in the block containing F, then the target is bound > in the block containing F. FYI, that's still not right, but I've been distracted by trying to convince myself that the manual actually defines what happens when absurdly deeply nested functions mix local values for a name at some levels with a `global` declaration of the name at other levels. I suspect that the above should be reworded to the simpler: - If the target is not declared `global` or `nonlocal` in the block containing F, then the target is bound in the block containing F. That makes "intuitive sense" because if the target is declared `global` or `nonlocal` the meaning of binding in the block is already defined to affect a not-local scope, while if it's not declared at all then binding in the block "should" establish that it's local.to the block (regardless of how containing scopes treat the same name) But whether that all follows from what the manual already says requires more staring at it ;-) Regardless, if anyone were to point it out, I'd agree that it _should_ count against this that establishing which names are local to a block may require searching top-level comprehensions in the block for assignment expressions. On a scale of minus a million to plus a million, I'd only weight that in the negative thousands, though ;-) From rosuav at gmail.com Sun May 13 17:59:44 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 14 May 2018 07:59:44 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180513210027.GA34554@cskk.homeip.net> References: <54daa379-bebc-23df-3cf7-dbd52c6627b6@trueblade.com> <20180513210027.GA34554@cskk.homeip.net> Message-ID: On Mon, May 14, 2018 at 7:00 AM, Cameron Simpson wrote: > While I can see the issue, on a personal basis I'd accept it: import and > except don't do this and they happily use "as". In my mind the problem lies > with "with" (and is a perfectly acceptable inconsistency there, given what > with does for us). The same problem happens with 'except', only less subtly. except Exception as e: # binds the caught exception to e except (Exception as e): # would bind the type Exception import doesn't put an expression on the left of 'as', so it's less likely to cause confusion; but all three of them do something special before binding to the target given with 'as'. By the way: do you know which of the three support arbitrary assignment targets and which support only names? No? Neither did I, till I checked the grammar. So there's no consistency there, other than a loose sense that "as" means "we're gonna toss something into somewhere". Not nearly enough to justify using that syntax for arbitrary name bindings, given how much hassle there is with 'with'. ChrisA From tim.peters at gmail.com Sun May 13 20:00:59 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 13 May 2018 19:00:59 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 Message-ID: [Tim] >> - If the target is not local to any function enclosing F, and is not >> declared `global` in the block containing F, then the target is bound >> in the block containing F. [also Tim] > FYI, that's still not right, ... > I suspect that the above should be reworded to the simpler: > > - If the target is not declared `global` or `nonlocal` in the block > containing F, then the target is bound in the block containing F. > ... I'm satisfied that captures the intent - but now it's misleadingly wordy. It should be the briefer: - The target is bound in the block containing F. Other text (in section 4.2.2) already covers the intended meanings for when a `global` or `nonlocal` declaration appears in the block too. And then it's short enough again that the bullet list isn't really helpful anymore. So, putting that all together: """ An assignment expression binds the target, except in a function F synthesized to implement a list comprehension or generator expression (see XXX). In the latter case[1], the target is bound in the block containing F, and errors may be detected: If the target also appears as an identifier target of a `for` loop header in F, a `SyntaxError` exception is raised. If the block containing F is a class block, a `SyntaxError` exception is raised. Footnote: [1] The intent is that runtime binding of the target occurs as if the binding were performed in the block containing F. Because that necessarily makes the target not local in F, it's an error if the target also appears in a `for` loop header, which is a local binding for the same target. If the containing block is a class block, F has no access to that block's scope, so it doesn't make sense to consider the containing block. The target is bound in the containing block, where it inherits that block's `global` or `nonlocal` declaration if one exists, else establishes that the target is local to that block. """ Along with the previous: > ... targets that are identifiers if occurring in an assignment, ... > > " statement" should be inserted before the comma. Of course there are other docs that need to change too. I viewed this as the hardest part (witness how long it took to reduce it to near-triviality ;-) ), so wanted to get it out of the way first. From rob.cliffe at btinternet.com Sun May 13 20:22:24 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 14 May 2018 01:22:24 +0100 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: On 13/05/2018 19:19, Guido van Rossum wrote: > As anyone still following the inline assignment discussion knows, a > problem with designing new syntax is that it's hard to introduce new > keywords into the language, since all the nice words seem to be used > as method names in popular packages. (E.g. we can't use 'where' > because there's numpy.where > , > and we can't use 'given' because it's used in Hypothesis > .) > > The idea I had (not for the first time :-) is that in many syntactic > positions we could just treat keywords as names, and that would free > up these keywords. > > For example, we could allow keywords after 'def' and after a period, > and then the following would become legal: > > class C: > ??? def and(self, other): > ??????? return ... > > a = C() > b = C() > print(a.and(b)) > > This does not create syntactic ambiguities because after 'def' and > after a period the grammar *always* requires a NAME. > > There are other positions where we could perhaps allow this, e.g. in a > decorator, immediately after '@' (the only keyword that's > *syntactically* legal here is 'not', though I'm not sure it would ever > be useful). > > This would not prevent code breakage when a new keyword was added. It would only reduce the amount of code broken.? Although in my unsubstantiated opinion not by very much; I suspect that most of the time an identifier is used in a module, it is used at least once in contexts where it would still be a SyntaxError if it were a keyword. Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun May 13 21:00:55 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 14 May 2018 13:00:55 +1200 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: <5AF8DFC7.909@canterbury.ac.nz> Guido van Rossum wrote: > Of course this would still not help for names of functions that might be > imported directly (do people write 'from numpy import where'?). Maybe things could be rigged so that if you use a reserved word as a name in an import statement, it's treated as a name everywhere else in that module. Then "from numpy import where" would Just Work. -- Greg From greg.ewing at canterbury.ac.nz Sun May 13 21:07:49 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 14 May 2018 13:07:49 +1200 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: <5AF8E165.6060806@canterbury.ac.nz> Elias Tarhini wrote: > The thought of writing "operator.not" is appealing, but > being forced to use *that* (with *from operator import not* being > non-allowable) may not be. Under the proposal I made in my last post, "from operator import not" would be fine -- you just wouldn't be able to use the "not" operator anywhere in the module then. :-) A more nuanced version would have the effect restricted to the scope the import appears in, so that you could write def f(): from operator import not # 'not' is now an ordinary name inside this # function, but it's business as usual elsewhere Someone else can figure out how to make the parser handle this, though. :-) -- Greg From greg.ewing at canterbury.ac.nz Sun May 13 21:28:51 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 14 May 2018 13:28:51 +1200 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: <78e8c1d4-0597-a3ae-1f52-b7e0d192df4e@btinternet.com> References: <78e8c1d4-0597-a3ae-1f52-b7e0d192df4e@btinternet.com> Message-ID: <5AF8E653.6000105@canterbury.ac.nz> Rob Cliffe via Python-ideas wrote: > def and(x, y): > return ... > > # and(1,2) # Oops, SyntaxError. Oh, I know: > globals()['and'](1,2) # Works! If the rule I proposed for "import" were extended to "def" then and(1,2) would work. The usual way of using "and" would no longer work in that module, but this just goes to show that redefining "and" is a silly thing to do in the first place. Redefining the existing keywords could perhaps be forbidden if you really want to protect people from shooting themselves in the kidneys this particular way. -- Greg From carl.input at gmail.com Sun May 13 22:30:59 2018 From: carl.input at gmail.com (Carl Smith) Date: Mon, 14 May 2018 03:30:59 +0100 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: <5AF8E653.6000105@canterbury.ac.nz> References: <78e8c1d4-0597-a3ae-1f52-b7e0d192df4e@btinternet.com> <5AF8E653.6000105@canterbury.ac.nz> Message-ID: New Python versions already use backwards incompatible syntax, but it has to be a strict superset so the new parser still parses old code. If a new Python version introduced syntax that would break old code, it would still work, so long as the parser knew which syntax the file contained, and was able to parse both versions. I personally think Python 3 should have always had a new file extension (mainly to help editors disambiguate), but some kind of directive at the top of the file would work as well, a bit like JavaScript's Strict Mode. The killer objection is that you have to maintain two parsers or some kind of mashup, but still, the old syntax would be frozen, there could be a lot of reuse in the front end, and there would still only be one VM. I'm not hoping this happens any time soon, but if was a choice between devs maintaining two parsers or users migrating to Python 4, as a user... -- Carl Smith carl.input at gmail.com On 14 May 2018 at 02:28, Greg Ewing wrote: > Rob Cliffe via Python-ideas wrote: > >> def and(x, y): >> return ... >> >> # and(1,2) # Oops, SyntaxError. Oh, I know: >> globals()['and'](1,2) # Works! >> > > If the rule I proposed for "import" were extended to "def" > then and(1,2) would work. The usual way of using "and" would > no longer work in that module, but this just goes to show > that redefining "and" is a silly thing to do in the first > place. > > Redefining the existing keywords could perhaps be forbidden > if you really want to protect people from shooting themselves > in the kidneys this particular way. > > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun May 13 22:38:03 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 13 May 2018 22:38:03 -0400 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: <5AF8DFC7.909@canterbury.ac.nz> References: <5AF8DFC7.909@canterbury.ac.nz> Message-ID: On Sun, May 13, 2018 at 9:00 PM, Greg Ewing wrote: > Guido van Rossum wrote: >> >> Of course this would still not help for names of functions that might be >> imported directly (do people write 'from numpy import where'?). > > > Maybe things could be rigged so that if you use a reserved word > as a name in an import statement, it's treated as a name everywhere > else in that module. Then "from numpy import where" would Just Work. 'from numpy import *' is also common. -n -- Nathaniel J. Smith -- https://vorpus.org From rob.cliffe at btinternet.com Sun May 13 22:47:07 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 14 May 2018 03:47:07 +0100 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: On 13/05/2018 19:19, Guido van Rossum wrote: > As anyone still following the inline assignment discussion knows, a > problem with designing new syntax is that it's hard to introduce new > keywords into the language, since all the nice words seem to be used > as method names in popular packages. (E.g. we can't use 'where' > because there's numpy.where > , > and we can't use 'given' because it's used in Hypothesis > .) > > The idea I had (not for the first time :-) is that in many syntactic > positions we could just treat keywords as names, and that would free > up these keywords. > > For example, we could allow keywords after 'def' and after a period, > and then the following would become legal: > > class C: > ??? def and(self, other): > ??????? return ... > > a = C() > b = C() > print(a.and(b)) > > This does not create syntactic ambiguities because after 'def' and > after a period the grammar *always* requires a NAME. > > There are other positions where we could perhaps allow this, e.g. in a > decorator, immediately after '@' (the only keyword that's > *syntactically* legal here is 'not', though I'm not sure it would ever > be useful). > > Please, imagine how you would write the documentation to explain this. Then, please, put yourself in the position of someone who teaches Python trying to explain it. Regards Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Sun May 13 22:38:21 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 14 May 2018 03:38:21 +0100 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: <5AF8E653.6000105@canterbury.ac.nz> References: <78e8c1d4-0597-a3ae-1f52-b7e0d192df4e@btinternet.com> <5AF8E653.6000105@canterbury.ac.nz> Message-ID: <01369ab4-cc72-72e5-cbde-fb782ad7f83b@btinternet.com> On 14/05/2018 02:28, Greg Ewing wrote: > Rob Cliffe via Python-ideas wrote: >> ??? def and(x, y): >> ??????? return ... >> >> #?? and(1,2)?????????? #? Oops, SyntaxError.? Oh, I know: >> ??? globals()['and'](1,2)??? # Works! > > If the rule I proposed for "import" were extended to "def" > then and(1,2) would work. The usual way of using "and" would > no longer work in that module, but this just goes to show > that redefining "and" is a silly thing to do in the first > place. > > Redefining the existing keywords could perhaps be forbidden > if you really want to protect people from shooting themselves > in the kidneys this particular way. > If you forbid redefining keywords, you remove the whole point of this proposal: to allow keywords to be sometimes used as bona fide keywords, sometimes as identifiers. I really do not intend to give offence (I know that that's counter-productive).? But I have formed a very strong opinion - which of course may be wrong, and certainly won't be universally shared - and I am honestly expressing that opinion: Nothing I have seen in this thread so far has persuaded me that this proposal has any merit. From carl.input at gmail.com Sun May 13 23:04:06 2018 From: carl.input at gmail.com (Carl Smith) Date: Mon, 14 May 2018 04:04:06 +0100 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: As long as any new syntax allowed us to still reference the occassional thing in the older libraries that used newly reserved names, it would not come up that often, and would avoid the biggest cost of a whole new version. If `foo` was a reserved word, then this could be allowed... import foo as bar ...but not these... import foo import bar as foo The same could be done with params, so this would be illegal... def f(foo): ... ...but this would be fine... f(foo=1) It would be illegal to define a property named `foo`, but you could still do `bar.foo` to use a library, etc. It could be done, but it's not especially relevant here, so I'll shut up now. -- Carl Smith carl.input at gmail.com On 14 May 2018 at 03:47, Rob Cliffe via Python-ideas < python-ideas at python.org> wrote: > > > On 13/05/2018 19:19, Guido van Rossum wrote: > > As anyone still following the inline assignment discussion knows, a > problem with designing new syntax is that it's hard to introduce new > keywords into the language, since all the nice words seem to be used as > method names in popular packages. (E.g. we can't use 'where' because > there's numpy.where > , > and we can't use 'given' because it's used in Hypothesis > .) > > The idea I had (not for the first time :-) is that in many syntactic > positions we could just treat keywords as names, and that would free up > these keywords. > > For example, we could allow keywords after 'def' and after a period, and > then the following would become legal: > > class C: > def and(self, other): > return ... > > a = C() > b = C() > print(a.and(b)) > > This does not create syntactic ambiguities because after 'def' and after a > period the grammar *always* requires a NAME. > > There are other positions where we could perhaps allow this, e.g. in a > decorator, immediately after '@' (the only keyword that's *syntactically* > legal here is 'not', though I'm not sure it would ever be useful). > > > Please, imagine how you would write the documentation to explain this. > Then, please, put yourself in the position of someone who teaches Python > trying to explain it. > Regards > Rob Cliffe > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun May 13 23:58:15 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 13 May 2018 23:58:15 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512172427.GF12683@ando.pearwood.info> Message-ID: On 12 May 2018 at 14:13, Tim Peters wrote: > Just clarifying a fine point here: > > [Steven D'Aprano ] > > ... > > average = 0 > > smooth_signal = [(average := (1-decay)*average + decay*x) for x in > signal] > > assert average == smooth_signal[-1] > > > The scope issues are logically independent of assignment-expression > spelling, but it's a pretty safe guess Nick is opposed to that example > ever "just working" regardless of spelling, while PEP 572 doesn't > currently support it anyway. I'm personally fine with that example working if there's an explicit nonlocal declaration on "average" in the nested scope - it's Guido that objected to requiring the explicit scoping declaration to access that behaviour. For the implicit version, my request is that any PEP proposing the idea of parent local scoping be held to the standard of *actually drafting the patch for the language specification*, rather than handwaving away the hard problems that it creates (i.e. what to do at class scope, what to do when multiple generators expressions reference the same nonlocal name, what to do with nested comprehensions, how to expand comprehensions using this kind of scoping to their statement form in a context independent way). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon May 14 00:02:21 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 14 May 2018 00:02:21 -0400 Subject: [Python-ideas] "given" vs ":=" in list comprehensions In-Reply-To: References: Message-ID: On 12 May 2018 at 20:34, Andre Roberge wrote: > Sorry for chiming in so late; I was lurking using google groups and had to > subscribe to post - hence this new thread. > > I gather that *where* has been discarded as a possible new keywords given > its use as a function in numpy (https://docs.scipy.org/doc/ > numpy-1.14.0/reference/generated/numpy.where.html) ... Still, I will > include it below for completeness (and as I think it reads better than the > other choices) > The same grammar adjustment that I believe will allow "given" to be used as both a postfix keyword and as a regular name would also work for "where". However, "where" still has the problem of semantically conflicting with SQL's use of it to introduce a filter clause, whereas Hypothesis uses "given" to bind names to values (just a little more indirectly than would be the case for assignment expressions) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon May 14 00:27:10 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 14 May 2018 00:27:10 -0400 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: On 13 May 2018 at 20:00, Tim Peters wrote: > [Tim] > >> - If the target is not local to any function enclosing F, and is not > >> declared `global` in the block containing F, then the target is bound > >> in the block containing F. > > [also Tim] > > FYI, that's still not right, ... > > I suspect that the above should be reworded to the simpler: > > > > - If the target is not declared `global` or `nonlocal` in the block > > containing F, then the target is bound in the block containing F. > > ... > > I'm satisfied that captures the intent - but now it's misleadingly > wordy. It should be the briefer: > > - The target is bound in the block containing F. > > Other text (in section 4.2.2) already covers the intended meanings for > when a `global` or `nonlocal` declaration appears in the block too. > > And then it's short enough again that the bullet list isn't really > helpful anymore. So, putting that all together: > > """ > An assignment expression binds the target, except in a function F > synthesized to implement a list comprehension or generator expression > (see XXX). In the latter case[1], the target is bound in the block > containing F, and errors may be detected: If the target also appears > as an identifier target of a `for` loop header in F, a `SyntaxError` > exception is raised. If the block containing F is a class block, a > `SyntaxError` exception is raised. > > Footnote: > [1] The intent is that runtime binding of the target occurs as if the > binding were performed in the block containing F. Because that > necessarily makes the target not local in F, it's an error if the > target also appears in a `for` loop header, which is a local binding > for the same target. If the containing block is a class block, F has > no access to that block's scope, so it doesn't make sense to consider > the containing block. The target is bound in the containing block, > where it inherits that block's `global` or `nonlocal` declaration if > one exists, else establishes that the target is local to that block. > """ > This is getting pretty close to being precise enough to be at least potentially implementable (thanks!), but there are still two cases that would need to be covered: - what happens inside a lambda expression? - what happens inside another comprehension or generator expression? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon May 14 00:36:11 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 14 May 2018 00:36:11 -0400 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: On 13 May 2018 at 14:19, Guido van Rossum wrote: > As anyone still following the inline assignment discussion knows, a > problem with designing new syntax is that it's hard to introduce new > keywords into the language, since all the nice words seem to be used as > method names in popular packages. (E.g. we can't use 'where' because > there's numpy.where > , > and we can't use 'given' because it's used in Hypothesis > .) > > The idea I had (not for the first time :-) is that in many syntactic > positions we could just treat keywords as names, and that would free up > these keywords. > While I think the "restricted use" idea would be confusing, I do like the idea of separating out "postfix keywords", which can't start a statement or expression, and hence can be used *unambiguously* as names everywhere that names are allowed. Adding such a capability is essential to proposing a keyword based approach to inline assignments, and would technically also allow "and", "or", "is", and "as" to be freed up for use as names. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon May 14 00:46:00 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 14 May 2018 00:46:00 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512172427.GF12683@ando.pearwood.info> Message-ID: On 11 May 2018 at 12:45, Tim Peters wrote: > [Nick Coghlan] > > I've been thinking about this problem, and I think for the If/elif/while > > cases it's actually possible to allow the "binding is the same as the > > condition" case to be simplified to: > > > > if command = pattern.match(the_string): > > ... > > elif command = other_pattern.match(the_string): > > ... > > > > while data = read_data(): > > Unless there's some weird font problem on my machine, that looks like > a single "equals sign". In which case we'd be reproducing C's > miserable confusion about whether: > > if (i = 1) > > was a too-hastily-typed spelling of the intended: > > if (i == 1) > > or whether they were thinking "equals" and typed "=" by mistake. > > If so, that would get an instant -1 from any number of core devs, who > have vivid painful memories of being burned by that in C. That's not > just speculation - it came up a number of times in the PEP 572 > threads. > I was one of those core devs, and would personally prefer to require that folks spell the inline binding completely unambiguously as "if i given i = 1:". However, if the repetition of "i" is considered a deal breaker relative to ":=" (even though the status quo already requires repetition of the target name in the condition), then I'd prefer to add this shorthand (which folks can then opt to prohibit in favour of the more explicit form in their style guides) over adding the cognitive complexity of deciding when to use "i = 1" and when to use "i := 1". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Mon May 14 01:39:56 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 14 May 2018 00:39:56 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512172427.GF12683@ando.pearwood.info> Message-ID: [Steven D'Aprano ] >>> ... >>> average = 0 >>> smooth_signal = [(average := (1-decay)*average + decay*x) for x in signal] >>> assert average == smooth_signal[-1] [Tim] >> The scope issues are logically independent of assignment-expression >> spelling, but it's a pretty safe guess Nick is opposed to that example >> ever "just working" regardless of spelling, while PEP 572 doesn't >> currently support it anyway. [Nick Coghlan ] > I'm personally fine with that example working "Just working" meant "exactly as written". "Regardless of spelling" meant whether binding is spelled via ":=", "given", "as", "where", "let ... in ...' "->", .. > if there's an explicit nonlocal declaration on "average" in the nested > scope - it's Guido that objected to requiring the explicit scoping > declaration to access that behaviour. I suspect, but don't know, that Guido would like that example to "just work" because it _looks like_ it should "just work". There's no visible function involved, and _needing_ to add scope declarations to make it work is pretty much inexplicable unless the user first learns more than most users "should" need to learn about how it's implemented. If so, no technical argument will change his mind - and especially not one based on "but the implementation today doesn't do that already". Recall that Python had no lexically nested scoping at first? If he wanted to _make_ users learn about nested lexical scopes to use Python features that don't _appear_ to use it, Python would have had it from the start ;-) I'd be happy enough with needing an explicit declaration too, but am _happiest_ with what I'm guessing Guido's view is. > For the implicit version, my request is that any PEP proposing the idea of > parent local scoping be held to the standard of *actually drafting the patch > for the language specification*, Of course. > rather than handwaving away the hard problems that it creates > (i.e. what to do at class scope, I finally looked at class scope and quickly decided it makes no sense there (a comprehension at class scope has no access to the class scope, so "same scope in the comprehension as in its immediately containing block" is incoherent in that case). > what to do when multiple generators expressions reference the same > nonlocal name, Then, as you said, they access the same nonlocal name. What of it? What if two generator functions reference (or even rebind) the same nonlocal name? What's unclear about that? If you were happy with explicit scope declarations, then exactly the same thing would happen as if the user were forced to explicitly declare the scopes chosen for them. > what to do with nested comprehensions, Again, if you were happy with explicit scope declarations, then exactly the same ... Where are the docs explaining how nested comprehensions work today? I haven't seen any. If any such exist, I'd bet nothing about them needs to be changed. If none such exist, I don't see a need to write any just for this PEP. How do nested expressions of any kind work? Same thing. The only thing the suggestion changes is the scope of assignment expression targets in synthetic functions created to implement comprehensions. That has nothing at all to do with the possibility of nesting, or with the structure of nesting. Why do you think it does - or might? > how to expand comprehensions using this kind of scoping to their statement > form in a context independent way). I've already explained why I view that as a non-issue (making a tedious manual process a relative handful of users undertake once in their life for self-education purposes slightly less tedious has approximately no value to me - and, to the contrary, the "context dependent" bits they may have to learn would make it _more_ educational). If that's a show-stopper for you, so be it. From greg.ewing at canterbury.ac.nz Mon May 14 01:57:05 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 14 May 2018 17:57:05 +1200 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: <01369ab4-cc72-72e5-cbde-fb782ad7f83b@btinternet.com> References: <78e8c1d4-0597-a3ae-1f52-b7e0d192df4e@btinternet.com> <5AF8E653.6000105@canterbury.ac.nz> <01369ab4-cc72-72e5-cbde-fb782ad7f83b@btinternet.com> Message-ID: <5AF92531.8090107@canterbury.ac.nz> Rob Cliffe via Python-ideas wrote: > If you forbid redefining keywords, you remove the whole point of this > proposal: I mean the keywords that are in the language as of now. There will never be a need to redefine those, since no current code uses them as names. -- Greg From greg.ewing at canterbury.ac.nz Mon May 14 02:13:48 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 14 May 2018 18:13:48 +1200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512172427.GF12683@ando.pearwood.info> Message-ID: <5AF9291C.1070301@canterbury.ac.nz> Can someone explain to me why it was considered a bad thing that for-clauses leaked names in comprehensions, but it will be a good thing for inline assignments to leak names from them? -- Greg From contact at brice.xyz Mon May 14 03:09:53 2018 From: contact at brice.xyz (Brice Parent) Date: Mon, 14 May 2018 09:09:53 +0200 Subject: [Python-ideas] PEP 572: about the operator precedence of := In-Reply-To: References: Message-ID: Le 10/05/2018 ? 09:32, Terry Reedy a ?crit?: > On 5/9/2018 11:33 PM, Guido van Rossum wrote: > >> I now think that the best way out is to rule `:=` in the top level >> expression of an expression statement completely > > I would like to be able to interactively enter > > >>> a: = f(2,4) > > to have 'a' echoed as well as bound. > This behaviour is reachable by using: print(a := f(2,4)) Otherwise, I can see coming a new wave of devs saying assignment with "=" is outdated, and we should use the new assignment operator ":=". Cleaning code by removing useless prints is something, having to go through the syntax to remove the ":" of some ":=", and having to test everything again to be sure we didn't break anything is another matter (long live unit tests! But still...). Side note: I wouldn't recommand to use `print(a := f(2,4))` either, as it would be a print-with-a-side-effect. Which would then also break the code if removed the usual way by deleting the line (or commenting it if you like messy). Maybe, for another proposal of course, we could allow the shell to be more verbose, by printing any explicit assignment (whether we used "=" or ":=" wouldn't matter)? -Brice From tim.peters at gmail.com Mon May 14 03:18:17 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 14 May 2018 02:18:17 -0500 Subject: [Python-ideas] A comprehension scope issue in PEP 572 In-Reply-To: References: Message-ID: [Tim] >> """ >> An assignment expression binds the target, except in a function F >> synthesized to implement a list comprehension or generator expression >> (see XXX). In the latter case[1], the target is bound in the block >> containing F, and errors may be detected: If the target also appears >> as an identifier target of a `for` loop header in F, a `SyntaxError` >> exception is raised. If the block containing F is a class block, a >> `SyntaxError` exception is raised. >> >> Footnote: >> [1] The intent is that runtime binding of the target occurs as if the >> binding were performed in the block containing F. Because that >> necessarily makes the target not local in F, it's an error if the >> target also appears in a `for` loop header, which is a local binding >> for the same target. If the containing block is a class block, F has >> no access to that block's scope, so it doesn't make sense to consider >> the containing block. The target is bound in the containing block, >> where it inherits that block's `global` or `nonlocal` declaration if >> one exists, else establishes that the target is local to that block. >> """ [Nick] > This is getting pretty close to being precise enough to be at least > potentially implementable (thanks!), but there are still two cases that > would need to be covered: > > - what happens inside a lambda expression? Section 4 of the Reference Manual doesn't contain the word "lambda", because there's no need to. "lambda" is just another way to create a function, and the behavior of functions is already specified. If you disagree, you mean something by "lambda expression" other than what I take it to mean, and the best way to illustrate what you do mean would be via giving a concrete example. As far as I'm concerned, ... (lambda...: expression) ... is exactly the same as def _hidden_unique_name(...): return expression ... (_hidden_unique_name) .... Even at class scope ;-) For example, def f(): g = lambda n: [(n := n+1) for i in range(1)] return g(10) is the same as: def f(): def g(n): return [(n := n+1) for i in range(1)] return g(10) When the listcomp synthesizes a function, g's code block will immediately contain it. The text at the top says `n` is bound in the containing block - which is g's. `n` is already local to `g`, so that part is just redundant in this case. The synthetic function will take 10 (via its nonlocal cell), add 1, and again via the cell rebind g's `n` to 11. The synthetic function returns [11] and the rebound `n` vanishes with its execution frame. But that's all about how functions behave; "lambda" is just incidental. > - what happens inside another comprehension or generator expression? I don't know what you're asking about, so I'll just copy part of a different reply: """ Where are the docs explaining how nested comprehensions work today? I haven't seen any. If any such exist, I'd bet nothing about them needs to be changed. If none such exist, I don't see a need to write any just for this PEP. How do nested expressions of any kind work? Same thing. The only thing the suggestion changes is the scope of assignment expression targets in synthetic functions created to implement comprehensions. That has nothing at all to do with the possibility of nesting, or with the structure of nesting. Why do you think it does - or might? """ Note that a day or two ago I posted a complete expansion of what list(i + sum((i := i+1) for j in range(i)) + i for i in range(5)) would do. There the inner genexp rebinds the outer genexp's local for-target. Atrocious. Here's how you can do the same: Replace `(i := i+1)` with `(LATER)`. Generate the complete expansion for how the assignment-expression-free derived statement is implemented, presumably following the detailed docs that don't appear to exist ;-) for how that's done. In the synthetic function created for the inner genexp, add nonlocal i at the top and replace yield (LATER) with yield (i := i+1) Now I didn't replace anything with "LATER", because I never thought adding a new binary operator had anything to do with this process to begin with ;-) All that's needed is to add cruft _after_ it's done to establish the intended scopes for assignment statement targets. If you ask how the inner genexp "knows" that it needs to access the outer genexp's `i`, it doesn't directly. It's simply that the synthetic inner genexp function is nested inside the synthetic outer genexp function, and the text at top says that the inner genexp binds `i` in its containing block - which is the block for the synthetic outer genexp. If the functions synthesized for nested comprehensions don't _already_ nest in this way, then they'd already screw up merely accessing outer comprehension names from within inner comprehensions. Is there a reason to suspect that there's anything inherently unique to that specific example? I did a bunch of these "by hand", _trying_ to create problems, but didn't manage to. The only times I got even slightly flustered were when brain fog temporarily blocked my ability to see how to generate the nested messes _entirely independent_ of that they happened to contain assignment expressions. As I also posted about earlier, the real problems I've seen were in corner cases _without_ the suggested change, stemming from that different pieces of a comprehension execute in different scopes. That can have ugly consequences when an assignment expression appears in the outermost for's iterable, and its target is also in the body of the comprehension. That doesn't even require nesting to become incoherent. [y for _ in range(y := 42)] [y for y in range(y := 42)] With the suggestion, a binding expression target resolves to the same scope regardless of where it appears in the genexp/listcomp, so that class of head-scratcher vanishes ("same scope as in the containing block" implies that all instances of the target resolve to the same scope, which, happily enough, is resolved in the very block the outermost iterable _is_ executed in; so in the first example above both instances of `y` are resolved in the block containing the listcomp, and the second example is a compile-time error for the coherent reason given in the text at the top). From tim.peters at gmail.com Mon May 14 03:43:56 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 14 May 2018 02:43:56 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AF9291C.1070301@canterbury.ac.nz> References: <20180512172427.GF12683@ando.pearwood.info> <5AF9291C.1070301@canterbury.ac.nz> Message-ID: [Greg Ewing '] > Can someone explain to me why it was considered a bad > thing that for-clauses leaked names in comprehensions, Because you can't write a list comprehension or generator expression AT ALL without specifying a `for` loop header, so whether its target(s) leaks is an issue in virtually every listcomp/genexp ever written, and that people tend to use short (typically 1-letter) names for for-targets, so unintentional stomping on names is exceptionally likely in this context. > but it will be a good thing for inline assignments to leak names > from them? Because you never _need_ to use an assignment expression to write a listcomp/genexp. You have to go out of your way to use it. Which will probably be rare in listcomps;genexps,, not virtually 100% of the time as with for-targets. Then you get what you went out of your way to explicitly ask for: a name for some subexpression result. Otherwise it's essentially impossible to explain why: total = 0 sums = [total := total + value for value in data] assert sums[-1] == total "blows up", despite that its intent is obvious, unless you first explain to a user how the listcomp is implemented via an invisible synthetic function created by magic, inside of which `total` has nothing to do with the `total` they see on the first line. UnboundLocalError - WTF? That's why leaking "is good". It works both directions: the outer name leaks _into_ the body too, not just _out_ of it. Things that "look like" they should obviously work do work then, and a user can remain blissfully unaware of the implementation.. Of course you can also find cases in which it's not wanted. If it doesn't leak the kind of use shown above can't be done at all via listcomps (at least not straightforwardly). If it does leak, the subset of cases where leaking is unwanted _of_ the subset of cases in which a listcomp//genexp uses an assignment expression at all are indeed inconvenienced. So - surprise! It's a tradeoff, something we've never faced before ;-) From j.van.dorp at deonet.nl Mon May 14 04:12:15 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Mon, 14 May 2018 10:12:15 +0200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512172427.GF12683@ando.pearwood.info> <5AF9291C.1070301@canterbury.ac.nz> Message-ID: I think we're approaching this from the wrong direction. My point is, expression assignments dont have the complex case as purpose - most coders wont try to maximize line information density. If you're doing magic, you might as well spell it out over multiple lines, because neither := nor given will be readable. However, in the simple case, it does matter a lot - and there := beats out given by a mile. If you're breaking your lines to avoid line length violations, you might as well put your assignments on a separate lines first. Im inclined to argue that if assignment expressions of any form forces to you make a multi-line statement, you're doing it wrong. (in the vast majority of cases). Consider it like the ternary operator - good to compact simple constructs, but while you can use it for black magic, you probably shouldn't. How about we just explicitly advice to keep it simple, stupid, and update PEP 8 to state that if assignment expressions take you to multi-line, split of the assignment and use statements instead. (truth be told, the general while (assignment expression isn't as clear cut as if-uses, but I think the idea holds.)) Jacco From greg.ewing at canterbury.ac.nz Mon May 14 05:17:03 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 14 May 2018 21:17:03 +1200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512172427.GF12683@ando.pearwood.info> <5AF9291C.1070301@canterbury.ac.nz> Message-ID: <5AF9540F.3030208@canterbury.ac.nz> Tim Peters wrote: > Because you never _need_ to use an assignment expression to write a > listcomp/genexp. This whole discussion started because someone wanted a way to bind a temporary result for use *within* a comprehension. Those use cases don't require leakage. > Otherwise it's essentially impossible to explain why: > > total = 0 > sums = [total := total + value for value in data] > assert sums[-1] == total > > "blows up", despite that its intent is obvious,unless you first > explain to a user how the listcomp is implemented via an invisible > synthetic function created by magic, inside of which `total` has > nothing to do with the `total` they see on the first line. It's no harder to explain that than it is to explain why x = 42 y = [x * x for x in range(5)] print(x) prints 42 rather than whatever value was last bound to the x in the comprehension. Seems to me it would be easier to explain that *all* names bound within a comprehension are local to the comprehension, than to have to say that some are and some aren't. -- Greg > UnboundLocalError - WTF? > > That's why leaking "is good". It works both directions: the outer > name leaks _into_ the body too, not just _out_ of it. Things that > "look like" they should obviously work do work then, and a user can > remain blissfully unaware of the implementation.. Of course you can > also find cases in which it's not wanted. > > If it doesn't leak the kind of use shown above can't be done at all > via listcomps (at least not straightforwardly). > > If it does leak, the subset of cases where leaking is unwanted _of_ > the subset of cases in which a listcomp//genexp uses an assignment > expression at all are indeed inconvenienced. > > So - surprise! It's a tradeoff, something we've never faced before ;-) From tim.peters at gmail.com Mon May 14 06:10:47 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 14 May 2018 05:10:47 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AF9540F.3030208@canterbury.ac.nz> References: <20180512172427.GF12683@ando.pearwood.info> <5AF9291C.1070301@canterbury.ac.nz> <5AF9540F.3030208@canterbury.ac.nz> Message-ID: [Greg Ewing '] > This whole discussion started because someone wanted a way > to bind a temporary result for use *within* a comprehension. It's been noted several times recently that the example PEP 572 gives as _not_ working: total = 0 progressive_sums = [total := total + value for value in data] was the original use case that prompted work on the PEP. You gotta admit that's ironic ;-) > Those use cases don't require leakage. I already said that - yes - some use cases don't want the leakage. Others do. And the vast majority of listcomps/genexps will likely never use an assignment expression anyway. If "no leak" wins, "want leak" can't do what they want at all. If "want leak" wins, the subset-of-a-subset "no leak" cases are inconvenienced. As I said, it's a tradeoff. You're giving 0 weight to one of the sides. >> Otherwise it's essentially impossible to explain why: >> >> total = 0 >> sums = [total := total + value for value in data] >> assert sums[-1] == total >> >> "blows up", despite that its intent is obvious,unless you first >> explain to a user how the listcomp is implemented via an invisible >> synthetic function created by magic, inside of which `total` has >> nothing to do with the `total` they see on the first line. > It's no harder to explain that than it is to explain > why > > x = 42 > y = [x * x for x in range(5)] > print(x) > > prints 42 rather than whatever value was last bound to > the x in the comprehension. You're overlooking the most relevant point: The example blows with an UnboundLocalError, which can't possibly be explained by looking at the example as it stands, or by reference to shallow tricks. You _need_ to drag in stuff about invisible (in the code) synthesized scopes in which `total` is in fact an unbound local name. You don't need that heavy machinery to explain why for-target names don't leak; indeed, shallow implementation tricks were used to achieve that well before synthetic functions _were_ used to implement listcomps. > Seems to me it would be easier to explain that *all* names > bound within a comprehension are local to the comprehension, > than to have to say that some are and some aren't. for-targets are local, assignment expression targets aren't. I agree that's harder to explain, but on a scale of 1 to 10? 1. It's a tradeoff. And I cheerfully just gave 1 point to the side you favor. By my accounting, "leak" is still wining by about 136 to 17 ;-) From dmoisset at machinalis.com Mon May 14 06:47:55 2018 From: dmoisset at machinalis.com (Daniel Moisset) Date: Mon, 14 May 2018 11:47:55 +0100 Subject: [Python-ideas] Syntax idea: escaping names to avoid keyword ambiguity Message-ID: Following up some of the discussions about the problems of adding keywords and Guido's proposal of making tokenization context-dependent, I wanted to propose an alternate way to go around the problem. My proposal essentially boils down to: 1. The character "$" can be used as a prefix of identifiers. formally, *identifier * ::= ["$"] xid_start xid_continue * 2. The "$" character is not part of the name. So the program "foo=3;print($foo)" prints 3. So does the program "$foo=3; print(foo)". Both set an entry to globals["foo"] and keep globals["$foo"] unset. 3. if "$" appears in a token, it's always an identifier. So "$with", "$if", "$return" are all identifiers. If you overcome the "yikes, that looks like awk/bash/perl/php, and I don't like those", and consider it as an escape for "unusual"/"deprecation" situations, I think it's not a bad chose, and allows to a simple solutions to many problems that have been in discussion recently and not so recently. [examples below] For me the benefits of this approach are: - It's very simple to explain how to use and its semantics - It (seems to me it) should be easy to explain to a python apprentice what a "$" means in code they read on a book/blogpost/manual - It's very easy to implement, minimal changes in the tokenizer - It's also easy to implement/integrate in other tools (editors with syntax highlighters, code formatters, etc) - It is easy to see that it's 100% backwards compatible (I understand that "$" has never been used in python before) - It is relatively unsurprising in the sense that other languages are already using $ to label names (there may be some point of confusion to people coming from javascript where "$" is a valid character in names and is not ignored). - It gives python devs and users a clear, easy and universal upgrade path when keywords are added (language designers: Add a __future__ import to enable keyword in python N+1, add warnings to change kw --> $kw in python N+2, and then turn it on by default in python N+3... ; developers: add the import when they want to upgrade , and fix their code with a search&replace when adding the import or after getting a warning). - It allows you to use new features even if some libraries were written for older python versions, depending the deprecation period (this could be improved with sth I'll write in another email, but that's the topic for another proposal) - When clashes occur, which they always do, there's one obvious way to disambiguate (see today the "class_" argument for gettext.translation, the "lambd" argument for random.expovariate, the "class_" filter in libraries like pyquery for CSS class, functions like pyquery, sqlalchemy.sql.operators.as_ , etc. Not counting all the "cls" argument to every classmethod ever) - If we're worried about over proliferation of "$" in code, I'm quite sure given past experience that just a notice in PEP 8 of "only with $ in names to prevent ambiguity" should be more than enough for the community What are the drawbacks that you find in this? Best, Daniel [The rest of this post is just examples] Example 1: Python 3.92 has just added a future import that makes "given" a keyword. Then you can do: # This works because we have no future import *from* hypothesis *import* given, strategies *as* st @given(st.integers()) *def* foo(i): x = f(i)**2 + f(i)**3 .... if you want to use the new feature (or upgraded to python 3.93 and started receiving warnings) you can then change it to: *from* __future__ *import* given_expression *from* hypothesis *import* $given, strategies *as* st @$given(st.integers()) *# If you forget the $ you get a SyntaxError* *def* foo(i): x = z**2 + z**3 *given* z = f(i) .... And also you could do: *from* __future__ *import* given_expression *import* hypothesis @hypothesis.$given(hypothesis.strategies.integers()) *def* foo(i): x = z**2 + z**3 *given* z = f(i) .... Or even, if you want to prevent the "$" all over your code: *from __future__ import given_expressionfrom* hypothesis *import* $given *as* hgiven, strategies *as* st @hgiven(st.integers()) *def* foo(i): x = z**2 + z**3 *given* z = f(i) .... If you have some library which uses a new keyword as a method name (you can't rename those with import ... as ...), it still works perfectly: *from* mylib *import* SomeClass instance = SomeClass() instance.$given("foo") This is also helpful as a universal way to solve name clashes between python keywords and libraries that use some external concept that overlaps with python (from https://pythonhosted.org/pyquery/attributes.html ): >>> *import* pyquery *as* pq >>> p = pq(*'

'*)(*'p'*) >>> p.attr(id=*'hello'*, $class=*'hello2'*) *[]* Or even nameclashes within python itself @classmethod *def *new_with_color($class, color): *# Instead of the usual cls or class_* result = $class() result.set_color(color) return result -- Daniel Moisset UK COUNTRY MANAGER A: 1 Fore Street, EC2Y 9DT London P: +44 7398 827139 <+44+7398+827139> M: dmoisset at machinalis.com | S: dmoisset Machinalis Limited is a company registered in England and Wales. Registered number: 10574987. -------------- next part -------------- An HTML attachment was scrubbed... URL: From leewangzhong+python at gmail.com Mon May 14 06:49:25 2018 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Mon, 14 May 2018 06:49:25 -0400 Subject: [Python-ideas] A cute Python implementation of itertools.tee In-Reply-To: References: Message-ID: The topic reminded me of Stephan Houben's streamtee. https://github.com/stephanh42/streamtee/blob/master/streamtee.py It was an attempt at a memory-efficient tee, but it turned out tee was efficient enough. It uses a thunk-like method and recursion to remove the need for an explicit linked list. In the spirit of the topic, I tried to strip it down to the bare essentials: def tee(iterable, n): stream = _streamiter(iter(iterable)) return tuple(_wrapper(stream) for _ in range(n)) def _streamiter(itr): result = (next(itr), _streamiter(itr)) del itr # Release unneeded reference. while True: yield result def _wrapper(stream): while True: result, stream = next(stream) yield result Unfortunately, this code, too, breaks with the new StopIteration propagation rules. On Sun, Apr 15, 2018 at 1:05 AM, Tim Peters wrote: > Just for fun - no complaint, no suggestion, just sharing a bit of code > that tickled me. > > The docs for `itertools.tee()` contain a Python work-alike, which is > easy to follow. It gives each derived generator its own deque, and > when a new value is obtained from the original iterator it pushes that > value onto each of those deques. > > Of course it's possible for them to share a single deque, but the code > gets more complicated. Is it possible to make it simpler instead? > > What it "really" needs is a shared singly-linked list of values, > pointing from oldest value to newest. Then each derived generator can > just follow the links, and yield its next result in time independent > of the number of derived generators. But how do you know when a new > value needs to be obtained from the original iterator, and how do you > know when space for an older value can be recycled (because all of the > derived generators have yielded it)? > > I ended up with almost a full page of code to do that, storing with > each value and link a count of the number of derived generators that > had yet to yield the value, effectively coding my own reference-count > scheme by hand, along with "head" and "tail" pointers to the ends of > the linked list that proved remarkably tricky to keep correct in all > cases. > > Then I thought "this is stupid! Python already does reference > counting." Voila! Vast swaths of tedious code vanished, giving this > remarkably simple implementation: > > def mytee(xs, n): > last = [None, None] > > def gen(it, mylast): > nonlocal last > while True: > mylast = mylast[1] > if not mylast: > mylast = last[1] = last = [next(it), None] > yield mylast[0] > > it = iter(xs) > return tuple(gen(it, last) for _ in range(n)) > > There's no need to keep a pointer to the start of the shared list at > all - we only need a pointer to the end of the list ("last"), and each > derived generator only needs a pointer to its own current position in > the list ("mylast"). > > What I find kind of hilarious is that it's no help at all as a > prototype for a C implementation: Python recycles stale `[next(it), > None]` pairs all by itself, when their internal refcounts fall to 0. > That's the hardest part. > > BTW, I certainly don't suggest adding this to the itertools docs > either. While it's short and elegant, it's too subtle to grasp easily > - if you think "it's obvious", you haven't yet thought hard enough > about the problem ;-) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From steve at pearwood.info Mon May 14 06:55:02 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 14 May 2018 20:55:02 +1000 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AF9540F.3030208@canterbury.ac.nz> References: <20180512172427.GF12683@ando.pearwood.info> <5AF9291C.1070301@canterbury.ac.nz> <5AF9540F.3030208@canterbury.ac.nz> Message-ID: <20180514105501.GP12683@ando.pearwood.info> On Mon, May 14, 2018 at 09:17:03PM +1200, Greg Ewing wrote: > Tim Peters wrote: > >Because you never _need_ to use an assignment expression to write a > >listcomp/genexp. > > This whole discussion started because someone wanted a way > to bind a temporary result for use *within* a comprehension. > Those use cases don't require leakage. To reiterate what Tim already pointed out, that original usecase required a way to feed values *into* the comprehension. https://mail.python.org/pipermail/python-ideas/2018-February/048971.html There's no need for dedicated syntax for that if we can just set an variable and have it show up in the comprehension. [...] > It's no harder to explain that than it is to explain > why > > x = 42 > y = [x * x for x in range(5)] > print(x) > > prints 42 rather than whatever value was last bound to > the x in the comprehension. But it *is* harder to explain why comprehensions are their own scope in the first place. They don't look like they are in their own scope. They look like any other expression. No other expression (apart from lambda) runs in its own scope. Every other scope relates to a clear lexical separation: - modules - classes - functions except for comprehensions, which just returns a plain old list. (Generator expressions are a little fuzzier: they at least are equivalent to a lambda-with-yield, if such a thing existed.) It is sometimes useful to be able to reach into a generator expression and manipulate a variable between calls. That would make it a coroutine (to use the pre-async language), and that's why yield is an expression that returns a value, not just a statement. And it is sometimes useful to be able to see the value of comprehension variables after they have finished running. We have given up all of that to allow ease of implementation to drive the semantics, and that's okay. Its a trade-off. But we can decide the other way too, and choose more useful semantics for assignment expressions over an easier implementation. > Seems to me it would be easier to explain that *all* names > bound within a comprehension are local to the comprehension, > than to have to say that some are and some aren't. Of course it would be easier to explain. It wouldn't be as useful. If we wanted "easy to explain", we'd be using BASIC circa 1974, we wouldn't have async, generators, comprehensions, exceptions, decorators, classes, metaclasses, descriptors, Unicode, or floating point numbers. -- Steve From rhodri at kynesim.co.uk Mon May 14 07:05:18 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Mon, 14 May 2018 12:05:18 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AF9540F.3030208@canterbury.ac.nz> References: <20180512172427.GF12683@ando.pearwood.info> <5AF9291C.1070301@canterbury.ac.nz> <5AF9540F.3030208@canterbury.ac.nz> Message-ID: On 14/05/18 10:17, Greg Ewing wrote: > Tim Peters wrote: >> Because you never _need_ to use an assignment expression to write a >> listcomp/genexp. > > This whole discussion started because someone wanted a way > to bind a temporary result for use *within* a comprehension. I still don't find that argument compelling. If you have a sufficiently complicated comprehension that you need a temporary result bound, IMHO you are well on the way to (if not far past) the point where converting it into a more traditional for-loop will be much clearer. I'm not sure we should be encouraging people to write less clear code. -- Rhodri James *-* Kynesim Ltd From rhodri at kynesim.co.uk Mon May 14 07:26:51 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Mon, 14 May 2018 12:26:51 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <5AF540CB.3040708@canterbury.ac.nz> <36274fa7-400d-280f-e680-b44aa8e61bba@kynesim.co.uk> <20180511161724.GX9562@ando.pearwood.info> Message-ID: <1ebcd6b4-7abd-bbe2-d432-cd1b8f0eede3@kynesim.co.uk> On 12/05/18 01:41, Juancarlo A?ez wrote: >>> while (cmd := get_command()).token != CMD_QUIT: >>> cmd.do_something() >> >> > while get_command() as cmd: > if cmd.token == CMD_QUIT: > break > cmd.do_something() This would be exactly one of the patterns Python currently forces on me that I don't like. This "while expr as name" form is not a win in my book. -- Rhodri James *-* Kynesim Ltd From rhodri at kynesim.co.uk Mon May 14 07:35:10 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Mon, 14 May 2018 12:35:10 +0100 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: On 13/05/18 19:19, Guido van Rossum wrote: > The idea I had (not for the first time:-) is that in many syntactic > positions we could just treat keywords as names, and that would free up > these keywords. I'm not familiar with the innards of the parser and it's waaaay too long since I sat through a parsing course, but can you turn that inside out? Are there times when the compiler knows it must be looking at a keyword, not a name? I suspect not, given that arbitrary expressions can be statements, but someone else may have a more knowledgeable opinion. -- Rhodri James *-* Kynesim Ltd From e+python-ideas at kellett.im Mon May 14 08:19:24 2018 From: e+python-ideas at kellett.im (Ed Kellett) Date: Mon, 14 May 2018 13:19:24 +0100 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: <2fd2b1a9-561c-bcd5-d737-5dea0a14ab25@kellett.im> On 2018-05-14 12:35, Rhodri James wrote: > On 13/05/18 19:19, Guido van Rossum wrote: >> The idea I had (not for the first time:-)? is that in many syntactic >> positions we could just treat keywords as names, and that would free up >> these keywords. > > I'm not familiar with the innards of the parser and it's waaaay too long > since I sat through a parsing course, but can you turn that inside out? > Are there times when the compiler knows it must be looking at a keyword, > not a name?? I suspect not, given that arbitrary expressions can be > statements, but someone else may have a more knowledgeable opinion. Sure are. For example, just after `1 `, a name would be an error, while a keyword would be fine. But, more to the point, there are times when the parser knows (or could know) it can't be looking at a certain keyword. Suppose you want to parse at the beginning of an expression when you see "and". Currently, you're duty-bound to explode, because "and" cannot begin an expression. You *could* equally well not explode and, knowing that "and" cannot begin an expression, interpret it completely unambiguously as a name. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From e+python-ideas at kellett.im Mon May 14 08:24:19 2018 From: e+python-ideas at kellett.im (Ed Kellett) Date: Mon, 14 May 2018 13:24:19 +0100 Subject: [Python-ideas] "given" vs ":=" in list comprehensions In-Reply-To: References: Message-ID: <1531c1ab-7ef5-10fa-8f6c-aeae769e90ab@kellett.im> On 2018-05-14 05:02, Nick Coghlan wrote: > The same grammar adjustment that I believe will allow "given" to be used as > both a postfix keyword and as a regular name would also work for "where". > However, "where" still has the problem of semantically conflicting with > SQL's use of it to introduce a filter clause, whereas Hypothesis uses > "given" to bind names to values (just a little more indirectly than would > be the case for assignment expressions) I suspect that SQL is not high on the list of languages people might confuse with Python. FWIW, as I'm sure will have been mentioned, Haskell uses "where", and people seem to manage fine with it. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From steve at pearwood.info Mon May 14 08:55:28 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 14 May 2018 22:55:28 +1000 Subject: [Python-ideas] Inline changes to context, or PEP572 In-Reply-To: References: Message-ID: <20180514125527.GR12683@ando.pearwood.info> On Sat, May 12, 2018 at 08:13:01PM -0400, Juancarlo A?ez wrote: > A new thread just to suggest taking the discussion about PEP572 well beyond > python-ideas (PyConn is good for that). > > The least anyone should want is a language change that immediately gets > tagged on the networks as "don't use", or "use only for...", etc. *Every* feature should be tagged as "use only for ...". Every feature has uses that it works for, and uses that are inappropriate. With old features, we rarely think about the bad uses. (When was the last time you needed to tell someone not to use a while loop when a for loop over an iterable was better? Apart from beginners reading "Learn Python The Hard Way", I've never seen anyone do that.) With new features, some people do nothing but think about the bad uses. > To be honest, I'll likely be on the "don't use :=, unless" band of pundits As we all should. Just as we didn't run out and change every if...else block to try to squeeze it into a ternary if operator, or turn every for loop into a comprehension, so we shouldn't use binding expressions unless they make the code better. > (already a filibuster). ":=" is like going back to "reduce()", which is > almost defunct thanks to.. us! I don't think that it is nice of you to gloat about (almost) taking away a useful, much-beloved functional programming idiom. For some people, this comes across as "I want to make your code worse". Fortunately, reduce is alive and well in the functools module. -- Steve From ncoghlan at gmail.com Mon May 14 09:13:26 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 14 May 2018 09:13:26 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512172427.GF12683@ando.pearwood.info> <5AF9291C.1070301@canterbury.ac.nz> <5AF9540F.3030208@canterbury.ac.nz> Message-ID: On 14 May 2018 at 06:10, Tim Peters wrote: > [Greg Ewing '] > > This whole discussion started because someone wanted a way > > to bind a temporary result for use *within* a comprehension. > > It's been noted several times recently that the example PEP 572 gives > as _not_ working: > > total = 0 > progressive_sums = [total := total + value for value in data] > > was the original use case that prompted work on the PEP. You gotta > admit that's ironic ;-) > After pondering this case further, I think it's also worth noting that that *particular* example could also be addressed by: 1. Allowing augmented assignment *expressions* 2. Changing the scoping rules for augmented assignment operations in general such that they *don't change the scope of the referenced name* Writing "i += n" without first declaring the scope of "i" with "i = 0", "nonlocal i" or "global i" is one of the most common sources of UnboundLocalError after all, so I'd be surprised to find anyone that considered the current augmented assignment scoping rules to be outside the realm of reconsideration. The accumulation example would then be written: total = 0 progressive_sums = [total += value for value in data] if progressive_sums: assert total == progressive_sums[-1] The question would then turn to "What if you just want to bind the target name, without considering the old value?". And then *that's* where "NAME : = EXPR" would come in: as an augmented assignment operator that used augmented assignment scoping semantics, rather than regular local name binding semantics. That would mean *directly* overturning PEP 3099's rejection of the idea of using "NAME := EXPR" to imply "nonlocal NAME" at function scope, but that's effectively on the table for implicit functions anyway (and I'd prefer to have ":=" be consistent everywhere, rather than having to special case the implicit scopes). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon May 14 09:19:57 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 14 May 2018 09:19:57 -0400 Subject: [Python-ideas] "given" vs ":=" in list comprehensions In-Reply-To: <1531c1ab-7ef5-10fa-8f6c-aeae769e90ab@kellett.im> References: <1531c1ab-7ef5-10fa-8f6c-aeae769e90ab@kellett.im> Message-ID: On 14 May 2018 at 08:24, Ed Kellett wrote: > On 2018-05-14 05:02, Nick Coghlan wrote: > > The same grammar adjustment that I believe will allow "given" to be used > as > > both a postfix keyword and as a regular name would also work for "where". > > However, "where" still has the problem of semantically conflicting with > > SQL's use of it to introduce a filter clause, whereas Hypothesis uses > > "given" to bind names to values (just a little more indirectly than would > > be the case for assignment expressions) > > I suspect that SQL is not high on the list of languages people might > confuse with Python. If we used "where" as a name binding keyword, ORM docs like http://docs.sqlalchemy.org/en/latest/orm/query.html, and https://docs.djangoproject.com/en/2.0/topics/db/queries/ would need to be modified to explain that "SQL WHERE" and "Python where" do very different things. It's better to just avoid the potential for that problem entirely (either by using a symbolic notation, or by using a different keyword) > FWIW, as I'm sure will have been mentioned, Haskell > uses "where", and people seem to manage fine with it. > Unfortunately, Haskell's adoption numbers and reputation as a difficult to learn language don't back up that assumption (I doubt that outcome has anything to do with their use of "where", it just means "Haskell uses it that way" can't be credited as evidence that something won't cause confusion) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon May 14 10:00:09 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 15 May 2018 02:00:09 +1200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512172427.GF12683@ando.pearwood.info> <5AF9291C.1070301@canterbury.ac.nz> <5AF9540F.3030208@canterbury.ac.nz> Message-ID: <5AF99669.1070300@canterbury.ac.nz> Tim Peters wrote: > total = 0 > progressive_sums = [total := total + value for value in data] I'm skeptical that it's a good idea to encourage this kind of thing in the first place. -- Greg From clint.hepner at gmail.com Mon May 14 10:02:34 2018 From: clint.hepner at gmail.com (Clint Hepner) Date: Mon, 14 May 2018 10:02:34 -0400 Subject: [Python-ideas] Syntax idea: escaping names to avoid keyword ambiguity In-Reply-To: References: Message-ID: > On 2018 May 14 , at 6:47 a, Daniel Moisset wrote: > > Following up some of the discussions about the problems of adding keywords and Guido's proposal of making tokenization context-dependent, I wanted to propose an alternate way to go around the problem. My main objection to what follows is that it doesn't seem to offer any benefit over the current practice of appending an underscore (_) to a keyword to make it a valid identifier. > > My proposal essentially boils down to: > ? The character "$" can be used as a prefix of identifiers. formally, > identifier ::= ["$"] xid_start xid_continue* > ? The "$" character is not part of the name. So the program "foo=3;print($foo)" prints 3. So does the program "$foo=3; print(foo)". Both set an entry to globals["foo"] and keep globals["$foo"] unset. What is the benefit here? ``globals`` returns a mapping that represents the global scope, but that doesn't mean the mapping *is* the global scope. Aside from dubious uses of globals()['class'] to sidestep parsing issues, '$class' would still be, for all practical purposes, the "real" name of the identifier. > ? if "$" appears in a token, it's always an identifier. So "$with", "$if", "$return" are all identifiers. > If you overcome the "yikes, that looks like awk/bash/perl/php, and I don't like those", and consider it as an escape for "unusual"/"deprecation" situations, I think it's not a bad chose, and allows to a simple solutions to many problems that have been in discussion recently and not so recently. [examples below] The same argument applies to the underscore (or could, since it would be trivial to promise that no future keyword will end with, or even contain, an underscore). Defining new *keywords* to take the $-prefix could be done in a backwards-compatible way, although IMO the $ is too ugly to be a realistic choice. There might be a Unicode character that would make a good prefix, but given the reluctance to extend the core grammar beyond 7-bit ASCII, I've haven't spent any time looking for good candidates. > For me the benefits of this approach are: > ? It's very simple to explain how to use and its semantics > ? It (seems to me it) should be easy to explain to a python apprentice what a "$" means in code they read on a book/blogpost/manual > ? It's very easy to implement, minimal changes in the tokenizer > ? It's also easy to implement/integrate in other tools (editors with syntax highlighters, code formatters, etc) > ? It is easy to see that it's 100% backwards compatible (I understand that "$" has never been used in python before) The above 5 points all apply to appending an underscore to a keyword to create a valid identifier. > ? It is relatively unsurprising in the sense that other languages are already using $ to label names (there may be some point of confusion to people coming from javascript where "$" is a valid character in names and is not ignored). Given the variation in how other languages use $, I'm not sure this is a point in favor. There are plenty of questions on Stack Overflow about how and when to use $ in bash, and much of the confusion appears to stem from how $ is used in Perl. And that ignores the cases where $ is either optional (arithmetic expressions) or *should not* (the read built-in, the -v conditional operator, etc) be used. For that matter, why make $ special and restrict it to prefix position, instead of simply allowing $ as a valid identifier character and declaring that no keyword will ever use it? > > ? It gives python devs and users a clear, easy and universal upgrade path when keywords are added (language designers: Add a __future__ import to enable keyword in python N+1, add warnings to change kw --> $kw in python N+2, and then turn it on by default in python N+3... ; developers: add the import when they want to upgrade , and fix their code with a search&replace when adding the import or after getting a warning). Nothing about this process is specific to the $-prefix; it applies just as well to the practice of appending _ when it becomes necessary. > ? It allows you to use new features even if some libraries were written for older python versions, depending the deprecation period (this could be improved with sth I'll write in another email, but that's the topic for another proposal) I'll withhold judgement on this point, then, but it's not obvious how this allows an old library that uses a newly-minted keyword as an identifier name to continue working. > ? When clashes occur, which they always do, there's one obvious way to disambiguate (see today the "class_" argument for gettext.translation, the "lambd" argument for random.expovariate, the "class_" filter in libraries like pyquery for CSS class, functions like pyquery, sqlalchemy.sql.operators.as_ , etc. Not counting all the "cls" argument to every classmethod ever) Any one of these practices could be declared the One True Way without modifying the grammar. You're just adding another one. (https://xkcd.com/927/) > ? If we're worried about over proliferation of "$" in code, I'm quite sure given past experience that just a notice in PEP 8 of "only with $ in names to prevent ambiguity" should be more than enough for the community And finally, this applies to the trailing underscore as well. -- Clint From greg.ewing at canterbury.ac.nz Mon May 14 10:07:04 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 15 May 2018 02:07:04 +1200 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <20180514105501.GP12683@ando.pearwood.info> References: <20180512172427.GF12683@ando.pearwood.info> <5AF9291C.1070301@canterbury.ac.nz> <5AF9540F.3030208@canterbury.ac.nz> <20180514105501.GP12683@ando.pearwood.info> Message-ID: <5AF99808.1040109@canterbury.ac.nz> Steven D'Aprano wrote: > To reiterate what Tim already pointed out, that original usecase > required a way to feed values *into* the comprehension. > > https://mail.python.org/pipermail/python-ideas/2018-February/048971.html > > There's no need for dedicated syntax for that if we can just set an > variable and have it show up in the comprehension. Yes, but doing it that way also allows things to come *out* of the comprehension, which is something we previously thought was such a bad idea that we went to considerable lengths to stop it happening. Now we seem to be seriously considering going to further lengths to allow it again for *some* things. Seems to me we should have a pretty compelling reason for doing that. I don't feel very compelled by what I've seen so far. -- Greg From dmoisset at machinalis.com Mon May 14 10:16:28 2018 From: dmoisset at machinalis.com (Daniel Moisset) Date: Mon, 14 May 2018 15:16:28 +0100 Subject: [Python-ideas] Syntax idea: escaping names to avoid keyword ambiguity In-Reply-To: References: Message-ID: On 14 May 2018 at 15:02, Clint Hepner wrote: > > > On 2018 May 14 , at 6:47 a, Daniel Moisset > wrote: > > > > Following up some of the discussions about the problems of adding > keywords and Guido's proposal of making tokenization context-dependent, I > wanted to propose an alternate way to go around the problem. > > My main objection to what follows is that it doesn't seem to offer any > benefit over the current practice of appending an underscore (_) to a > keyword to make it a valid identifier. > There is a key difference: if an optional keyword is added (through a "from __future__ import some_keyword"), I still can use in a simple ways names from modules that do not have the future import enabled. Using just an underscore suffix or similar name *change* is fine when the language is static. But when the language changes, and I can not modify all the third party libraries, being able to refer to *the original name* instead of a modified one is a significant need that can not be covered by "just add a _ suffix". Best, D. -- Daniel Moisset UK COUNTRY MANAGER A: 1 Fore Street, EC2Y 9DT London P: +44 7398 827139 <+44+7398+827139> M: dmoisset at machinalis.com | S: dmoisset Machinalis Limited is a company registered in England and Wales. Registered number: 10574987. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmoisset at machinalis.com Mon May 14 10:44:20 2018 From: dmoisset at machinalis.com (Daniel Moisset) Date: Mon, 14 May 2018 15:44:20 +0100 Subject: [Python-ideas] Syntax idea: escaping names to avoid keyword ambiguity In-Reply-To: References: Message-ID: On 14 May 2018 at 15:02, Clint Hepner wrote: > > > On 2018 May 14 , at 6:47 a, Daniel Moisset > wrote: > > > > Following up some of the discussions about the problems of adding > keywords and Guido's proposal of making tokenization context-dependent, I > wanted to propose an alternate way to go around the problem. > > My main objection to what follows is that it doesn't seem to offer any > benefit over the current practice of appending an underscore (_) to a > keyword to make it a valid identifier. > > A secondary benefit I forgot to mention in my previous reply (and which is not covered by adding an underscore suffix) is that the original name being preserved it's better for interfacing with external systems that may use those names: For example, this code: https://github.com/dsc/pyquery/blob/fd725469701a8f47840f142df5c9b6d4479ea58e/pyquery/pyquery.py#L584 (and some code below that) is essentially a workaround that is required because they used a "_" suffix ibecause it's hard to pass "class" or "for" as keys in **kwargs -- Daniel Moisset UK COUNTRY MANAGER A: 1 Fore Street, EC2Y 9DT London P: +44 7398 827139 <+44+7398+827139> M: dmoisset at machinalis.com | S: dmoisset Machinalis Limited is a company registered in England and Wales. Registered number: 10574987. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abrault at mapgears.com Mon May 14 11:28:36 2018 From: abrault at mapgears.com (Alexandre Brault) Date: Mon, 14 May 2018 11:28:36 -0400 Subject: [Python-ideas] Syntax idea: escaping names to avoid keyword ambiguity In-Reply-To: References: Message-ID: On 2018-05-14 06:47 AM, Daniel Moisset wrote: > Following up some of the discussions about the problems of adding > keywords and Guido's proposal of making tokenization > context-dependent, I wanted to propose an alternate way to go around > the problem. > > My proposal essentially boils down to: > > 1. The character "$" can be used as a prefix of identifiers. formally,? > *identifier * ::= ["$"] |xid_start| > |xid_continue| > * > 2. The "$" character is not part of the name. So the program > "foo=3;print($foo)" prints 3. So does the program "$foo=3; > print(foo)". Both set an entry to globals["foo"] and keep > globals["$foo"] unset.? > 3. if "$" appears in a token, it's always an identifier. So "$with", > "$if", "$return" are all identifiers. > > If you overcome the "yikes, that looks like awk/bash/perl/php, and I > don't like those", and consider it as an escape for > "unusual"/"deprecation" situations, I think it's not a bad chose, and > allows to a simple solutions to many problems that have been in > discussion recently and not so recently. [examples below] > > For me the benefits of this approach are: > > * It's very simple to explain how to use and its semantics > * It (seems to me it) should be easy to explain to a python > apprentice what a "$" means in code they read on a > book/blogpost/manual > * It's very easy to implement, minimal changes in the tokenizer > * It's also easy to implement/integrate in other tools (editors with > syntax highlighters, code formatters, etc) > * It is easy to see that it's 100% backwards compatible (I > understand that "$" has never been used in python before) > * It is relatively unsurprising in the sense that other languages > are already using $ to label names (there may be some point of > confusion to people coming from javascript where "$" is a valid > character in names and is not ignored).? > * It gives python devs and users a clear, easy and universal upgrade > path when keywords are added (language designers: Add a __future__ > import to enable keyword in python N+1, add warnings to change kw > --> $kw in python N+2, and then turn it on by default in python > N+3... ; developers: add the import when they want to upgrade , > and fix their code with a search&replace when adding the import or > after getting a warning). > * It allows you to use new features even if some libraries were > written for older python versions, depending the deprecation > period (this could be improved with sth I'll write in another > email, but that's the topic for another proposal) > * When clashes occur, which they always do, there's one obvious way > to disambiguate (see today the "class_" argument for > gettext.translation, the "lambd" argument for random.expovariate, > the "class_" filter in libraries like pyquery for CSS class, > functions like pyquery,?sqlalchemy.sql.operators.as_ , etc. Not > counting all the "cls" argument to every classmethod ever) > * If we're worried about over proliferation of "$" in code, I'm > quite sure given past experience that just a notice in PEP 8 of > "only with $ in names to prevent ambiguity" should be more than > enough for the community > > > What are the drawbacks that you find in this? > Best, > ? ?Daniel > For the record, C# does something similar to help interface with other CLR languages with different keywords: any token starting with @ is an identifier even if the unprefixed token would be a reserved keyword. More on that: https://ericlippert.com/2013/09/09/verbatim-identifiers/ Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon May 14 11:48:41 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 14 May 2018 11:48:41 -0400 Subject: [Python-ideas] Sorry for yet another self discussion In-Reply-To: <20180510223348.GQ9562@ando.pearwood.info> References: <20180510223348.GQ9562@ando.pearwood.info> Message-ID: > So I was thinking: why not do define the methods > > like: "def self.whatevermethod(par1, par2, etc)" instead of "def > > whatevermethod(self, par1, par2, etc)"? > because "self" in this case is a class instance, passed in at method call time. but "whatevermethod" is a class attribute. note the distinction between the class object (shared by all instances) and the instance object. I'm not saying the compiler couldn't make it work, but I am saying that it would confuse the issue -- as it would not be defining an attribute of self, even though it looks like it is. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon May 14 12:05:08 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 14 May 2018 12:05:08 -0400 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: On Thu, May 10, 2018 at 6:13 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > Is there interest in a PEP for extending time, datetime / timedelta for > arbitrary or extended precision fractional seconds? > > Having seen the utter disaster that similar ideas brought to numpy, I would > say: no. > I'm not sure the "disaster" was due to this idea.... nor, frankly, is datetime64 a disaster at all, though certainly far from perfect. But my question is whether high precision timedeltas belongs with "calendar time" at all. What with UTC and leap seconds, and all that, it gets pretty ugly, when down to the second or sub-second, what a given datetime really means. If I were to work with high precision measurements, experiments, etc, I'd use a "nanoseconds since" representation, where the "epoch" would likely be the beginning of the experiment, of something relevant. Note that this issued in netcdf CF formats, datetimes are expressed in things like: "hours since 1970-01-01:00:00" granted, it's mostly so that the values can be stored as an array of a simple scalars, but it does allow precision and an epoch that are suited to the data at hand. NOTE: One source of the "disaster" of numpy's datetime64 is you can set teh precision, but NOT the epoch -- which is kind of problematic if you really want femtosecond precision for something not in 1970 :-) -CHB > On the other hand, nanoseconds are slowly making their way to the stdlib > and to add nanoseconds to datetime we only need a fully backward compatible > implementation, not even a PEP. > > See . > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon May 14 12:17:05 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 15 May 2018 02:17:05 +1000 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: On Tue, May 15, 2018 at 2:05 AM, Chris Barker via Python-ideas wrote: > But my question is whether high precision timedeltas belongs with "calendar > time" at all. > > What with UTC and leap seconds, and all that, it gets pretty ugly, when down > to the second or sub-second, what a given datetime really means. UTC and leap seconds aren't a problem. When there's a leap second, you have 23:59:60 (or you repeat 23:59:59, if you can't handle second #60). That's pretty straight-forward, perfectly well-defined. No, the REAL problems come from relativity..... > If I were to work with high precision measurements, experiments, etc, I'd > use a "nanoseconds since" representation, where the "epoch" would likely be > the beginning of the experiment, of something relevant. That's an unrelated form of time calculation. For that kind of thing, you probably want to ignore calendars and use some form of monotonic time; but also, if you want to go to (or below) nanosecond resolution, you'll need your clock to actually be that accurate, which most likely means you're not using a computer's clock. Femtosecond timestamping would basically be just taking numbers given to you by an external device and using them as sequence points - clocks and calendars become irrelevant. The numbers might as well be frame numbers in a super-high-speed filming of the event. ChrisA From tim.peters at gmail.com Mon May 14 12:47:58 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 14 May 2018 11:47:58 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <5AF99669.1070300@canterbury.ac.nz> References: <20180512172427.GF12683@ando.pearwood.info> <5AF9291C.1070301@canterbury.ac.nz> <5AF9540F.3030208@canterbury.ac.nz> <5AF99669.1070300@canterbury.ac.nz> Message-ID: >> total = 0 >> progressive_sums = [total := total + value for value in data] [Greg Ewing ] > I'm skeptical that it's a good idea to encourage this kind of thing > in the first place. Adding a feature is creating possibility for its use, not encouraging its use. I'd hate to, ,e.g., see people encouraging use of ternary-if either, _Sometimes_ ternary-if is exactly right, both shortening code and increasing clarity. But not often. Ditto for metaclasses, the ability to nest blocks 20 levels deep, on & on. That they can be abused, and sometimes are abused, is a very weak argument against them. On a scale of 1 to 1000000? Maybe 23 ;-) The specific use case that grabbed Guido's attention on this one wasn't the above, but: while any(n % p == 0 for p in small_primes): // divide p out of n - but which p succeeded? _General_ features turn out to have multiple good uses, not all of which are anticipated. I think this is an example of that. It's easy to miss that the argument to `any()` runs in an entirely different scope, so that its local `p` vanishes when `any()` completes. It's not proposed to change that either, because people overwhelmingly seem to want `for` targets not to leak. Fine by me. Under the proposal, it's straightforward to use an assignment expression to "export" p's last materialized value: while any(n % (factor := p) == 0 for p in small_primes): n //= factor To my eyes, the most surprising thing about that is why it's necessary to "export" p's winning value to begin with. "Because for-targets always vanish, so there's another form of explicit binding that doesn't" is a reasonably satisfying explanation. Nick would be happiest, and I'd be "happy enough", if all forms of binding remained "local" and instead we fiddled the syntax to allow explicitly declaring the desired scope. Then, e.g., while any( n % p == 0 for p in small_primes): n //= p wouldn't need an assignment expression at all. But Guido doesn't like that. I think he wants to hide, so far as humanly possible, that listcomps and genexps happen to implemented now by synthesizing functions. From tjreedy at udel.edu Mon May 14 14:04:49 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 14 May 2018 14:04:49 -0400 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: On 5/13/2018 2:19 PM, Guido van Rossum wrote: > The idea I had (not for the first time :-) is that in many syntactic > positions we could just treat keywords as names, and that would free up > these keywords. This trades the simplicity of 'using a keyword as an identifier always fails immediately' for the flexibility of 'using a keyword as an identifier may work, or may fail later'. The failure would be predictable in the sense of being being deterministic for any particular code string but unpredictable in the sense of not knowing what code would be written. It would also be unpredictable when people use a keyword as variable without knowing that they had done so. The proposal reminds me of the current tkinter situation when using tcl/tk compiled without thread support: calling a tkinter function (method) from worker threads may work, at least initially, but may eventually fail. The main problem here is that this is not documented. But I suspect that if and when it is (https://bugs.python.org/issue33479), most will 'not do that' and only a few intrepid souls will do the necessary experiments. The impact on real-time syntax coloring should be considered. An re-based colorizer, like IDLE's, tags every appearance of keywords outside of strings and comments, syntactically correct or not. For instance, the first two 'and's in "a and b.and # and error" are tagged. To not tag the second would require full parsing, presumably by compiling to AST, and more support code. I suspect this would would make coloring slower. Whether too slow for existing machines, I don't know. -- Terry Jan Reedy From tjreedy at udel.edu Mon May 14 14:11:54 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 14 May 2018 14:11:54 -0400 Subject: [Python-ideas] Syntax idea: escaping names to avoid keyword ambiguity In-Reply-To: References: Message-ID: On 5/14/2018 10:02 AM, Clint Hepner wrote: > >> On 2018 May 14 , at 6:47 a, Daniel Moisset wrote: >> >> Following up some of the discussions about the problems of adding keywords and Guido's proposal of making tokenization context-dependent, I wanted to propose an alternate way to go around the problem. > > My main objection to what follows is that it doesn't seem to offer any benefit over the current practice of appending an underscore (_) to a keyword to make it a valid identifier. Tkinter uses this convention for a few option names that clash. -- Terry Jan Reedy From tjreedy at udel.edu Mon May 14 14:29:24 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 14 May 2018 14:29:24 -0400 Subject: [Python-ideas] Sorry for yet another self discussion In-Reply-To: References: Message-ID: On 5/10/2018 3:58 PM, stefano wrote: > I know that "self" parameter have been discussed a lot, but still I didn't > find this proposal. If it was instead take my sincere apologies and please > forget this mail. > > The disturbing part of the "self parameter" is the asymmetry of the > definition and the call. You are free to use the full signature of the method by calling the method on the class, where it is defined. Self *is* a parameter of the function and therefore *is* part of its signature. >>> l = [] >>> l.append(1) >>> list.append(l, 2) >>> l [1, 2] Note that you can make a function defined outside of any class a method of some class by making it an attribute of the class. An intelligent editor like IDLE changes the call signature tip according to the call form. "l.append(" brings up "(object, /)" while "list.append(" brings up "(self, object, /)", along with a note explaining that '/' marks the preceding arguments as positional-only. > So I was thinking: why not do define the methods > like: "def self.whatevermethod(par1, par2, etc)" instead of "def > whatevermethod(self, par1, par2, etc)"? This or something like it has been proposed and rejected before. > This will allow the call and the definition to be written exactly in the > same way, still leaving the "clarity" of the additional input for the > function. When you call 'self.method', you are not calling type(self).method. You are instead calling the bound method 'self.method', which has the abbreviated signature. This proposal would create an asymmetry between the function definition and the call of the function. -- Terry Jan Reedy From tim.peters at gmail.com Mon May 14 15:38:48 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 14 May 2018 14:38:48 -0500 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: [Terry Reedy ] > ... > The impact on real-time syntax coloring should be considered. An re-based > colorizer, like IDLE's, tags every appearance of keywords outside of strings > and comments, syntactically correct or not. For instance, the first two > 'and's in "a and b.and # and error" are tagged. To not tag the second would > require full parsing, presumably by compiling to AST, and more support code. > I suspect this would would make coloring slower. Whether too slow for > existing machines, I don't know. I can pretty much guarantee it would be slower - way slower. But regexps could still do the bulk of it, provided the contexts remain as simple-minded as the specific ones Guido suggested: For example, we could allow keywords after 'def' and after a period, So when the IDLE colorizer regexp finds a match with the "KEYWORD" group name, don't color it at once. Instead look back from the start of the _possible_ keyword match, to see whether it's preceded by period maybe_whitespace or "def"-starting-at-a-word-boundary at_least_one_whitespace and if so skip coloring. There are many ways to code that, but - ya - they're annoying. Alas, the re module's negative lookbehind assertion isn't powerful enough for it, else those checks could be added directly to the keyword part of the colorizing regexp. I expect (but don't know) that lookbehinds in MRAB's "regex" module are strong enough for it. Of course that's just IDLE. It was remarkably easy to teach my primary editor (Source Insight) all sorts of stuff about .py files, but its "keywords" subsystem is restricted to specifying literal strings to look for. The kinds of tricks above probably require that the tool have access to a real programming language (like IDLE and Emacs enjoy). From ethan at stoneleaf.us Mon May 14 16:09:58 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 14 May 2018 13:09:58 -0700 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: <5AF92531.8090107@canterbury.ac.nz> References: <78e8c1d4-0597-a3ae-1f52-b7e0d192df4e@btinternet.com> <5AF8E653.6000105@canterbury.ac.nz> <01369ab4-cc72-72e5-cbde-fb782ad7f83b@btinternet.com> <5AF92531.8090107@canterbury.ac.nz> Message-ID: <5AF9ED16.3010008@stoneleaf.us> On 05/13/2018 10:57 PM, Greg Ewing wrote: > Rob Cliffe via Python-ideas wrote: >> If you forbid redefining keywords, you remove the whole point of this proposal: > > I mean the keywords that are in the language as of now. > There will never be a need to redefine those, since no > current code uses them as names. Part of the point of the proposal is to be able to use existing keywords (at least, I thought it was). -- ~Ethan~ From carl.input at gmail.com Mon May 14 16:52:52 2018 From: carl.input at gmail.com (Carl Smith) Date: Mon, 14 May 2018 21:52:52 +0100 Subject: [Python-ideas] Syntax idea: escaping names to avoid keyword ambiguity In-Reply-To: References: Message-ID: I can only think of three ways to reference a name defined in a different file: In an import statement, as properties of objects and as keyword arguments. Import statements are implicit assignments, so if Python allowed the following grammar, you could still import the odd thing that had a reserved name, without bringing that name into your local namespace. from import as Property names always follow a dot, where only a name is valid, so Python could allow this too: . Keyword arguments are also generally unambiguous, as they have to appear within the parens of an invocation, before the equals sign: (=) If Python allowed those three examples (but still prevented users from *defining* names that are keywords) new keywords could be introduced without breaking old code , and the language would only require relatively minor tweaking. -- Carl Smith carl.input at gmail.com On 14 May 2018 at 19:11, Terry Reedy wrote: > On 5/14/2018 10:02 AM, Clint Hepner wrote: > >> >> On 2018 May 14 , at 6:47 a, Daniel Moisset >>> wrote: >>> >>> Following up some of the discussions about the problems of adding >>> keywords and Guido's proposal of making tokenization context-dependent, I >>> wanted to propose an alternate way to go around the problem. >>> >> >> My main objection to what follows is that it doesn't seem to offer any >> benefit over the current practice of appending an underscore (_) to a >> keyword to make it a valid identifier. >> > > Tkinter uses this convention for a few option names that clash. > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.input at gmail.com Mon May 14 17:06:57 2018 From: carl.input at gmail.com (Carl Smith) Date: Mon, 14 May 2018 22:06:57 +0100 Subject: [Python-ideas] Syntax idea: escaping names to avoid keyword ambiguity In-Reply-To: References: Message-ID: Just to be clear, if `foo` was introduced as a new infix operator, projects that used `foo` as a name would not be able to also use `foo` as an infix operator in the file that defines `foo` as a name, but could use the operator throughout the rest of their project. -- Carl Smith carl.input at gmail.com On 14 May 2018 at 21:52, Carl Smith wrote: > I can only think of three ways to reference a name defined in a different > file: In an import > statement, as properties of objects and as keyword arguments. > > Import statements are implicit assignments, so if Python allowed the > following grammar, > you could still import the odd thing that had a reserved name, without > bringing that name > into your local namespace. > > from import as > > Property names always follow a dot, where only a name is valid, so Python > could allow > this too: > > . > > Keyword arguments are also generally unambiguous, as they have to appear > within the > parens of an invocation, before the equals sign: > > (=) > > If Python allowed those three examples (but still prevented users from > *defining* names > that are keywords) new keywords could be introduced without breaking old > code , and the > language would only require relatively minor tweaking. > > -- Carl Smith > carl.input at gmail.com > > On 14 May 2018 at 19:11, Terry Reedy wrote: > >> On 5/14/2018 10:02 AM, Clint Hepner wrote: >> >>> >>> On 2018 May 14 , at 6:47 a, Daniel Moisset >>>> wrote: >>>> >>>> Following up some of the discussions about the problems of adding >>>> keywords and Guido's proposal of making tokenization context-dependent, I >>>> wanted to propose an alternate way to go around the problem. >>>> >>> >>> My main objection to what follows is that it doesn't seem to offer any >>> benefit over the current practice of appending an underscore (_) to a >>> keyword to make it a valid identifier. >>> >> >> Tkinter uses this convention for a few option names that clash. >> >> -- >> Terry Jan Reedy >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.input at gmail.com Mon May 14 17:25:59 2018 From: carl.input at gmail.com (Carl Smith) Date: Mon, 14 May 2018 22:25:59 +0100 Subject: [Python-ideas] Syntax idea: escaping names to avoid keyword ambiguity In-Reply-To: References: Message-ID: Sorry to think out loud, but if the lexer marked `foo` as a generic `Word` token, that could be a keyword or a name, then the parser could look at the value of each `Word` token, and if the context is `import foo`, `class foo...`, `def foo...` or `foo = ...`, then `foo` is a name there and thereafter (and `x foo y` is a SyntaxError) else... you get the idea. -- Carl Smith carl.input at gmail.com On 14 May 2018 at 22:06, Carl Smith wrote: > Just to be clear, if `foo` was introduced as a new infix operator, > projects that used `foo` > as a name would not be able to also use `foo` as an infix operator in the > file that defines > `foo` as a name, but could use the operator throughout the rest of their > project. > > -- Carl Smith > carl.input at gmail.com > > On 14 May 2018 at 21:52, Carl Smith wrote: > >> I can only think of three ways to reference a name defined in a different >> file: In an import >> statement, as properties of objects and as keyword arguments. >> >> Import statements are implicit assignments, so if Python allowed the >> following grammar, >> you could still import the odd thing that had a reserved name, without >> bringing that name >> into your local namespace. >> >> from import as >> >> Property names always follow a dot, where only a name is valid, so Python >> could allow >> this too: >> >> . >> >> Keyword arguments are also generally unambiguous, as they have to appear >> within the >> parens of an invocation, before the equals sign: >> >> (=) >> >> If Python allowed those three examples (but still prevented users from >> *defining* names >> that are keywords) new keywords could be introduced without breaking old >> code , and the >> language would only require relatively minor tweaking. >> >> -- Carl Smith >> carl.input at gmail.com >> >> On 14 May 2018 at 19:11, Terry Reedy wrote: >> >>> On 5/14/2018 10:02 AM, Clint Hepner wrote: >>> >>>> >>>> On 2018 May 14 , at 6:47 a, Daniel Moisset >>>>> wrote: >>>>> >>>>> Following up some of the discussions about the problems of adding >>>>> keywords and Guido's proposal of making tokenization context-dependent, I >>>>> wanted to propose an alternate way to go around the problem. >>>>> >>>> >>>> My main objection to what follows is that it doesn't seem to offer any >>>> benefit over the current practice of appending an underscore (_) to a >>>> keyword to make it a valid identifier. >>>> >>> >>> Tkinter uses this convention for a few option names that clash. >>> >>> -- >>> Terry Jan Reedy >>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Mon May 14 17:26:16 2018 From: mertz at gnosis.cx (David Mertz) Date: Mon, 14 May 2018 17:26:16 -0400 Subject: [Python-ideas] Syntax idea: escaping names to avoid keyword ambiguity In-Reply-To: References: Message-ID: So does NumPy and sckit-learn use the trailing underscore convention. Albeit, sklearn uses it for (almost) all the model attributes, not just those it thinks might clash. On Mon, May 14, 2018, 2:12 PM Terry Reedy wrote: > On 5/14/2018 10:02 AM, Clint Hepner wrote: > > > >> On 2018 May 14 , at 6:47 a, Daniel Moisset > wrote: > >> > >> Following up some of the discussions about the problems of adding > keywords and Guido's proposal of making tokenization context-dependent, I > wanted to propose an alternate way to go around the problem. > > > > My main objection to what follows is that it doesn't seem to offer any > benefit over the current practice of appending an underscore (_) to a > keyword to make it a valid identifier. > > Tkinter uses this convention for a few option names that clash. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon May 14 18:28:08 2018 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 14 May 2018 18:28:08 -0400 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: > > UTC and leap seconds aren't a problem. Of course they are a problem? why else would they not be implemented in datetime? But my point if that given datetimestamp or calculation could be off by a second or so depending on whether and how leap seconds are implemented. It just doesn?t seem like a good idea to be handling months and femptoseconds with the same ?encoding? -CHB From mertz at gnosis.cx Mon May 14 18:48:01 2018 From: mertz at gnosis.cx (David Mertz) Date: Mon, 14 May 2018 18:48:01 -0400 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: Chris is certainly right. A program that deals with femtosecond intervals should almost surely start by defining a "start of experiment" epoch where microseconds are fine. Then within that epoch, events should be monotonic integers for when measured or calculated times are marked. I can easily see reasons why a specialized wrapped int for FemtosecondsFromStart could be useful. But that's still a specialized need for a third party library. One possible use of this class might be to interoperate with datetimes or timedeltas. Conceivably sick interoperability could be dealing with leap seconds when needed. But "experiment time" should be a simple monotonic and uniform counter. On Mon, May 14, 2018, 6:35 PM Chris Barker - NOAA Federal via Python-ideas < python-ideas at python.org> wrote: > > > > UTC and leap seconds aren't a problem. > > Of course they are a problem? why else would they not be implemented > in datetime? > > But my point if that given datetimestamp or calculation could be off > by a second or so depending on whether and how leap seconds are > implemented. > > It just doesn?t seem like a good idea to be handling months and > femptoseconds with the same ?encoding? > > -CHB > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Mon May 14 18:56:07 2018 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 14 May 2018 18:56:07 -0400 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: >From "[Python-Dev] PEP 564: Add new time functions with nanosecond resolution" (2017-10-16 hh:mm ss[...] -Z) https://groups.google.com/forum/m/#!topic/dev-python/lLJuW_asYa0 : > Maybe that's why we haven't found any CTCs (closed timelike curves) yet. > > Aligning simulation data in context to other events may be enlightening: is there a good library for handing high precision time units in Python (and/or CFFI)? There's not yet an ISO8601-like standard for this level of time/date precision. Correlating particle events between experiments does require date+time. On Monday, May 14, 2018, David Mertz wrote: > Chris is certainly right. A program that deals with femtosecond intervals > should almost surely start by defining a "start of experiment" epoch where > microseconds are fine. Then within that epoch, events should be monotonic > integers for when measured or calculated times are marked. > > I can easily see reasons why a specialized wrapped int for > FemtosecondsFromStart could be useful. But that's still a specialized need > for a third party library. One possible use of this class might be to > interoperate with datetimes or timedeltas. Conceivably sick > interoperability could be dealing with leap seconds when needed. But > "experiment time" should be a simple monotonic and uniform counter. > > On Mon, May 14, 2018, 6:35 PM Chris Barker - NOAA Federal via Python-ideas > wrote: > >> > >> > UTC and leap seconds aren't a problem. >> >> Of course they are a problem? why else would they not be implemented >> in datetime? >> >> But my point if that given datetimestamp or calculation could be off >> by a second or so depending on whether and how leap seconds are >> implemented. >> >> It just doesn?t seem like a good idea to be handling months and >> femptoseconds with the same ?encoding? >> >> -CHB >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon May 14 18:58:39 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 15 May 2018 10:58:39 +1200 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: <5AFA149F.3060202@canterbury.ac.nz> Terry Reedy wrote: > the first two 'and's in "a and b.and # and error" are tagged. > To not tag the second would require full parsing, That particular case could be handled by just not colouring any word following a dot. Some other situations might result in false positives, but there are cases like that already. Many Python colourisers colour any occurrence of a builtin function name, even if it has been shadowed, and people seem to live with that. It might even be useful, if it alerts people that a name they're using is reserved in some contexts, and so might be best avoided. -- Greg From steve at pearwood.info Mon May 14 20:35:58 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 15 May 2018 10:35:58 +1000 Subject: [Python-ideas] A partial (wacky?) alternative to assignment expressions Message-ID: <20180515003557.GS12683@ando.pearwood.info> I'm hoping that the arguments for assignment expressions will be over by Christmas *wink* so as a partial (and hopefully less controversial) alternative, what do people think of the idea of flagging certain expressions as "pure functions" so the compiler can automatically cache results from it? Let me explain: one of the use-cases for assignment expressions is to reduce repetition of code which may be expensive. A toy example: func(arg) + func(arg)*2 + func(arg)**2 If func() is a pure function with no side-effects, that is three times as costly as it ought to be: (f := func(arg)) + f*2 + f**2 Functional languages like Haskell can and do make this optimization all the time (or so I am lead to believe), because the compiler knows that func must be a pure, side-effect-free function. But the Python interpreter cannot do this optimization for us, because it has no way of knowing whether func() is a pure function. Now for the wacky idea: suppose we could tell the interpreter to cache the result of some sub-expression, and re-use it within the current expression? That would satisfy one use-case for assignment operators, and perhaps weaken the need for := operator. Good idea? Dumb idea? Good idea, but you want the assignment operator regardless? I don't have a suggestion for syntax yet, so I'm going to make up syntax which is *clearly and obviously rubbish*, a mere placeholder, so don't bother telling me all the myriad ways it sucks. I know it sucks, that's deliberate. Please focus on the *concept*, not the syntax. We would need to flag which expression can be cached because it is PURE, and tag how far the CACHE operates over: func(arg) + func(arg)*2 + func(arg)**2 This would tell the compiler to only evaluate the sub-expression "func(arg)" once, cache the result, and re-use it each other time it sees that same sub-expression within the surrounding expression. To be clear: it doesn't matter whether or not the sub-expression actually is pure. And it doesn't have to be a function call: it could be anything legal in an expression. If we had this, with appropriately awesome syntax, would that negate the usefulness of assignment expresions in your mind? -- Steve From rosuav at gmail.com Mon May 14 20:40:35 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 15 May 2018 10:40:35 +1000 Subject: [Python-ideas] A partial (wacky?) alternative to assignment expressions In-Reply-To: <20180515003557.GS12683@ando.pearwood.info> References: <20180515003557.GS12683@ando.pearwood.info> Message-ID: On Tue, May 15, 2018 at 10:35 AM, Steven D'Aprano wrote: > We would need to flag which expression can be cached because it is PURE, > and tag how far the CACHE operates over: > > > > func(arg) > > + func(arg)*2 + func(arg)**2 > > > This would tell the compiler to only evaluate the sub-expression > "func(arg)" once, cache the result, and re-use it each other time it > sees that same sub-expression within the surrounding expression. > > To be clear: it doesn't matter whether or not the sub-expression > actually is pure. And it doesn't have to be a function call: it could be > anything legal in an expression. > > If we had this, with appropriately awesome syntax, would that negate the > usefulness of assignment expresions in your mind? > Not for me, no. It doesn't eliminate lengthy expressions, only costly ones. It also doesn't deduplicate for convenience of editing. So that's one goal achieved, two not achieved. If the function is actually pure, all you need is lru_cache and you get that result. If it isn't, you've hit one of the two hardest problems in computing, good luck. ChrisA From mertz at gnosis.cx Mon May 14 20:46:42 2018 From: mertz at gnosis.cx (David Mertz) Date: Mon, 14 May 2018 20:46:42 -0400 Subject: [Python-ideas] A partial (wacky?) alternative to assignment expressions In-Reply-To: <20180515003557.GS12683@ando.pearwood.info> References: <20180515003557.GS12683@ando.pearwood.info> Message-ID: The time machine is used again! We HAVE a spelling: @functions.lru_cache() The problem is that there's still a time/space trade-off. If you might call a (pure) function with millions of different values, you likely don't want to cache them all. The current spelling makes this configurable, but there's no universal right answer. On Mon, May 14, 2018, 8:36 PM Steven D'Aprano wrote: > I'm hoping that the arguments for assignment expressions will be over by > Christmas *wink* so as a partial (and hopefully less controversial) > alternative, what do people think of the idea of flagging certain > expressions as "pure functions" so the compiler can automatically cache > results from it? > > Let me explain: one of the use-cases for assignment expressions is to > reduce repetition of code which may be expensive. A toy example: > > func(arg) + func(arg)*2 + func(arg)**2 > > If func() is a pure function with no side-effects, that is three times > as costly as it ought to be: > > (f := func(arg)) + f*2 + f**2 > > Functional languages like Haskell can and do make this optimization all > the time (or so I am lead to believe), because the compiler knows that > func must be a pure, side-effect-free function. But the Python > interpreter cannot do this optimization for us, because it has no way of > knowing whether func() is a pure function. > > Now for the wacky idea: suppose we could tell the interpreter to cache > the result of some sub-expression, and re-use it within the current > expression? That would satisfy one use-case for assignment operators, > and perhaps weaken the need for := operator. > > Good idea? Dumb idea? > > Good idea, but you want the assignment operator regardless? > > I don't have a suggestion for syntax yet, so I'm going to make up syntax > which is *clearly and obviously rubbish*, a mere placeholder, so don't > bother telling me all the myriad ways it sucks. I know it sucks, that's > deliberate. Please focus on the *concept*, not the syntax. > > We would need to flag which expression can be cached because it is PURE, > and tag how far the CACHE operates over: > > > > func(arg) > > + func(arg)*2 + func(arg)**2 > > > This would tell the compiler to only evaluate the sub-expression > "func(arg)" once, cache the result, and re-use it each other time it > sees that same sub-expression within the surrounding expression. > > To be clear: it doesn't matter whether or not the sub-expression > actually is pure. And it doesn't have to be a function call: it could be > anything legal in an expression. > > If we had this, with appropriately awesome syntax, would that negate the > usefulness of assignment expresions in your mind? > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Mon May 14 21:58:04 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 14 May 2018 20:58:04 -0500 Subject: [Python-ideas] A partial (wacky?) alternative to assignment expressions In-Reply-To: <20180515003557.GS12683@ando.pearwood.info> References: <20180515003557.GS12683@ando.pearwood.info> Message-ID: [Steven D'Aprano ] > I'm hoping that the arguments for assignment expressions will be over by > Christmas *wink* so as a partial (and hopefully less controversial) > alternative, what do people think of the idea of flagging certain > expressions as "pure functions" so the compiler can automatically cache > results from it? > > Let me explain: one of the use-cases for assignment expressions is to > reduce repetition of code which may be expensive. A toy example: > > func(arg) + func(arg)*2 + func(arg)**2 > > If func() is a pure function with no side-effects, that is three times > as costly as it ought to be: > > (f := func(arg)) + f*2 + f**2 > > Functional languages like Haskell can and do make this optimization all > the time (or so I am lead to believe), because the compiler knows that > func must be a pure, side-effect-free function. But the Python > interpreter cannot do this optimization for us, because it has no way of > knowing whether func() is a pure function. > > Now for the wacky idea: suppose we could tell the interpreter to cache > the result of some sub-expression, and re-use it within the current > expression? That would satisfy one use-case for assignment operators, > and perhaps weaken the need for := operator. > > Good idea? Dumb idea? Despite that Haskell can do optimizations like this , its "let ... in ..." and "... where ..." constructs (which create names for expressions, for use in another expression or code block) are widely used anyway. They don't care about the optimization (they already get it), but about improving clarity. In Haskell they'd spell it like, e.g., (mixing Python with Haskell keywords in UPPERCASE) :LET fa = func(arg) IN fa + fa*2 + fa**2 which the compiler may (but almost certainly won't) optimize further to LET fa = func(arg) IN fa * (3 + fa) if it knows that fa is of a type for which that makes sense. In Python today, I expect most people would do it as t = func(arg) t + 2*t + t*t # or t*(3+t) because they also know that multiplying t by itself once is usually faster than squaring ;-) And they wouldn't _want_ all the redundant typing in func(arg) + func(arg)*2 + func(arg)**2 anyway. So I'm not saying "good" or "bad", but that it needs a more compelling use case. > Good idea, but you want the assignment operator regardless? I'd probably write the example the way "I expect most people would do it" above even if we do get assignment expressions. > I don't have a suggestion for syntax yet, so I'm going to make up syntax > which is *clearly and obviously rubbish*, a mere placeholder, so don't > bother telling me all the myriad ways it sucks. I know it sucks, that's > deliberate. Please focus on the *concept*, not the syntax. > > We would need to flag which expression can be cached because it is PURE, > and tag how far the CACHE operates over: > > > > func(arg) > > + func(arg)*2 + func(arg)**2 > That syntax is clearly and obviously rubbish! It sucks. You're welcome ;-) > This would tell the compiler to only evaluate the sub-expression > "func(arg)" once, cache the result, and re-use it each other time it > sees that same sub-expression within the surrounding expression. > > To be clear: it doesn't matter whether or not the sub-expression > actually is pure. And it doesn't have to be a function call: it could be > anything legal in an expression. > > If we had this, with appropriately awesome syntax, would that negate the > usefulness of assignment expresions in your mind? The use cases I envision for that have no intersection with use cases I have for assignment expressions, so, no. My first thought about where it might be handy probably has no intersection with what you were thinking of either ;-) math.ceil math.floor def int_away_from_zero(x): if x >= 0: return math.ceil(x) else: return math.floor(x) The body of `int_away_from_zero()` is the way I _want_ to write it. But in heavily used functions it's expensive to look up "math", then look up its "ceil" (or "floor") attribute, on every call. So stuff like this often abuses default arguments instead: def int_away_from_zero(x, mc=math.ceil, mf=math.floor): if x >= 0: return mc(x) else: return mf(x) As the function grows over time, the default arg abuse grows, and the body of the function gets more obscure as more-&-more "tiny names" are introduced to save on repeated global and module attribute lookups. Indeed, in many cases I'd like to wrap an entire module in .... , with oodles of "module.attribute" thingies in the block. _Most_ of my code gets no benefit from Python's "extremely dynamic" treatment of module.attribute. It would be great if Python could do those lookups once at module import time, then generate some kind of `LOAD_GLOBAL_FAST index` opcode to fetch the results whenever they're used anywhere inside the module. Which would doubtless delight all the people struggling to cut Python's startup time - "Jeez Louise - now he wants Python to do even _more_ at import time?!" ;-) There are, e.g., other cases where invariant values of the form `n+1` or `n-1` are frequently used in a long function, and - cheap as each one is - it can actually make a time difference if they're pre-computed outside a loop. I'm ashamed of how many variables I have named "np1" and "nm1" :-( So there's some interesting stuff to ponder here! From leewangzhong+python at gmail.com Mon May 14 22:07:45 2018 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Mon, 14 May 2018 22:07:45 -0400 Subject: [Python-ideas] A partial (wacky?) alternative to assignment expressions In-Reply-To: <20180515003557.GS12683@ando.pearwood.info> References: <20180515003557.GS12683@ando.pearwood.info> Message-ID: On Mon, May 14, 2018 at 8:35 PM, Steven D'Aprano wrote: > I'm hoping that the arguments for assignment expressions will be over by > Christmas *wink* so as a partial (and hopefully less controversial) > alternative, what do people think of the idea of flagging certain > expressions as "pure functions" so the compiler can automatically cache > results from it? > > Let me explain: one of the use-cases for assignment expressions is to > reduce repetition of code which may be expensive. A toy example: > > func(arg) + func(arg)*2 + func(arg)**2 > > If func() is a pure function with no side-effects, that is three times > as costly as it ought to be: > > (f := func(arg)) + f*2 + f**2 > > Functional languages like Haskell can and do make this optimization all > the time (or so I am lead to believe), because the compiler knows that > func must be a pure, side-effect-free function. But the Python > interpreter cannot do this optimization for us, because it has no way of > knowing whether func() is a pure function. While I support the idea of marking pure functions somehow, automating this optimization is not a simple matter. Haskell (or at least its standard compiler) usually doesn't do it for you: """ GHC doesn't actually perform CSE as often as you might expect. The trouble is, performing CSE can affect the strictness/laziness of the program. So GHC does do CSE, but only in specific circumstances --- see the GHC manual. (Section??) Long story short: "If you care about CSE, do it by hand." """ https://wiki.haskell.org/GHC_optimisations#Common_subexpression_elimination Pure functions do, however, help with constant propagation during compilation. A PyPy blog post goes into a bit of detail about how pure functions can be helpful: https://morepypy.blogspot.com/2011/03/controlling-tracing-of-interpreter-with_15.html From tjreedy at udel.edu Mon May 14 22:49:49 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 14 May 2018 22:49:49 -0400 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: <5AFA149F.3060202@canterbury.ac.nz> References: <5AFA149F.3060202@canterbury.ac.nz> Message-ID: On 5/14/2018 6:58 PM, Greg Ewing wrote: > Terry Reedy wrote: >> the first two 'and's in "a and b.and # and error" are tagged. To not >> tag the second would require full parsing, > > That particular case could be handled by just not colouring > any word following a dot. OK, more parsing, even if not complete. Other cases require looking at several other things before or after, depending on the exact proposal. Okay, more parsing, depending on the exact rules. Some of the variations ( > Some other situations might result in false positives, but > there are cases like that already. Many Python colourisers > colour any occurrence of a builtin function name, even if > it has been shadowed, and people seem to live with that. For IDLE: >>> def int(): return 0 # int colored as defined name >>> int() # int colored as builtin. 0 Perhaps I should add a check whether defined names are builtins, and if so, color them as builtins even after 'def' and 'class'. > It might even be useful, if it alerts people that a name > they're using is reserved in some contexts, and so might > be best avoided. Since IDLE is especially aimed at beginners, I could think the same about shadowing keywords, and leave them always marked as such. Terry Jan Reedy From tim.peters at gmail.com Mon May 14 23:32:20 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 14 May 2018 22:32:20 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512172427.GF12683@ando.pearwood.info> Message-ID: Just noting some real code I typed today where `given` works great if it allows unpacking syntax, and assignment expressions don't: while True: head, matched, s = s.partition(sep) if not matched: break Using `given`: while matched given head, matched, s = s.partition(sep): Typing "matched " twice still sucks, though;-) It doesn't work as well with assignment expressions even if they were (re)generalized to allow unpacking syntax. In this specific case, I'd keep the original loop-and-a-half rather than do: while (t := s.partition(sep})[1]: head, matched, s = t With unpacking syntax restored, same answer: while (head, matched, s := s.partition(sep})[1]: or while [(head, matched, s := s.partition(sep}), matched][-1]: to combine the worst annoyances of everything and add another ;-) From dan at tombstonezero.net Mon May 14 23:59:04 2018 From: dan at tombstonezero.net (Dan Sommers) Date: Tue, 15 May 2018 03:59:04 +0000 (UTC) Subject: [Python-ideas] A partial (wacky?) alternative to assignment expressions References: <20180515003557.GS12683@ando.pearwood.info> Message-ID: On Tue, 15 May 2018 10:35:58 +1000, Steven D'Aprano wrote: > We would need to flag which expression can be cached because it is > PURE, and tag how far the CACHE operates over: > > > > func(arg) > > + func(arg)*2 + func(arg)**2 > > > This would tell the compiler to only evaluate the sub-expression > "func(arg)" once, cache the result, and re-use it each other time it > sees that same sub-expression within the surrounding expression. Does anyone else remember an ancient Python recipe that looked something like what follows? SENTINEL = object() class Magic: def __call__(self, arg=SENTINEL): if arg != SENTINEL: self.value = arg return self.value magic = Magic() And then you'd use it something like this: if magic(g(x)) != 4: print("probably not 4: ", magic()) else: print("should be 4: ", magic()) magic(f(x)) + magic()*2 + magic()**2 consume_the_value_again(magic()) It's not perfect, but it's not too much worse than some of the other suggestions floating around. Sort of. Dan From tim.peters at gmail.com Tue May 15 01:53:02 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 15 May 2018 00:53:02 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512172427.GF12683@ando.pearwood.info> <5AF9291C.1070301@canterbury.ac.nz> <5AF9540F.3030208@canterbury.ac.nz> Message-ID: [Tim] >> It's been noted several times recently that the example PEP 572 gives >> as _not_ working: >> >> total = 0 >> progressive_sums = [total := total + value for value in data] >> >> was the original use case that prompted work on the PEP. You gotta >> admit that's ironic ;-) [Nick] > After pondering this case further, I think it's also worth noting that that > *particular* example could also be addressed by: > > 1. Allowing augmented assignment *expressions* > 2. Changing the scoping rules for augmented assignment operations in general > such that they *don't change the scope of the referenced name* > > Writing "i += n" without first declaring the scope of "i" with "i = 0", > "nonlocal i" or "global i" is one of the most common sources of > UnboundLocalError after all, so I'd be surprised to find anyone that > considered the current augmented assignment scoping rules to be outside the > realm of reconsideration. Yes, it's worth considering. In my experience, I don't believe I've ever had an UnboundLocalError for which a correct fix was to add `nonlocal`. Usually the correct fix was to add `global`, but that's mostly due to an old habit of using piles of globals to count trips through various code paths, used by a watchdog thread that periodically wakes up to display (the global) counts. > The accumulation example would then be written: > > total = 0 > progressive_sums = [total += value for value in data] > if progressive_sums: > assert total == progressive_sums[-1] And the divide-out-small-primes example could be factor = -42 while any(n % (factor += p - factor) == 0 for p in small_primes): n //= factor Heh ;-) > The question would then turn to "What if you just want to bind the target > name, without considering the old value?". And then *that's* where "NAME : = > EXPR" would come in: as an augmented assignment operator that used augmented > assignment scoping semantics, rather than regular local name binding > semantics. Plain old ":=" would somehow be viewed as being an augmented assignment operator too? ... OK, the meaning is that augmented assignment _and_ "::=" would resolve the target's scope in the way the containing block resolves it. > That would mean *directly* overturning PEP 3099's rejection of the idea of > using "NAME := EXPR" to imply "nonlocal NAME" at function scope, but that's > effectively on the table for implicit functions anyway (and I'd prefer to > have ":=" be consistent everywhere, rather than having to special case the > implicit scopes). Creating a local in a containing scope by magic is never done by Python today. Extending that beyond "need" seems potentially perilous. For example, it can already be tedious to figure out which names _are_ local to a function by staring at the function's code, but people quickly get better at that over time; change the rules so that they _also_ have to stare at all immediately contained functions too to figure it out, and it may become significantly harder (OK, I didn't declare `x`, and a contained function did `x := 3.14` but `x` isn't declared there either - I guess it's my `x` now). Then again, if they're doing that much function nesting they deserve whatever they get ;-) Restrict it to that only synthetically generated functions can pull off this trick by magic (where there are real use cases to motivate it), and they still don't have to look outside the body of a function's text to figure it out. Visually, there's no distinction between the code running in the function's scope and in scopes synthesized to implement comprehensions appearing in the function's text. The comprehensions aren't even indented more. So, offhand, I'm not sure that the right way to address something you view as a wart is to vastly expand its reach to 12 operators that impose it on everyone everywhere every time they're used ;-) Seriously, I do suspect that in def f(...): ... no instances of `s` ... s += f"START {time.time():.2f}" it's overwhelmingly more likely that they simply forgot to do s = "" earlier in `f` than they actually wanted to append to whatever `s` means in f's parent block.. That's a radical change to what people have come to expect `NAME +=` to do. OTOH, I don't (yet?) see a way the change could break code that currently works, so it remains worth thinking about. BTW, would def f(): x := 3.14 x = 3.14 be a compile-time error? Everyone agreed the analogous case would be in synthetic functions. Fine by me! From python-ideas at shalmirane.com Tue May 15 02:41:36 2018 From: python-ideas at shalmirane.com (Ken Kundert) Date: Mon, 14 May 2018 23:41:36 -0700 Subject: [Python-ideas] Escaped braces in format specifiers Message-ID: <20180515064136.GA29222@kundert.designers-guide.com> The syntax for formatted string literals is given here: https://docs.python.org/3/reference/lexical_analysis.html#f-strings If you were to examine this carefully, you would see that a format_spec (the part within the braces but after the colon) can be empty or it can consist of literal characters and replacement fields. For example: f"result: {value:{width}.{precision}}" In this case '{width}.{precision}' is the format_spec, it consists of two replacement fields, {width} and {precision}, and one literal character, '.'. The definition of literal character includes all characters except braces. Presumably excluding braces makes it easier to distinguish the replacement fields. However, replacement fields are also used in the f_string itself, but in that case escaped braces are explicitly added to the list of valid symbols. I think the fact that one cannot include braces in the format_spec is an unnecessary and undesirable restriction. As an example of where it would be useful to allow braces in the format_spec, consider this simple example: >>> class mydict(dict): ... def __format__(self, template): ... return ''.join( ... template.format(v, k=k, v=v) for k, v in self.items() ... ) >>> accounts = mydict(checking=4256.78, savings=12000, brokerage=24685.5) >>> print('Accounts:\n {0:{{k:>9s}}: ${{v:>9,.2f}}\n }'.format(accounts)) Accounts: checking: $ 4,256.78 savings: $12,000.00 brokerage: $24,685.50 In this example, the format_spec is itself treated as a format string, which allows a recursive specification of the string format. If you try this example in Python3.6 you will find that it works, but it should not because the format_spec contains escaped braces. However, the fact that it works appears be be a happy accident. Things break if the escaped braces are not balanced (thanks to Thomas Jollans for finding this). Furthermore, things break in a different manner when an f-string is used. For example: >>> print(f'Accounts:\n {accounts:{{k:>9s}}: ${{v:>9,.2f}}\n }') File "", line 1 ({k:>9s}) ^ SyntaxError: invalid syntax or: >>> print(f'Accounts:\n {accounts:{{k}}: ${{v}}\n }') File "tryit", line 12, in print(f'Accounts:\n {accounts:{{k}}: ${{v}}\n }') NameError: name 'k' is not defined Oddly, the f-string still raises the NameError even if both k and v are explicitly defined. At a minimum it would be good to improve the error messages that are produced when escaped braces are included in the format_spec. Most of the error messages that are given, if they are given at all, are misleading. None are as simple as: SyntaxError: escaped braces are not allowed in a format spec. But rather than improve the error messages, I think we should consider simply allowing escaped braces in the format_spec. Doing so enables the neat idea of recursive format strings. But even if you don't like that idea, it would be nice to remove this rather rather odd restriction and would make the behavior of f-strings and the format method more consistent. -Ken From greg.ewing at canterbury.ac.nz Tue May 15 05:51:20 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 15 May 2018 21:51:20 +1200 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: <5AF9ED16.3010008@stoneleaf.us> References: <78e8c1d4-0597-a3ae-1f52-b7e0d192df4e@btinternet.com> <5AF8E653.6000105@canterbury.ac.nz> <01369ab4-cc72-72e5-cbde-fb782ad7f83b@btinternet.com> <5AF92531.8090107@canterbury.ac.nz> <5AF9ED16.3010008@stoneleaf.us> Message-ID: <5AFAAD98.2040804@canterbury.ac.nz> Ethan Furman wrote: > Part of the point of the proposal is to be able to use existing keywords > (at least, I thought it was). Mainly it's so that *new* keywords can be added to the language without breaking old code. Nobody is going to want to turn one of the currently existing keywords into a name. However, it might be desirable for calling APIs of another language which has a different set of keywords. Personally I don't really care if "from foreignlib import and" works and clobbers the "and" operator for the rest of the module. I was just suggesting a milder version of the feature that some people might be more comfortable with. -- Greg From solipsis at pitrou.net Tue May 15 06:07:43 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 15 May 2018 12:07:43 +0200 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions References: <78e8c1d4-0597-a3ae-1f52-b7e0d192df4e@btinternet.com> <5AF8E653.6000105@canterbury.ac.nz> <01369ab4-cc72-72e5-cbde-fb782ad7f83b@btinternet.com> <5AF92531.8090107@canterbury.ac.nz> <5AF9ED16.3010008@stoneleaf.us> <5AFAAD98.2040804@canterbury.ac.nz> Message-ID: <20180515120743.38e9cb57@fsol> On Tue, 15 May 2018 21:51:20 +1200 Greg Ewing wrote: > Ethan Furman wrote: > > Part of the point of the proposal is to be able to use existing keywords > > (at least, I thought it was). > > Mainly it's so that *new* keywords can be added to the language > without breaking old code. Nobody is going to want to turn one > of the currently existing keywords into a name. I'm a worried that we're speaking about making it *easier* to add new keywords. The language doesn't need a C++-like destiny, IMHO, where the language definition becomes increasingly bloated by piling up syntactical features. Regards Antoine. From p.f.moore at gmail.com Tue May 15 06:36:29 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 15 May 2018 11:36:29 +0100 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: <20180515120743.38e9cb57@fsol> References: <78e8c1d4-0597-a3ae-1f52-b7e0d192df4e@btinternet.com> <5AF8E653.6000105@canterbury.ac.nz> <01369ab4-cc72-72e5-cbde-fb782ad7f83b@btinternet.com> <5AF92531.8090107@canterbury.ac.nz> <5AF9ED16.3010008@stoneleaf.us> <5AFAAD98.2040804@canterbury.ac.nz> <20180515120743.38e9cb57@fsol> Message-ID: On 15 May 2018 at 11:07, Antoine Pitrou wrote: > On Tue, 15 May 2018 21:51:20 +1200 > Greg Ewing wrote: >> Ethan Furman wrote: >> > Part of the point of the proposal is to be able to use existing keywords >> > (at least, I thought it was). >> >> Mainly it's so that *new* keywords can be added to the language >> without breaking old code. Nobody is going to want to turn one >> of the currently existing keywords into a name. > > I'm a worried that we're speaking about making it *easier* to add new > keywords. The language doesn't need a C++-like destiny, IMHO, where > the language definition becomes increasingly bloated by piling up > syntactical features. Agreed - but there also seems to be a trend at the moment to propose punctuation-based syntax, with keyword-based alternatives being rejected on the basis that keywords break backward compatibility (because of existing use as variables). I'm not sure a Perl-like destiny (where we pile up extra punctuation) is that much better. I'm not in favour of this idea (it seems likely there will be too many odd corner cases and exceptions) but I can understand the motivation behind it. As you say, the real solution is of course to exercise restraint in what new syntax gets added. But looking at the recent "what's new in Python" docs, the reality is actually a lot more restrained than reading python-ideas might imply - so I'm not sure things are as bad as they initially seem. Paul From j.van.dorp at deonet.nl Tue May 15 07:25:04 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Tue, 15 May 2018 13:25:04 +0200 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: <78e8c1d4-0597-a3ae-1f52-b7e0d192df4e@btinternet.com> <5AF8E653.6000105@canterbury.ac.nz> <01369ab4-cc72-72e5-cbde-fb782ad7f83b@btinternet.com> <5AF92531.8090107@canterbury.ac.nz> <5AF9ED16.3010008@stoneleaf.us> <5AFAAD98.2040804@canterbury.ac.nz> <20180515120743.38e9cb57@fsol> Message-ID: While I understand the attraction, I think the clarity cost might be to high. If it's a pain to explain it to an IDE, it's an even bigger pain to explain it to people new to the language. From steve at pearwood.info Tue May 15 07:51:09 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 15 May 2018 21:51:09 +1000 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: <5AF8E653.6000105@canterbury.ac.nz> References: <78e8c1d4-0597-a3ae-1f52-b7e0d192df4e@btinternet.com> <5AF8E653.6000105@canterbury.ac.nz> Message-ID: <20180515115109.GT12683@ando.pearwood.info> On Mon, May 14, 2018 at 01:28:51PM +1200, Greg Ewing wrote: > Redefining the existing keywords could perhaps be forbidden > if you really want to protect people from shooting themselves > in the kidneys this particular way. So instead of having two classes of identifiers - those that can be used anywhere; - those that can only be used as dotted names; you would have two classes of keywords: - those that cannot be shadowed; - those that can be shadowed. -- Steve From guido at python.org Tue May 15 08:22:24 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 15 May 2018 08:22:24 -0400 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: <20180515115109.GT12683@ando.pearwood.info> References: <78e8c1d4-0597-a3ae-1f52-b7e0d192df4e@btinternet.com> <5AF8E653.6000105@canterbury.ac.nz> <20180515115109.GT12683@ando.pearwood.info> Message-ID: I hereby withdraw the idea. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Tue May 15 13:10:56 2018 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 15 May 2018 18:10:56 +0100 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512172427.GF12683@ando.pearwood.info> Message-ID: <96962743-b3dd-4143-45f2-726c38ca4e6a@mrabarnett.plus.com> On 2018-05-15 04:32, Tim Peters wrote: > Just noting some real code I typed today where `given` works great if > it allows unpacking syntax, and assignment expressions don't: > > while True: > head, matched, s = s.partition(sep) > if not matched: > break > > Using `given`: > > while matched given head, matched, s = s.partition(sep): > > Typing "matched " twice still sucks, though;-) > [snip] If you're using .partition multiple times, you might as well use .split instead! From tim.peters at gmail.com Tue May 15 13:57:07 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 15 May 2018 12:57:07 -0500 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: <96962743-b3dd-4143-45f2-726c38ca4e6a@mrabarnett.plus.com> References: <20180512172427.GF12683@ando.pearwood.info> <96962743-b3dd-4143-45f2-726c38ca4e6a@mrabarnett.plus.com> Message-ID: [Tim] >> Just noting some real code I typed today where `given` works great if >> it allows unpacking syntax, and assignment expressions don't: >> >> while True: >> head, matched, s = s.partition(sep) >> if not matched: >> break >> >> Using `given`: >> >> while matched given head, matched, s = s.partition(sep): >> >> Typing "matched " twice still sucks, though;-) >> > [snip] [MRAB ] > If you're using .partition multiple times, you might as well use .split > instead! Possibly - depends on what the rest of the loop is doing. In this specific case, there are paths in the loop body that change what `sep` is bound to, so I'd have to pass `maxsplit=1` to get a clumsier way to do what `partition()` does directly, For example, >>> "a+b".split("+", maxsplit=1) # separator wholly contained ['a', 'b'] >>> "a-b".split("+", maxsplit=1) # separator doesn't appear at all ['a-b'] I'ts more convenient that `partition()` always returns a 3-tuple, not sometimes a 1-list and other times a 2-list. From rspeer at luminoso.com Tue May 15 14:03:03 2018 From: rspeer at luminoso.com (Rob Speer) Date: Tue, 15 May 2018 14:03:03 -0400 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: While importlib.resources looks very good, I'm certain that it can't replace every use of __file__ for accessing files relative to your Python code. Consider a mini-Web-server written in Python (there are, of course, lots of these) that needs to serve static files. Users of the Web server will expect to be able to place these static files somewhere relative to the directory their code is in, because the files are version-controlled along with the code. If you make developers configure an absolute path, they'll probably use __file__ anyway to get that path, so that it works on more systems than their own without an installer or a layer of configuration management. If I understand the importlib.resources documentation, it won't give you a way of accessing your static files directory unless you place an '__init__.py' file in each subdirectory, and convert conventional locations such as "assets/css/main.css" into path(mypackage.assets.css, 'main.css'). That's already a bit awkward. But do you even want __init__.py to be in your static directory? Even if you tell the mini-server to ignore __init__.py, when you upgrade to a production-ready server like Nginx and point it at the same directory, it won't know anything about this and it'll serve your __init__.py files as static files, leaking details of your system. So you probably wouldn't do this. This is one example; there are other examples of non-Python directories that you need to be able to access from Python code, where adding a file named __init__.py to the directory would cause undesired changes in behavior. Again, importlib.resources is a good idea. I will look into using it in the cases where it applies. But the retort of "well, you shouldn't be using __file__" doesn't hold up when sometimes you do need to use __file__, and there's no universal replacement for it. (Also, every Python programmer I've met who's faced with the decision would choose "well, we need to use __file__, so don't zip things" over "well, we need to zip things, so don't use __file__". Yes, it's bad that Python programmers even have to make this choice, and then on top of that they make the un-recommended choice, but that's how things are.) On Mon, 7 May 2018 at 22:09 Barry Warsaw wrote: > Yuval Greenfield wrote: > > > I often need to reference a script's current directory. I end up writing: > > > > import os > > SRC_DIR = os.path.dirname(__file__) > > The question I have is, why do you want to reference the script's > current directory? > > If the answer is to access other files in that directory, then please > consider using importlib.resources (for Python 3.7) and > importlib_resources (for Python 2.7, 3.4-3.6). > > __file__ simply isn't safe, and pkg_resources can be a performance > killer. The problem of course is that if you're writing an application > and *any* of your dependencies use either technique, you are going to > pay for it. This is exactly why Brett and I wrote importlib.resources. > We wanted a consistent API, that allows custom loaders to play along, > and which is about as efficient as possible, uses Python's import > machinery, and is safe for uses like zipapps. > > now-you-don't-have-to-attend-my-pycon-talk-ly y'rs, > -Barry > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rspeer at luminoso.com Tue May 15 14:21:46 2018 From: rspeer at luminoso.com (Rob Speer) Date: Tue, 15 May 2018 14:21:46 -0400 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: On Mon, 14 May 2018 at 12:17 Chris Angelico wrote: > On Tue, May 15, 2018 at 2:05 AM, Chris Barker via Python-ideas > wrote: > > But my question is whether high precision timedeltas belongs with > "calendar > > time" at all. > > > > What with UTC and leap seconds, and all that, it gets pretty ugly, when > down > > to the second or sub-second, what a given datetime really means. > > UTC and leap seconds aren't a problem. When there's a leap second, you > have 23:59:60 (or you repeat 23:59:59, if you can't handle second > #60). That's pretty straight-forward, perfectly well-defined. > I'm sure that the issue of "what do you call the leap second itself" is not the problem that Chris Barker is referring to. The problem with leap seconds is that they create unpredictable differences between UTC and real elapsed time. You can represent a timedelta of exactly 10^8 seconds, but if you add it to the current time, what should you get? What UTC time will it be in 10^8 real-time seconds? You don't know, and neither does anybody else, because you don't know how many leap seconds will occur in that time. The ways to resolve this problem are: (1) fudge the definition of "exactly 10^8 seconds" to disregard any leap seconds that occur in that time interval in the real world, making it not so exact anymore (2) use TAI instead of UTC, as GPS systems do (3) leave the relationship between time deltas and calendar time undefined, as some in this thread are suggesting -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Tue May 15 16:23:26 2018 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 15 May 2018 16:23:26 -0400 Subject: [Python-ideas] Escaped braces in format specifiers In-Reply-To: <20180515064136.GA29222@kundert.designers-guide.com> References: <20180515064136.GA29222@kundert.designers-guide.com> Message-ID: I'm busy at the sprints, so I don't have a lot of time to think about this. However, let me just say that recursive format specs are supported, to a depth of 1. >>> width=10 >>> f'{"test":{width}}' 'test ' So first the string is basically expanded to: f'{"test":10}' Then the string is formatted again to produce the final result. That is why the braces must match: they're being used for recursive format specs. There's no mechanism for having braces that aren't inspected by the f-string machinery. Eric On 5/15/18 2:41 AM, Ken Kundert wrote: > The syntax for formatted string literals is given here: > https://docs.python.org/3/reference/lexical_analysis.html#f-strings > > If you were to examine this carefully, you would see that a format_spec (the > part within the braces but after the colon) can be empty or it can consist of > literal characters and replacement fields. For example: > > f"result: {value:{width}.{precision}}" > > In this case '{width}.{precision}' is the format_spec, it consists of two > replacement fields, {width} and {precision}, and one literal character, '.'. > > The definition of literal character includes all characters except braces. > Presumably excluding braces makes it easier to distinguish the replacement > fields. However, replacement fields are also used in the f_string itself, but > in that case escaped braces are explicitly added to the list of valid symbols. > > I think the fact that one cannot include braces in the format_spec is an > unnecessary and undesirable restriction. > > As an example of where it would be useful to allow braces in the format_spec, > consider this simple example: > > >>> class mydict(dict): > ... def __format__(self, template): > ... return ''.join( > ... template.format(v, k=k, v=v) for k, v in self.items() > ... ) > > >>> accounts = mydict(checking=4256.78, savings=12000, brokerage=24685.5) > >>> print('Accounts:\n {0:{{k:>9s}}: ${{v:>9,.2f}}\n }'.format(accounts)) > Accounts: > checking: $ 4,256.78 > savings: $12,000.00 > brokerage: $24,685.50 > > In this example, the format_spec is itself treated as a format string, which > allows a recursive specification of the string format. > > If you try this example in Python3.6 you will find that it works, but it should > not because the format_spec contains escaped braces. However, the fact that it > works appears be be a happy accident. Things break if the escaped braces are not > balanced (thanks to Thomas Jollans for finding this). Furthermore, things break > in a different manner when an f-string is used. For example: > > >>> print(f'Accounts:\n {accounts:{{k:>9s}}: ${{v:>9,.2f}}\n }') > File "", line 1 > ({k:>9s}) > ^ > SyntaxError: invalid syntax > > or: > > >>> print(f'Accounts:\n {accounts:{{k}}: ${{v}}\n }') > File "tryit", line 12, in > print(f'Accounts:\n {accounts:{{k}}: ${{v}}\n }') > NameError: name 'k' is not defined > > Oddly, the f-string still raises the NameError even if both k and v are > explicitly defined. > > At a minimum it would be good to improve the error messages that are produced > when escaped braces are included in the format_spec. Most of the error messages > that are given, if they are given at all, are misleading. None are as simple as: > > SyntaxError: escaped braces are not allowed in a format spec. > > But rather than improve the error messages, I think we should consider simply > allowing escaped braces in the format_spec. Doing so enables the neat idea of > recursive format strings. But even if you don't like that idea, it would be nice > to remove this rather rather odd restriction and would make the behavior of > f-strings and the format method more consistent. > > -Ken > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From barry at python.org Tue May 15 16:30:13 2018 From: barry at python.org (Barry Warsaw) Date: Tue, 15 May 2018 16:30:13 -0400 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: Message-ID: <598EB1E5-AD1E-47C6-ADBC-A09036CFFA11@python.org> On May 15, 2018, at 14:03, Rob Speer wrote: > Consider a mini-Web-server written in Python (there are, of course, lots of these) that needs to serve static files. Users of the Web server will expect to be able to place these static files somewhere relative to the directory their code is in, because the files are version-controlled along with the code. If you make developers configure an absolute path, they'll probably use __file__ anyway to get that path, so that it works on more systems than their own without an installer or a layer of configuration management. You don?t need an absolute path, since you don?t pass file system paths to importlib.resources, and even if you relative import a module, you can pass that module to the APIs and it will still work, since the loaders know where they got the modules from. > If I understand the importlib.resources documentation, it won't give you a way of accessing your static files directory unless you place an '__init__.py' file in each subdirectory, and convert conventional locations such as "assets/css/main.css" into path(mypackage.assets.css, 'main.css?). That is correct. Note that we?re not necessarily saying that we won?t add hierarchical path support to the `resource` attributes of the various APIs, but they do complicate the semantics and implementation. It?s also easier to add features if the use cases warrant, than remove features that are YAGNI. > That's already a bit awkward. But do you even want __init__.py to be in your static directory? Even if you tell the mini-server to ignore __init__.py, when you upgrade to a production-ready server like Nginx and point it at the same directory, it won't know anything about this and it'll serve your __init__.py files as static files, leaking details of your system. So you probably wouldn't do this. Are you saying that servers like Nginx or whatever your mini-server uses don?t have a way to blanket ignore files? That would surprise me, and it seems like a lurking security vulnerability regardless of importlib.resources or __init__.py files. I would think that you?d want to whitelist file extensions, and that `.py` would not be in that list. Is this a problem you?ve actually encountered or is it theoretical? > This is one example; there are other examples of non-Python directories that you need to be able to access from Python code, where adding a file named __init__.py to the directory would cause undesired changes in behavior. Can you provide more examples? > Again, importlib.resources is a good idea. I will look into using it in the cases where it applies. But the retort of "well, you shouldn't be using __file__" doesn't hold up when sometimes you do need to use __file__, and there's no universal replacement for it. > > (Also, every Python programmer I've met who's faced with the decision would choose "well, we need to use __file__, so don't zip things" over "well, we need to zip things, so don't use __file__". Yes, it's bad that Python programmers even have to make this choice, and then on top of that they make the un-recommended choice, but that's how things are.) We certainly see a ton of __file__ usage, but I?m not sure whether it?s the case because most developers aren?t aware of the implications, don?t know of the alternatives, or just use the simplest thing possible. Using __file__ in your application, personal web service, or private library is fine. The problem is exacerbated when you use __file__ in your publicly released libraries, because not only can?t *you* use them in zip files, but nothing that depends on your library can use zip files. Given how popular pex is (and hopefully shiv will be), that will cause pain up the Python food chain, and it may mean that other people won?t be able to use your library. It?s certainly a trade-off, but it?s important to keep this in mind. If hierarchical resource paths are important to you, I invite you to submit an issue to our GitLab project: https://gitlab.com/python-devs/importlib_resources/issues Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From greg.ewing at canterbury.ac.nz Tue May 15 18:59:06 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 16 May 2018 10:59:06 +1200 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: <20180515115109.GT12683@ando.pearwood.info> References: <78e8c1d4-0597-a3ae-1f52-b7e0d192df4e@btinternet.com> <5AF8E653.6000105@canterbury.ac.nz> <20180515115109.GT12683@ando.pearwood.info> Message-ID: <5AFB663A.6040100@canterbury.ac.nz> Steven D'Aprano wrote: > So instead of having two classes of identifiers > > - those that can be used anywhere; > - those that can only be used as dotted names; > > you would have two classes of keywords: > > - those that cannot be shadowed; > - those that can be shadowed. Yes. But we would always recommend that people not shadow any keywords, just as we recommend that tney not shadow builtins. All of this is purely to allow old code to keep working and old APIs to be used. -- Greg From steve at pearwood.info Tue May 15 20:41:53 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 16 May 2018 10:41:53 +1000 Subject: [Python-ideas] Verbatim names (allowing keywords as names) Message-ID: <20180516004153.GU12683@ando.pearwood.info> Inspired by Alex Brault's post: https://mail.python.org/pipermail/python-ideas/2018-May/050750.html I'd like to suggest we copy C#'s idea of verbatim identifiers, but using a backslash rather than @ sign: \name would allow "name" to be used as an identifier, even if it clashes with a keyword. It would *not* allow the use of characters that aren't valid in identifiers, e.g. this is out: \na!me # still not legal See usage #1 here: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/tokens/verbatim If "verbatim name" is too long, we could call them "raw names", by analogy with raw strings. I believe that \ is currently illegal in any Python expression, except inside strings and at the very end of the line, so this ought to be syntactically unambgiguous. We should still include a (mild?) recommendation against using keywords unless necessary, and a (strong?) preference for the trailing underscore convention. But I think this doesn't look too bad: of = 'output.txt' \if = 'input.txt' with open(\if, 'r'): with open(of, 'w'): of.write(\if.read()) maybe even nicer than if_. Some examples: result = \except + 1 result = something.\except result = \except.\finally -- Steve From timothy.c.delaney at gmail.com Tue May 15 20:53:26 2018 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 16 May 2018 10:53:26 +1000 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <20180516004153.GU12683@ando.pearwood.info> References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: On Wed, 16 May 2018 at 10:42, Steven D'Aprano wrote: > Inspired by Alex Brault's post: > > https://mail.python.org/pipermail/python-ideas/2018-May/050750.html > > I'd like to suggest we copy C#'s idea of verbatim identifiers, but using > a backslash rather than @ sign: > > \name > ?Personally, I prefer $name as originally suggested in that thread. It has the advantage of precedence from other languages as a variable indicator (even as far back as BASIC for me) and is more visible (which IMO is a positive, but others may see as a negative).? ?In either case?, I'm +1 for some way of indicating a verbatiim identifier. But I think this doesn't look too bad: > > ?? > of = 'output.txt' > \if = 'input.txt' > with open(\if, 'r'): > with open(of, 'w'): > of.write(\if.read()) > ?? of = 'output.txt' $if = 'input.txt' with open($if, 'r'): with open(of, 'w'): of.write($if.read()) ? > maybe even nicer than if_. > > Some examples: > > result = \except + 1 > ?result = $except + 1? > result = something.\except > ?result = somthing.$except? result = \except.\finally > ?result = $except.$finally @\return def func(): pass @$return def func(): pass Tim Delaney? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue May 15 21:09:03 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 15 May 2018 21:09:03 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <20180516004153.GU12683@ando.pearwood.info> References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: I like it. I much prefer \ to $ since in most languages that use $ that I know of (Perl, shell) there's a world of difference between $foo and foo whenever they occur (basically they never mean the same thing), whereas at least in shell, \foo means the same thing as foo *unless* foo would otherwise have a special meaning. I also recall that in some Fortran dialect I once used, $ was treated as the 27th letter of the alphabet, but not in the language standard. See e.g. https://gcc.gnu.org/onlinedocs/gcc-3.4.6/g77/Dollar-Signs.html. Apparently it has a similar role in Java ( https://stackoverflow.com/questions/7484210/what-is-the-meaning-of-in-a-variable-name ). On Tue, May 15, 2018 at 8:41 PM, Steven D'Aprano wrote: > Inspired by Alex Brault's post: > > https://mail.python.org/pipermail/python-ideas/2018-May/050750.html > > I'd like to suggest we copy C#'s idea of verbatim identifiers, but using > a backslash rather than @ sign: > > \name > > would allow "name" to be used as an identifier, even if it clashes with > a keyword. > > It would *not* allow the use of characters that aren't valid in > identifiers, e.g. this is out: \na!me # still not legal > > See usage #1 here: > > https://docs.microsoft.com/en-us/dotnet/csharp/language- > reference/tokens/verbatim > > > If "verbatim name" is too long, we could call them "raw names", by > analogy with raw strings. > > I believe that \ is currently illegal in any Python expression, except > inside strings and at the very end of the line, so this ought to be > syntactically unambgiguous. > > We should still include a (mild?) recommendation against using keywords > unless necessary, and a (strong?) preference for the trailing underscore > convention. But I think this doesn't look too bad: > > of = 'output.txt' > \if = 'input.txt' > with open(\if, 'r'): > with open(of, 'w'): > of.write(\if.read()) > > maybe even nicer than if_. > > Some examples: > > result = \except + 1 > > result = something.\except > > result = \except.\finally > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From leewangzhong+python at gmail.com Tue May 15 21:53:16 2018 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Tue, 15 May 2018 21:53:16 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: I assume there can't be space between the backslash and the name, to prevent ambiguity like in the following: # Is this `foo = not [1]` or `foo = \not[1]`? foo = (\ not[1]) A sampling of \ in other languages, for consideration: - Haskell: A lambda. E.g. `\x -> x+1` - TeX: A command. E.g. `\bold{Some text.}`. - Matlab: Matrix division operator. - PHP: Namespace delimiter. Uh. While LaTeX users have some intersection with Python users (Scipy!), I think there are enough differences in the languages that this one more won't hurt. On Tue, May 15, 2018 at 9:09 PM, Guido van Rossum wrote: > I like it. I much prefer \ to $ since in most languages that use $ that I > know of (Perl, shell) there's a world of difference between $foo and foo > whenever they occur (basically they never mean the same thing), whereas at > least in shell, \foo means the same thing as foo *unless* foo would > otherwise have a special meaning. > > I also recall that in some Fortran dialect I once used, $ was treated as the > 27th letter of the alphabet, but not in the language standard. See e.g. > https://gcc.gnu.org/onlinedocs/gcc-3.4.6/g77/Dollar-Signs.html. Apparently > it has a similar role in Java > (https://stackoverflow.com/questions/7484210/what-is-the-meaning-of-in-a-variable-name). > > On Tue, May 15, 2018 at 8:41 PM, Steven D'Aprano > wrote: >> >> Inspired by Alex Brault's post: >> >> https://mail.python.org/pipermail/python-ideas/2018-May/050750.html >> >> I'd like to suggest we copy C#'s idea of verbatim identifiers, but using >> a backslash rather than @ sign: >> >> \name >> >> would allow "name" to be used as an identifier, even if it clashes with >> a keyword. >> >> It would *not* allow the use of characters that aren't valid in >> identifiers, e.g. this is out: \na!me # still not legal >> >> See usage #1 here: >> >> >> https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/tokens/verbatim >> >> >> If "verbatim name" is too long, we could call them "raw names", by >> analogy with raw strings. >> >> I believe that \ is currently illegal in any Python expression, except >> inside strings and at the very end of the line, so this ought to be >> syntactically unambgiguous. >> >> We should still include a (mild?) recommendation against using keywords >> unless necessary, and a (strong?) preference for the trailing underscore >> convention. But I think this doesn't look too bad: >> >> of = 'output.txt' >> \if = 'input.txt' >> with open(\if, 'r'): >> with open(of, 'w'): >> of.write(\if.read()) >> >> maybe even nicer than if_. >> >> Some examples: >> >> result = \except + 1 >> >> result = something.\except >> >> result = \except.\finally >> >> >> -- >> Steve >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From carl.input at gmail.com Tue May 15 23:03:29 2018 From: carl.input at gmail.com (Carl Smith) Date: Wed, 16 May 2018 04:03:29 +0100 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: > On Tue, May 15, 2018 at 8:41 PM, Steven D'Aprano > wrote: > >> Inspired by Alex Brault's post: >> >> https://mail.python.org/pipermail/python-ideas/2018-May/050750.html >> >> I'd like to suggest we copy C#'s idea of verbatim identifiers, but using >> a backslash rather than @ sign: >> >> \name >> >> would allow "name" to be used as an identifier, even if it clashes with >> a keyword. > > I strongly disagree, but can't seem to get anyone ? to bite. We want to be able to introduce a keyword that was formally a name, still allow it to be used as a name, still allow code that uses it as a keyword to interoperate ? ? with code that uses it as a name , without changing the language or implementation too much. ?Ideally, Python would still not allow the keyword to be used as a name and a keyword in the same file?? The lexer could class the tokens as *keynames*, and the parser could use the context of the first instance of each keyname to determine if it's a name or keyword for the rest of that file. Projects that used the word as a name would only be prevented from also using it as a keyword in the same file. It's really then a question of whether users could elegantly and naturally reference a name in another module without introducing the name to the current module's namespace. We only reference external names (as syntactic names) in import statements, as properties after the dot operator, and as keyword arguments. If code that used the word as a keyword was still allowed to use the word as a name after the dot operator and as a keyword argument *in an invocation*, it would only change the language in a subtle way. If we could reference keynames in import statements, but not import the name, so basically allow `from keyname import keyname as name`, but not allow `import keyname`, we could still easily import things that used the keyname as a name. This wouldn't change the language too dramatically either. Maybe I'm just being dumb, but it seems like three subtle changes to the language would allow for everything we want to have, with only minor limitations on the rare occasion that you want to use the new keyword with a library that is also using the same keyword as a name. I promise not to push this idea again, but would really appreciate someone taking a couple of minutes to explain why it's not worth responding to. I'm not offended, but would like to know what I'm doing wrong. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue May 15 23:20:34 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 15 May 2018 23:20:34 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <20180516004153.GU12683@ando.pearwood.info> References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: On 5/15/2018 8:41 PM, Steven D'Aprano wrote: > Inspired by Alex Brault's post: > > https://mail.python.org/pipermail/python-ideas/2018-May/050750.html > > I'd like to suggest we copy C#'s idea of verbatim identifiers, but using > a backslash rather than @ sign: Not quite as heavy. > \name > > would allow "name" to be used as an identifier, even if it clashes with > a keyword. > > It would *not* allow the use of characters that aren't valid in > identifiers, e.g. this is out: \na!me # still not legal > > See usage #1 here: > > https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/tokens/verbatim > If "verbatim name" is too long, we could call them "raw names", by > analogy with raw strings. > I believe that \ is currently illegal in any Python expression, except > inside strings and at the very end of the line, so this ought to be > syntactically unambgiguous. > > We should still include a (mild?) recommendation against using keywords > unless necessary, and a (strong?) preference for the trailing underscore > convention. But I think this doesn't look too bad: I think it is just ugly enough to discourage wild use. > of = 'output.txt' > \if = 'input.txt' > with open(\if, 'r'): > with open(of, 'w'): > of.write(\if.read()) > > maybe even nicer than if_. > > Some examples: > > result = \except + 1 > > result = something.\except > > result = \except.\finally I believe avoiding tagging raw names as keywords could be done by adjusting the re for keywords and that addition of '\' could be done by re.sub. (The details should be in the doc.) -- Terry Jan Reedy From tim.peters at gmail.com Tue May 15 23:45:47 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 15 May 2018 22:45:47 -0500 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: [Terry Reedy] > ... > I believe avoiding tagging raw names as keywords could be done by adjusting > the re for keywords Yup - it should just require adding a negative lookbehind assertion; e.g., >>> import re >>> keypat = r"(?>> re.search(keypat, r"yup! while") <_sre.SRE_Match object; span=(5, 10), match='while'> >>> re.search(keypat, r"nope! \while") # None >>> The "(? References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: <5AFBAD96.9010100@stoneleaf.us> On 05/15/2018 08:03 PM, Carl Smith wrote: > On Tue, May 15, 2018 at 8:41 PM, Steven D'Aprano wrote: >> I'd like to suggest we copy C#'s idea of verbatim identifiers, but using >> a backslash rather than @ sign: >> >> \name >> >> would allow "name" to be used as an identifier, even if it clashes with >> a keyword. > > I strongly disagree, but can't seem to get anyone to bite. Sometimes it's like that, and yes it is frustrating. > We want to be able to introduce a keyword that was formally a name, still > allow > it to be used as a name, still allow code that uses it as a keyword to > interoperate > ? ? > with code that uses it as a name, without changing the language > or implementation too much. > > ?Ideally, Python would still not allow the keyword to be used as a name and a > keyword in the same file?? For me at least, this is a deal breaker. My libraries tend to be single-file packages, so all my code is in one place -- only being able to use the keyname as one or the other in my multi-thousand line file does me no good. -- ~Ethan~ From rspeer at luminoso.com Wed May 16 01:11:37 2018 From: rspeer at luminoso.com (Rob Speer) Date: Wed, 16 May 2018 01:11:37 -0400 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: <598EB1E5-AD1E-47C6-ADBC-A09036CFFA11@python.org> References: <598EB1E5-AD1E-47C6-ADBC-A09036CFFA11@python.org> Message-ID: > Are you saying that servers like Nginx or whatever your mini-server uses don?t have a way to blanket ignore files? That would surprise me, and it seems like a lurking security vulnerability regardless of importlib.resources or __init__.py files. I would think that you?d want to whitelist file extensions, and that `.py` would not be in that list. "Whitelisting file extensions" is very uncommon. You just put the files you intend to serve in your static directory, and don't put the files you don't intend to serve there. Mixing code and static data is usually seen as a sign of muddy PHP-like thinking. >From what I can tell, if you wanted to exclude '__init__.py' from Nginx in particular, you would have to write an unconventional Nginx configuration, where you determine whether a path refers to a static file according to a regex that excludes things that end in '__init__.py'. Anything is possible, but this would be a significant discouragement to using importlib. In practice, Flask's built-in server has its own logic about where to find files (which doesn't involve importlib, and I don't know what it actually does). Tornado appears to ask for an absolute path, so users mostly use __file__ to discover that path. > Is this a problem you?ve actually encountered or is it theoretical? I had a situation where I wanted to have files that were both served by Flask as static files, and resources that I could load in my tests. Making this work with pkg_resources took a few tries. It sounds like importlib won't really improve the situation. On Tue, 15 May 2018 at 16:30 Barry Warsaw wrote: > On May 15, 2018, at 14:03, Rob Speer wrote: > > > Consider a mini-Web-server written in Python (there are, of course, lots > of these) that needs to serve static files. Users of the Web server will > expect to be able to place these static files somewhere relative to the > directory their code is in, because the files are version-controlled along > with the code. If you make developers configure an absolute path, they'll > probably use __file__ anyway to get that path, so that it works on more > systems than their own without an installer or a layer of configuration > management. > > You don?t need an absolute path, since you don?t pass file system paths to > importlib.resources, and even if you relative import a module, you can pass > that module to the APIs and it will still work, since the loaders know > where they got the modules from. > > > If I understand the importlib.resources documentation, it won't give you > a way of accessing your static files directory unless you place an > '__init__.py' file in each subdirectory, and convert conventional locations > such as "assets/css/main.css" into path(mypackage.assets.css, 'main.css?). > > That is correct. Note that we?re not necessarily saying that we won?t add > hierarchical path support to the `resource` attributes of the various APIs, > but they do complicate the semantics and implementation. It?s also easier > to add features if the use cases warrant, than remove features that are > YAGNI. > > > That's already a bit awkward. But do you even want __init__.py to be in > your static directory? Even if you tell the mini-server to ignore > __init__.py, when you upgrade to a production-ready server like Nginx and > point it at the same directory, it won't know anything about this and it'll > serve your __init__.py files as static files, leaking details of your > system. So you probably wouldn't do this. > > Are you saying that servers like Nginx or whatever your mini-server uses > don?t have a way to blanket ignore files? That would surprise me, and it > seems like a lurking security vulnerability regardless of > importlib.resources or __init__.py files. I would think that you?d want to > whitelist file extensions, and that `.py` would not be in that list. > > Is this a problem you?ve actually encountered or is it theoretical? > > > This is one example; there are other examples of non-Python directories > that you need to be able to access from Python code, where adding a file > named __init__.py to the directory would cause undesired changes in > behavior. > > Can you provide more examples? > > > Again, importlib.resources is a good idea. I will look into using it in > the cases where it applies. But the retort of "well, you shouldn't be using > __file__" doesn't hold up when sometimes you do need to use __file__, and > there's no universal replacement for it. > > > > (Also, every Python programmer I've met who's faced with the decision > would choose "well, we need to use __file__, so don't zip things" over > "well, we need to zip things, so don't use __file__". Yes, it's bad that > Python programmers even have to make this choice, and then on top of that > they make the un-recommended choice, but that's how things are.) > > We certainly see a ton of __file__ usage, but I?m not sure whether it?s > the case because most developers aren?t aware of the implications, don?t > know of the alternatives, or just use the simplest thing possible. > > Using __file__ in your application, personal web service, or private > library is fine. The problem is exacerbated when you use __file__ in your > publicly released libraries, because not only can?t *you* use them in zip > files, but nothing that depends on your library can use zip files. Given > how popular pex is (and hopefully shiv will be), that will cause pain up > the Python food chain, and it may mean that other people won?t be able to > use your library. > > It?s certainly a trade-off, but it?s important to keep this in mind. > > If hierarchical resource paths are important to you, I invite you to > submit an issue to our GitLab project: > > https://gitlab.com/python-devs/importlib_resources/issues > > Cheers, > -Barry > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed May 16 04:13:52 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 16 May 2018 09:13:52 +0100 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <20180516004153.GU12683@ando.pearwood.info> References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: On 16 May 2018 at 01:41, Steven D'Aprano wrote: > Inspired by Alex Brault's post: > > https://mail.python.org/pipermail/python-ideas/2018-May/050750.html > > I'd like to suggest we copy C#'s idea of verbatim identifiers, but using > a backslash rather than @ sign: > > \name > > would allow "name" to be used as an identifier, even if it clashes with > a keyword. I'm missing something. How is that different from using a trailing underscore (like if_ or while_) at the moment? I understand that foo and \foo are the same name, whereas foo and foo_ are different, but how would that help? Can you give a worked example of how this would help if we wanted to introduce a new keyword? For example, if we intended to make "where" a keyword, what would numpy and its users need to do to continue using `numpy.where`? Paul From eric at trueblade.com Wed May 16 04:47:55 2018 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 16 May 2018 04:47:55 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: <9e152416-9afb-5b6b-e035-3dea6f37c32c@trueblade.com> On 5/16/18 4:13 AM, Paul Moore wrote: > On 16 May 2018 at 01:41, Steven D'Aprano wrote: >> Inspired by Alex Brault's post: >> >> https://mail.python.org/pipermail/python-ideas/2018-May/050750.html >> >> I'd like to suggest we copy C#'s idea of verbatim identifiers, but using >> a backslash rather than @ sign: >> >> \name >> >> would allow "name" to be used as an identifier, even if it clashes with >> a keyword. > > I'm missing something. How is that different from using a trailing > underscore (like if_ or while_) at the moment? I understand that foo > and \foo are the same name, whereas foo and foo_ are different, but > how would that help? Presumably things that know their name (like def'd functions and classes, off the top of my head) would be able to figure out their keyword-like name. Although exactly how that would work is debatable. >>> def \if(): pass ... Is \if.__name__ equal to r"\if", or "if"? Probably just "if". Not that it really matters, but I can see code generators adding backslashes everywhere. For example, in dataclasses I'd probably generate __init__ for this: \for=float @dataclass class C: \if: int only: \for as: def __init__(self, \if:\int, \only:\for): That is, I'd add a backslash in front of every identifier, instead of trying to figure out if I need to or not. I think a lot of code generators (such as attrs) would need to be modified. Not a show-stopper, but something to think about. > Can you give a worked example of how this would > help if we wanted to introduce a new keyword? For example, if we > intended to make "where" a keyword, what would numpy and its users > need to do to continue using `numpy.where`? I think they'd have to change to `numpy.\where` when `where` became a keyword. Another thought: I'm sure f-strings would have fun with this. This code is currently illegal: >>> f'{\if}' File "", line 1 SyntaxError: f-string expression part cannot include a backslash That would be a bear to fix, and would require all code that looks at f-strings, even if only to ignore them, to change. Currently you can just say "any string that has an f in front of it can be lexed (as a unit) the same way 'r', 'u', and 'b' strings are". But this would break that, and mean that instead of a simple tokenizer to find the end of an f-string, you'd probably need a full expression parser. Again, just something to think about. Eric From eric at trueblade.com Wed May 16 04:56:22 2018 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 16 May 2018 04:56:22 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <9e152416-9afb-5b6b-e035-3dea6f37c32c@trueblade.com> References: <20180516004153.GU12683@ando.pearwood.info> <9e152416-9afb-5b6b-e035-3dea6f37c32c@trueblade.com> Message-ID: <439b03f0-050b-0ee2-3466-208328909f1f@trueblade.com> On 5/16/18 4:47 AM, Eric V. Smith wrote: > On 5/16/18 4:13 AM, Paul Moore wrote: >> Can you give a worked example of how this would >> help if we wanted to introduce a new keyword? For example, if we >> intended to make "where" a keyword, what would numpy and its users >> need to do to continue using `numpy.where`? > > I think they'd have to change to `numpy.\where` when `where` became a > keyword. To be clear: this would apply to any code that uses numpy.where, not just the code that defines it. The only way to bullet-proof your code so that it would never need any modifications in the future would be to put a backslash in front of every identifier. Or maybe just all-lowercase identifiers, since we're unlikely to make a keyword with uppercase chars in it. And since no one in their right mind would do that, there's still the risk of your code breaking in the future. But at least there would be a way of fixing it in a way that would work both with old versions of python where the identifier isn't a keyword, and for versions where it is. That is, once "old versions" include ones that support verbatim names. Eric From p.f.moore at gmail.com Wed May 16 05:03:47 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 16 May 2018 10:03:47 +0100 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <439b03f0-050b-0ee2-3466-208328909f1f@trueblade.com> References: <20180516004153.GU12683@ando.pearwood.info> <9e152416-9afb-5b6b-e035-3dea6f37c32c@trueblade.com> <439b03f0-050b-0ee2-3466-208328909f1f@trueblade.com> Message-ID: On 16 May 2018 at 09:56, Eric V. Smith wrote: > On 5/16/18 4:47 AM, Eric V. Smith wrote: >> >> On 5/16/18 4:13 AM, Paul Moore wrote: > > >>> Can you give a worked example of how this would >>> help if we wanted to introduce a new keyword? For example, if we >>> intended to make "where" a keyword, what would numpy and its users >>> need to do to continue using `numpy.where`? >> >> >> I think they'd have to change to `numpy.\where` when `where` became a >> keyword. > > > To be clear: this would apply to any code that uses numpy.where, not just > the code that defines it. > > The only way to bullet-proof your code so that it would never need any > modifications in the future would be to put a backslash in front of every > identifier. Or maybe just all-lowercase identifiers, since we're unlikely to > make a keyword with uppercase chars in it. > > And since no one in their right mind would do that, there's still the risk > of your code breaking in the future. But at least there would be a way of > fixing it in a way that would work both with old versions of python where > the identifier isn't a keyword, and for versions where it is. That is, once > "old versions" include ones that support verbatim names. That's about what I thought - thanks. Paul From stephanh42 at gmail.com Wed May 16 05:12:29 2018 From: stephanh42 at gmail.com (Stephan Houben) Date: Wed, 16 May 2018 11:12:29 +0200 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <9e152416-9afb-5b6b-e035-3dea6f37c32c@trueblade.com> <439b03f0-050b-0ee2-3466-208328909f1f@trueblade.com> Message-ID: Hi all, One problem already alluded to with the \identifier syntax is that it only works if the old Python version is sufficiently recent to understand \. What about using parentheses to allow a keyword to be used as an identifier: (where)(x, y) This then in combination with allowing keywords in the following unambiguous locations: 1. After dot ("numpy.where") 2. After def and class ("def where") 3. After "as". This should make it possible to write code which works in a hypothetical future Python version where "where" is a keyword, and which also works with current Python versions. Stephan 2018-05-16 11:03 GMT+02:00 Paul Moore : > On 16 May 2018 at 09:56, Eric V. Smith wrote: > > On 5/16/18 4:47 AM, Eric V. Smith wrote: > >> > >> On 5/16/18 4:13 AM, Paul Moore wrote: > > > > > >>> Can you give a worked example of how this would > >>> help if we wanted to introduce a new keyword? For example, if we > >>> intended to make "where" a keyword, what would numpy and its users > >>> need to do to continue using `numpy.where`? > >> > >> > >> I think they'd have to change to `numpy.\where` when `where` became a > >> keyword. > > > > > > To be clear: this would apply to any code that uses numpy.where, not just > > the code that defines it. > > > > The only way to bullet-proof your code so that it would never need any > > modifications in the future would be to put a backslash in front of every > > identifier. Or maybe just all-lowercase identifiers, since we're > unlikely to > > make a keyword with uppercase chars in it. > > > > And since no one in their right mind would do that, there's still the > risk > > of your code breaking in the future. But at least there would be a way of > > fixing it in a way that would work both with old versions of python where > > the identifier isn't a keyword, and for versions where it is. That is, > once > > "old versions" include ones that support verbatim names. > > That's about what I thought - thanks. > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at brice.xyz Wed May 16 05:22:42 2018 From: contact at brice.xyz (Brice Parent) Date: Wed, 16 May 2018 11:22:42 +0200 Subject: [Python-ideas] Crazy idea: allow keywords as names in certain positions In-Reply-To: References: Message-ID: I'm not half aware of the way everything works under the hood as the vast majority here, but I have an idea. It possibly is technically stupid when you really know how things work, but as I'm not sure, I'm submitting it anyways: Why not add the compatibility done at setup.py's level? Like having something like this: setup( ??? name="LegacyLib", ??? version="8.4.1", ??? ... ??? max_compatibility="3.4"? # Defining here Python's max version for which this lib has been upgraded ) Of course, we may use any other word instead of "max_compatibility", like "designed_for", "python_version", or anything a better English speaker could think of. The point is, it would either: - when you install the library, rename all variables that are now keywords (we'd know the exact list thanks to max_compatiblity) by suffixing them with "_" - or set a flag that will do that when creating the *.pyc files. Possible problems/limitations I can already find: - There would still be possible errors when using variable names that are generated on the fly (I have no clue how this could ever be addressed) - It might get complicated at some point to know what to do, like when we have lib_a in some version depending on lib_b (with or without a max_compatibility version number), it is obvious that lib_a will use lib_b's original variable names (without the appended "_"), but our code which might also want to interact with lib_b would have to. Is it plain stupid? Are there lots of things I didn't think of? Could it be a possibility? -Brice Le 13/05/2018 ? 20:19, Guido van Rossum a ?crit?: > As anyone still following the inline assignment discussion knows, a > problem with designing new syntax is that it's hard to introduce new > keywords into the language, since all the nice words seem to be used > as method names in popular packages. (E.g. we can't use 'where' > because there's numpy.where > , > and we can't use 'given' because it's used in Hypothesis > .) > > The idea I had (not for the first time :-) is that in many syntactic > positions we could just treat keywords as names, and that would free > up these keywords. > > For example, we could allow keywords after 'def' and after a period, > and then the following would become legal: > > class C: > ??? def and(self, other): > ??????? return ... > > a = C() > b = C() > print(a.and(b)) > > This does not create syntactic ambiguities because after 'def' and > after a period the grammar *always* requires a NAME. > > There are other positions where we could perhaps allow this, e.g. in a > decorator, immediately after '@' (the only keyword that's > *syntactically* legal here is 'not', though I'm not sure it would ever > be useful). > > Of course this would still not help for names of functions that might > be imported directly (do people write 'from numpy import where'?). And > it would probably cause certain typos be harder to diagnose. > > I should also mention that this was inspired from some messages where > Tim Peters berated the fashion of using "reserved words", waxing > nostalgically about the old days of Fortran (sorry, FORTRAN), which > doesn't (didn't?) have reserved words at all (nor significant > whitespace, apart from the "start in column 7" rule). > > Anyway, just throwing this out. Please tear it apart! > > -- > --Guido van Rossum (python.org/~guido ) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed May 16 05:59:32 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 16 May 2018 11:59:32 +0200 Subject: [Python-ideas] Verbatim names (allowing keywords as names) References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: <20180516115932.56670bfd@fsol> On Wed, 16 May 2018 09:13:52 +0100 Paul Moore wrote: > On 16 May 2018 at 01:41, Steven D'Aprano wrote: > > Inspired by Alex Brault's post: > > > > https://mail.python.org/pipermail/python-ideas/2018-May/050750.html > > > > I'd like to suggest we copy C#'s idea of verbatim identifiers, but using > > a backslash rather than @ sign: > > > > \name > > > > would allow "name" to be used as an identifier, even if it clashes with > > a keyword. > > I'm missing something. How is that different from using a trailing > underscore (like if_ or while_) at the moment? I understand that foo > and \foo are the same name, whereas foo and foo_ are different, but > how would that help? I think it could help in cases like namedtuple, where names can be part of a data description (e.g. coming from a database) and then used for attribute access. I do not find it extremely pretty, but I like it much better still than the "allowing keywords as names" proposal. It also has the nice side-effect that it doesn't make it easier to add new keywords, since the common spelling (e.g. `np.where`) would still become a syntax error and therefore break compatibility with existing code. Regards Antoine. From Eloi.Gaudry at fft.be Wed May 16 04:29:00 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Wed, 16 May 2018 08:29:00 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <1525793916.5522.5.camel@fft.be> References: <1525765072.24469.5.camel@fft.be> <0dd9cf67-08cc-6c57-0570-98507290c483@trueblade.com> <1525787426.5522.1.camel@fft.be> <1525793916.5522.5.camel@fft.be> Message-ID: <1526459338.23739.1.camel@fft.be> Is there some interest in the proposal or should I finally close this thread ? Thanks for your feedback, Eloi > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From steve at pearwood.info Wed May 16 07:15:38 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 16 May 2018 21:15:38 +1000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <1526459338.23739.1.camel@fft.be> References: <1525765072.24469.5.camel@fft.be> <0dd9cf67-08cc-6c57-0570-98507290c483@trueblade.com> <1525787426.5522.1.camel@fft.be> <1525793916.5522.5.camel@fft.be> <1526459338.23739.1.camel@fft.be> Message-ID: <20180516111538.GY12683@ando.pearwood.info> On Wed, May 16, 2018 at 08:29:00AM +0000, Eloi Gaudry wrote: > Is there some interest in the proposal or should I finally close this > thread ? I'm definitely interested in the concept, not the suggested syntax or semantics. -- Steve From wolfgang.maier at biologie.uni-freiburg.de Wed May 16 08:13:45 2018 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Wed, 16 May 2018 14:13:45 +0200 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <20180516004153.GU12683@ando.pearwood.info> References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> On 16.05.2018 02:41, Steven D'Aprano wrote: > > Some examples: > > result = \except + 1 > > result = something.\except > > result = \except.\finally > Maybe that could get combined with Guido's original suggestion by making the \ optional after a .? Example: class A (): \global = 'Hello' def __init__(self): self.except = 0 def \finally(self): return 'bye' print(A.global) a = A() a.except += 1 print(a.finally()) or with a module, in my_module.py: \except = 0 elsewhere: import my_module print(my_module.except) or from my_module import \except print(\except) Best, Wolfgang From adelfino at gmail.com Wed May 16 09:05:33 2018 From: adelfino at gmail.com (=?UTF-8?Q?Andr=C3=A9s_Delfino?=) Date: Wed, 16 May 2018 10:05:33 -0300 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> References: <20180516004153.GU12683@ando.pearwood.info> <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> Message-ID: IMHO, it would be much easier to learn and understand if keywords can only be used by escaping them, instead of depending where they occur. On Wed, May 16, 2018 at 9:13 AM, Wolfgang Maier < wolfgang.maier at biologie.uni-freiburg.de> wrote: > On 16.05.2018 02:41, Steven D'Aprano wrote: > >> >> Some examples: >> >> result = \except + 1 >> >> result = something.\except >> >> result = \except.\finally >> >> > Maybe that could get combined with Guido's original suggestion by making > the \ optional after a .? > > Example: > > class A (): > \global = 'Hello' > def __init__(self): > self.except = 0 > > def \finally(self): > return 'bye' > > print(A.global) > a = A() > a.except += 1 > print(a.finally()) > > or with a module, in my_module.py: > > \except = 0 > > elsewhere: > > import my_module > print(my_module.except) > > or > > from my_module import \except > print(\except) > > Best, > Wolfgang > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Wed May 16 09:46:43 2018 From: toddrjen at gmail.com (Todd) Date: Wed, 16 May 2018 09:46:43 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: On Tue, May 15, 2018, 23:03 Carl Smith wrote: > > >> On Tue, May 15, 2018 at 8:41 PM, Steven D'Aprano >> wrote: >> >>> Inspired by Alex Brault's post: >>> >>> https://mail.python.org/pipermail/python-ideas/2018-May/050750.html >>> >>> I'd like to suggest we copy C#'s idea of verbatim identifiers, but using >>> a backslash rather than @ sign: >>> >>> \name >>> >>> would allow "name" to be used as an identifier, even if it clashes with >>> a keyword. >> >> > I strongly disagree, but can't seem to get anyone > ? to bite. > > We want to be able to introduce a keyword that was formally a name, still > allow > it to be used as a name, still allow code that uses it as a keyword to > interoperate > ? ? > with code that uses it as a name > , without changing the language > or implementation > too much. > > ?Ideally, Python would still not allow the keyword to be used as a name > and a > keyword in the same file?? > > The lexer could class the tokens as *keynames*, and the parser could use > the > context of the first instance of each keyname to determine if it's a name > or > keyword for the rest of that file. Projects that used the word as a name > would > only be prevented from also using it as a keyword in the same file. > > It's really then a question of whether users could elegantly and naturally > reference a name in another module without introducing the name to the > current module's namespace. > > We only reference external names (as syntactic names) in import statements, > as properties after the dot operator, and as keyword arguments. > > If code that used the word as a keyword was still allowed to use the word > as > a name after the dot operator and as a keyword argument *in an invocation*, > it would only change the language in a subtle way. > > If we could reference keynames in import statements, but not import the > name, > so basically allow `from keyname import keyname as name`, but not allow > `import keyname`, we could still easily import things that used the keyname > as a name. This wouldn't change the language too dramatically either. > > Maybe I'm just being dumb, but it seems like three subtle changes to the > language would allow for everything we want to have, with only minor > limitations > on the rare occasion that you want to use the new keyword with a library > that is > also using the same keyword as a name. > > I promise not to push this idea again, but would really appreciate someone > taking > a couple of minutes to explain why it's not worth responding to. I'm not > offended, > but would like to know what I'm doing wrong. > > Thanks. > I think your idea would work okay if everyone followed good programming practices. But when you have files that are tens of thousands of ugly code written by dozens of non-programmers over a dozen years it sounds like a recipe for a nightmare. For example someone you never met that left your group ten years ago could have made "True" be "np.bool_(1)" on a whim that makes your code break later in very hard-to-debug ways. To put it simply, I think it encourages people to take convenient shortcuts with implications they don't understand. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericfahlgren at gmail.com Wed May 16 10:00:00 2018 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Wed, 16 May 2018 07:00:00 -0700 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: References: <598EB1E5-AD1E-47C6-ADBC-A09036CFFA11@python.org> Message-ID: On Tue, May 15, 2018 at 10:11 PM Rob Speer wrote: > From what I can tell, if you wanted to exclude '__init__.py' from Nginx in > particular, you would have to write an unconventional Nginx configuration, > where you determine whether a path refers to a static file according to a > regex that excludes things that end in '__init__.py'. Anything is possible, > but this would be a significant discouragement to using importlib. > ? Ok, I haven't dug into the details ("It should be easy!" :) ), but couldn't you implement a Finder that based its search on, say, 'data.toc' instead of '__init__.py' and graft it into importlib.resources? -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.input at gmail.com Wed May 16 10:26:42 2018 From: carl.input at gmail.com (Carl Smith) Date: Wed, 16 May 2018 15:26:42 +0100 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: Thanks for the reply Todd. If `True` was redefined somewhere else, it would still be `True` for you. You could do `from oldlib import True as true` and have `true` equal ` np.bool_(1)`. You could reference `oldlib.True` or do `oldlib.function(True=x)` to interact with the name in the old library. None of this would actually apply to `True`, as it's a reserved word in all versions. The proposal only applies to new keywords that are used as names in other libraries. Again, thanks for taking the time. -- Carl Smith carl.input at gmail.com On 16 May 2018 at 14:46, Todd wrote: > On Tue, May 15, 2018, 23:03 Carl Smith wrote: > >> >> >>> On Tue, May 15, 2018 at 8:41 PM, Steven D'Aprano >>> wrote: >>> >>>> Inspired by Alex Brault's post: >>>> >>>> https://mail.python.org/pipermail/python-ideas/2018-May/050750.html >>>> >>>> I'd like to suggest we copy C#'s idea of verbatim identifiers, but >>>> using >>>> a backslash rather than @ sign: >>>> >>>> \name >>>> >>>> would allow "name" to be used as an identifier, even if it clashes with >>>> a keyword. >>> >>> >> I strongly disagree, but can't seem to get anyone >> ? to bite. >> >> We want to be able to introduce a keyword that was formally a name, still >> allow >> it to be used as a name, still allow code that uses it as a keyword to >> interoperate >> ? ? >> with code that uses it as a name >> , without changing the language >> or implementation >> too much. >> >> ?Ideally, Python would still not allow the keyword to be used as a name >> and a >> keyword in the same file?? >> >> The lexer could class the tokens as *keynames*, and the parser could use >> the >> context of the first instance of each keyname to determine if it's a name >> or >> keyword for the rest of that file. Projects that used the word as a name >> would >> only be prevented from also using it as a keyword in the same file. >> >> It's really then a question of whether users could elegantly and naturally >> reference a name in another module without introducing the name to the >> current module's namespace. >> >> We only reference external names (as syntactic names) in import >> statements, >> as properties after the dot operator, and as keyword arguments. >> >> If code that used the word as a keyword was still allowed to use the word >> as >> a name after the dot operator and as a keyword argument *in an >> invocation*, >> it would only change the language in a subtle way. >> >> If we could reference keynames in import statements, but not import the >> name, >> so basically allow `from keyname import keyname as name`, but not allow >> `import keyname`, we could still easily import things that used the >> keyname >> as a name. This wouldn't change the language too dramatically either. >> >> Maybe I'm just being dumb, but it seems like three subtle changes to the >> language would allow for everything we want to have, with only minor >> limitations >> on the rare occasion that you want to use the new keyword with a library >> that is >> also using the same keyword as a name. >> >> I promise not to push this idea again, but would really appreciate >> someone taking >> a couple of minutes to explain why it's not worth responding to. I'm not >> offended, >> but would like to know what I'm doing wrong. >> >> Thanks. >> > > > I think your idea would work okay if everyone followed good programming > practices. But when you have files that are tens of thousands of ugly code > written by dozens of non-programmers over a dozen years it sounds like a > recipe for a nightmare. > > For example someone you never met that left your group ten years ago could > have made "True" be "np.bool_(1)" on a whim that makes your code break > later in very hard-to-debug ways. > > To put it simply, I think it encourages people to take convenient > shortcuts with implications they don't understand. > >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.input at gmail.com Wed May 16 10:54:49 2018 From: carl.input at gmail.com (Carl Smith) Date: Wed, 16 May 2018 15:54:49 +0100 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: One problem with my proposal is with assignments to properties (`name.keyword = something`) and regular assignments (including class and def statements) inside the body of a class that subclasses and externally defined class would all need to be allowed, so that inherited names can be reassigned to and inherited methods can be overridden. As there is no way to know from static analysis whether the code is (legally) overriding something or (illegally) creating a name that is also a keyword in that file, doing so would need to be handled by a runtime exception, something like `NameError: cannot create names that are keywords in the same context`. Runtime errors still seem preferable to making keywords legally names in the same file (especially if we have to escape the names). -- Carl Smith carl.input at gmail.com On 16 May 2018 at 15:26, Carl Smith wrote: > Thanks for the reply Todd. > > If `True` was redefined somewhere else, it would still be `True` for you. > You could do `from oldlib import True as true` and have `true` equal ` > np.bool_(1)`. You could reference `oldlib.True` or do > `oldlib.function(True=x)` to interact with the name in the old library. > > None of this would actually apply to `True`, as it's a reserved word in > all versions. The proposal only applies to new keywords that are used as > names in other libraries. > > Again, thanks for taking the time. > > -- Carl Smith > carl.input at gmail.com > > On 16 May 2018 at 14:46, Todd wrote: > >> On Tue, May 15, 2018, 23:03 Carl Smith wrote: >> >>> >>> >>>> On Tue, May 15, 2018 at 8:41 PM, Steven D'Aprano >>>> wrote: >>>> >>>>> Inspired by Alex Brault's post: >>>>> >>>>> https://mail.python.org/pipermail/python-ideas/2018-May/050750.html >>>>> >>>>> I'd like to suggest we copy C#'s idea of verbatim identifiers, but >>>>> using >>>>> a backslash rather than @ sign: >>>>> >>>>> \name >>>>> >>>>> would allow "name" to be used as an identifier, even if it clashes >>>>> with >>>>> a keyword. >>>> >>>> >>> I strongly disagree, but can't seem to get anyone >>> ? to bite. >>> >>> We want to be able to introduce a keyword that was formally a name, still >>> allow >>> it to be used as a name, still allow code that uses it as a keyword to >>> interoperate >>> ? ? >>> with code that uses it as a name >>> , without changing the language >>> or implementation >>> too much. >>> >>> ?Ideally, Python would still not allow the keyword to be used as a name >>> and a >>> keyword in the same file?? >>> >>> The lexer could class the tokens as *keynames*, and the parser could >>> use the >>> context of the first instance of each keyname to determine if it's a >>> name or >>> keyword for the rest of that file. Projects that used the word as a name >>> would >>> only be prevented from also using it as a keyword in the same file. >>> >>> It's really then a question of whether users could elegantly and >>> naturally >>> reference a name in another module without introducing the name to the >>> current module's namespace. >>> >>> We only reference external names (as syntactic names) in import >>> statements, >>> as properties after the dot operator, and as keyword arguments. >>> >>> If code that used the word as a keyword was still allowed to use the >>> word as >>> a name after the dot operator and as a keyword argument *in an >>> invocation*, >>> it would only change the language in a subtle way. >>> >>> If we could reference keynames in import statements, but not import the >>> name, >>> so basically allow `from keyname import keyname as name`, but not allow >>> `import keyname`, we could still easily import things that used the >>> keyname >>> as a name. This wouldn't change the language too dramatically either. >>> >>> Maybe I'm just being dumb, but it seems like three subtle changes to the >>> language would allow for everything we want to have, with only minor >>> limitations >>> on the rare occasion that you want to use the new keyword with a library >>> that is >>> also using the same keyword as a name. >>> >>> I promise not to push this idea again, but would really appreciate >>> someone taking >>> a couple of minutes to explain why it's not worth responding to. I'm not >>> offended, >>> but would like to know what I'm doing wrong. >>> >>> Thanks. >>> >> >> >> I think your idea would work okay if everyone followed good programming >> practices. But when you have files that are tens of thousands of ugly code >> written by dozens of non-programmers over a dozen years it sounds like a >> recipe for a nightmare. >> >> For example someone you never met that left your group ten years ago >> could have made "True" be "np.bool_(1)" on a whim that makes your code >> break later in very hard-to-debug ways. >> >> To put it simply, I think it encourages people to take convenient >> shortcuts with implications they don't understand. >> >>> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Wed May 16 11:22:33 2018 From: toddrjen at gmail.com (Todd) Date: Wed, 16 May 2018 11:22:33 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: On Wed, May 16, 2018 at 10:26 AM, Carl Smith wrote: > Thanks for the reply Todd. > > If `True` was redefined somewhere else, it would still be `True` for you. > You could do `from oldlib import True as true` and have `true` equal ` > np.bool_(1)`. You could reference `oldlib.True` or do > `oldlib.function(True=x)` to interact with the name in the old library. > > Not if you need to make changes in the same tens of thousands of lines file. > None of this would actually apply to `True`, as it's a reserved word in > all versions. The proposal only applies to new keywords that are used as > names in other libraries. > No it isn't. It was added in Python 2.3 and only became a keyword in Python 3. Prior to that lots of other packages defined their own "True" (or "true" or "TRUE", etc), which is why it wasn't made a keyword for such a long time. But this is just an example of the sort of problem that can come up with your approach. The overall issue is that python has no way of knowing if the keyword is being used for legitimate backwards-compatibility purposes or someone intentionally overrode after it was made a keyword because they somehow thought it was a good idea. That is why being explicit about overriding the keyword is so important. > Again, thanks for taking the time. > > -- Carl Smith > carl.input at gmail.com > > On 16 May 2018 at 14:46, Todd wrote: > >> On Tue, May 15, 2018, 23:03 Carl Smith wrote: >> >>> >>> >>>> On Tue, May 15, 2018 at 8:41 PM, Steven D'Aprano >>>> wrote: >>>> >>>>> Inspired by Alex Brault's post: >>>>> >>>>> https://mail.python.org/pipermail/python-ideas/2018-May/050750.html >>>>> >>>>> I'd like to suggest we copy C#'s idea of verbatim identifiers, but >>>>> using >>>>> a backslash rather than @ sign: >>>>> >>>>> \name >>>>> >>>>> would allow "name" to be used as an identifier, even if it clashes >>>>> with >>>>> a keyword. >>>> >>>> >>> I strongly disagree, but can't seem to get anyone >>> ? to bite. >>> >>> We want to be able to introduce a keyword that was formally a name, still >>> allow >>> it to be used as a name, still allow code that uses it as a keyword to >>> interoperate >>> ? ? >>> with code that uses it as a name >>> , without changing the language >>> or implementation >>> too much. >>> >>> ?Ideally, Python would still not allow the keyword to be used as a name >>> and a >>> keyword in the same file?? >>> >>> The lexer could class the tokens as *keynames*, and the parser could >>> use the >>> context of the first instance of each keyname to determine if it's a >>> name or >>> keyword for the rest of that file. Projects that used the word as a name >>> would >>> only be prevented from also using it as a keyword in the same file. >>> >>> It's really then a question of whether users could elegantly and >>> naturally >>> reference a name in another module without introducing the name to the >>> current module's namespace. >>> >>> We only reference external names (as syntactic names) in import >>> statements, >>> as properties after the dot operator, and as keyword arguments. >>> >>> If code that used the word as a keyword was still allowed to use the >>> word as >>> a name after the dot operator and as a keyword argument *in an >>> invocation*, >>> it would only change the language in a subtle way. >>> >>> If we could reference keynames in import statements, but not import the >>> name, >>> so basically allow `from keyname import keyname as name`, but not allow >>> `import keyname`, we could still easily import things that used the >>> keyname >>> as a name. This wouldn't change the language too dramatically either. >>> >>> Maybe I'm just being dumb, but it seems like three subtle changes to the >>> language would allow for everything we want to have, with only minor >>> limitations >>> on the rare occasion that you want to use the new keyword with a library >>> that is >>> also using the same keyword as a name. >>> >>> I promise not to push this idea again, but would really appreciate >>> someone taking >>> a couple of minutes to explain why it's not worth responding to. I'm not >>> offended, >>> but would like to know what I'm doing wrong. >>> >>> Thanks. >>> >> >> >> I think your idea would work okay if everyone followed good programming >> practices. But when you have files that are tens of thousands of ugly code >> written by dozens of non-programmers over a dozen years it sounds like a >> recipe for a nightmare. >> >> For example someone you never met that left your group ten years ago >> could have made "True" be "np.bool_(1)" on a whim that makes your code >> break later in very hard-to-debug ways. >> >> To put it simply, I think it encourages people to take convenient >> shortcuts with implications they don't understand. >> >>> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From niki.spahiev at gmail.com Wed May 16 11:40:27 2018 From: niki.spahiev at gmail.com (Niki Spahiev) Date: Wed, 16 May 2018 18:40:27 +0300 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> Message-ID: On 16.05.2018 16:05, Andr?s Delfino wrote: > IMHO, it would be much easier to learn and understand if keywords can only > be used by escaping them, instead of depending where they occur. There can be 2 escape characters '\' and '.' Niki From Eloi.Gaudry at fft.be Wed May 16 09:27:50 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Wed, 16 May 2018 13:27:50 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <20180516111538.GY12683@ando.pearwood.info> References: <1525765072.24469.5.camel@fft.be> <0dd9cf67-08cc-6c57-0570-98507290c483@trueblade.com> <1525787426.5522.1.camel@fft.be> <1525793916.5522.5.camel@fft.be> <1526459338.23739.1.camel@fft.be> <20180516111538.GY12683@ando.pearwood.info> Message-ID: <1526477268.24366.1.camel@fft.be> On Wed, 2018-05-16 at 21:15 +1000, Steven D'Aprano wrote: > On Wed, May 16, 2018 at 08:29:00AM +0000, Eloi Gaudry wrote: > > Is there some interest in the proposal or should I finally close > > this > > thread ? > > I'm definitely interested in the concept, not the suggested syntax > or? > semantics. Would you briefly describe a syntax that would better fit this concept ? From steve at pearwood.info Wed May 16 13:09:31 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 17 May 2018 03:09:31 +1000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <1526477268.24366.1.camel@fft.be> References: <1525765072.24469.5.camel@fft.be> <0dd9cf67-08cc-6c57-0570-98507290c483@trueblade.com> <1525787426.5522.1.camel@fft.be> <1525793916.5522.5.camel@fft.be> <1526459338.23739.1.camel@fft.be> <20180516111538.GY12683@ando.pearwood.info> <1526477268.24366.1.camel@fft.be> Message-ID: <20180516170931.GZ12683@ando.pearwood.info> On Wed, May 16, 2018 at 01:27:50PM +0000, Eloi Gaudry wrote: > On Wed, 2018-05-16 at 21:15 +1000, Steven D'Aprano wrote: > > On Wed, May 16, 2018 at 08:29:00AM +0000, Eloi Gaudry wrote: > > > Is there some interest in the proposal or should I finally close > > > this > > > thread ? > > > > I'm definitely interested in the concept, not the suggested syntax > > or?semantics. > > Would you briefly describe a syntax that would better fit this concept > ? The syntax is the minor point: you give is an ungainly name, "runtime_assert", and your proposed PEP shows it requiring parentheses as if it were an ordinary function. The bigger problem is the semantics. As I already said in an earlier email, you don't explain what "runtime_assert_active" is (is it a per-module global variable? a single application-wide super-global? a local variable? something else?) or how we are supposed to set it. That too is an ungainly name, and the fact that there's only *one* such flag (whether it is per module or not) makes this proposal useless for my needs. Another problem is that your runtime_assert *prints* the error message instead of raising an exception, and there's no way to customise the message. But the most important reason is that I'm not really interested in adding a new keyword for this. I would much prefer to explore ways to allow ordinary functions to receive arguments and be able to delay evaluation of those arguments until needed. -- Steve From drekin at gmail.com Wed May 16 13:24:19 2018 From: drekin at gmail.com (=?UTF-8?B?QWRhbSBCYXJ0b8Wh?=) Date: Wed, 16 May 2018 19:24:19 +0200 Subject: [Python-ideas] Keyword declarations Message-ID: Hello, I have yet another idea regarding the the clashes between new keywords and already used names. How about introducing two new keywords *wink* that would serve as lexical keyword/nonkeyword declarations, similarly to nonlocal and global declarations? def f(): nonkeyword if if = 2 # we use 'if' as an identifier def g(): keyword if if x > 0: pass # now 'if' again introduces a conditional statement This allows to have a name as both identifier and keyword in a single module, and since it's lexical, it could be in principle syntax-highlighted correctly. When a new keyword is added to the list of standard keywords like 'given' or 'where', a module that uses the name as identifier could be easily fixed by a global declaration 'nonkeyword given'. Maybe even exception messages pointing to this could be added. If 'nonkeyword keyword' is allowed, we can also fix code using the name 'keyword' as an identifier, but doing so in the global scope couldn't be undone. On the other hand, new language features depending on new keywords could be made provisionary by not adding the keywords to the standard list, so people who would like to use them would need to opt in by e.g. 'keyword given'. Surely, this provisional mechanism isn't robust at all since new features may just extend the usage of current keywords. Declaring a keyword that has no meaning in the language would result in an exception: keyword foo # SyntaxError: undefined keyword 'foo' It should be possible to use a keyword as a parameter name without need to declare it in the surrounding scope, the local declaration would suffice: # nonkeyword if # not needed def f(if=3): # ok nonkeyword if Other option is to interpret parameters always as nonkeywords or to raise a special syntax error when a keyword occurs at a place of a formal parameter (similarly to 'def f(x): nonlocal x'). Clearly, even if this proposal diminished the cost of adding new keywords, the cost would still be high. Best regards, Adam Barto? -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.input at gmail.com Wed May 16 14:17:30 2018 From: carl.input at gmail.com (Carl Smith) Date: Wed, 16 May 2018 19:17:30 +0100 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> Message-ID: > Not if you need to make changes in the same tens of thousands of lines file. But what has that got to do with the the syntax of the new code? The old code is what it is. I did think after I replied that `True` wasn't actually reserved until more recently, but the point still stands: You would be able to reference the name *as defined* in an external library, and yeah, it could refer to anything, but that's kinda the point. We have to assume the library does something sane with the name. We can't preempt an employee sabotaging `True`. As a more realistic example (if not for Python), say `until` became a keyword, then you could end up with lines like this: from oldlib import until as upto dance(until="am") event.until = time(9, 30) > The overall issue is that python has no way of knowing if the keyword is being used for legitimate > backwards-compatibility purposes or someone intentionally overrode after it was made a keyword > because they somehow thought it was a good idea. I only said that Python does not know *until runtime*, and I was wrong when I described that as a problem. A runtime NameError actually makes perfect sense. Assigning to `self.until` or assigning to `until` inside a subclass should not be a syntax error. A NameError would be correct. It worth mentioning that the cost of checking only applies to cases where the name in question is also keyword, so almost never. -- Carl Smith carl.input at gmail.com On 16 May 2018 at 16:40, Niki Spahiev wrote: > On 16.05.2018 16:05, Andr?s Delfino wrote: > >> IMHO, it would be much easier to learn and understand if keywords can only >> be used by escaping them, instead of depending where they occur. >> > > There can be 2 escape characters '\' and '.' > > Niki > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.input at gmail.com Wed May 16 14:20:35 2018 From: carl.input at gmail.com (Carl Smith) Date: Wed, 16 May 2018 19:20:35 +0100 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> Message-ID: ?> There can be 2 escape characters '\' and '.' That's clever, but then we have to put a slash in front of names in imports, assignments and keyword arguments, but not properties. -- Carl Smith carl.input at gmail.com On 16 May 2018 at 19:17, Carl Smith wrote: > > Not if you need to make changes in the same tens of thousands of lines > file. > > But what has that got to do with the the syntax of the new code? The old > code is > what it is. > > I did think after I replied that `True` wasn't actually reserved until > more recently, but > the point still stands: You would be able to reference the name *as > defined* in an > external library, and yeah, it could refer to anything, but that's kinda > the point. We > have to assume the library does something sane with the name. We can't > preempt > an employee sabotaging `True`. > > As a more realistic example (if not for Python), say `until` became a > keyword, then > you could end up with lines like this: > > from oldlib import until as upto > > dance(until="am") > > event.until = time(9, 30) > > > The overall issue is that python has no way of knowing if the keyword > is being used for legitimate > > backwards-compatibility purposes or someone intentionally overrode after > it was made a keyword > > because they somehow thought it was a good idea. > > I only said that Python does not know *until runtime*, and I was wrong > when I described that as a > problem. A runtime NameError actually makes perfect sense. Assigning to `self.until` > or assigning > to `until` inside a subclass should not be a syntax error. A NameError > would be correct. > > It worth mentioning that the cost of checking only applies to cases where > the name in question is also > keyword, so almost never. > > > -- Carl Smith > carl.input at gmail.com > > On 16 May 2018 at 16:40, Niki Spahiev wrote: > >> On 16.05.2018 16:05, Andr?s Delfino wrote: >> >>> IMHO, it would be much easier to learn and understand if keywords can >>> only >>> be used by escaping them, instead of depending where they occur. >>> >> >> There can be 2 escape characters '\' and '.' >> >> Niki >> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed May 16 14:37:24 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 16 May 2018 14:37:24 -0400 Subject: [Python-ideas] Keyword declarations In-Reply-To: References: Message-ID: On 5/16/2018 1:24 PM, Adam Barto? wrote: > Hello, > > I have yet another idea regarding the the clashes between new keywords > and already used names. How about introducing two new keywords *wink* > that would serve as lexical keyword/nonkeyword declarations, similarly > to nonlocal and global declarations? > > def f(): > ??? nonkeyword if > ??? if = 2 # we use 'if' as an identifier > ??? def g(): > ??????? keyword if > ??????? if x > 0: pass # now 'if' again introduces a conditional statement These are not at all similar to 'global' and 'nonlocal'. These would make Python into a syntactically context-dependent language, rather than a restricted (ll(1)) context-free language. I am pretty sure that the current parser generator could not handle this. 'Global' and 'nonlocal' have NO effect on what is syntactically legal and therefore no effect on syntax parsing. They only affect the meaning (semantics) of other statements within the same function. The compiler handles this by making two passes over a function body. -- Terry Jan Reedy From python at mrabarnett.plus.com Wed May 16 15:02:47 2018 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 16 May 2018 20:02:47 +0100 Subject: [Python-ideas] Specifying Python version Message-ID: <864f9b35-8a84-00a5-35ef-dfa692864b43@mrabarnett.plus.com> Instead of verbatim identifiers, how about a special comment giving the Python version in which the file was written? There could then be a tool similar to 2to3 that converts the file to a more recent version of Python that might have new reserved words. In most cases the new file would merely be a copy of the original, but with an updated Python version. From carl.input at gmail.com Wed May 16 15:03:18 2018 From: carl.input at gmail.com (Carl Smith) Date: Wed, 16 May 2018 20:03:18 +0100 Subject: [Python-ideas] Keyword declarations In-Reply-To: References: Message-ID: If `def(if=3)...` works implicitly, then why not make `if = 3`, `x.if = 3`, `import if`, `def if` and `class if` implicit too? Another issue is what happens here: keyword if import if f(if=3) f.if = 3 The keyword will be a valid name in old code, so you need to be able to reference it as a name in code that uses it as a keyword. -- Carl Smith carl.input at gmail.com On 16 May 2018 at 18:24, Adam Barto? wrote: > Hello, > > I have yet another idea regarding the the clashes between new keywords and > already used names. How about introducing two new keywords *wink* that > would serve as lexical keyword/nonkeyword declarations, similarly to > nonlocal and global declarations? > > def f(): > nonkeyword if > if = 2 # we use 'if' as an identifier > def g(): > keyword if > if x > 0: pass # now 'if' again introduces a conditional statement > > This allows to have a name as both identifier and keyword in a single > module, and since it's lexical, it could be in principle syntax-highlighted > correctly. > > When a new keyword is added to the list of standard keywords like 'given' > or 'where', a module that uses the name as identifier could be easily fixed > by a global declaration 'nonkeyword given'. Maybe even exception messages > pointing to this could be added. If 'nonkeyword keyword' is allowed, we can > also fix code using the name 'keyword' as an identifier, but doing so in > the global scope couldn't be undone. > > On the other hand, new language features depending on new keywords could > be made provisionary by not adding the keywords to the standard list, so > people who would like to use them would need to opt in by e.g. 'keyword > given'. Surely, this provisional mechanism isn't robust at all since new > features may just extend the usage of current keywords. > > Declaring a keyword that has no meaning in the language would result in an > exception: > > keyword foo # SyntaxError: undefined keyword 'foo' > > It should be possible to use a keyword as a parameter name without need to > declare it in the surrounding scope, the local declaration would suffice: > > # nonkeyword if # not needed > def f(if=3): # ok > nonkeyword if > > Other option is to interpret parameters always as nonkeywords or to raise > a special syntax error when a keyword occurs at a place of a formal > parameter (similarly to 'def f(x): nonlocal x'). > > Clearly, even if this proposal diminished the cost of adding new keywords, > the cost would still be high. > > > Best regards, > Adam Barto? > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.input at gmail.com Wed May 16 15:22:28 2018 From: carl.input at gmail.com (Carl Smith) Date: Wed, 16 May 2018 20:22:28 +0100 Subject: [Python-ideas] Keyword declarations In-Reply-To: References: Message-ID: My proposal assumes we want to be able to reference the name as defined in external libraries, but never have it be a name and a keyword in the same namespace. Your proposal (and the others I've seen) seem to be deliberately aiming to allow that. Do you want to have keywords that are names in the same namespace because that would be nice in itself, or are you only tolerating that to solve the original problem? If being free to do `\if if not \not else \else or \or` is something people want for its own upsides, my proposal is useless, but I thought we only wanted a way to introduce keywords that are already names. -- Carl Smith carl.input at gmail.com On 16 May 2018 at 20:03, Carl Smith wrote: > If `def(if=3)...` works implicitly, then why not make `if = 3`, `x.if = > 3`, `import if`, `def if` and `class if` implicit too? > > Another issue is what happens here: > > keyword if > import if > f(if=3) > f.if = 3 > > The keyword will be a valid name in old code, so you need to be able to > reference it as a name in code that uses it as a keyword. > > > -- Carl Smith > carl.input at gmail.com > > On 16 May 2018 at 18:24, Adam Barto? wrote: > >> Hello, >> >> I have yet another idea regarding the the clashes between new keywords >> and already used names. How about introducing two new keywords *wink* that >> would serve as lexical keyword/nonkeyword declarations, similarly to >> nonlocal and global declarations? >> >> def f(): >> nonkeyword if >> if = 2 # we use 'if' as an identifier >> def g(): >> keyword if >> if x > 0: pass # now 'if' again introduces a conditional statement >> >> This allows to have a name as both identifier and keyword in a single >> module, and since it's lexical, it could be in principle syntax-highlighted >> correctly. >> >> When a new keyword is added to the list of standard keywords like 'given' >> or 'where', a module that uses the name as identifier could be easily fixed >> by a global declaration 'nonkeyword given'. Maybe even exception messages >> pointing to this could be added. If 'nonkeyword keyword' is allowed, we can >> also fix code using the name 'keyword' as an identifier, but doing so in >> the global scope couldn't be undone. >> >> On the other hand, new language features depending on new keywords could >> be made provisionary by not adding the keywords to the standard list, so >> people who would like to use them would need to opt in by e.g. 'keyword >> given'. Surely, this provisional mechanism isn't robust at all since new >> features may just extend the usage of current keywords. >> >> Declaring a keyword that has no meaning in the language would result in >> an exception: >> >> keyword foo # SyntaxError: undefined keyword 'foo' >> >> It should be possible to use a keyword as a parameter name without need >> to declare it in the surrounding scope, the local declaration would suffice: >> >> # nonkeyword if # not needed >> def f(if=3): # ok >> nonkeyword if >> >> Other option is to interpret parameters always as nonkeywords or to raise >> a special syntax error when a keyword occurs at a place of a formal >> parameter (similarly to 'def f(x): nonlocal x'). >> >> Clearly, even if this proposal diminished the cost of adding new >> keywords, the cost would still be high. >> >> >> Best regards, >> Adam Barto? >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Wed May 16 15:40:40 2018 From: toddrjen at gmail.com (Todd) Date: Wed, 16 May 2018 15:40:40 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> Message-ID: On Wed, May 16, 2018 at 2:17 PM, Carl Smith wrote: > > Not if you need to make changes in the same tens of thousands of lines > file. > > But what has that got to do with the the syntax of the new code? The old > code is > what it is. > > Again, because you end up with hard-to-debug issues through no fault of your own. > I did think after I replied that `True` wasn't actually reserved until > more recently, but > the point still stands: You would be able to reference the name *as > defined* in an > external library, and yeah, it could refer to anything, but that's kinda > the point. We > have to assume the library does something sane with the name. We can't > preempt > an employee sabotaging `True`. > > We can and do preempt someone sabotaging a keywords by not letting anyone override them. That is the whole point of using reserved keywords. Some languages allow you to change important words, some don't. Guido made a conscious decision to make certain words keywords, and to not let anyone change them, I believe to avoid the sorts of issues I have brought up. You are talking about removing one of the most important and long-standing protections the language has in place. That is not a small change. > -- Carl Smith > carl.input at gmail.com > > On 16 May 2018 at 16:40, Niki Spahiev wrote: > >> On 16.05.2018 16:05, Andr?s Delfino wrote: >> >>> IMHO, it would be much easier to learn and understand if keywords can >>> only >>> be used by escaping them, instead of depending where they occur. >>> >> >> There can be 2 escape characters '\' and '.' >> >> Niki >> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.input at gmail.com Wed May 16 17:09:56 2018 From: carl.input at gmail.com (Carl Smith) Date: Wed, 16 May 2018 22:09:56 +0100 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> Message-ID: > We can and do preempt someone sabotaging keywords by not letting anyone override them. > That is the whole point of using reserved keywords. Some languages allow you to change > important words, some don't. Guido made a conscious decision to make certain words keywords, > and to not let anyone change them, I believe to avoid the sorts of issues I have brought up. > You are talking about removing one of the most important and long-standing protections the > language has in place. That is not a small change. Making names keywords requires that keywords also be names. If Guido is open to introducing keywords that are currently names, it cannot be lost on him that some code will use names that are now keywords. If your position is that Guido shouldn't introduce keywords that are currently used as names at all, fair enough; that'd be my first choice too. But, if we are going to do it, I have a strong preference for a specific approach. -- Carl Smith carl.input at gmail.com On 16 May 2018 at 20:40, Todd wrote: > On Wed, May 16, 2018 at 2:17 PM, Carl Smith wrote: > >> > Not if you need to make changes in the same tens of thousands of lines >> file. >> >> But what has that got to do with the the syntax of the new code? The old >> code is >> what it is. >> >> > Again, because you end up with hard-to-debug issues through no fault of > your own. > > >> I did think after I replied that `True` wasn't actually reserved until >> more recently, but >> the point still stands: You would be able to reference the name *as >> defined* in an >> external library, and yeah, it could refer to anything, but that's kinda >> the point. We >> have to assume the library does something sane with the name. We can't >> preempt >> an employee sabotaging `True`. >> >> > We can and do preempt someone sabotaging a keywords by not letting anyone > override them. That is the whole point of using reserved keywords. Some > languages allow you to change important words, some don't. Guido made a > conscious decision to make certain words keywords, and to not let anyone > change them, I believe to avoid the sorts of issues I have brought up. You > are talking about removing one of the most important and long-standing > protections the language has in place. That is not a small change. > > >> -- Carl Smith >> carl.input at gmail.com >> >> On 16 May 2018 at 16:40, Niki Spahiev wrote: >> >>> On 16.05.2018 16:05, Andr?s Delfino wrote: >>> >>>> IMHO, it would be much easier to learn and understand if keywords can >>>> only >>>> be used by escaping them, instead of depending where they occur. >>>> >>> >>> There can be 2 escape characters '\' and '.' >>> >>> Niki >>> >>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed May 16 18:58:34 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 17 May 2018 10:58:34 +1200 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> Message-ID: <5AFCB79A.2000703@canterbury.ac.nz> Todd wrote: > The overall issue is that python has no way of knowing > if the keyword is being used for legitimate backwards-compatibility > purposes or someone intentionally overrode after it was made a keyword > because they somehow thought it was a good idea. That is why being > explicit about overriding the keyword is so important. The trouble with explicitly overriding keywords is that it still requires old code to be changed whenever a new keyword is added, which as far as I can see almost competely defeats the purpose. If e.g. you need to change all uses of given to \given in order for your code to keep working in Python 3.x for some x, you might just as well change it to given_ or some other already-legal name. The only remotely legitimate use I can think of is for calling APIs that come from a different language, but the same thing applies there -- names in the Python binding can always be modified somehow to make them legal. As far as I can see, any mechanism allowing keywords to be used as names has to be completely transparent to existing code, otherwise there's no point in it. -- Greg From steve at pearwood.info Wed May 16 20:21:55 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 17 May 2018 10:21:55 +1000 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <5AFCB79A.2000703@canterbury.ac.nz> References: <20180516004153.GU12683@ando.pearwood.info> <5AFCB79A.2000703@canterbury.ac.nz> Message-ID: <20180517002153.GA12683@ando.pearwood.info> On Thu, May 17, 2018 at 10:58:34AM +1200, Greg Ewing wrote: > The trouble with explicitly overriding keywords is that it > still requires old code to be changed whenever a new keyword > is added, which as far as I can see almost competely defeats > the purpose. If e.g. you need to change all uses of given > to \given in order for your code to keep working in > Python 3.x for some x, you might just as well change it > to given_ or some other already-legal name. Well, maybe. Certainly using name_ is a possible solution, and it is one which has worked for over a quarter century. We can argue about whether \name or name_ looks nicer, but \name has one advantage: the key used in the namespace is actually "name". That's important: see below. > The only remotely legitimate use I can think of is for > calling APIs that come from a different language, but the > same thing applies there -- names in the Python binding can > always be modified somehow to make them legal. Of course they can be modified. But having to do so is a pain. With the status quo, when dealing with external data which may include names which are keywords, we have to: - add an underscore when we read keywords from external data - add an underscore when used as obj.kw literals - add an underscore when used as getattr("kw") literals - conditionally remove trailing underscore when writing to external APIs to: - do nothing special when we read keywords from external data - add a backslash when used as obj.kw literals - do nothing special when used as getattr("kw") literals - do nothing special when writing to external APIs I think that overall this pushes it from a mere matter of visual preference \kw versus kw_ to a significant win for verbatim names. Let's say you're reading from a CSV file, creating an object from each row, and processing it: # untested reader = csv.reader(infile) header = next(reader) header = [name + "_" if name in keywords.kwlist() else name for name in header] for row in reader: obj = SimpleNamespace(*zip(header, row)) process(obj) The consumer of these objects, process(), has to reverse the transformation: def process(obj): for name, value in vars(obj): if name.endswith("_") and name[:-1] in keywords.kwlist(): name = name[:-1] write_to_external_API(name, value) Verbatim names lets us skip both of these boilerplate steps. An interesting case is when you are using the keywords as hard-coded names for attribute access. In the status quo, we write: obj.name_ obj.getattr("name_") In the first line, if you neglect the _ the compiler will complain and you get a syntax error. In the second line, if you neglect the _ you'll get no warning, only a runtime failure. With verbatim names, we can write: obj.\name obj.getattr("name") # Don't escape it here! In this case, the failure modes are similar: - if you forget the backslash in the first line, you get a SyntaxError at compile time, so there's no change here. - if you wrongly include the backslash in the second line, there are two cases: * if the next character matches a string escape, say \n or \t, you'll get no error but a runtime failure; (but linters could warn about that) * if it doesn't match, say \k, you'll now get a warning and eventually a failure as we depreciate silently ignoring backslashes. -- Steve From steve at pearwood.info Wed May 16 20:33:25 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 17 May 2018 10:33:25 +1000 Subject: [Python-ideas] Specifying Python version In-Reply-To: <864f9b35-8a84-00a5-35ef-dfa692864b43@mrabarnett.plus.com> References: <864f9b35-8a84-00a5-35ef-dfa692864b43@mrabarnett.plus.com> Message-ID: <20180517003325.GC12683@ando.pearwood.info> On Wed, May 16, 2018 at 08:02:47PM +0100, MRAB wrote: > Instead of verbatim identifiers, how about a special comment giving the > Python version in which the file was written? > > There could then be a tool similar to 2to3 that converts the file to a > more recent version of Python that might have new reserved words. In > most cases the new file would merely be a copy of the original, but with > an updated Python version. That would be a serious PITA for modules which have to work with more than one version of Python. -- Steve From cs at cskk.id.au Wed May 16 20:48:43 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Thu, 17 May 2018 10:48:43 +1000 Subject: [Python-ideas] Specifying Python version In-Reply-To: <20180517003325.GC12683@ando.pearwood.info> References: <20180517003325.GC12683@ando.pearwood.info> Message-ID: <20180517004843.GA74456@cskk.homeip.net> On 17May2018 10:33, Steven D'Aprano wrote: >On Wed, May 16, 2018 at 08:02:47PM +0100, MRAB wrote: >> Instead of verbatim identifiers, how about a special comment giving the >> Python version in which the file was written? >> >> There could then be a tool similar to 2to3 that converts the file to a >> more recent version of Python that might have new reserved words. In >> most cases the new file would merely be a copy of the original, but with >> an updated Python version. > >That would be a serious PITA for modules which have to work with more >than one version of Python. It also feels rather fragile. Though I'm having a little trouble formulating a concrete example, I just feel like this is the kind of thing that is easily mangled. For example in a revision control merge (pull new change from branch using older Python, whose older Python got slightly changed in the special comment - in the diff the special comment looks new, easy to accidentally roll over the target branch whose special comment has been updated: main branch: # ##pyver 3.3## maintenance branch: # ##pyver 2.4## maintenance branch: update to # ##pyver 2.5## and add a feature or fix main branch: merge changeset from maintenance, tread on the "3.3" with "2.5" Cheers, Cameron Simpson From steve at pearwood.info Wed May 16 20:53:46 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 17 May 2018 10:53:46 +1000 Subject: [Python-ideas] Keyword declarations In-Reply-To: References: Message-ID: <20180517005345.GD12683@ando.pearwood.info> On Wed, May 16, 2018 at 07:24:19PM +0200, Adam Barto? wrote: > Hello, > > I have yet another idea regarding the the clashes between new keywords and > already used names. How about introducing two new keywords *wink* that > would serve as lexical keyword/nonkeyword declarations, similarly to > nonlocal and global declarations? > > def f(): > nonkeyword if > if = 2 # we use 'if' as an identifier > def g(): > keyword if > if x > 0: pass # now 'if' again introduces a conditional statement This is absolutely no help at all for the common case that we have an identifier that is a keyword and want to use it as a keyword in the same block. For example, we can currently write: try: value = data.except_ except: value = data.missing() say. We're using "except_" because the data comes from some external interface where it uses "except", but we can't use that because its a keyword. I also challenge to think about how you will document the complicated rules for when you can and can't use keywords as names, especially think about explaining them to beginners: def spam(None=42): print(None) # What will this print? x = None # Fine, but what does it do? None = 999 # Is this an error or not? Remember the KISS principle. -- Steve From greg.ewing at canterbury.ac.nz Thu May 17 02:03:32 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 17 May 2018 18:03:32 +1200 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <20180517002153.GA12683@ando.pearwood.info> References: <20180516004153.GU12683@ando.pearwood.info> <5AFCB79A.2000703@canterbury.ac.nz> <20180517002153.GA12683@ando.pearwood.info> Message-ID: <5AFD1B34.6030203@canterbury.ac.nz> Steven D'Aprano wrote: > Let's say you're reading from a CSV file, creating an object from each > row, and processing it: Okay, I can see it could be useful for situations like that. But this is still a completely different use case from the one that started this discussion, which was making it less painful to add new keywords to the language. The backslash idea doesn't help at all with that. -- Greg From Eloi.Gaudry at fft.be Thu May 17 03:46:41 2018 From: Eloi.Gaudry at fft.be (Eloi Gaudry) Date: Thu, 17 May 2018 07:46:41 +0000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <20180516170931.GZ12683@ando.pearwood.info> References: <1525765072.24469.5.camel@fft.be> <0dd9cf67-08cc-6c57-0570-98507290c483@trueblade.com> <1525787426.5522.1.camel@fft.be> <1525793916.5522.5.camel@fft.be> <1526459338.23739.1.camel@fft.be> <20180516111538.GY12683@ando.pearwood.info> <1526477268.24366.1.camel@fft.be> <20180516170931.GZ12683@ando.pearwood.info> Message-ID: <1526543198.32395.1.camel@fft.be> On Thu, 2018-05-17 at 03:09 +1000, Steven D'Aprano wrote: > On Wed, May 16, 2018 at 01:27:50PM +0000, Eloi Gaudry wrote: > > On Wed, 2018-05-16 at 21:15 +1000, Steven D'Aprano wrote: > > > On Wed, May 16, 2018 at 08:29:00AM +0000, Eloi Gaudry wrote: > > > > Is there some interest in the proposal or should I finally > > > > close > > > > this > > > > thread ? > > > > > > I'm definitely interested in the concept, not the suggested > > > syntax > > > or?semantics. > > > > Would you briefly describe a syntax that would better fit this > > concept > > ? > > The syntax is the minor point: you give is an ungainly name,? > "runtime_assert", and your proposed PEP shows it requiring > parentheses? > as if it were an ordinary function. > > The bigger problem is the semantics. As I already said in an earlier? > email, you don't explain what "runtime_assert_active" is (is it a? > per-module global variable? a single application-wide super-global? > a? > local variable? something else?) or how we are supposed to set it. I proposed to have several ways to set it (see previous answers) : one would be extension based and the other would rely on having a builtin value that one would be able to change using a set function of the python core. > That? > too is an ungainly name, and the fact that there's only *one* such > flag? > (whether it is per module or not) makes this proposal useless for my? > needs. I agree with you on this : the main issue with my proposal is that having only one assert (function with deferred evaluation) more is not enough : we would need to cover several other scenarii such as an extension wanting to activate its own asserts without activating other extensions checks. > Another problem is that your runtime_assert *prints* the error? > message instead of raising an exception, and there's no way to? > customise the message. It actually raise an AssertionError. I do agree that it should allow to set/display a message though. > > But the most important reason is that I'm not really interested in? > adding a new keyword for this. I would much prefer to explore ways > to? > allow ordinary functions to receive arguments and be able to delay? > evaluation of those arguments until needed. What would then be solution for implementing the runtime_assert with the current python api ? A lambda ? From rosuav at gmail.com Thu May 17 04:38:31 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 17 May 2018 18:38:31 +1000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <1526543198.32395.1.camel@fft.be> References: <1525765072.24469.5.camel@fft.be> <0dd9cf67-08cc-6c57-0570-98507290c483@trueblade.com> <1525787426.5522.1.camel@fft.be> <1525793916.5522.5.camel@fft.be> <1526459338.23739.1.camel@fft.be> <20180516111538.GY12683@ando.pearwood.info> <1526477268.24366.1.camel@fft.be> <20180516170931.GZ12683@ando.pearwood.info> <1526543198.32395.1.camel@fft.be> Message-ID: On Thu, May 17, 2018 at 5:46 PM, Eloi Gaudry wrote: >> But the most important reason is that I'm not really interested in >> adding a new keyword for this. I would much prefer to explore ways >> to >> allow ordinary functions to receive arguments and be able to delay >> evaluation of those arguments until needed. > What would then be solution for implementing the runtime_assert with > the current python api ? A lambda ? if __runtime_assert__ and long_expensive_calculation: raise AssertionError There's minimal run-time cost, but you lose the convenience of having the expression shown in the error's message. There is, however, a lot of text to this. So if it could be wrapped up, that would be useful. Hence the idea of exploring ways of doing this: def rtassert(expr): if not __runtime_assert__: return if expr.eval(): return raise AssertionError(expr.text) rtassert(&long_expensive_calculation) where the ampersand (or something of the sort) means "pass this expression along unevaluated". The only way to do this currently is with a lambda function, but they don't get their texts captured. It'd take either syntactic support, or something like MacroPy, to make this happen. ChrisA From drekin at gmail.com Thu May 17 04:40:11 2018 From: drekin at gmail.com (=?UTF-8?B?QWRhbSBCYXJ0b8Wh?=) Date: Thu, 17 May 2018 10:40:11 +0200 Subject: [Python-ideas] Keyword declarations Message-ID: On Wed May 16 20:53:46 EDT 2018, Steven D'Aprano wrote: > On Wed, May 16, 2018 at 07:24:19PM +0200, Adam Barto? wrote: >> Hello, >> >> I have yet another idea regarding the the clashes between new keywords and >> already used names. How about introducing two new keywords *wink* that >> would serve as lexical keyword/nonkeyword declarations, similarly to >> nonlocal and global declarations? >> >> def f(): >> nonkeyword if >> if = 2 # we use 'if' as an identifier >> def g(): >> keyword if >> if x > 0: pass # now 'if' again introduces a conditional statement > > This is absolutely no help at all for the common case that we have an > identifier that is a keyword and want to use it as a keyword in the same > block. For example, we can currently write: > > try: > value = data.except_ > except: > value = data.missing() > > say. We're using "except_" because the data comes from some external > interface where it uses "except", but we can't use that because its a > keyword. If it was possible to consider any syntactical block for the declarations, this use case could be handled like: try: nonkeyword except value = data.except except: value = data.missing() Alternatively, data.except could be allowed even without a declaration as another idea suggests. There are syntactical positions where it is easy to distinguish between a keyword and an identifier, there are positions where it is harder, or even impossible due to limitations on the parser complexity. So both ideas may be viewed as complementary, and the base case of my proposal would be just a per module opt out from a keyword, so it is easier to fix such module when a new keyword is introduced in the language. > I also challenge to think about how you will document the complicated > rules for when you can and can't use keywords as names, especially think > about explaining them to beginners: > > def spam(None=42): > print(None) # What will this print? > x = None # Fine, but what does it do? > None = 999 # Is this an error or not? None is a different kind of keyword. Ordinary keywords that make the statements of the language have no value, so 'x = if' makes no sense. On the other hand, None, True, and False are singleton values (like Ellispsis and NotImplemented) that are additionally protected from be redefined, which makes them similar to keywords. I think that the barrier for adding new keyword constants is even higher than the barrier for adding new ordinary keywords, and so may be omited when trying to solve the keyword/identifier problem. nonkeyword None # SyntaxError: a keyword constant 'None' cannot be an identifier > Remember the KISS principle. It seemed to me that the syntactical block based declarations are quite simple in principle, but you are right, there are many issues. Best regards, Adam Barto? -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at brice.xyz Thu May 17 05:07:27 2018 From: contact at brice.xyz (Brice Parent) Date: Thu, 17 May 2018 11:07:27 +0200 Subject: [Python-ideas] Specifying Python version In-Reply-To: <864f9b35-8a84-00a5-35ef-dfa692864b43@mrabarnett.plus.com> References: <864f9b35-8a84-00a5-35ef-dfa692864b43@mrabarnett.plus.com> Message-ID: It's in the same spirit as what I proposed in my answer to Guido's crazy idea, but set at a different level (in code instead of in setup.py in my proposal). I think it belongs to the same discussion, so I'm copying it here (sorry for the inconvenience if you already read it there): Le 16/05/2018 ? 11:22, Brice Parent a ?crit?: > > Why not have the compatibility be done at setup.py's level? > > Like having something like this: > > setup( > ??? name="LegacyLib", > ??? version="8.4.1", > ??? ... > ??? max_compatibility="3.4"? # Defining here Python's max version for > which this lib has been upgraded > ) > > Of course, we may use any other word instead of "max_compatibility", > like "designed_for", "python_version", or anything a better English > speaker could think of. > > The point is, it would either: > > - when you install the library, rename all variables that are now > keywords (we'd know the exact list thanks to max_compatiblity) by > suffixing them with "_" > > - or set a flag that will do that when creating the *.pyc files. > > Possible problems/limitations I can already find: > > - There would still be possible errors when using variable names that > are generated on the fly (I have no clue how this could ever be addressed) > - It might get complicated at some point to know what to do, like when > we have lib_a in some version depending on lib_b (with or without a > max_compatibility version number), it is obvious that lib_a will use > lib_b's original variable names (without the appended "_"), but our > code which might also want to interact with lib_b would have to. > -Brice Le 16/05/2018 ? 21:02, MRAB a ?crit?: > Instead of verbatim identifiers, how about a special comment giving > the Python version in which the file was written? > > There could then be a tool similar to 2to3 that converts the file to a > more recent version of Python that might have new reserved words. In > most cases the new file would merely be a copy of the original, but > with an updated Python version. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From kenlhilton at gmail.com Thu May 17 06:53:32 2018 From: kenlhilton at gmail.com (Ken Hilton) Date: Thu, 17 May 2018 18:53:32 +0800 Subject: [Python-ideas] String and bytes bitwise operations Message-ID: Hi all, We all know the bitwise operators: & (and), | (or), ^ (xor), and ~ (not). We know how they work with numbers: 420 ^ 502 110100100 111110110 == XOR == 001010010 = 82 But it might be useful in some cases to (let's say) xor a string (or bytestring): HELLO ^ world 01001000 01000101 01001100 01001100 01001111 01110111 01101111 01110010 01101100 01100100 =================== XOR ==================== 00111111 00101010 00111110 00100000 00101011 = ?*> + Currently, that's done with this expression for strings: >>> ''.join(chr(ord(a) ^ ord(b)) for a, b in zip('HELLO', 'world')) '?*> +' and this expression for bytestrings: >>> bytes(a ^ b for a, b in zip(b'HELLO', b'world')) b'?*> +' It would be much more convenient, however, to allow a simple xor of a string: >>> 'HELLO' ^ 'world' '?*> +' or bytestring: >>> b'HELLO' ^ b'world' b'?*> +' (All of this applies to other bitwise operators, of course.) Compatibility issues are a no-brainer - currently, bitwise operators for strings raise TypeErrors. Thanks. Suggesting, Ken ? Hilton? ; -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Thu May 17 07:06:59 2018 From: stephanh42 at gmail.com (Stephan Houben) Date: Thu, 17 May 2018 13:06:59 +0200 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: References: Message-ID: Seems you want numpy: >>> import numpy >>> numpy.frombuffer(b"Hello", dtype=numpy.uint8) ^ numpy.frombuffer(b"World", dtype=numpy.uint8) array([31, 10, 30, 0, 11], dtype=uint8) Stephan 2018-05-17 12:53 GMT+02:00 Ken Hilton : > Hi all, > > We all know the bitwise operators: & (and), | (or), ^ (xor), and ~ (not). > We know how they work with numbers: > > 420 ^ 502 > > 110100100 > 111110110 > == XOR == > 001010010 > = 82 > > But it might be useful in some cases to (let's say) xor a string (or > bytestring): > > HELLO ^ world > > 01001000 01000101 01001100 01001100 01001111 > 01110111 01101111 01110010 01101100 01100100 > =================== XOR ==================== > 00111111 00101010 00111110 00100000 00101011 > = ?*> + > > Currently, that's done with this expression for strings: > > >>> ''.join(chr(ord(a) ^ ord(b)) for a, b in zip('HELLO', 'world')) > '?*> +' > > and this expression for bytestrings: > > >>> bytes(a ^ b for a, b in zip(b'HELLO', b'world')) > b'?*> +' > > It would be much more convenient, however, to allow a simple xor of a > string: > > >>> 'HELLO' ^ 'world' > '?*> +' > > or bytestring: > > >>> b'HELLO' ^ b'world' > b'?*> +' > > (All of this applies to other bitwise operators, of course.) > Compatibility issues are a no-brainer - currently, bitwise operators for > strings raise TypeErrors. > > Thanks. > > Suggesting, > Ken > ? Hilton? > ; > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu May 17 07:14:10 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 17 May 2018 14:14:10 +0300 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: References: Message-ID: 17.05.18 13:53, Ken Hilton ????: > We all know the bitwise operators: & (and), | (or), ^ (xor), and ~ > (not). We know how they work with numbers: > > 420 ^ 502 > > 110100100 > 111110110 > == XOR == > 001010010 > = 82 > > But it might be useful in some cases to (let's say) xor a string (or > bytestring): The question is how common a need of these operations? If it is not common enough, they are better be implemented as functions in a third-party library. > Currently, that's done with this expression for strings: > > ? ? >>> ''.join(chr(ord(a) ^ ord(b)) for a, b in zip('HELLO', 'world')) > ? ? '?*> +' Are you aware that this can raise a ValueError for some input strings? For example for '\U00100000' and '\U00010000'. From steve at pearwood.info Thu May 17 08:20:43 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 17 May 2018 22:20:43 +1000 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: References: Message-ID: <20180517122043.GF12683@ando.pearwood.info> On Thu, May 17, 2018 at 02:14:10PM +0300, Serhiy Storchaka wrote: > 17.05.18 13:53, Ken Hilton ????: > >We all know the bitwise operators: & (and), | (or), ^ (xor), and ~ > >(not). We know how they work with numbers: > > > >420 ^ 502 > > > >110100100 > >111110110 > >== XOR == > >001010010 > >= 82 > > > >But it might be useful in some cases to (let's say) xor a string (or > >bytestring): > > The question is how common a need of these operations? If it is not > common enough, they are better be implemented as functions in a > third-party library. The real question is, what does it mean to XOR a text string? What is '?' XOR '?'? I can think of at least three possibilities: '-\t' # treat the strings as UTF-8 '?' # treat them as UTF-16 or UTF-32 '\x14' # treat them as MacRoman What if the strings are unequal lengths? But XORing equal length byte strings makes sense to me. There's no ambiguity: it is equivalent to just XORing each byte with the corresponding byte. > >Currently, that's done with this expression for strings: > > > > ? ? >>> ''.join(chr(ord(a) ^ ord(b)) for a, b in zip('HELLO', 'world')) > > ? ? '?*> +' > > Are you aware that this can raise a ValueError for some input strings? > For example for '\U00100000' and '\U00010000'. That works for me. py> ''.join(chr(ord(a) ^ ord(b)) for a, b in zip('\U00100000', '\U00100000')) '\x00' -- Steve From storchaka at gmail.com Thu May 17 08:49:02 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 17 May 2018 15:49:02 +0300 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: <20180517122043.GF12683@ando.pearwood.info> References: <20180517122043.GF12683@ando.pearwood.info> Message-ID: 17.05.18 15:20, Steven D'Aprano ????: > On Thu, May 17, 2018 at 02:14:10PM +0300, Serhiy Storchaka wrote: >> 17.05.18 13:53, Ken Hilton ????: >>> But it might be useful in some cases to (let's say) xor a string (or >>> bytestring): >> >> The question is how common a need of these operations? If it is not >> common enough, they are better be implemented as functions in a >> third-party library. > > The real question is, what does it mean to XOR a text string? The OP explained this meaning with a sample implementation. >> Are you aware that this can raise a ValueError for some input strings? >> For example for '\U00100000' and '\U00010000'. > > That works for me. > > py> ''.join(chr(ord(a) ^ ord(b)) for a, b in zip('\U00100000', '\U00100000')) > '\x00' Try with '\U00100000' and '\U00010000', not with '\U00100000' and '\U00100000'. From steve at pearwood.info Thu May 17 08:51:49 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 17 May 2018 22:51:49 +1000 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <5AFD1B34.6030203@canterbury.ac.nz> References: <20180516004153.GU12683@ando.pearwood.info> <5AFCB79A.2000703@canterbury.ac.nz> <20180517002153.GA12683@ando.pearwood.info> <5AFD1B34.6030203@canterbury.ac.nz> Message-ID: <20180517125148.GG12683@ando.pearwood.info> On Thu, May 17, 2018 at 06:03:32PM +1200, Greg Ewing wrote: > Steven D'Aprano wrote: > >Let's say you're reading from a CSV file, creating an object from each > >row, and processing it: > > Okay, I can see it could be useful for situations like that. > > But this is still a completely different use case from the > one that started this discussion, which was making it less > painful to add new keywords to the language. The backslash > idea doesn't help at all with that. Doesn't help *at all*? Sure it does. It's Python 3.8, and I learn that in 4.0 "spam" is going to become a keyword. I simply take my code and change all the references spam to \spam, and I've future-proofed the code for 4.0 while still keeping compatibility with 3.8 and 3.9. (But not 3.7 of course. But we can't have everything.) If my code is a library which others will use, the benefit is even bigger. (For a purely internal project, I could just replace spam with spam_ and be done with it.) But for a library, I already have public documentation saying that my API is the function spam(), and I don't want to have to change the public API. As far as my library's users are concerned, nothing has changed. -- Steve From abrault at mapgears.com Thu May 17 08:57:12 2018 From: abrault at mapgears.com (Alexandre Brault) Date: Thu, 17 May 2018 08:57:12 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <5AFD1B34.6030203@canterbury.ac.nz> References: <20180516004153.GU12683@ando.pearwood.info> <5AFCB79A.2000703@canterbury.ac.nz> <20180517002153.GA12683@ando.pearwood.info> <5AFD1B34.6030203@canterbury.ac.nz> Message-ID: On 2018-05-17 2:03 AM, Greg Ewing wrote: > Steven D'Aprano wrote: >> Let's say you're reading from a CSV file, creating an object from >> each row, and processing it: > > Okay, I can see it could be useful for situations like that. > > But this is still a completely different use case from the > one that started this discussion, which was making it less > painful to add new keywords to the language. The backslash > idea doesn't help at all with that. > I don't think there will a solution that makes it less painful to add keywords to the language, nor that finding one should be something we aim for. What this proposal accomplishes is help interoperation with other languages that have different keywords, simplify code generators by letting them blindly escape names and avoid mangling/demangling keywords, and as a distant third, an easy out if the language adds a keyword you use as a name. Alex From steve at pearwood.info Thu May 17 09:13:22 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 17 May 2018 23:13:22 +1000 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: References: <20180517122043.GF12683@ando.pearwood.info> Message-ID: <20180517131321.GH12683@ando.pearwood.info> On Thu, May 17, 2018 at 03:49:02PM +0300, Serhiy Storchaka wrote: > 17.05.18 15:20, Steven D'Aprano ????: > >On Thu, May 17, 2018 at 02:14:10PM +0300, Serhiy Storchaka wrote: > >>17.05.18 13:53, Ken Hilton ????: > >>>But it might be useful in some cases to (let's say) xor a string (or > >>>bytestring): > >> > >>The question is how common a need of these operations? If it is not > >>common enough, they are better be implemented as functions in a > >>third-party library. > > > >The real question is, what does it mean to XOR a text string? > > The OP explained this meaning with a sample implementation. No, he didn't explain the meaning. He gave an example, but not a reason why it should do what he showed. Why should the *abstract character* 'H' XORed with the abstract character 'w' return the abstract character '?'? Why shouldn't the result be '>' instead? (For the record that's using 'EBCDIC-CP-BE'.) The point is, XORing abstract characters seems meaningless to me. If the OP has an explanation for why 'H'^'?' must mean '?', he should explain it. XORing code points could easily generate invalid Unicode sequences containing lone surrogates, say, or undefined characters. Or as you point out, out of range values. But XORing bytes seems perfectly reasonable. Bytes are numbers, even if we display them as ASCII characters. > >>Are you aware that this can raise a ValueError for some input strings? > >>For example for '\U00100000' and '\U00010000'. > > > >That works for me. > > > >py> ''.join(chr(ord(a) ^ ord(b)) for a, b in zip('\U00100000', > >'\U00100000')) > >'\x00' > > Try with '\U00100000' and '\U00010000', not with '\U00100000' and > '\U00100000'. Oops, sorry. I misread your post. -- Steve From ncoghlan at gmail.com Thu May 17 09:20:33 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 17 May 2018 09:20:33 -0400 Subject: [Python-ideas] Inline assignments using "given" clauses In-Reply-To: References: <20180512172427.GF12683@ando.pearwood.info> <5AF9291C.1070301@canterbury.ac.nz> <5AF9540F.3030208@canterbury.ac.nz> Message-ID: On 15 May 2018 at 01:53, Tim Peters wrote: > [Nick] > > The question would then turn to "What if you just want to bind the target > > name, without considering the old value?". And then *that's* where "NAME > : = > > EXPR" would come in: as an augmented assignment operator that used > augmented > > assignment scoping semantics, rather than regular local name binding > > semantics. > > Plain old ":=" would somehow be viewed as being an augmented > assignment operator too? ... OK, the meaning is that augmented > assignment _and_ ":=" would resolve the target's scope in the way the > containing block resolves it. > > > That would mean *directly* overturning PEP 3099's rejection of the idea > of > > using "NAME := EXPR" to imply "nonlocal NAME" at function scope, but > that's > > effectively on the table for implicit functions anyway (and I'd prefer to > > have ":=" be consistent everywhere, rather than having to special case > the > > implicit scopes). > > Creating a local in a containing scope by magic is never done by > Python today. Extending that beyond "need" seems potentially > perilous. For example, it can already be tedious to figure out which > names _are_ local to a function by staring at the function's code, but > people quickly get better at that over time; change the rules so that > they _also_ have to stare at all immediately contained functions too > to figure it out, and it may become significantly harder (OK, I didn't > declare `x`, and a contained function did `x := 3.14` but `x` isn't > declared there either - I guess it's my `x` now). Then again, if > they're doing that much function nesting they deserve whatever they > get ;-) > More likely they'd get a compile time error complaining that the compiler couldn't figure out what they meant, and asking them to be clearer about the intended scoping. Restrict it to that only synthetically generated functions can pull > off this trick by magic (where there are real use cases to motivate > it), and they still don't have to look outside the body of a > function's text to figure it out. Visually, there's no distinction > between the code running in the function's scope and in scopes > synthesized to implement comprehensions appearing in the function's > text. The comprehensions aren't even indented more. > > So, offhand, I'm not sure that the right way to address something you > view as a wart is to vastly expand its reach to 12 operators that > impose it on everyone everywhere every time they're used ;-) > Once I reframed the idea as being like an augmented assignment, your proposed semantics seemed a lot less magical to me, since I was able to define them in terms of "find the assignment or declaration that already exists", rather than implicitly creating a new one. If the compiler can't find a suitable target scope, then it can throw AmbiguousTargetError (which would be an improvement over the runtime UnboundLocalError you typically get today). > Seriously, I do suspect that in > > def f(...): > ... no instances of `s` ... > s += f"START {time.time():.2f}" > > it's overwhelmingly more likely that they simply forgot to do > > s = "" > > earlier in `f` than they actually wanted to append to whatever `s` > means in f's parent block.. That's a radical change to what people > have come to expect `NAME +=` to do. > I think this is the key argument in favour of only allowing the "implicitly nonlocal rebinding" behaviour in lambda expressions, generator expressions, and comprehensions, as that way the search for a target to bind would always terminate at the containing block (just as it does today). BTW, would > > def f(): > x := 3.14 > x = 3.14 > > be a compile-time error? Everyone agreed the analogous case would be > in synthetic functions. Fine by me! > Yeah, I think that would be an AmbiguousTargetError, as when the compiler saw "x := 3.14", it wouldn't have seen "x = 3.14" yet. For other augmented assignments, it would be a DeprecationWarning for the time being, and become an AmbiguousTargetError at a later date. (This also relates to the previous point: if "x := 3.14" can be implicitly nonlocal, then I couldn't answer that question without knowing which names were defined in outer scopes. By contrast, if the implicit access to outer scopes is limited to inline scopes accessing their containing scope, then this example becomes precisely analagous to the current syntax error for binding a name as a local before declaring it as global or nonlocal. The statement of ambiguity would arise from the fact that when we see "TARGET := EXPR" at statement level, we don't know if the missing prior statement is a local variable assignment, a type declaration, or a global or nonlocal declaration) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu May 17 09:25:25 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 17 May 2018 23:25:25 +1000 Subject: [Python-ideas] Runtime assertion with no overhead when not active In-Reply-To: <1526543198.32395.1.camel@fft.be> References: <1525765072.24469.5.camel@fft.be> <0dd9cf67-08cc-6c57-0570-98507290c483@trueblade.com> <1525787426.5522.1.camel@fft.be> <1525793916.5522.5.camel@fft.be> <1526459338.23739.1.camel@fft.be> <20180516111538.GY12683@ando.pearwood.info> <1526477268.24366.1.camel@fft.be> <20180516170931.GZ12683@ando.pearwood.info> <1526543198.32395.1.camel@fft.be> Message-ID: <20180517132525.GI12683@ando.pearwood.info> On Thu, May 17, 2018 at 07:46:41AM +0000, Eloi Gaudry wrote: > On Thu, 2018-05-17 at 03:09 +1000, Steven D'Aprano wrote: > I proposed to have several ways to set it (see previous answers) : one > would be extension based and the other would rely on having a builtin > value that one would be able to change using a set function of the > python core. Sorry, I had forgetten the extension answer: Include/pydebug.h : int Py_RuntimeAssertFlag = 1; Your_extension/main.c :?SetFlag(Py_RuntimeAssertFlag); and I had missed your answer to have a builtin. In any case, I don't think that having only a single flag for the entire Python application is sufficiently useful to bother with this. At the *absolute least* I would expect individual modules to be able to independently enable or disable their own "runtime assertions". > What would then be solution for implementing the runtime_assert with > the current python api ? A lambda ? Earlier in this thread I mentioned that I had a few thoughts on that, and I was exploring those ideas. I asked for anyone interested to contact me off list. Basically I am looking at how we could get delayed evaluation of expressions, which has a number of good use-cases. (So far I have four: this is only one of them.) And yes, wrapping the expression in a function (lambda) seems like the most promising approach. -- Steve From solipsis at pitrou.net Thu May 17 09:49:09 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 17 May 2018 15:49:09 +0200 Subject: [Python-ideas] String and bytes bitwise operations References: Message-ID: <20180517154909.1b1efbab@fsol> Indeed, at this point Numpy sounds like the ideal solution for such use cases. Regards Antoine. On Thu, 17 May 2018 13:06:59 +0200 Stephan Houben wrote: > Seems you want numpy: > > >>> import numpy > >>> numpy.frombuffer(b"Hello", dtype=numpy.uint8) ^ > numpy.frombuffer(b"World", dtype=numpy.uint8) > array([31, 10, 30, 0, 11], dtype=uint8) > > Stephan > > 2018-05-17 12:53 GMT+02:00 Ken Hilton : > > > Hi all, > > > > We all know the bitwise operators: & (and), | (or), ^ (xor), and ~ (not). > > We know how they work with numbers: > > > > 420 ^ 502 > > > > 110100100 > > 111110110 > > == XOR == > > 001010010 > > = 82 > > > > But it might be useful in some cases to (let's say) xor a string (or > > bytestring): > > > > HELLO ^ world > > > > 01001000 01000101 01001100 01001100 01001111 > > 01110111 01101111 01110010 01101100 01100100 > > =================== XOR ==================== > > 00111111 00101010 00111110 00100000 00101011 > > = ?*> + > > > > Currently, that's done with this expression for strings: > > > > >>> ''.join(chr(ord(a) ^ ord(b)) for a, b in zip('HELLO', 'world')) > > '?*> +' > > > > and this expression for bytestrings: > > > > >>> bytes(a ^ b for a, b in zip(b'HELLO', b'world')) > > b'?*> +' > > > > It would be much more convenient, however, to allow a simple xor of a > > string: > > > > >>> 'HELLO' ^ 'world' > > '?*> +' > > > > or bytestring: > > > > >>> b'HELLO' ^ b'world' > > b'?*> +' > > > > (All of this applies to other bitwise operators, of course.) > > Compatibility issues are a no-brainer - currently, bitwise operators for > > strings raise TypeErrors. > > > > Thanks. > > > > Suggesting, > > Ken > > ? Hilton? > > ; > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > From carl.input at gmail.com Thu May 17 09:54:43 2018 From: carl.input at gmail.com (Carl Smith) Date: Thu, 17 May 2018 14:54:43 +0100 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <20180517002153.GA12683@ando.pearwood.info> References: <20180516004153.GU12683@ando.pearwood.info> <5AFCB79A.2000703@canterbury.ac.nz> <20180517002153.GA12683@ando.pearwood.info> Message-ID: > The trouble with explicitly overriding keywords is that it still requires old code to > be changed whenever a new keyword is added... Nope. That's the exact thing my proposal avoids. I'm not sure it also does everything everyone else wants it to do, so it may be a bad idea, but you would not have to change old code, ever. For the purpose of this discussion, let's say that if any code implicitly enables a new feature (by simply using it), all the code in that file is 'new code' in the context of the specific feature (a bit like how `yield` works). If `until` was a new keyword, any file that used it as a keyword would be new code. Any other files are 'old code'. New code could still import an object named `until` from old code. It just could not import it as `until`. So `import until as upto` is fine, but `import until` is a NameError *in new code*. In old code, `until` is still just a name. We would also allow `name.until` and `dance(until="am")` in new code, so that we can still reference names in old code (without them becoming names in new code). If `self.until = something` appeared anywhere in new code, or any local assignment to `until` (including class and def statements) appeared inside a subclass definition within new code, that would need to be checked for a runtime NameError (not very often, but sometimes). In practice, many libraries would alias names that became keywords, so new code could use the new name without restrictions, but old code would carry on working with the old name. TLDR: The syntax and semantics of old code would remain totally unchanged. -- Carl Smith carl.input at gmail.com On 17 May 2018 at 01:21, Steven D'Aprano wrote: > On Thu, May 17, 2018 at 10:58:34AM +1200, Greg Ewing wrote: > > > The trouble with explicitly overriding keywords is that it > > still requires old code to be changed whenever a new keyword > > is added, which as far as I can see almost competely defeats > > the purpose. If e.g. you need to change all uses of given > > to \given in order for your code to keep working in > > Python 3.x for some x, you might just as well change it > > to given_ or some other already-legal name. > > Well, maybe. Certainly using name_ is a possible solution, and it is one > which has worked for over a quarter century. > > We can argue about whether \name or name_ looks nicer, but \name > has one advantage: the key used in the namespace is actually "name". > That's important: see below. > > > > The only remotely legitimate use I can think of is for > > calling APIs that come from a different language, but the > > same thing applies there -- names in the Python binding can > > always be modified somehow to make them legal. > > Of course they can be modified. But having to do so is a pain. > > With the status quo, when dealing with external data which may include > names which are keywords, we have to: > > - add an underscore when we read keywords from external data > - add an underscore when used as obj.kw literals > - add an underscore when used as getattr("kw") literals > - conditionally remove trailing underscore when writing to external APIs > > to: > > - do nothing special when we read keywords from external data > - add a backslash when used as obj.kw literals > - do nothing special when used as getattr("kw") literals > - do nothing special when writing to external APIs > > > I think that overall this pushes it from a mere matter of visual > preference \kw versus kw_ to a significant win for verbatim names. > > Let's say you're reading from a CSV file, creating an object from each > row, and processing it: > > # untested > reader = csv.reader(infile) > header = next(reader) > header = [name + "_" if name in keywords.kwlist() else name for name in > header] > for row in reader: > obj = SimpleNamespace(*zip(header, row)) > process(obj) > > > The consumer of these objects, process(), has to reverse the > transformation: > > def process(obj): > for name, value in vars(obj): > if name.endswith("_") and name[:-1] in keywords.kwlist(): > name = name[:-1] > write_to_external_API(name, value) > > > Verbatim names lets us skip both of these boilerplate steps. > > An interesting case is when you are using the keywords as hard-coded > names for attribute access. In the status quo, we write: > > obj.name_ > obj.getattr("name_") > > In the first line, if you neglect the _ the compiler will complain and > you get a syntax error. In the second line, if you neglect the _ you'll > get no warning, only a runtime failure. > > With verbatim names, we can write: > > obj.\name > obj.getattr("name") # Don't escape it here! > > In this case, the failure modes are similar: > > - if you forget the backslash in the first line, you get a > SyntaxError at compile time, so there's no change here. > > - if you wrongly include the backslash in the second line, > there are two cases: > > * if the next character matches a string escape, say \n > or \t, you'll get no error but a runtime failure; > > (but linters could warn about that) > > * if it doesn't match, say \k, you'll now get a warning > and eventually a failure as we depreciate silently > ignoring backslashes. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu May 17 09:56:47 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 17 May 2018 09:56:47 -0400 Subject: [Python-ideas] Escaped braces in format specifiers In-Reply-To: References: <20180515064136.GA29222@kundert.designers-guide.com> Message-ID: On 15 May 2018 at 16:23, Eric V. Smith wrote: > I'm busy at the sprints, so I don't have a lot of time to think about this. > > However, let me just say that recursive format specs are supported, to a > depth of 1. > > >>> width=10 > >>> f'{"test":{width}}' > 'test ' > > So first the string is basically expanded to: > f'{"test":10}' > Then the string is formatted again to produce the final result. > > That is why the braces must match: they're being used for recursive format > specs. There's no mechanism for having braces that aren't inspected by the > f-string machinery. https://www.python.org/dev/peps/pep-0536/ also seems worth noting (I don't actually understand the specifics of that PEP myself, just making sure that Ken's aware of its existence if this is an area of interest) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenlhilton at gmail.com Thu May 17 10:58:08 2018 From: kenlhilton at gmail.com (Ken Hilton) Date: Thu, 17 May 2018 22:58:08 +0800 Subject: [Python-ideas] String and bytes bitwise operations Message-ID: On Thu, 17 May 2018 23:13:22 +1000, Steven D'Aprano wrote: > No, he didn't explain the meaning. He gave an example, but not a reason why it should do what he showed. > > Why should the *abstract character* 'H' XORed with the abstract character 'w' return the abstract character '?'? Why shouldn't the result be '>' instead? My initial thought was that 'H' ^ 'w' -> '?' because when I was experimenting with the idea, ord('H') ^ ord('w') -> ord('?'). However, I do see your point that different encodings give different results, so I'll drop the idea of bitwise operations on strings. > XORing code points could easily generate invalid Unicode sequences containing lone surrogates, say, or undefined characters. Or as you point out, out of range values. Invalid Unicode sequences, lone surrogates, and undefined characters, IMO, are simply consequences of misusing the operators. I hadn't anticipated the ValueError for '\U00100000' and '\U00010000', though, which is another reason for me to drop bitwise operations on strings. > But XORing bytes seems perfectly reasonable. Bytes are numbers, even if we display them as ASCII characters. My thought exactly. On Thu, 17 May 2018 22:20:43 +1000, Steven D'Aprano wrote: > What if the strings are unequal lengths? (out-of-order quote lol) Then the operators would raise a ValueError. (Assuming bytestrings, since again, I'm dropping text strings.) ?Sharing ideas? , Ken ? Hilton? ; -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu May 17 11:11:45 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 17 May 2018 17:11:45 +0200 Subject: [Python-ideas] String and bytes bitwise operations References: <20180517122043.GF12683@ando.pearwood.info> <20180517131321.GH12683@ando.pearwood.info> Message-ID: <20180517171145.7f4ee5fa@fsol> I agree with Steven. XORing unicode strings doesn't make sense, and is pointless anyway. The only interesting question is whether we want to add bytewise operations to the stdlib. Regards Antoine. On Thu, 17 May 2018 23:13:22 +1000 Steven D'Aprano wrote: > On Thu, May 17, 2018 at 03:49:02PM +0300, Serhiy Storchaka wrote: > > 17.05.18 15:20, Steven D'Aprano ????: > > >On Thu, May 17, 2018 at 02:14:10PM +0300, Serhiy Storchaka wrote: > > >>17.05.18 13:53, Ken Hilton ????: > > >>>But it might be useful in some cases to (let's say) xor a string (or > > >>>bytestring): > > >> > > >>The question is how common a need of these operations? If it is not > > >>common enough, they are better be implemented as functions in a > > >>third-party library. > > > > > >The real question is, what does it mean to XOR a text string? > > > > The OP explained this meaning with a sample implementation. > > No, he didn't explain the meaning. He gave an example, but not a reason > why it should do what he showed. > > Why should the *abstract character* 'H' XORed with the abstract > character 'w' return the abstract character '?'? Why shouldn't the > result be '>' instead? > > (For the record that's using 'EBCDIC-CP-BE'.) > > The point is, XORing abstract characters seems meaningless to me. If the > OP has an explanation for why 'H'^'?' must mean '?', he should explain > it. > > XORing code points could easily generate invalid Unicode sequences > containing lone surrogates, say, or undefined characters. Or as you > point out, out of range values. > > > But XORing bytes seems perfectly reasonable. Bytes are numbers, even if > we display them as ASCII characters. > > > > > >>Are you aware that this can raise a ValueError for some input strings? > > >>For example for '\U00100000' and '\U00010000'. > > > > > >That works for me. > > > > > >py> ''.join(chr(ord(a) ^ ord(b)) for a, b in zip('\U00100000', > > >'\U00100000')) > > >'\x00' > > > > Try with '\U00100000' and '\U00010000', not with '\U00100000' and > > '\U00100000'. > > Oops, sorry. I misread your post. > > From tjreedy at udel.edu Thu May 17 11:57:26 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 17 May 2018 11:57:26 -0400 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: References: Message-ID: On 5/17/2018 6:53 AM, Ken Hilton wrote: > Hi all, > > We all know the bitwise operators: & (and), | (or), ^ (xor), and ~ > (not). We know how they work with numbers: > > 420 ^ 502 > > 110100100 > 111110110 > == XOR == > 001010010 > = 82 > > But it might be useful in some cases to (let's say) xor a string (or > bytestring): > > HELLO ^ world > > 01001000 01000101 01001100 01001100 01001111 > 01110111 01101111 01110010 01101100 01100100 > =================== XOR ==================== > 00111111 00101010 00111110 00100000 00101011 > = ?*> + > > Currently, that's done with this expression for strings: > > ? ? >>> ''.join(chr(ord(a) ^ ord(b)) for a, b in zip('HELLO', 'world')) > ? ? '?*> +' > > and this expression for bytestrings: > > ? ? >>> bytes(a ^ b for a, b in zip(b'HELLO', b'world')) > ? ? b'?*> +' > > It would be much more convenient, however, to allow a simple xor of a > string: > > ? ? >>> 'HELLO' ^ 'world' > ? ? '?*> +' > > or bytestring: > > ? ? >>> b'HELLO' ^ b'world' > ? ? b'?*> +' > > (All of this applies to other bitwise operators, of course.) > Compatibility issues are a no-brainer - currently, bitwise operators for > strings raise TypeErrors. https://bugs.python.org/issue19251 bitwise ops for bytes of equal length -- Terry Jan Reedy From eric at trueblade.com Thu May 17 12:08:25 2018 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 17 May 2018 12:08:25 -0400 Subject: [Python-ideas] Escaped braces in format specifiers In-Reply-To: References: <20180515064136.GA29222@kundert.designers-guide.com> Message-ID: <94C60405-2D11-4764-87F9-062A524A5861@trueblade.com> > On May 17, 2018, at 9:56 AM, Nick Coghlan wrote: > >> On 15 May 2018 at 16:23, Eric V. Smith wrote: >> I'm busy at the sprints, so I don't have a lot of time to think about this. >> >> However, let me just say that recursive format specs are supported, to a depth of 1. >> >> >>> width=10 >> >>> f'{"test":{width}}' >> 'test ' >> >> So first the string is basically expanded to: >> f'{"test":10}' >> Then the string is formatted again to produce the final result. >> >> That is why the braces must match: they're being used for recursive format specs. There's no mechanism for having braces that aren't inspected by the f-string machinery. > > https://www.python.org/dev/peps/pep-0536/ also seems worth noting (I don't actually understand the specifics of that PEP myself, just making sure that Ken's aware of its existence if this is an area of interest) Yes, that?s a good point. Thanks for remembering this PEP. I disagree with much of the PEP and it?s language, such as ?These limitations serve no purpose ...?. And I especially object to the fact that it doesn?t take in to account the effect of its proposed changes on other python implementations or on tools like editors, linters, static type checkers, etc., all of which would have to be modified. Not that those are insurmountable issues or mean that we should never improve f-strings, but they do need to be carefully considered and given weight in the decision process. Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu May 17 12:49:09 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 17 May 2018 09:49:09 -0700 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: On Tue, May 15, 2018 at 11:21 AM, Rob Speer wrote: > > I'm sure that the issue of "what do you call the leap second itself" is > not the problem that Chris Barker is referring to. The problem with leap > seconds is that they create unpredictable differences between UTC and real > elapsed time. > > You can represent a timedelta of exactly 10^8 seconds, but if you add it > to the current time, what should you get? What UTC time will it be in 10^8 > real-time seconds? You don't know, and neither does anybody else, because > you don't know how many leap seconds will occur in that time. > indeed -- even if you only care about the past, where you *could* know the leap seconds -- they are, by their very nature, of second precision -- which means right before leap second occurs, your "time" could be off by up to a second (or a half second?) It's kind of like using a carpenter's tape measure to to locate points from a electron microscope scan :-) The other issue with leap-seconds is that python's datetime doesn't support them :-) And neither do most date-time libraries. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu May 17 13:10:10 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 17 May 2018 10:10:10 -0700 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> Message-ID: On Wed, May 16, 2018 at 2:09 PM, Carl Smith wrote: > If your position is that Guido shouldn't introduce keywords that are > currently used as names at all, > Exactly -- which is why I'm wondering my no one (that I've seen -- long thread) is presenting the backwards option: Any new keywords introduced will be non-legal as regular names. \new_key_word for instance. Makes me think that it may have been good to have ALL keywords somehow non-legal as user-defined names -- maybe ugly syntax, but it would make a clear distinction. how ugly would this be? \for i in range(n): \while \True: ... pretty ugly :-( But maybe not so much if only a handful of new ones.... Or is there another currently illegal character that could be used that would be less ugly? I'm actually confused as to what the point is to the \ prefix idea for names: * It would still require people to change their code when a new keyword was introduced * It would be no easier / harder than adding a conventional legal character -- trailing underscore, or ??? * but now the changed code would no longer run on older versions of python. I guess it comes down to why you'd want to call out: "this is a name that is almost like a keyword" Seems like a meh, meh, lose proposal to me. OK, I see one advantage -- one could have code that already has BOTH word and word_ names in it. So when word becomes a keyword, a tool that automatically added an underscore would break the code. whereas if it automatically added an currently illegal character, it wouldn't shadow anything. But a sufficiently smart tool could get around that, too. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu May 17 13:14:06 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 17 May 2018 13:14:06 -0400 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: On Thu, May 17, 2018 at 12:56 PM Chris Barker via Python-ideas < python-ideas at python.org> wrote: > The other issue with leap-seconds is that python's datetime doesn't support them :-) That's not entirely true. Since the implementation of PEP 495, it is possible to represent the 23:59:60 as 23:59:59 with the "fold" bit set. Of course, the repeated 23:59:59 will be displayed and behave exactly the same as the first 23:59:59, but a 3rd party library can be written to take the "fold" bit into account in temporal operations. From chris.barker at noaa.gov Thu May 17 13:30:03 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 17 May 2018 10:30:03 -0700 Subject: [Python-ideas] __dir__ in which folder is this py file In-Reply-To: <20180507161732.GX9562@ando.pearwood.info> References: <20180507104436.GU9562@ando.pearwood.info> <20180507161732.GX9562@ando.pearwood.info> Message-ID: On Mon, May 7, 2018 at 9:17 AM, Steven D'Aprano wrote: > I'm arguing that for some people, your preferred syntax *is* more > distracting and hard to comprehend than the more self-descriptive > version with named functions. then use Path.joinpath() if you want. > From that perspective, using / to mean something kinda-sorta like string > concatenation, only path separator aware, is precisely the sort of thing > that makes some people dislike operator overloading. > The time for this argument was when the pathlib API was designed -- and I"m sure there was plenty of argument -- but using "/" to join paths was jsut too nifty to ignore :-) But we are doing everyone a disservice if we essentially say: This very useful standard library API was poorly designed, so let's stick with the old, ugly painful way... (OK, I'm being a bit hyperbolic there ...) TOOWTDI is a really good principle -- we never should have added pathlib if we weren't going to try to make it as useful and standard as possible. felt to me awfully close to > > "pathlib! it's the future!" > > I know that's not what you said, or even meant, but I felt it was > important to remind people that not everyone knows pathlib or finds its > API clearer than the explicitly named functions of os.path. > no -- but it IS clearer an easier once we get all the common functionality in there, as opposed to having to poke around in os.path, os, and shutil for what you need. so I'll say, it even if Nathaniel didn't: pathlib! it's the future! :-) - CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu May 17 13:32:40 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 17 May 2018 10:32:40 -0700 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: On Thu, May 17, 2018 at 10:14 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > The other issue with leap-seconds is that python's datetime doesn't > support them :-) > > That's not entirely true. Since the implementation of PEP 495, it is > possible to represent the 23:59:60 as 23:59:59 with the "fold" bit set. Of > course, the repeated 23:59:59 will be displayed and behave exactly the same > as the first 23:59:59, but a 3rd party library can be written to take the > "fold" bit into account in temporal operations. > Does that support the other way -- or do we never lose a leap second anyway? (showing ignorance here) But still, now datetime *could* support leap seconds (which is nice, because before, 23:59:60 was illegal, so it couldn't even be done at all), but that doesn't mean that it DOES support leap seconds.... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Thu May 17 13:53:36 2018 From: stephanh42 at gmail.com (Stephan Houben) Date: Thu, 17 May 2018 19:53:36 +0200 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> Message-ID: Fortunately we have Unicode bold characters nowadays ?? if ?? in: ?????? return Look ma! No syntactic ambiguity! Stephan 2018-05-17 19:10 GMT+02:00 Chris Barker via Python-ideas < python-ideas at python.org>: > On Wed, May 16, 2018 at 2:09 PM, Carl Smith wrote: > >> If your position is that Guido shouldn't introduce keywords that are >> currently used as names at all, >> > > Exactly -- which is why I'm wondering my no one (that I've seen -- long > thread) is presenting the backwards option: > > Any new keywords introduced will be non-legal as regular names. > > \new_key_word > > for instance. > > Makes me think that it may have been good to have ALL keywords somehow > non-legal as user-defined names -- maybe ugly syntax, but it would make a > clear distinction. > > how ugly would this be? > > \for i in range(n): > \while \True: > ... > > pretty ugly :-( > > But maybe not so much if only a handful of new ones.... > > Or is there another currently illegal character that could be used that > would be less ugly? > > I'm actually confused as to what the point is to the \ prefix idea for > names: > > * It would still require people to change their code when a new keyword > was introduced > > * It would be no easier / harder than adding a conventional legal > character -- trailing underscore, or ??? > > * but now the changed code would no longer run on older versions of python. > > I guess it comes down to why you'd want to call out: > > "this is a name that is almost like a keyword" > > Seems like a meh, meh, lose proposal to me. > > OK, I see one advantage -- one could have code that already has BOTH word > and word_ names in it. So when word becomes a keyword, a tool that > automatically added an underscore would break the code. whereas if it > automatically added an currently illegal character, it wouldn't shadow > anything. > > But a sufficiently smart tool could get around that, too. > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Thu May 17 14:01:46 2018 From: stephanh42 at gmail.com (Stephan Houben) Date: Thu, 17 May 2018 20:01:46 +0200 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> Message-ID: OK, that was a silly joke, but did you realize that the following already WORKS TODAY: >>> class Foo: ... pass >>> f = Foo() >>> f.__dict__["if"] = 42 >>> f.?? 42 That's right, I can access a member whose name is a keyword by simply using Unicode bold. Because Python normalizes Unicode fonts but this is apparently AFTER keywords have been recognized. Stephan 2018-05-17 19:53 GMT+02:00 Stephan Houben : > Fortunately we have Unicode bold characters nowadays > > ?? if ?? in: > ?????? return > > Look ma! No syntactic ambiguity! > > Stephan > > 2018-05-17 19:10 GMT+02:00 Chris Barker via Python-ideas < > python-ideas at python.org>: > >> On Wed, May 16, 2018 at 2:09 PM, Carl Smith wrote: >> >>> If your position is that Guido shouldn't introduce keywords that are >>> currently used as names at all, >>> >> >> Exactly -- which is why I'm wondering my no one (that I've seen -- long >> thread) is presenting the backwards option: >> >> Any new keywords introduced will be non-legal as regular names. >> >> \new_key_word >> >> for instance. >> >> Makes me think that it may have been good to have ALL keywords somehow >> non-legal as user-defined names -- maybe ugly syntax, but it would make a >> clear distinction. >> >> how ugly would this be? >> >> \for i in range(n): >> \while \True: >> ... >> >> pretty ugly :-( >> >> But maybe not so much if only a handful of new ones.... >> >> Or is there another currently illegal character that could be used that >> would be less ugly? >> >> I'm actually confused as to what the point is to the \ prefix idea for >> names: >> >> * It would still require people to change their code when a new keyword >> was introduced >> >> * It would be no easier / harder than adding a conventional legal >> character -- trailing underscore, or ??? >> >> * but now the changed code would no longer run on older versions of >> python. >> >> I guess it comes down to why you'd want to call out: >> >> "this is a name that is almost like a keyword" >> >> Seems like a meh, meh, lose proposal to me. >> >> OK, I see one advantage -- one could have code that already has BOTH word >> and word_ names in it. So when word becomes a keyword, a tool that >> automatically added an underscore would break the code. whereas if it >> automatically added an currently illegal character, it wouldn't shadow >> anything. >> >> But a sufficiently smart tool could get around that, too. >> >> -CHB >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Thu May 17 14:41:33 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 17 May 2018 11:41:33 -0700 (PDT) Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> Message-ID: My preference is to do nothing. If you end up making "where" a keyword in Python 3.8, numpy will probably: * rename their where function to "where_" in 3.8 * add a where_ alias in Python < 3.8. And then people will have to fix their code in 3.8 anyway. Only instead of learning a new verbatim syntax, they will just add the familiar underscore. One thing that should ideally be done is to improve the SyntaxError processing to special case use of keywords in places that identifiers are used. Instead of: In [1]: for if in range(10): File "", line 1 for if in range(10): ^ SyntaxError: invalid syntax How about In [1]: for if in range(10): File "", line 1 for if in range(10): ^ SyntaxError: "if" was used where a variable belongs, but "if" is a keyword. Consider using "if_" instead. Similarly, In [2]: int.if File "", line 1 int.if ^ SyntaxError: "if" was used where an attribute name belongs. Did you mean "if_"? SyntaxError doesn't need to quickly generate its error strings. So there is only an upside to having clear error messages. On Thursday, May 17, 2018 at 1:54:42 PM UTC-4, Stephan Houben wrote: > > Fortunately we have Unicode bold characters nowadays > > ?? if ?? in: > ?????? return > > Look ma! No syntactic ambiguity! > > That's hilarious :) > Stephan > > 2018-05-17 19:10 GMT+02:00 Chris Barker via Python-ideas < > python... at python.org >: > >> On Wed, May 16, 2018 at 2:09 PM, Carl Smith > > wrote: >> >>> If your position is that Guido shouldn't introduce keywords that are >>> currently used as names at all, >>> >> >> Exactly -- which is why I'm wondering my no one (that I've seen -- long >> thread) is presenting the backwards option: >> >> Any new keywords introduced will be non-legal as regular names. >> >> \new_key_word >> >> for instance. >> >> Makes me think that it may have been good to have ALL keywords somehow >> non-legal as user-defined names -- maybe ugly syntax, but it would make a >> clear distinction. >> >> how ugly would this be? >> >> \for i in range(n): >> \while \True: >> ... >> >> pretty ugly :-( >> >> But maybe not so much if only a handful of new ones.... >> >> Or is there another currently illegal character that could be used that >> would be less ugly? >> >> I'm actually confused as to what the point is to the \ prefix idea for >> names: >> >> * It would still require people to change their code when a new keyword >> was introduced >> >> * It would be no easier / harder than adding a conventional legal >> character -- trailing underscore, or ??? >> >> * but now the changed code would no longer run on older versions of >> python. >> >> I guess it comes down to why you'd want to call out: >> >> "this is a name that is almost like a keyword" >> >> Seems like a meh, meh, lose proposal to me. >> >> OK, I see one advantage -- one could have code that already has BOTH word >> and word_ names in it. So when word becomes a keyword, a tool that >> automatically added an underscore would break the code. whereas if it >> automatically added an currently illegal character, it wouldn't shadow >> anything. >> >> But a sufficiently smart tool could get around that, too. >> >> -CHB >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.... at noaa.gov >> >> _______________________________________________ >> Python-ideas mailing list >> Python... at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu May 17 14:51:59 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 17 May 2018 14:51:59 -0400 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: On Thu, May 17, 2018 at 1:33 PM Chris Barker wrote: > > On Thu, May 17, 2018 at 10:14 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: >> [...] Since the implementation of PEP 495, it is >> possible to represent the 23:59:60 as 23:59:59 with the "fold" bit set. Of >> course, the repeated 23:59:59 will be displayed and behave exactly the same >> as the first 23:59:59, but a 3rd party library can be written to take the >> "fold" bit into account in temporal operations. > > > Does that support the other way -- or do we never lose a leap second anyway? (showing ignorance here) > I am not sure I understand your question. All I said was that since PEP 495, it became possible to write a pair of functions to convert between TAI and UTC timestamps without any loss of information. For example, around the insertion of the last leap second at the end of 2016, we had the following sequence of seconds: TAI | UTC ---------------------+-------------------- 2016-12-31T23:59:35 | 2016-12-31T23:59:59 2016-12-31T23:59:36 | 2016-12-31T23:59:60 2016-12-31T23:59:37 | 2016-01-01T00:00:00 this correspondence can be implemented in Python using the following datetime objects: TAI | UTC -------------------------------+------------------------------------------- datetime(2016,12,31,23,59,35) | datetime(2016,12,31,23,59,59) datetime(2016,12,31,23,59,36) | datetime(2016,12,31,23,59,59,fold=1) datetime(2016,12,31,23,59,37) | datetime(2016,1,1,0,0,0) Of course, Python will treat datetime(2016,12,31,23,59,59) and datetime( 2016,12,31,23,59,59,fold=1)as equal, but you should be able to use your utc_to_tai(t) function to translate to TAI, do the arithmetic there and translate back with the tai_to_utc(t) function. Wherever tai_to_utc(t) returns a datetime instance with fold=1, you should add that to the seconds field before displaying. > But still, now datetime *could* support leap seconds (which is nice, because before, 23:59:60 was illegal, so it couldn't even be done at all), but that doesn't mean that it DOES support leap seconds.... By the same logic the standard library datetime does not support any local time because it does not include the timezone database. This is where the 3rd party developers should fill the gap. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu May 17 15:00:17 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 17 May 2018 15:00:17 -0400 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: On Thu, May 17, 2018 at 2:51 PM Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > TAI | UTC > ---------------------+-------------------- > 2016-12-31T23:59:35 | 2016-12-31T23:59:59 > 2016-12-31T23:59:36 | 2016-12-31T23:59:60 > 2016-12-31T23:59:37 | 2017-01-01T00:00:00 > > this correspondence can be implemented in Python using the following > datetime objects: > > TAI | UTC > -------------------------------+------------------------------------------- > datetime(2016,12,31,23,59,35) | datetime(2016,12,31,23,59,59) > datetime(2016,12,31,23,59,36) | datetime(2016,12,31,23,59,59,fold=1) > datetime(2016,12,31,23,59,37) | datetime(2017,1,1,0,0,0) > > > Correction: 2016-01-01 in the tables I presented before should be read as 2017-01-01 and similarly for the datetime fields. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Thu May 17 15:13:20 2018 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 17 May 2018 14:13:20 -0500 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: [Chris Barker] > Does that support the other way -- or do we never lose a leap second anyway? > (showing ignorance here) Alexander covered the Python part of this, so I'll answer the possible higher-level question: we haven't yet needed a "negative" leap second, and it's considered unlikely (but not impossible) that we ever will. That's because the Earth's rotation is inexorably slowing ,so the mean solar day inexorably lengthens when measured by SI seconds. Other things can cause the Earth's rotation to speed up temporarily (like some major geological events), but they've only been able to overcome factors acting to slow rotation for brief periods, and never yet got near to overcoming them by a full second. From ethan at stoneleaf.us Thu May 17 15:53:03 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 17 May 2018 12:53:03 -0700 Subject: [Python-ideas] OT: slowing rotation [was: High Precision datetime] In-Reply-To: References: Message-ID: <5AFDDD9F.1080502@stoneleaf.us> On 05/17/2018 12:13 PM, Tim Peters wrote: > Other things can cause the Earth's rotation to speed up temporarily > (like some major geological events), but they've only been able to > overcome factors acting to slow rotation for brief periods, and never > yet got near to overcoming them by a full second. How long before the earth stops rotating? When it does, will we be tide-locked with the sun, or will an earth day become an earth year? Inquiring-minds-want-to-know'ly yrs; -- ~Ethan~ From rosuav at gmail.com Thu May 17 15:54:28 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 18 May 2018 05:54:28 +1000 Subject: [Python-ideas] OT: slowing rotation [was: High Precision datetime] In-Reply-To: <5AFDDD9F.1080502@stoneleaf.us> References: <5AFDDD9F.1080502@stoneleaf.us> Message-ID: On Fri, May 18, 2018 at 5:53 AM, Ethan Furman wrote: > On 05/17/2018 12:13 PM, Tim Peters wrote: > >> Other things can cause the Earth's rotation to speed up temporarily >> (like some major geological events), but they've only been able to >> overcome factors acting to slow rotation for brief periods, and never >> yet got near to overcoming them by a full second. > > > How long before the earth stops rotating? When it does, will we be > tide-locked with the sun, or will an earth day become an earth year? > > Inquiring-minds-want-to-know'ly yrs; Won't ever happen. A few thousand years ago, the planet heard the adage "one good turn deserves another", and interpreted it as an infinite loop. ChrisA From mbarkhau at gmail.com Thu May 17 15:55:08 2018 From: mbarkhau at gmail.com (Manuel Barkhau) Date: Thu, 17 May 2018 21:55:08 +0200 Subject: [Python-ideas] Syntactic Sugar: Post-Conditions for Guard Clauses Message-ID: Hello, has there been consideration for implementing the following new syntax: def my_fun(elements): results = [] for elem in elements: ... continue if not is_valid(elem) ... results.append(result) break if len(results) == max_results return results Or similarly: def iter_fun(elements): num_results = 0 for elem in elements: ... continue if not is_valid(elem) ... yield result num_results += 1 return if num_results == max_results When there is an expression involved for the case of a `return` or `raise` statement, I don't think it's such a great style, because the conditional gets hidden off to the right. def myfun(val): return default_result if val not in known_vals ... return result Alternatively, is there a good way I can implement this as a preprocessor for myself? Thanks Manuel Barkhau From chris.barker at noaa.gov Thu May 17 16:18:59 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 17 May 2018 13:18:59 -0700 Subject: [Python-ideas] Syntactic Sugar: Post-Conditions for Guard Clauses In-Reply-To: References: Message-ID: On Thu, May 17, 2018 at 12:55 PM, Manuel Barkhau wrote: > continue if not is_valid(elem) > ... > how is this better than: if not is_valid(elem): continue ? But even if it is, that might be a consideration for a new language, but adding it to python now is pretty darn unlikely. > When there is an expression involved for the case of a `return` > or `raise` statement, I don't think it's such a great style, > because the conditional gets hidden off to the right. > now I"m confused -- if it's not a great sytle, why suggest it? and I was thinking that the one good thing is that the "return" or "break" is a bit more prominent -- which could be a good thing. Alternatively, is there a good way I can implement this as a > preprocessor for myself? > I'm pretty sure not -- python is very dynamic, but not let you redefine the language semantics. Other than as a import hook that re-wrote the code on the fly. but really -- don't do that -- then no one else will understand your code. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu May 17 16:22:09 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 17 May 2018 13:22:09 -0700 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: now we really have gotten OT... But thanks! that was my question! -CHB Alexander covered the Python part of this, so I'll answer the possible > higher-level question: we haven't yet needed a "negative" leap > second, and it's considered unlikely (but not impossible) that we ever > will. That's because the Earth's rotation is inexorably slowing ,so > the mean solar day inexorably lengthens when measured by SI seconds. > > Other things can cause the Earth's rotation to speed up temporarily > (like some major geological events), but they've only been able to > overcome factors acting to slow rotation for brief periods, and never > yet got near to overcoming them by a full second. > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu May 17 17:00:41 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 17 May 2018 17:00:41 -0400 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: On Thu, May 17, 2018 at 3:13 PM Tim Peters wrote: > [Chris Barker] > > Does that support the other way -- or do we never lose a leap second > anyway? > > (showing ignorance here) > > Alexander covered the Python part of this, ... > No, I did not. I did not realize that the question was about skipping a second instead of inserting it. Yes, regardless of whether it is possible given the physics of Earth rotation, negative leap seconds can be supported. They simply become "gaps" in PEP 495 terminology. Check out PEP 495 and read "second" whenever you see "hour". :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Thu May 17 17:37:45 2018 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 17 May 2018 17:37:45 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> Message-ID: [Resending due to Google Groups getting involved and giving me an error] On 5/17/2018 2:41 PM, Neil Girdhar wrote: > My preference is to do nothing.? If you end up making "where" a keyword > in Python 3.8, numpy will probably: > * rename their where function to "where_" in 3.8 > * add a where_ alias in Python < 3.8. > > And then people will have to fix their code in 3.8 anyway.? Only instead > of learning a new verbatim syntax, they will just add the familiar > underscore. I'm not saying this applies to numpy, but one bonus of using \where would be that existing 3.7 pickles would work in 3.8 (or I think so, it's all obviously underspecified at this point). With renaming, pickles would break. Eric From rob.cliffe at btinternet.com Thu May 17 17:38:18 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 17 May 2018 22:38:18 +0100 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <9e152416-9afb-5b6b-e035-3dea6f37c32c@trueblade.com> <439b03f0-050b-0ee2-3466-208328909f1f@trueblade.com> Message-ID: <0e66063c-625a-492f-c0fa-d32e696fe814@btinternet.com> On 16/05/2018 10:12, Stephan Houben wrote: > Hi all, > > One problem already alluded to with the \identifier syntax is that it > only works > if the old Python version is sufficiently recent to understand \. > > What about using parentheses to allow a keyword to be used as an > identifier: > (where)(x, y) > > I believe this is the first proposal that allows future-proofing of new code while preserving complete backward compatibility.? As far as I know,??? ( keyword )??? is never legal syntax. Of course, putting brackets round every occurrence of every identifier that you think might become an identifier in the next century is a bit of a chore.? There is no perfect solution. Best wishes Rob Cliffe From carl.input at gmail.com Thu May 17 17:54:25 2018 From: carl.input at gmail.com (Carl Smith) Date: Thu, 17 May 2018 22:54:25 +0100 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <0e66063c-625a-492f-c0fa-d32e696fe814@btinternet.com> References: <20180516004153.GU12683@ando.pearwood.info> <9e152416-9afb-5b6b-e035-3dea6f37c32c@trueblade.com> <439b03f0-050b-0ee2-3466-208328909f1f@trueblade.com> <0e66063c-625a-492f-c0fa-d32e696fe814@btinternet.com> Message-ID: > I believe this is the first proposal that allows future-proofing of new code while preserving > complete backward compatibility. My proposal removes the need to future proof anything, and only requires subtle changes to the syntax (nothing visually different). It also preserves perfect backwards compatibility. Just saying :) -- Carl Smith carl.input at gmail.com On 17 May 2018 at 22:38, Rob Cliffe via Python-ideas < python-ideas at python.org> wrote: > > > On 16/05/2018 10:12, Stephan Houben wrote: > >> Hi all, >> >> One problem already alluded to with the \identifier syntax is that it >> only works >> if the old Python version is sufficiently recent to understand \. >> >> What about using parentheses to allow a keyword to be used as an >> identifier: >> (where)(x, y) >> >> >> I believe this is the first proposal that allows future-proofing of new > code while preserving complete backward compatibility. As far as I > know, ( keyword ) is never legal syntax. > Of course, putting brackets round every occurrence of every identifier > that you think might become an identifier in the next century is a bit of a > chore. There is no perfect solution. > Best wishes > Rob Cliffe > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Thu May 17 18:07:27 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 17 May 2018 15:07:27 -0700 (PDT) Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> Message-ID: On Thursday, May 17, 2018 at 5:40:05 PM UTC-4, Carl Smith wrote: > > > My preference is to do nothing. If you end up making "where" a keyword > in Python 3.8, numpy will probably: > > * rename their where function to "where_" in 3.8 > > * add a where_ alias in Python < 3.8. > > This assumes that every library that defines something named `where` (and > every library that references it) > are maintained?. If the parser just starts treating `where` as a keyword, > working code will not parse. That > may be acceptable (I don't know what the policy is), but it makes > introducing keywords more costly than > an approach that allows old code to continue working. > Fair enough. If that solution is taken, I want the linters to complain about verbatim uses as a stopgap measure that ultimately should be removed. I don't think interfaces should be naming functions and arguments: "if", "while", etc. I think PEP 08 should discourage its use as well except when it's necessary to continue to use an unmaintained interface. > -- Carl Smith > carl.... at gmail.com > > On 17 May 2018 at 19:41, Neil Girdhar > > wrote: > >> My preference is to do nothing. If you end up making "where" a keyword >> in Python 3.8, numpy will probably: >> * rename their where function to "where_" in 3.8 >> * add a where_ alias in Python < 3.8. >> >> And then people will have to fix their code in 3.8 anyway. Only instead >> of learning a new verbatim syntax, they will just add the familiar >> underscore. >> >> One thing that should ideally be done is to improve the SyntaxError >> processing to special case use of keywords in places that identifiers are >> used. Instead of: >> >> In [1]: for if in range(10): >> File "", line 1 >> for if in range(10): >> ^ >> SyntaxError: invalid syntax >> >> How about >> >> In [1]: for if in range(10): >> File "", line 1 >> for if in range(10): >> ^ >> SyntaxError: "if" was used where a variable belongs, but "if" is a >> keyword. Consider using "if_" instead. >> >> Similarly, >> >> In [2]: int.if >> File "", line 1 >> int.if >> ^ >> SyntaxError: "if" was used where an attribute name belongs. Did you mean >> "if_"? >> >> SyntaxError doesn't need to quickly generate its error strings. So there >> is only an upside to having clear error messages. >> >> On Thursday, May 17, 2018 at 1:54:42 PM UTC-4, Stephan Houben wrote: >>> >>> Fortunately we have Unicode bold characters nowadays >>> >>> ?? if ?? in: >>> ?????? return >>> >>> Look ma! No syntactic ambiguity! >>> >>> That's hilarious :) >> >>> Stephan >>> >>> 2018-05-17 19:10 GMT+02:00 Chris Barker via Python-ideas < >>> python... at python.org>: >>> >>>> On Wed, May 16, 2018 at 2:09 PM, Carl Smith wrote: >>>> >>>>> If your position is that Guido shouldn't introduce keywords that are >>>>> currently used as names at all, >>>>> >>>> >>>> Exactly -- which is why I'm wondering my no one (that I've seen -- long >>>> thread) is presenting the backwards option: >>>> >>>> Any new keywords introduced will be non-legal as regular names. >>>> >>>> \new_key_word >>>> >>>> for instance. >>>> >>>> Makes me think that it may have been good to have ALL keywords somehow >>>> non-legal as user-defined names -- maybe ugly syntax, but it would make a >>>> clear distinction. >>>> >>>> how ugly would this be? >>>> >>>> \for i in range(n): >>>> \while \True: >>>> ... >>>> >>>> pretty ugly :-( >>>> >>>> But maybe not so much if only a handful of new ones.... >>>> >>>> Or is there another currently illegal character that could be used that >>>> would be less ugly? >>>> >>>> I'm actually confused as to what the point is to the \ prefix idea for >>>> names: >>>> >>>> * It would still require people to change their code when a new keyword >>>> was introduced >>>> >>>> * It would be no easier / harder than adding a conventional legal >>>> character -- trailing underscore, or ??? >>>> >>>> * but now the changed code would no longer run on older versions of >>>> python. >>>> >>>> I guess it comes down to why you'd want to call out: >>>> >>>> "this is a name that is almost like a keyword" >>>> >>>> Seems like a meh, meh, lose proposal to me. >>>> >>>> OK, I see one advantage -- one could have code that already has BOTH >>>> word and word_ names in it. So when word becomes a keyword, a tool that >>>> automatically added an underscore would break the code. whereas if it >>>> automatically added an currently illegal character, it wouldn't shadow >>>> anything. >>>> >>>> But a sufficiently smart tool could get around that, too. >>>> >>>> -CHB >>>> >>>> >>>> -- >>>> >>>> Christopher Barker, Ph.D. >>>> Oceanographer >>>> >>>> Emergency Response Division >>>> NOAA/NOS/OR&R (206) 526-6959 voice >>>> 7600 Sand Point Way NE (206) 526-6329 fax >>>> Seattle, WA 98115 (206) 526-6317 main reception >>>> >>>> Chris.... at noaa.gov >>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python... at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>>> >>> >> _______________________________________________ >> Python-ideas mailing list >> Python... at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Thu May 17 18:11:14 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 17 May 2018 15:11:14 -0700 (PDT) Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <9e152416-9afb-5b6b-e035-3dea6f37c32c@trueblade.com> <439b03f0-050b-0ee2-3466-208328909f1f@trueblade.com> <0e66063c-625a-492f-c0fa-d32e696fe814@btinternet.com> Message-ID: On Thursday, May 17, 2018 at 5:55:30 PM UTC-4, Carl Smith wrote: > > > I believe this is the first proposal that allows future-proofing of new > code while preserving > > complete backward compatibility. > > My proposal removes the need to future proof anything, and only requires > subtle changes to the syntax (nothing visually different). It also > preserves > perfect backwards compatibility. Just saying :) > > Maybe I misunderstood, but it seems like your solution places a small burden on new code that uses "given" or "where" or whatever in the form of a special import or statement enabling it. I love that we're instead making it easy to keep old code working while protecting Python's beautiful future with no special imports or statements to use the core language. > > > -- Carl Smith > carl.... at gmail.com > > On 17 May 2018 at 22:38, Rob Cliffe via Python-ideas > wrote: > >> >> >> On 16/05/2018 10:12, Stephan Houben wrote: >> >>> Hi all, >>> >>> One problem already alluded to with the \identifier syntax is that it >>> only works >>> if the old Python version is sufficiently recent to understand \. >>> >>> What about using parentheses to allow a keyword to be used as an >>> identifier: >>> (where)(x, y) >>> >>> >>> I believe this is the first proposal that allows future-proofing of new >> code while preserving complete backward compatibility. As far as I >> know, ( keyword ) is never legal syntax. >> Of course, putting brackets round every occurrence of every identifier >> that you think might become an identifier in the next century is a bit of a >> chore. There is no perfect solution. >> Best wishes >> Rob Cliffe >> >> _______________________________________________ >> Python-ideas mailing list >> Python... at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu May 17 19:12:44 2018 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 17 May 2018 19:12:44 -0400 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: AstroPy solves for leap seconds [1][2] according to the IAU ERFA (SOFA) library [3] and the IERS-B and IERS-A tables [4]. IERS-B tables ship with AstroPy. The latest IERS-A tables ("from 1973 though one year into the future") auto-download on first use [5]. [1] http://docs.astropy.org/en/stable/time/#time-scales-for-time-deltas [2] http://docs.astropy.org/en/stable/time/#writing-a-custom-format [3] "Leap second day utc2tai interpolation" https://github.com/astropy/astropy/issues/5369 [4] https://github.com/astropy/astropy/pull/4436 [5] http://docs.astropy.org/en/stable/utils/iers.html On Thursday, May 17, 2018, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > > On Thu, May 17, 2018 at 3:13 PM Tim Peters wrote: > >> [Chris Barker] >> > Does that support the other way -- or do we never lose a leap second >> anyway? >> > (showing ignorance here) >> >> Alexander covered the Python part of this, ... >> > > No, I did not. I did not realize that the question was about skipping a > second instead of inserting it. Yes, regardless of whether it is possible > given the physics of Earth rotation, negative leap seconds can be > supported. They simply become "gaps" in PEP 495 terminology. Check out > PEP 495 and read "second" whenever you see "hour". :-) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Thu May 17 19:18:26 2018 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 18 May 2018 00:18:26 +0100 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <0e66063c-625a-492f-c0fa-d32e696fe814@btinternet.com> References: <20180516004153.GU12683@ando.pearwood.info> <9e152416-9afb-5b6b-e035-3dea6f37c32c@trueblade.com> <439b03f0-050b-0ee2-3466-208328909f1f@trueblade.com> <0e66063c-625a-492f-c0fa-d32e696fe814@btinternet.com> Message-ID: On 2018-05-17 22:38, Rob Cliffe via Python-ideas wrote: > > > On 16/05/2018 10:12, Stephan Houben wrote: >> Hi all, >> >> One problem already alluded to with the \identifier syntax is that it >> only works >> if the old Python version is sufficiently recent to understand \. >> >> What about using parentheses to allow a keyword to be used as an >> identifier: >> (where)(x, y) >> >> > I believe this is the first proposal that allows future-proofing of new > code while preserving complete backward compatibility.? As far as I > know,??? ( keyword )??? is never legal syntax. Apart from (False), (True) and (None), there's also (yield). > Of course, putting brackets round every occurrence of every identifier > that you think might become an identifier in the next century is a bit > of a chore.? There is no perfect solution. > From wes.turner at gmail.com Thu May 17 19:19:18 2018 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 17 May 2018 19:19:18 -0400 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: https://en.wikipedia.org/wiki/Leap_second : > Insertion of each UTC leap second is usually decided about six months in advance by the International Earth Rotation and Reference Systems Service (IERS), when needed to ensure that the difference between the UTC and UT1 readings will never exceed 0.9 seconds On Thursday, May 17, 2018, Wes Turner wrote: > AstroPy solves for leap seconds [1][2] according to the IAU ERFA (SOFA) > library [3] and the IERS-B and IERS-A tables [4]. IERS-B tables ship with > AstroPy. The latest IERS-A tables ("from 1973 though one year into the > future") auto-download on first use [5]. > > [1] http://docs.astropy.org/en/stable/time/#time-scales-for-time-deltas > [2] http://docs.astropy.org/en/stable/time/#writing-a-custom-format > [3] "Leap second day utc2tai interpolation" > https://github.com/astropy/astropy/issues/5369 > [4] https://github.com/astropy/astropy/pull/4436 > [5] http://docs.astropy.org/en/stable/utils/iers.html > > > On Thursday, May 17, 2018, Alexander Belopolsky < > alexander.belopolsky at gmail.com> wrote: > >> >> >> On Thu, May 17, 2018 at 3:13 PM Tim Peters wrote: >> >>> [Chris Barker] >>> > Does that support the other way -- or do we never lose a leap second >>> anyway? >>> > (showing ignorance here) >>> >>> Alexander covered the Python part of this, ... >>> >> >> No, I did not. I did not realize that the question was about skipping a >> second instead of inserting it. Yes, regardless of whether it is possible >> given the physics of Earth rotation, negative leap seconds can be >> supported. They simply become "gaps" in PEP 495 terminology. Check out >> PEP 495 and read "second" whenever you see "hour". :-) >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu May 17 19:41:19 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 17 May 2018 19:41:19 -0400 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: On Thu, May 17, 2018 at 7:12 PM Wes Turner wrote: > AstroPy solves for leap seconds [1][2] according to the IAU ERFA (SOFA) > library [3] and the IERS-B and IERS-A tables [4]. IERS-B tables ship with > AstroPy. The latest IERS-A tables ("from 1973 though one year into the > future") auto-download on first use [5]. > I've just tried it. Unfortunately, it does not seem to be compatible with PEP 495 datetime yet: >>> t = astropy.time.Time('2016-12-31T23:59:60') >>> t.to_datetime() Traceback (most recent call last): ... ValueError: Time (array(2016, dtype=int32), array(12, dtype=int32), array(31, dtype=int32), array(23, dtype=int32), array(59, dtype=int32), array(60, dtype=int32), array(0, dtype=int32)) is within a leap second but datetime does not support leap seconds Maybe someone can propose a feature for astropy to return datetime(2016,12,31,23,59,59,fold=1) in this case. > > [1] http://docs.astropy.org/en/stable/time/#time-scales-for-time-deltas > [2] http://docs.astropy.org/en/stable/time/#writing-a-custom-format > [3] "Leap second day utc2tai interpolation" > https://github.com/astropy/astropy/issues/5369 > [4] https://github.com/astropy/astropy/pull/4436 > [5] http://docs.astropy.org/en/stable/utils/iers.html > >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu May 17 20:09:35 2018 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 17 May 2018 17:09:35 -0700 Subject: [Python-ideas] High Precision datetime In-Reply-To: References: Message-ID: On Thu, May 17, 2018 at 9:49 AM, Chris Barker via Python-ideas wrote: > On Tue, May 15, 2018 at 11:21 AM, Rob Speer wrote: >> >> >> I'm sure that the issue of "what do you call the leap second itself" is >> not the problem that Chris Barker is referring to. The problem with leap >> seconds is that they create unpredictable differences between UTC and real >> elapsed time. >> >> You can represent a timedelta of exactly 10^8 seconds, but if you add it >> to the current time, what should you get? What UTC time will it be in 10^8 >> real-time seconds? You don't know, and neither does anybody else, because >> you don't know how many leap seconds will occur in that time. > > > indeed -- even if you only care about the past, where you *could* know the > leap seconds -- they are, by their very nature, of second precision -- which > means right before leap second occurs, your "time" could be off by up to a > second (or a half second?) Not really. There are multiple time standards in use. Atomic clocks count the duration of time ? from their point of view, every second is the same (modulo relativistic effects). TAI is the international standard based on using atomic clocks to count seconds since a fixed starting point, at mean sea level on Earth. Another approach is to declare that each day (defined as "the time between the sun passing directly overhead the Greenwich Observatory twice") is 24 * 60 * 60 seconds long. This is what UT1 does. The downside is that since the earth's rotation varies over time, this means that the duration of a UT1 second varies from day to day in ways that are hard to estimate precisely. UTC is defined as a hybrid of these two approaches: it uses the same seconds as TAI, but every once in a while we add or remove a leap second to keep it roughly aligned with UT1. This is the time standard that computers use the vast majority of the time. Importantly, since we only ever add or remove an integer number of seconds, and only at the boundary in between seconds, UTC is defined just as precisely as TAI. So if you're trying to measure time using UT1 then yeah, your computer clock is wrong all the time by up to 0.9 seconds, and we don't even know what UT1 is more precisely than ~milliseconds. Generally it gets slightly more accurate just after a leap second, but it's not very precise either before or after. Which is why no-one does this. But if you're trying to measure time using UTC, then computers with the appropriate setup (e.g. at CERN, or in HFT data centers) routinely have clocks accurate to <1 microsecond, and leap seconds don't affect that at all. The datetime module still isn't appropriate for doing precise calculations over periods long enough to include a leap second though, e.g. Python simply doesn't know how many seconds passed between two arbitrary UTC timestamps, even if they were in the past. -n -- Nathaniel J. Smith -- https://vorpus.org From steve at pearwood.info Thu May 17 21:14:19 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 18 May 2018 11:14:19 +1000 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <9e152416-9afb-5b6b-e035-3dea6f37c32c@trueblade.com> <439b03f0-050b-0ee2-3466-208328909f1f@trueblade.com> <0e66063c-625a-492f-c0fa-d32e696fe814@btinternet.com> Message-ID: <20180518011418.GL12683@ando.pearwood.info> On Thu, May 17, 2018 at 10:54:25PM +0100, Carl Smith wrote: > My proposal removes the need to future proof anything, and only requires > subtle changes to the syntax (nothing visually different). It also preserves > perfect backwards compatibility. Just saying :) I must admit, I don't understand your proposal. Can you summarise it again? -- Steve From steve at pearwood.info Thu May 17 21:34:26 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 18 May 2018 11:34:26 +1000 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> Message-ID: <20180518013426.GM12683@ando.pearwood.info> On Thu, May 17, 2018 at 11:41:33AM -0700, Neil Girdhar wrote: > My preference is to do nothing. If you end up making "where" a keyword in > Python 3.8, numpy will probably: This is only incidently about "where". I'm hoping that the "where" (or "given") proposal is rejected. It is so verbose and redundantly repetitious in the common case that rather than being an improvement over the status quo, it is worse. But that's by-the-by. > * rename their where function to "where_" in 3.8 > * add a where_ alias in Python < 3.8. > > And then people will have to fix their code in 3.8 anyway. Only instead of > learning a new verbatim syntax, they will just add the familiar underscore. You should be thinking forward two or three versions from now, when \name is the familiar syntax and name_ looks like you started to write an identifier using the underscore_words_convention but got distracted halfway through. Remember that (if approved) verbatim names will not be "that new syntax" for long. We don't still talk about "that new fangled list comprehension syntax" or "that new yield keyword". That was the problem with the "old versus new style classes" terminology: at the point that "new-style classes" had been around for six releases, approaching a decade, they weren't new any more. > One thing that should ideally be done is to improve the SyntaxError > processing to special case use of keywords in places that identifiers are > used. This is worth doing regardless of whether or not we get verbatim strings or some other alternative. You ought to raise it on the bug tracker. -- Steve From mistersheik at gmail.com Thu May 17 23:02:23 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 17 May 2018 23:02:23 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <20180518013426.GM12683@ando.pearwood.info> References: <20180516004153.GU12683@ando.pearwood.info> <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> <20180518013426.GM12683@ando.pearwood.info> Message-ID: On Thu, May 17, 2018 at 9:35 PM Steven D'Aprano wrote: > On Thu, May 17, 2018 at 11:41:33AM -0700, Neil Girdhar wrote: > > My preference is to do nothing. If you end up making "where" a keyword > in > > Python 3.8, numpy will probably: > > This is only incidently about "where". I'm hoping that the "where" (or > "given") proposal is rejected. It is so verbose and redundantly > repetitious in the common case that rather than being an improvement > over the status quo, it is worse. But that's by-the-by. > > > > * rename their where function to "where_" in 3.8 > > * add a where_ alias in Python < 3.8. > > > > And then people will have to fix their code in 3.8 anyway. Only instead > of > > learning a new verbatim syntax, they will just add the familiar > underscore. > > You should be thinking forward two or three versions from now, when > \name is the familiar syntax and name_ looks like you started to write > an identifier using the underscore_words_convention but got distracted > halfway through. > > Remember that (if approved) verbatim names will not be "that new syntax" > for long. We don't still talk about "that new fangled list comprehension > syntax" or "that new yield keyword". That was the problem with the "old > versus new style classes" terminology: at the point that "new-style > classes" had been around for six releases, approaching a decade, they > weren't new any more. > > I can get behind the benefit of main benefit of backslash, which is keeping code working with old libraries. However, the difference between the backslash syntax and comprehensions and generator functions is that comprehensions and generator functions make the language more expressive. The backslash is no more expressive than trailing underscore. It's no more succinct, and no more clear. Adding it to the language constitutes more that the user needs to learn, which makes Python slightly less accessible. I don't like multiple ways of doing the same thing. There is already probably billions of lines of code that use trailing underscore to avoid collisions. If the backslash syntax is added, then there will be a natural push towards the "right way" to avoid collisions. If that's backslashes, then projects are typically going to have code written with backslashes and code written with underscore. When you go access a variable named "where", you'll wonder: was it called "\where" or "where_"? Maybe the "\where" is pretty enough that it's worth it like you say. Maybe a function like: def f(\in=0, out=1): is prettier than def f(in_=0, out=1): but I'm already so used the current way of doing things, my aesthetic is that it's not worth the variability. For that reason, I'd like to make a more modest proposal to *only* add a verbatim versions of keywords as necessary, e.g., "\where" or "\given". That way, there will be no temptation to use that syntax in any other place. If a new version of Python comes out with a new keyword, say "abc", then all of the old Python versions can get a minor revision that knows about "\abc". This would ensure that the backslash syntax is only used to avoid collisions with new keywords. When 3.7 hits end-of-life, the "\given" (or whatever) can be deprecated. > > One thing that should ideally be done is to improve the SyntaxError > > processing to special case use of keywords in places that identifiers > are > > used. > > This is worth doing regardless of whether or not we get verbatim strings > or some other alternative. You ought to raise it on the bug tracker. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/r1kFC8mYEKk/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Fri May 18 02:07:41 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 18 May 2018 18:07:41 +1200 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: <20180517131321.GH12683@ando.pearwood.info> References: <20180517122043.GF12683@ando.pearwood.info> <20180517131321.GH12683@ando.pearwood.info> Message-ID: <5AFE6DAD.4000203@canterbury.ac.nz> Steven D'Aprano wrote: > But XORing bytes seems perfectly reasonable. Bytes are numbers, even if > we display them as ASCII characters. Yep. Implement it for bytes, then the user can decode/encode as appropriate for the application. -- Greg From abrault at mapgears.com Fri May 18 02:34:34 2018 From: abrault at mapgears.com (Alexandre Brault) Date: Fri, 18 May 2018 02:34:34 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <584f24a5-3fee-c8bd-131a-f657c72bbaaf@biologie.uni-freiburg.de> <20180518013426.GM12683@ando.pearwood.info> Message-ID: <3a8eac75-377d-7c72-7598-c386130aff72@mapgears.com> On 2018-05-17 11:02 PM, Neil Girdhar wrote: > > For that reason, I'd like to make a more modest proposal to *only* add > a verbatim versions of keywords as necessary, e.g., "\where" or > "\given".? That way, there will be no temptation to use that syntax in > any other place.? If a new version of Python comes out with a new > keyword, say "abc", then all of the old Python versions can get a > minor revision that knows about "\abc".? This would ensure that the > backslash syntax is only used to avoid collisions with new keywords. > > When 3.7 hits end-of-life, the "\given" (or whatever) can be deprecated. -1. This would add an extra maintenance and mental ("which keywords are allowed as verbatim and which not") cost to the feature while limiting its utility to the one use case it's only incidentally addressing. PEP8 can warn people not to use verbatim names frivolously in handwritten code. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Fri May 18 02:45:58 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 18 May 2018 18:45:58 +1200 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180516004153.GU12683@ando.pearwood.info> <5AFCB79A.2000703@canterbury.ac.nz> <20180517002153.GA12683@ando.pearwood.info> Message-ID: <5AFE76A6.5070206@canterbury.ac.nz> Carl Smith wrote: > I wrote: >> The trouble with explicitly overriding keywords is that it still requires old code to >> be changed whenever a new keyword is added... > > For the purpose of this discussion, let's say that if any code implicitly > enables a new feature (by simply using it), all the code in that file is 'new > code' in the context of the specific feature (a bit like how `yield` works). > If `until` was a new keyword, any file that used it as a keyword would be new > code. Any other files are 'old code'. Okay, that works because it *doesn't* require old code to explicitly say "I'm using this word the old way". My comment was about the idea of having to use a backslash to escape keywords used as names, or similar schemes. > We would also allow `name.until` and `dance(until="am")` in new code, so that > we can still reference names in old code (without them becoming names in new > code). Actually it would be fine if new code had to say "name.\until" etc. The only problem I can see is that it would probably be near-impossible to implement using the current parser generator. It might be doable by keeping multiple versions of the grammar -- try to parse using the most recent grammar, if that doesn't work, try the next most recent, etc. But that would be pretty horrible, and it would require keeping old cruft in the implementation forever, which we don't like doing. -- Greg From steve at pearwood.info Fri May 18 02:47:30 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 18 May 2018 16:47:30 +1000 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180518013426.GM12683@ando.pearwood.info> Message-ID: <20180518064729.GN12683@ando.pearwood.info> On Thu, May 17, 2018 at 11:02:23PM -0400, Neil Girdhar wrote: > However, the difference between the backslash syntax and comprehensions and > generator functions is that comprehensions and generator functions make the > language more expressive. The backslash is no more expressive than > trailing underscore. It's no more succinct, and no more clear. Adding it > to the language constitutes more that the user needs to learn, which makes > Python slightly less accessible. On the contrary: it removes a pain point when dealing with external libraries. No longer will we have to *transform* the name on both input and output. Instead, we only need to *escape* the name when written as a literal. > I don't like multiple ways of doing the same thing. Ah, like when people want to use "class" as an identifier, and since they can't, they write: klass cls Class and maybe even occasionally class_ :-) Or they use a synonym: kind, clade, type (with or without trailing underscore). I've seen *every one of those choices* in real code. Except "clade", I just added that one now. Remind me again what the "one (obvious) way to do it" is? > There is already > probably billions of lines of code that use trailing underscore to avoid > collisions. Indeed, and if this proposal is accepted, that will remain legal, and if people want to write class_ instead of \class or klass, or if_ instead of \in or infile, they are permitted to do so. You can even have your own in-house style rules mandating whatever style you prefer. > If the backslash syntax is added, then there will be a natural > push towards the "right way" to avoid collisions. If that's backslashes, > then projects are typically going to have code written with backslashes and > code written with underscore. When you go access a variable named "where", > you'll wonder: was it called "\where" or "where_"? Yes? Why is this more of a problem than what we have now? Is it called (in context of PEP 572) "where" or "given"? In general, is it called: where, place, location, loc, locus, position, pos, x, xy, locality, locale, coordinates, coord or some other synonym? In order to successfully use a library's API, one needs to actually know what that API *is*. That means you need to know the name of things. Adding verbatim names won't change that. > Maybe the "\where" is pretty enough that it's worth it like you say. Maybe > a function like: > > def f(\in=0, out=1): > > is prettier than > > def f(in_=0, out=1): > > but I'm already so used the current way of doing things, my aesthetic is > that it's not worth the variability. Being able to use "in" as an identifier as in that example is not the driving motivation for adding this feature. The driving motivation is to remove a pain point when dealing with external APIs that use keywords as regular identifiers, and to make it simpler to future-proof code when a new keyword is due to be introduced. Nobody is going to recommend that folks rush to deprecate their name_ APIs and replace them with \name. I'm sure most library maintainers will have better things to do. in_ will stay in_ for most existing code. It is only new code that doesn't have to care about 3.7 or older than can even consider this. > For that reason, I'd like to make a more modest proposal to *only* add a > verbatim versions of keywords as necessary, Because "special cases are special enough to break the rules, complicate the documentation and the implementation, and give rise to a thousand Stackoverflow posts asking why we can escape some keywords but not others". > e.g., "\where" or "\given". > That way, there will be no temptation to use that syntax in any other > place. Just because you have no use-case for using "except", say, as an identifier doesn't mean nobody has. You are not arbiter of which keywords are acceptable to use verbatim and which ones are forbidden. > When 3.7 hits end-of-life, the "\given" (or whatever) can be deprecated. Having a white list of "Permitted keywords you may escape" is horrible enough without baking in a policy of continued code churn by removing them from the whitelist every few releases. -- Steve From greg.ewing at canterbury.ac.nz Fri May 18 04:05:47 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 18 May 2018 20:05:47 +1200 Subject: [Python-ideas] OT: slowing rotation [was: High Precision datetime] In-Reply-To: <5AFDDD9F.1080502@stoneleaf.us> References: <5AFDDD9F.1080502@stoneleaf.us> Message-ID: <5AFE895B.1040905@canterbury.ac.nz> Ethan Furman wrote: > How long before the earth stops rotating? Apparently about 1.9 trillion years. > When it does, will we be > tide-locked with the sun, or will an earth day become an earth year? Wikipedia says the main cause of the slowing is tidal effects from the moon, so probably it would become tide-locked with the moon and then not slow any further. Having a month-long day ought to make our current fears about climate change look like a lot of panic over nothing. However, the good news is that we won't have to worry about it. The sun will become a red giant and swallow the earth long before then. -- Greg From stefan_ml at behnel.de Fri May 18 04:27:25 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 18 May 2018 10:27:25 +0200 Subject: [Python-ideas] OT: slowing rotation [was: High Precision datetime] In-Reply-To: <5AFE895B.1040905@canterbury.ac.nz> References: <5AFDDD9F.1080502@stoneleaf.us> <5AFE895B.1040905@canterbury.ac.nz> Message-ID: Greg Ewing schrieb am 18.05.2018 um 10:05: > Ethan Furman wrote: >> How long before the earth stops rotating?? > > Apparently about 1.9 trillion years. So, does that mean we now need to hold our breath for 1.9 british trillion years or 1.9 american trillion years? Assuming you were referring to the French-Latin-Arabic based numbers and naming systems at all, that is... And anyway, what's that point doing there, right between the "1" and the "9" ? Stefan From greg.ewing at canterbury.ac.nz Fri May 18 02:05:05 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 18 May 2018 18:05:05 +1200 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <20180517125148.GG12683@ando.pearwood.info> References: <20180516004153.GU12683@ando.pearwood.info> <5AFCB79A.2000703@canterbury.ac.nz> <20180517002153.GA12683@ando.pearwood.info> <5AFD1B34.6030203@canterbury.ac.nz> <20180517125148.GG12683@ando.pearwood.info> Message-ID: <5AFE6D11.4060907@canterbury.ac.nz> Steven D'Aprano wrote: > It's Python 3.8, and I learn that in 4.0 "spam" is going to become a > keyword. I simply take my code and change all the references spam to > \spam, and I've future-proofed the code for 4.0 while still keeping > compatibility with 3.8 and 3.9. Okay, maybe it helps a little bit, but not very much. There will still be a lot of reluctance to add new keywords, because of the disruption it will cause to existing code. If we've learned nothing else from the Python 3 changeover, it's that many people work in an environment where it's extremely difficult to update working code. -- Greg From rosuav at gmail.com Fri May 18 04:56:09 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 18 May 2018 18:56:09 +1000 Subject: [Python-ideas] OT: slowing rotation [was: High Precision datetime] In-Reply-To: References: <5AFDDD9F.1080502@stoneleaf.us> <5AFE895B.1040905@canterbury.ac.nz> Message-ID: On Fri, May 18, 2018 at 6:27 PM, Stefan Behnel wrote: > Greg Ewing schrieb am 18.05.2018 um 10:05: >> Ethan Furman wrote: >>> How long before the earth stops rotating? >> >> Apparently about 1.9 trillion years. > > So, does that mean we now need to hold our breath for 1.9 british trillion > years or 1.9 american trillion years? I'm not sure. How long will it take for people to agree on a meaning for "trillion"? Oh wait, that's even longer. ChrisA From stephanh42 at gmail.com Fri May 18 05:17:13 2018 From: stephanh42 at gmail.com (Stephan Houben) Date: Fri, 18 May 2018 11:17:13 +0200 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <5AFE6D11.4060907@canterbury.ac.nz> References: <20180516004153.GU12683@ando.pearwood.info> <5AFCB79A.2000703@canterbury.ac.nz> <20180517002153.GA12683@ando.pearwood.info> <5AFD1B34.6030203@canterbury.ac.nz> <20180517125148.GG12683@ando.pearwood.info> <5AFE6D11.4060907@canterbury.ac.nz> Message-ID: 2018-05-18 8:05 GMT+02:00 Greg Ewing : > Steven D'Aprano wrote: > >> It's Python 3.8, and I learn that in 4.0 "spam" is going to become a >> keyword. I simply take my code and change all the references spam to \spam, >> and I've future-proofed the code for 4.0 while still keeping compatibility >> with 3.8 and 3.9. >> > > Okay, maybe it helps a little bit, but not very much. There > will still be a lot of reluctance to add new keywords, because > of the disruption it will cause to existing code And the alternative is to replace all occurrences of spam with ???? , which has the same effect and also is backward-compatible with 3.x for x < 8. So there is already a kind of solution available, albeit an ugly one. Stephan If we've learned nothing else from the Python 3 changeover, > it's that many people work in an environment where it's > extremely difficult to update working code. > > -- > Greg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Fri May 18 05:29:21 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 18 May 2018 21:29:21 +1200 Subject: [Python-ideas] OT: slowing rotation [was: High Precision datetime] In-Reply-To: References: <5AFDDD9F.1080502@stoneleaf.us> <5AFE895B.1040905@canterbury.ac.nz> Message-ID: <5AFE9CF1.6000106@canterbury.ac.nz> Stefan Behnel wrote: > So, does that mean we now need to hold our breath for 1.9 british trillion > years or 1.9 american trillion years? Seeing as the time-to-red-giant is only about 5e9 years, I don't think it matters much either way. -- Greg From greg.ewing at canterbury.ac.nz Fri May 18 05:30:26 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 18 May 2018 21:30:26 +1200 Subject: [Python-ideas] OT: slowing rotation [was: High Precision datetime] In-Reply-To: References: <5AFDDD9F.1080502@stoneleaf.us> <5AFE895B.1040905@canterbury.ac.nz> Message-ID: <5AFE9D32.7010508@canterbury.ac.nz> Chris Angelico wrote: > I'm not sure. How long will it take for people to agree on a meaning > for "trillion"? About a trillion years, I estimate. :-) -- Greg From kenlhilton at gmail.com Fri May 18 07:22:13 2018 From: kenlhilton at gmail.com (Ken Hilton) Date: Fri, 18 May 2018 19:22:13 +0800 Subject: [Python-ideas] Make keywords KEYwords only in places they would have syntactical meaning Message-ID: Hi all, Yes, this is another idea for avoiding breaking existing code when introducing new keywords. I'm not sure if this is too similar to Guido's previous "allow keywords in certain places" idea, but here goes: Only treat keywords as having any special meaning when they are in places with syntactical significance. So, currently, let's say someone set the variable "and_" to some value. The following lines are both SyntaxErrors: True and_ False obj.and = value And the following are both correct: True and False obj.and_ = value My idea is to only treat keywords as having special meaning when they're in the right place. So the following would all be legal: >>> from operator import and >>> var = and(True, False) >>> var False >>> var = True and False >>> var False >>> def except(exc, def): ... try: ... return def() ... except exc as e: ... return e ... >>> except(ZeroDivisionError, lambda: 1/0) ZeroDivisionError('division by zero',) >>> except(ZeroDivisionError, lambda: 0/1) 0.0 >>> import asyncio as await #this is already currently legal, but will not be in the __future__ >>> async def async(def): ... return await await.get_event_loop().run_in_executor(None, def) ... >>> And so on. What are your thoughts? ?Sharing, ?Ken Hilton ; -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhodri at kynesim.co.uk Fri May 18 07:29:12 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Fri, 18 May 2018 12:29:12 +0100 Subject: [Python-ideas] Make keywords KEYwords only in places they would have syntactical meaning In-Reply-To: References: Message-ID: On 18/05/18 12:22, Ken Hilton wrote: > My idea is to only treat keywords as having special meaning when they're in > the right place. So the following would all be legal: > > >>> from operator import and > >>> var = and(True, False) > >>> var > False > >>> var = True and False > >>> var > False > >>> def except(exc, def): > ... try: > ... return def() > ... except exc as e: > ... return e > ... > >>> except(ZeroDivisionError, lambda: 1/0) > ZeroDivisionError('division by zero',) > >>> except(ZeroDivisionError, lambda: 0/1) > 0.0 > >>> import asyncio as await #this is already currently legal, but will > not be in the __future__ > >>> async def async(def): > ... return await await.get_event_loop().run_in_executor(None, def) > ... > >>> > > And so on. > > What are your thoughts? I asked about this earlier, much less clearly, and didn't get a helpful answer. I haven't had the spare time to look at the parser since then to see if it's plausible. Though seriously, your example with "except()" makes me want to recant! -- Rhodri James *-* Kynesim Ltd From steve at pearwood.info Fri May 18 07:35:43 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 18 May 2018 21:35:43 +1000 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <5AFE6D11.4060907@canterbury.ac.nz> References: <5AFCB79A.2000703@canterbury.ac.nz> <20180517002153.GA12683@ando.pearwood.info> <5AFD1B34.6030203@canterbury.ac.nz> <20180517125148.GG12683@ando.pearwood.info> <5AFE6D11.4060907@canterbury.ac.nz> Message-ID: <20180518113543.GP12683@ando.pearwood.info> On Fri, May 18, 2018 at 06:05:05PM +1200, Greg Ewing wrote: > Steven D'Aprano wrote: > >It's Python 3.8, and I learn that in 4.0 "spam" is going to become a > >keyword. I simply take my code and change all the references spam to > >\spam, and I've future-proofed the code for 4.0 while still keeping > >compatibility with 3.8 and 3.9. > > Okay, maybe it helps a little bit, but not very much. There > will still be a lot of reluctance to add new keywords, because > of the disruption it will cause to existing code. That's okay, in fact there *ought* to be reluctance to add new keywords. The aim of the exercise is not to add dozens of new keywords to the language, just to make it easier to deal with the situation when we do. -- Steve From steve at pearwood.info Fri May 18 07:37:59 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 18 May 2018 21:37:59 +1000 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <5AFCB79A.2000703@canterbury.ac.nz> <20180517002153.GA12683@ando.pearwood.info> <5AFD1B34.6030203@canterbury.ac.nz> <20180517125148.GG12683@ando.pearwood.info> <5AFE6D11.4060907@canterbury.ac.nz> Message-ID: <20180518113759.GQ12683@ando.pearwood.info> On Fri, May 18, 2018 at 11:17:13AM +0200, Stephan Houben wrote: > And the alternative is to replace all occurrences of > spam with ???? , which has the same effect and also is > backward-compatible with 3.x for x < 8. > > So there is already a kind of solution available, albeit an ugly one. You are kidding, I hope. If that works at all, I don't think its something we want to guarantee will work. And for what it's worth, what I see is eight empty boxes (missing glyph symbols). -- Steve From steve at pearwood.info Fri May 18 07:59:46 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 18 May 2018 21:59:46 +1000 Subject: [Python-ideas] Make keywords KEYwords only in places they would have syntactical meaning In-Reply-To: References: Message-ID: <20180518115945.GS12683@ando.pearwood.info> On Fri, May 18, 2018 at 07:22:13PM +0800, Ken Hilton wrote: [...] > Only treat keywords as having any special meaning when they are in places > with syntactical significance. [...] > My idea is to only treat keywords as having special meaning when they're in > the right place. Chris discussed his experience with REXX, which works like that: The problem is that you can go a long way down the road of using a particular name, only to find that suddenly you can't use it in some particular context. [...] So the question is: Is it better to be able to use a keyword as an identifier for a while, and then run into trouble later, or would you prefer to be told straight away "no, sorry, pick a different name"? https://mail.python.org/pipermail/python-ideas/2018-May/050694.html I think its a no-brainer: I'd much rather get an error earlier, when I first decide its a good idea to call a method "while", rather than six months later when I stumble across a corner case that stops me from using it. I'm sure that everyone is absolutely champing at the bit to write code like: for = if if while else else or = [in or for for in in for if if] *wink* but I think we should be conservative about allowing keywords as identifiers. Requiring a clear and obvious escaping mechanism (verbatim strings \name) or a clear and obvious renaming convension (name_) is better than a context-sensitive grammar. That's even assuming such a context-sensitive grammar would be possible within the requirement that Python's parser is LL(1). -- Steve From rosuav at gmail.com Fri May 18 08:03:17 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 18 May 2018 22:03:17 +1000 Subject: [Python-ideas] Make keywords KEYwords only in places they would have syntactical meaning In-Reply-To: <20180518115945.GS12683@ando.pearwood.info> References: <20180518115945.GS12683@ando.pearwood.info> Message-ID: On Fri, May 18, 2018 at 9:59 PM, Steven D'Aprano wrote: > That's even assuming such a context-sensitive grammar would be possible > within the requirement that Python's parser is LL(1). I'm sure it could be done, but it may impact existing Python grammar. So backward compatibility might mean making certain words "keywords" in contexts where they otherwise wouldn't be, and then just throwing SyntaxError because it would have been ambiguous. But even if it's possible, I don't think it's desirable. ChrisA From mistersheik at gmail.com Fri May 18 08:31:36 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 18 May 2018 08:31:36 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <20180518064729.GN12683@ando.pearwood.info> References: <20180518013426.GM12683@ando.pearwood.info> <20180518064729.GN12683@ando.pearwood.info> Message-ID: On Fri, May 18, 2018 at 2:48 AM Steven D'Aprano wrote: > On Thu, May 17, 2018 at 11:02:23PM -0400, Neil Girdhar wrote: > > > However, the difference between the backslash syntax and comprehensions > and > > generator functions is that comprehensions and generator functions make > the > > language more expressive. The backslash is no more expressive than > > trailing underscore. It's no more succinct, and no more clear. Adding > it > > to the language constitutes more that the user needs to learn, which > makes > > Python slightly less accessible. > > On the contrary: it removes a pain point when dealing with external > libraries. No longer will we have to *transform* the name on both input > and output. Instead, we only need to *escape* the name when written as a > literal. > > > > I don't like multiple ways of doing the same thing. > > Ah, like when people want to use "class" as an identifier, and since > they can't, they write: > > klass cls Class > > and maybe even occasionally class_ :-) > > Or they use a synonym: > > kind, clade, type (with or without trailing underscore). > > I've seen *every one of those choices* in real code. Except "clade", I > just added that one now. > > Remind me again what the "one (obvious) way to do it" is? > In most cases: cls https://www.python.org/dev/peps/pep-0008/#function-and-method-arguments > > > > There is already > > probably billions of lines of code that use trailing underscore to avoid > > collisions. > > Indeed, and if this proposal is accepted, that will remain legal, and if > people want to write class_ instead of \class or klass, or if_ instead > of \in or infile, they are permitted to do so. > You can even have your own in-house style rules mandating whatever style > you prefer. > > > > If the backslash syntax is added, then there will be a natural > > push towards the "right way" to avoid collisions. If that's > backslashes, > > then projects are typically going to have code written with backslashes > and > > code written with underscore. When you go access a variable named > "where", > > you'll wonder: was it called "\where" or "where_"? > > Yes? Why is this more of a problem than what we have now? Is it called > (in context of PEP 572) "where" or "given"? In general, is it called: > > where, place, location, loc, locus, position, pos, x, xy, > locality, locale, coordinates, coord > > or some other synonym? > > In order to successfully use a library's API, one needs to actually > know what that API *is*. That means you need to know the name of things. > Adding verbatim names won't change that. > > > > Maybe the "\where" is pretty enough that it's worth it like you say. > Maybe > > a function like: > > > > def f(\in=0, out=1): > > > > is prettier than > > > > def f(in_=0, out=1): > > > > but I'm already so used the current way of doing things, my aesthetic is > > that it's not worth the variability. > > Being able to use "in" as an identifier as in that example is not the > driving motivation for adding this feature. The driving motivation is to > remove a pain point when dealing with external APIs that use keywords as > regular identifiers, and to make it simpler to future-proof code when a > new keyword is due to be introduced. > > Nobody is going to recommend that folks rush to deprecate their name_ > APIs and replace them with \name. I'm sure most library maintainers > will have better things to do. in_ will stay in_ for most existing code. > It is only new code that doesn't have to care about 3.7 or older than > can even consider this. > > > > For that reason, I'd like to make a more modest proposal to *only* add a > > verbatim versions of keywords as necessary, > > Because "special cases are special enough to break the rules, complicate > the documentation and the implementation, and give rise to a thousand > Stackoverflow posts asking why we can escape some keywords but not > others". > > > > > e.g., "\where" or "\given". > > That way, there will be no temptation to use that syntax in any other > > place. > > Just because you have no use-case for using "except", say, as an > identifier doesn't mean nobody has. You are not arbiter of which > keywords are acceptable to use verbatim and which ones are forbidden. > All of your arguments would have applied to a keyword escaping proposal had it been proposed before "given" was even considered. The only reason we're even considered considering escaping is to keep code that uses "given" as an identifier working. That's why I prefer the most modest solution of only being able to escape given. After all, there wasn't any big need to escape other keywords last year. > > > When 3.7 hits end-of-life, the "\given" (or whatever) can be deprecated. > > Having a white list of "Permitted keywords you may escape" is horrible > enough without baking in a policy of continued code churn by removing > them from the whitelist every few releases. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/r1kFC8mYEKk/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From clint.hepner at gmail.com Fri May 18 08:50:47 2018 From: clint.hepner at gmail.com (Clint Hepner) Date: Fri, 18 May 2018 08:50:47 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <20180518113759.GQ12683@ando.pearwood.info> References: <5AFCB79A.2000703@canterbury.ac.nz> <20180517002153.GA12683@ando.pearwood.info> <5AFD1B34.6030203@canterbury.ac.nz> <20180517125148.GG12683@ando.pearwood.info> <5AFE6D11.4060907@canterbury.ac.nz> <20180518113759.GQ12683@ando.pearwood.info> Message-ID: <7CD08AD0-3753-4CAD-A8D1-5C7FCED1097A@gmail.com> > On 2018 May 18 , at 7:37 a, Steven D'Aprano wrote: > > On Fri, May 18, 2018 at 11:17:13AM +0200, Stephan Houben wrote: > >> And the alternative is to replace all occurrences of >> spam with ???? , which has the same effect and also is >> backward-compatible with 3.x for x < 8. >> >> So there is already a kind of solution available, albeit an ugly one. > > You are kidding, I hope. > > If that works at all, I don't think its something we want to guarantee > will work. And for what it's worth, what I see is eight empty boxes > (missing glyph symbols). It could be worse. At least ???? ("\U0001d42c\U0001d429\U0001d41a\U0001d426") is visually distinct with the right font support. You could (ab?)use the Unicode Tags block (https://en.wikipedia.org/wiki/Tags_(Unicode_block)) and use something like 'spam\U000e0069\U000e0064' (spam). (Luckily, that's not even valid Python, as the tag characters aren't valid for identifiers.) -- Clint From stephanh42 at gmail.com Fri May 18 09:09:57 2018 From: stephanh42 at gmail.com (Stephan Houben) Date: Fri, 18 May 2018 15:09:57 +0200 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <20180518113759.GQ12683@ando.pearwood.info> References: <5AFCB79A.2000703@canterbury.ac.nz> <20180517002153.GA12683@ando.pearwood.info> <5AFD1B34.6030203@canterbury.ac.nz> <20180517125148.GG12683@ando.pearwood.info> <5AFE6D11.4060907@canterbury.ac.nz> <20180518113759.GQ12683@ando.pearwood.info> Message-ID: 2018-05-18 13:37 GMT+02:00 Steven D'Aprano : > On Fri, May 18, 2018 at 11:17:13AM +0200, Stephan Houben wrote: > > > And the alternative is to replace all occurrences of > > spam with ???? , which has the same effect and also is > > backward-compatible with 3.x for x < 8. > > > > So there is already a kind of solution available, albeit an ugly one. > > You are kidding, I hope. > I am not kidding; I am merely defending the status quo. I demonstrate how the intended behavior can be achieved using features available in current Python versions. The approach has at least the following two technical advantages. 1. It requires no change to Python 2. It provides backwards compatibility all the way back to 3.0. The spelling is arguably ugly, but this should be weighted against the, IMHO, extremely rare use of this feature. > > If that works at all, I don't think its something we want to guarantee > will work. It is guaranteed to work by PEP-3131: https://www.python.org/dev/peps/pep-3131 "All identifiers are converted into the normal form NFKC while parsing; comparison of identifiers is based on NFKC." NFKC normalization means spam must be considered the same identifier as ???? . Note that the choice for NFKC normalization was apparently explicitly discussed and decided upon at the time. Since the difference between NFC and NFKC is exactly that identifiers like spam and ???? are different under the former and identical under the latter, I take it this is all quite intentional. > And for what it's worth, what I see is eight empty boxes > (missing glyph symbols). > > I am afraid that mostly shows that your mailer has a bug in handling non-BMP unicode characters; you should be seeing FOUR missing glyph symbols. Stephan > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri May 18 09:37:51 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 18 May 2018 23:37:51 +1000 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <5AFCB79A.2000703@canterbury.ac.nz> <20180517002153.GA12683@ando.pearwood.info> <5AFD1B34.6030203@canterbury.ac.nz> <20180517125148.GG12683@ando.pearwood.info> <5AFE6D11.4060907@canterbury.ac.nz> <20180518113759.GQ12683@ando.pearwood.info> Message-ID: <20180518133751.GT12683@ando.pearwood.info> On Fri, May 18, 2018 at 03:09:57PM +0200, Stephan Houben wrote: > 2018-05-18 13:37 GMT+02:00 Steven D'Aprano : > > > On Fri, May 18, 2018 at 11:17:13AM +0200, Stephan Houben wrote: > > > > > And the alternative is to replace all occurrences of > > > spam with ???? , which has the same effect and also is > > > backward-compatible with 3.x for x < 8. > > > > > > So there is already a kind of solution available, albeit an ugly one. > > > > You are kidding, I hope. > > > > > I am not kidding; Earlier you described this suggestion as "a silly joke". https://mail.python.org/pipermail/python-ideas/2018-May/050861.html I think you were right then. > I am merely defending the status quo. > I demonstrate how the intended behavior can be achieved using features > available in current Python versions. Aside from the fact that font, editor and keyboard support for such non-BMP Unicode characters is very spotty, it isn't the intended behaviour. As you point out, the intended behaviour is that obj.?? and obj.if ought to be identical. Since the later is a syntax error, so should be the former. > It is guaranteed to work by PEP-3131: > https://www.python.org/dev/peps/pep-3131 > > "All identifiers are converted into the normal form NFKC while parsing; > comparison of identifiers is based on NFKC." > > NFKC normalization means spam must be considered the same identifier as > ???? . It's not the NFKC normalization that I'm questioning. Its the fact that it is done too late to catch the use of a keyword. -- Steve From stephanh42 at gmail.com Fri May 18 10:31:49 2018 From: stephanh42 at gmail.com (Stephan Houben) Date: Fri, 18 May 2018 16:31:49 +0200 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <20180518133751.GT12683@ando.pearwood.info> References: <5AFCB79A.2000703@canterbury.ac.nz> <20180517002153.GA12683@ando.pearwood.info> <5AFD1B34.6030203@canterbury.ac.nz> <20180517125148.GG12683@ando.pearwood.info> <5AFE6D11.4060907@canterbury.ac.nz> <20180518113759.GQ12683@ando.pearwood.info> <20180518133751.GT12683@ando.pearwood.info> Message-ID: 2018-05-18 15:37 GMT+02:00 Steven D'Aprano : > > Earlier you described this suggestion as "a silly joke". > > https://mail.python.org/pipermail/python-ideas/2018-May/050861.html The joke proposal was to write all keywords in Python using bold font variation, as a reaction to a similar joke proposal to precede all keywords in Python with \. In contrast this isn't even a proposal, it is merely a description of an existing feature. Practically speaking, suppose "spam" becomes a keyword in 3.8, and I have a module which I want to make compatible with 3.8 AND I want to preserve the API for pre-3.8 versions, then I will first update my module to use some alternative spelling spam_ throughout, and then, in a single place, write: ???? = spam_ # exploit NFKC normalization to set identifier "spam" for backward compatibility Even if this single line shows up as mojibake in somebody's editor, it shouldn't inconvenience them too much. > I think you were right then. > > > > I am merely defending the status quo. > > I demonstrate how the intended behavior can be achieved using features > > available in current Python versions. > > Aside from the fact that font, editor and keyboard support for such > non-BMP Unicode characters is very spotty, it isn't the intended > behaviour. > I am not sure from what you conclude that. There seem to be three design possibilities here: 1. ?? is an alternative spelling for the keyword if 2. ?? is an identifier 3. ?? is an error I am pretty sure option 1 (non-ASCII spelling of keywords) was not intended (doc says about keywords: "They must be spelled exactly as written here:") So it is either 2 or 3. Option 3 would only make sense if we conclude that it is a bad idea to have an identifier with the same name as a keyword. Whereas this whole thread so far has been about introducing such a feature. So that leaves 2, which happens to be the implemented behavior. As an aside: A general observation of PEP-3131 and Unicode identifiers in Python: from the PEP it becomes clear that there have been several proposals of making it more restricted (e.g. requiring source code to be already in NFKC normal form, which would make ?? illegal, disallowing confusables, etc.) Ultimately this has been rejected and the result is that we have a rather liberal definition of Unicode identifiers in Python. I feel that ?? being a valid identifier fits into that pattern, just as various confusable spellings of if would be legal identifiers. In theory this could lead to all kinds of sneaky attacks where code appears to do one thing but does another, but it just doesn't seem so big an issue in practice. > As you point out, the intended behaviour is that obj.?? and > obj.if ought to be identical. Since the later is a syntax error, so > should be the former. > NFKC normalization is restricted to identifiers. Keywords "must be spelled exactly as written here." > > > > It is guaranteed to work by PEP-3131: > > https://www.python.org/dev/peps/pep-3131 > > > > "All identifiers are converted into the normal form NFKC while parsing; > > comparison of identifiers is based on NFKC." > > > > NFKC normalization means spam must be considered the same identifier as > > ???? . > > > It's not the NFKC normalization that I'm questioning. Its the fact that > it is done too late to catch the use of a keyword. > > See above. Stephan > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri May 18 10:48:28 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 19 May 2018 00:48:28 +1000 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180518013426.GM12683@ando.pearwood.info> <20180518064729.GN12683@ando.pearwood.info> Message-ID: <20180518144827.GU12683@ando.pearwood.info> On Fri, May 18, 2018 at 08:31:36AM -0400, Neil Girdhar wrote: [...] > > > I don't like multiple ways of doing the same thing. > > > > Ah, like when people want to use "class" as an identifier, and since > > they can't, they write: > > > > klass cls Class [...] > > Remind me again what the "one (obvious) way to do it" is? > > > > In most cases: cls > > https://www.python.org/dev/peps/pep-0008/#function-and-method-arguments The immediate next sentence goes on to say: If a function argument's name clashes with a reserved keyword, it is generally better to append a single trailing underscore rather than use an abbreviation or spelling corruption. Thus class_ is better than clss. (Perhaps better is to avoid such clashes by using a synonym.) So PEP 8 already torpedoes your preference for a single way to spell words that clash with keywords: - use a synonym; - or a trailing underscore - except for the first argument of class methods, where we use the misspelling (or abbreviation) "cls". If you object to this proposed verbatim names on the basis of disliking multiple ways to spell identifies that clash with keywords, that ship sailed long ago. There has always been multiple ways. But I like to think that verbatim names could become the one OBVIOUS way to do it in the future. [...] > All of your arguments would have applied to a keyword escaping proposal had > it been proposed before "given" was even considered. Every new idea has to be thought of for the first time. Otherwise it would have been in the language from day zero and it wouldn't be a new idea. If it wasn't "given", it could have been for "async" and "await", if not for them it could have been for "with", if not for "with" it might have been "yield", and so on. There had to be a first time for any idea. I would have suggested this twenty years ago if I thought of it twenty years ago, but I didn't, so I didn't. Dismissing the idea because I didn't suggest it earlier when other keywords were suggested is illogical. > The only reason we're > even considered considering escaping is to keep code that uses "given" as > an identifier working. That might be the only reason YOU are considing it, but it definitely isn't the only reason for me. And probably not for others. In fact, since I strongly dislike the "given" syntax, and really want that idea to be rejected, anything that increases its chances are a negative to me. Nevertheless, identifiers clashing with keywords isn't something brand new that only occurs thanks to "given". It has been a pain point forever. A small one, true, but still a nuisance. Verbatim names has the possibility to cut that nuisance value even further. But not if we short-sightedly limit it to a single case. > That's why I prefer the most modest solution of > only being able to escape given. Limiting a general method of mitigating the keyword clashing problem to a single keyword is as sensible as limiting the pathlib library to only work with a single hard-coded pathname. -- Steve From Richard at Damon-Family.org Fri May 18 11:20:02 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Fri, 18 May 2018 11:20:02 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <5AFCB79A.2000703@canterbury.ac.nz> <20180517002153.GA12683@ando.pearwood.info> <5AFD1B34.6030203@canterbury.ac.nz> <20180517125148.GG12683@ando.pearwood.info> <5AFE6D11.4060907@canterbury.ac.nz> <20180518113759.GQ12683@ando.pearwood.info> <20180518133751.GT12683@ando.pearwood.info> Message-ID: <5d5e4d70-513e-26a3-26b2-6d5178d2c505@Damon-Family.org> On 5/18/18 10:31 AM, Stephan Houben wrote: > > NFKC normalization is restricted to identifiers. > Keywords "must be spelled exactly as written here." > ? > > > > > It is guaranteed to work by PEP-3131: > > https://www.python.org/dev/peps/pep-3131 > > > > > "All identifiers are converted into the normal form NFKC while > parsing; > > comparison of identifiers is based on NFKC." > > > > NFKC normalization means spam must be considered the same > identifier as > > ???? . > > > It's not the NFKC normalization that I'm questioning. Its the fact > that > it is done too late to catch the use of a keyword. > > > See above. I would think that the rule that normalization is restricted to identifiers says that it needs to happen AFTER keyword identification (otherwise it would have applied to the keyword).? To follow the rules and flag identifiers that normalize to keywords, either you need to normalize early and tag text that had been changed by normalization so keywords could flag errors (but late enough that you don't normalize inside strings and such), or you need to normalize late (as is done) but then add a check to see if the text became the same as a keyword. Seems a shame to go to extra work to flag as an error something that really could only have been done intentionally, removing a 'feature' to help with backwards compatibility. -- Richard Damon From chris.barker at noaa.gov Fri May 18 12:32:10 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 18 May 2018 09:32:10 -0700 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: <5AFE6DAD.4000203@canterbury.ac.nz> References: <20180517122043.GF12683@ando.pearwood.info> <20180517131321.GH12683@ando.pearwood.info> <5AFE6DAD.4000203@canterbury.ac.nz> Message-ID: On Thu, May 17, 2018 at 11:07 PM, Greg Ewing wrote: > Steven D'Aprano wrote: > >> But XORing bytes seems perfectly reasonable. Bytes are numbers, even if >> we display them as ASCII characters. > > actually, bytes are, well, bytes ;-) -- that is, 8 bits. But the point is that "bitwise" operations make all the sense in the world for bytes, but not for unicode text -- did anyone have a use case for bitwise operation on unicode strings? I can't imagine one, even if you could agree on a definition... Yep. Implement it for bytes, What exactly would be implemented? bytes is a sequence type, so would a bitwise operator perform the operation "elementwise"? (i.e. like numpy) or would it operate on the whole sequence as a single collection of bits? Would it be any different? Without thinking hard, it seems some operations, like AND and OR and XOR would be the same, but bit shifting would be different. And then what do you do if the two bytes objects are not the same length? If "elementwise", then we should think carefully about that -- no where else does Python do things elementwise in the standard library -- and we already have numpy if you want to do it now. and a bytes object can be representing any type of data -- so one byte at a time might make sense, but maybe two bytes at a time makes more sense -- and if the data is representing, say an integer, then endian-ness matters... All this is handled by numpy by having multiple data types that can be "mapped" to a buffer. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From facundobatista at gmail.com Fri May 18 12:49:58 2018 From: facundobatista at gmail.com (Facundo Batista) Date: Fri, 18 May 2018 13:49:58 -0300 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: References: <20180517122043.GF12683@ando.pearwood.info> <20180517131321.GH12683@ando.pearwood.info> <5AFE6DAD.4000203@canterbury.ac.nz> Message-ID: 2018-05-18 13:32 GMT-03:00 Chris Barker via Python-ideas : > or would it operate on the whole sequence as a single collection of bits? Once you think as the whole sequence of bytes as a sequence of bits (eight times longer, of course), all questions are easly answered, see below... > Would it be any different? Without thinking hard, it seems some operations, > like AND and OR and XOR would be the same, but bit shifting would be > different. AND, XOR and OR are simple. Bit shifting will shift bits for the whole bit sequence. > And then what do you do if the two bytes objects are not the same length? Pad with 0s to the left. Exactly the same that happen in >>> 1 ^ 1231312312 1231312313 Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From Richard at Damon-Family.org Fri May 18 12:54:20 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Fri, 18 May 2018 12:54:20 -0400 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: References: <20180517122043.GF12683@ando.pearwood.info> <20180517131321.GH12683@ando.pearwood.info> <5AFE6DAD.4000203@canterbury.ac.nz> Message-ID: <1b3baa5e-16d3-08fb-0bc6-ef5689533a54@Damon-Family.org> On 5/18/18 12:32 PM, Chris Barker via Python-ideas wrote: > actually, bytes are, well, bytes ;-) -- that is, 8 bits. But the point > is that "bitwise" operations make all the sense in the world for > bytes, but not for unicode text -- did anyone have a use case for > bitwise operation on unicode strings? I can't imagine one, even if you > could agree on a definition... > The one case I could think of is a very crude form of encryption. Probably most makes sense if you assume the string is really just ASCII. Not sure if that is a good enough case to be worth adding it, as often you special case some characters (like control characters) to avoid messing things up too much for display. -- Richard Damon From python at mrabarnett.plus.com Fri May 18 12:56:33 2018 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 18 May 2018 17:56:33 +0100 Subject: [Python-ideas] Make keywords KEYwords only in places they would have syntactical meaning In-Reply-To: References: Message-ID: <200b7139-4025-4d9e-d3d7-64475eab59ab@mrabarnett.plus.com> On 2018-05-18 12:22, Ken Hilton wrote: > Hi all, > > Yes, this is another idea for avoiding breaking existing code when > introducing new keywords. I'm not sure if this is too similar to Guido's > previous "allow keywords in certain places" idea, but here goes: > > Only treat keywords as having any special meaning when they are in > places with syntactical significance. > So, currently, let's say someone set the variable "and_" to some value. > The following lines are both SyntaxErrors: > > ? ? True and_ False > ? ? obj.and = value > > And the following are both correct: > > ? ? True and False > ? ? obj.and_ = value > > My idea is to only treat keywords as having special meaning when they're > in the right place. So the following would all be legal: > > ? ? >>> from operator import and > ? ? >>> var = and(True, False) > ? ? >>> var > ? ? False > ? ? >>> var = True and False > ? ? >>> var > ? ? False > ? ? >>> def except(exc, def): > ? ? ...? ? ?try: > ? ? ...? ? ? ? ?return def() > ? ? ...? ? ?except exc as e: > ? ? ...? ? ? ? ?return e > ? ? ... > ? ? >>> except(ZeroDivisionError, lambda: 1/0) > ? ? ZeroDivisionError('division by zero',) > ? ? >>> except(ZeroDivisionError, lambda: 0/1) > ? ? 0.0 > ? ? >>> import asyncio as await #this is already currently legal, but > will not be in the __future__ > ? ? >>> async def async(def): > ? ? ...? ? ?return await await.get_event_loop().run_in_executor(None, def) > ? ? ... > ? ? >>> > > And so on. > > What are your thoughts? > This would be legal, but what would happen? def yield(x): print('YIELD') def foo(): yield('FOO') f = foo() From rosuav at gmail.com Fri May 18 14:15:52 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 19 May 2018 04:15:52 +1000 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: References: <20180517122043.GF12683@ando.pearwood.info> <20180517131321.GH12683@ando.pearwood.info> <5AFE6DAD.4000203@canterbury.ac.nz> Message-ID: On Sat, May 19, 2018 at 2:32 AM, Chris Barker via Python-ideas wrote: > On Thu, May 17, 2018 at 11:07 PM, Greg Ewing > wrote: >> >> Steven D'Aprano wrote: >>> >>> But XORing bytes seems perfectly reasonable. Bytes are numbers, even if >>> we display them as ASCII characters. > > > actually, bytes are, well, bytes ;-) -- that is, 8 bits. Grammatically, you appear to be disagreeing with the assertion that bytes are numbers. Is that the case? If you want to be extremely technical, an "octet" is a group of eight bits (or eight musicians, but I haven't yet figured out how to send musicians down an ethernet cable), and a "byte" isn't as rigidly defined. But on modern PCs, you can fairly safely assume that they're synonymous. I suppose you could argue that a "byte" is a patch of storage capable of holding a number from 0 to 255, as opposed to being the number itself, but that's getting rather existential :) In Python, a "bytes" object represents a sequence of eight-bit units. When you subscript a bytes [1], you get back an integer with the value at that position. So if a collection of them is called a "bytes" and one of them is an integer in range(0, 256), doesn't it stand to reason that a byte is a number? Maybe I'm completely misunderstanding your statement here. ChrisA [1] Yes, I'm aware that older versions of Python behaved differently. [2] [2] I'm also aware that putting square brackets after the word "bytes" could be interpreted as subscripting the 'bytes' type itself, rather than creating a reference to a footnote. Ain't contextual grammar fun? From rosuav at gmail.com Fri May 18 14:34:24 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 19 May 2018 04:34:24 +1000 Subject: [Python-ideas] Make keywords KEYwords only in places they would have syntactical meaning In-Reply-To: <200b7139-4025-4d9e-d3d7-64475eab59ab@mrabarnett.plus.com> References: <200b7139-4025-4d9e-d3d7-64475eab59ab@mrabarnett.plus.com> Message-ID: On Sat, May 19, 2018 at 2:56 AM, MRAB wrote: > On 2018-05-18 12:22, Ken Hilton wrote: >> >> Hi all, >> >> Yes, this is another idea for avoiding breaking existing code when >> introducing new keywords. I'm not sure if this is too similar to Guido's >> previous "allow keywords in certain places" idea, but here goes: >> >> Only treat keywords as having any special meaning when they are in places >> with syntactical significance. >> So, currently, let's say someone set the variable "and_" to some value. >> The following lines are both SyntaxErrors: >> >> True and_ False >> obj.and = value >> >> And the following are both correct: >> >> True and False >> obj.and_ = value >> >> My idea is to only treat keywords as having special meaning when they're >> in the right place. So the following would all be legal: >> >> >>> from operator import and >> >>> var = and(True, False) >> >>> var >> False >> >>> var = True and False >> >>> var >> False >> >>> def except(exc, def): >> ... try: >> ... return def() >> ... except exc as e: >> ... return e >> ... >> >>> except(ZeroDivisionError, lambda: 1/0) >> ZeroDivisionError('division by zero',) >> >>> except(ZeroDivisionError, lambda: 0/1) >> 0.0 >> >>> import asyncio as await #this is already currently legal, but >> will not be in the __future__ >> >>> async def async(def): >> ... return await await.get_event_loop().run_in_executor(None, >> def) >> ... >> >>> >> >> And so on. >> >> What are your thoughts? >> > This would be legal, but what would happen? > > def yield(x): > print('YIELD') > > def foo(): > yield('FOO') > > f = foo() "yield" would have to be a keyword in any context where an expression is valid. Which, in turn, makes it utterly useless as a function name, or any other identifier. ChrisA From mistersheik at gmail.com Fri May 18 14:54:13 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 18 May 2018 14:54:13 -0400 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: <20180518144827.GU12683@ando.pearwood.info> References: <20180518013426.GM12683@ando.pearwood.info> <20180518064729.GN12683@ando.pearwood.info> <20180518144827.GU12683@ando.pearwood.info> Message-ID: On Fri, May 18, 2018 at 10:49 AM Steven D'Aprano wrote: > On Fri, May 18, 2018 at 08:31:36AM -0400, Neil Girdhar wrote: > [...] > > > > I don't like multiple ways of doing the same thing. > > > > > > Ah, like when people want to use "class" as an identifier, and since > > > they can't, they write: > > > > > > klass cls Class > [...] > > > Remind me again what the "one (obvious) way to do it" is? > > > > > > > In most cases: cls > > > > https://www.python.org/dev/peps/pep-0008/#function-and-method-arguments > > The immediate next sentence goes on to say: > > If a function argument's name clashes with a reserved keyword, > it is generally better to append a single trailing underscore > rather than use an abbreviation or spelling corruption. Thus > class_ is better than clss. (Perhaps better is to avoid such > clashes by using a synonym.) > > So PEP 8 already torpedoes your preference for a single way to spell > words that clash with keywords: > > - use a synonym; > > - or a trailing underscore > > - except for the first argument of class methods, where we > use the misspelling (or abbreviation) "cls". > > If you object to this proposed verbatim names on the basis of disliking > multiple ways to spell identifies that clash with keywords, that ship > sailed long ago. There has always been multiple ways. > Right. I think it's pretty clear that there is one way to avoid naming conflict and keep the same name: use a trailing underscore except when it's the first argument to a class method. > > But I like to think that verbatim names could become the one OBVIOUS way > to do it in the future. > That's what I don't want. > > > [...] > > All of your arguments would have applied to a keyword escaping proposal > had > > it been proposed before "given" was even considered. > > Every new idea has to be thought of for the first time. Otherwise it > would have been in the language from day zero and it wouldn't be a new > idea. If it wasn't "given", it could have been for "async" and "await", > if not for them it could have been for "with", if not for "with" it > might have been "yield", and so on. > > There had to be a first time for any idea. I would have > suggested this twenty years ago if I thought of it twenty years ago, but > I didn't, so I didn't. Dismissing the idea because I didn't suggest it > earlier when other keywords were suggested is illogical. > > > The only reason we're > > even considered considering escaping is to keep code that uses "given" as > > an identifier working. > > That might be the only reason YOU are considing it, but it definitely > isn't the only reason for me. And probably not for others. > > In fact, since I strongly dislike the "given" syntax, and really want > that idea to be rejected, anything that increases its chances are a > negative to me. > > Nevertheless, identifiers clashing with keywords isn't something brand > new that only occurs thanks to "given". It has been a pain point > forever. A small one, true, but still a nuisance. > > Verbatim names has the possibility to cut that nuisance value even > further. But not if we short-sightedly limit it to a single case. > > > That's why I prefer the most modest solution of > > only being able to escape given. > > Limiting a general method of mitigating the keyword clashing problem to > a single keyword is as sensible as limiting the pathlib library to only > work with a single hard-coded pathname. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/r1kFC8mYEKk/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From noahs2003 at gmail.com Fri May 18 16:53:38 2018 From: noahs2003 at gmail.com (Noah Simon) Date: Fri, 18 May 2018 16:53:38 -0400 Subject: [Python-ideas] Async Lambda syntax In-Reply-To: <24B64AF1-A3B6-41C4-8BAE-B11102F55E3A@gmail.com> References: <24B64AF1-A3B6-41C4-8BAE-B11102F55E3A@gmail.com> Message-ID: <7ED94226-9104-4DFC-84E1-2CB3FAB3A422@gmail.com> Hello all, I was developing a script using an asyncio-based API, when I came across the need to define an asynchronous lambda. I found this syntax does not currently exist. Obviously I could have (and did) just write a regular coroutine, but for simple one-line functions and such, I think an asynchronous lambda syntax would be useful. I do not have the experience to write a PEP or implement this, so I was wondering what you all think of the idea. What I was thinking: foo = async lambda a, b: b + await bar(a I posted an issue on the issue tracker suggesting this, and was directed to this mailing list. Thanks, Noah -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri May 18 17:44:15 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 18 May 2018 17:44:15 -0400 Subject: [Python-ideas] Async Lambda syntax In-Reply-To: <7ED94226-9104-4DFC-84E1-2CB3FAB3A422@gmail.com> References: <24B64AF1-A3B6-41C4-8BAE-B11102F55E3A@gmail.com> <7ED94226-9104-4DFC-84E1-2CB3FAB3A422@gmail.com> Message-ID: On 5/18/2018 4:53 PM, Noah Simon wrote: > Hello all, > I was developing a script using an asyncio-based API, when I came across > the need to define an asynchronous lambda. Not really. > I found this syntax does not > currently exist. Obviously I could have (and did) just write a regular > coroutine, but for simple one-line functions and such, I think an > asynchronous lambda syntax would be useful. I do not have the experience > to write a PEP or implement this, so I was wondering what you all think > of the idea. > > What I was thinking: > > foo = async lambda a, b: b + await bar(a PEP8 intentionally and properly discourages 'name = lambda ...' as inferior to 'def name(...'. For the above, async def foo(a, b): return b + await bar(a) > I posted an issue ?on the issue > tracker suggesting this, and was directed to this mailing list. -- Terry Jan Reedy From chris.barker at noaa.gov Fri May 18 18:25:19 2018 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 18 May 2018 18:25:19 -0400 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: References: <20180517122043.GF12683@ando.pearwood.info> <20180517131321.GH12683@ando.pearwood.info> <5AFE6DAD.4000203@canterbury.ac.nz> Message-ID: >> actually, bytes are, well, bytes ;-) -- that is, 8 bits. > > Grammatically, you appear to be disagreeing with the assertion that > bytes are numbers. Is that the case? Um, yes. Though I think for the rest of the conversation, it?s a distinction that doesn?t matter. > If you want to be extremely technical, an "octet" is a group of eight > bits (or eight musicians, but I haven't yet figured out how to send > musicians down an ethernet cable), and a "byte" isn't as rigidly > defined. But on modern PCs, you can fairly safely assume that they're > synonymous. Sure ? a byte, I think, is the smallest unit of memory that can be addressed. It could be more or less that 8 bytes, but that wasn?t the point either. > I suppose you could argue that a "byte" is a patch of > storage capable of holding a number from 0 to 255, as opposed to being > the number itself, but that's getting rather existential :) No, I?m making the distinction that an eight bit byte is, well, eight bits, that CAN represent a number from 0 to 255, or it can represent any other data type ? like one eighth of the bits in a float, for instance. Or a bit field, or 1/2 a 16 bit int. > In Python, a "bytes" object represents a sequence of eight-bit units. > When you subscript a bytes [1], you get back an integer with the value > at that position. And when you print it, you get the ascii characters corresponding to each byte.... So one element in a bytes object is no more an integer than a character.... > So if a collection of them is called a "bytes" and > one of them is an integer in range(0, 256), doesn't it stand to reason > that a byte is a number? We use a decimal number (and ascii) to represent the bytes, as it?s more human readable and consistent with other python types. > Maybe I'm completely misunderstanding your statement here. Again, it doesn?t much matter, until you get to deciding how to bitshift an entire bytes object. -CHB From rosuav at gmail.com Fri May 18 20:14:07 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 19 May 2018 10:14:07 +1000 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: References: <20180517122043.GF12683@ando.pearwood.info> <20180517131321.GH12683@ando.pearwood.info> <5AFE6DAD.4000203@canterbury.ac.nz> Message-ID: On Sat, May 19, 2018 at 8:25 AM, Chris Barker - NOAA Federal wrote: >> I suppose you could argue that a "byte" is a patch of >> storage capable of holding a number from 0 to 255, as opposed to being >> the number itself, but that's getting rather existential :) > > No, I?m making the distinction that an eight bit byte is, well, eight > bits, that CAN represent a number from 0 to 255, or it can represent > any other data type ? like one eighth of the bits in a float, for > instance. Or a bit field, or 1/2 a 16 bit int. Since "bit" simply means "binary digit", that's like saying that a four-digit number isn't a number; it MIGHT represent a number, but it might represent one quarter of your credit card. Is "4564" less of a number for that reason? >> In Python, a "bytes" object represents a sequence of eight-bit units. >> When you subscript a bytes [1], you get back an integer with the value >> at that position. > > And when you print it, you get the ascii characters corresponding to > each byte.... That's because those numbers can often be used to represent characters. But they are really and truly numbers. (If you want to get down to brass tacks, a Unicode string could be treated as a sequence of 21-bit numbers. And in some languages, the "string" type is actually a highly-optimized version of 21-bit-number-array - or 32-bit, perhaps - with fully supported use-cases involving numerical (non-textual) data.) > So one element in a bytes object is no more an integer than a character.... Except that the bytestring b"\x00\x80\xff\x99" very clearly represents four numbers, but doesn't clearly represent any characters. >> Maybe I'm completely misunderstanding your statement here. > > Again, it doesn?t much matter, until you get to deciding how to > bitshift an entire bytes object. Bitshifting a sequence of bytes has nothing whatsoever to do with characters. It has to do with the individual numbers, and then you have to decide how you represent those as a collective: little-endian or big-endian. That's still a matter of numbers, not characters. ChrisA From python-ideas at mgmiller.net Fri May 18 20:54:04 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Fri, 18 May 2018 17:54:04 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements Message-ID: Background: While the previous discussions about assignment-expressions (PEP?572) (abbreviated AE below) have been raging one thing that was noticeable is that folks have been looking back to C for a solution. But how are newer languages solving the problem today? Believe Ryan brought this up first on the list, but it had been in the back of my mind as well. Finally have compiled my research, corrections welcome. In alphabetical order: Dart, 2011: Design goal: Less confusing, conservative replacement for JavaScript. - Assignment is a statement, not an expression. - JS allows AE to used freely, Dart doesn't support them. - Only option is to write simple, mildly redundant code. Details: https://www.dartlang.org/guides/language/language-tour#assignment-operators https://github.com/dart-lang/sdk/issues/55 Golang, 2009: Design goal: Simple, maintainable replacement for C++/Java/C#. - Assignment is a statement, not an expression. - Precursors allowed AE to be used freely - Golang doesn't them but allows assignment inside the if statement: if x := f(); x < y { ? } - No assignment in while (spelled for) currently allowed. Use mildly redundant code in other locations. Details: https://stackoverflow.com/questions/13127929/assign-and-compare-in-gos-while-equivalent https://clipperhouse.com/statements-are-statements-and-expressions-are-expressions-in-go-4087d103e3b7 Kotlin, 2011: Design goal: Terse, modern replacement for Java with high interop. - Assignment is a statement, not an expression. - Java allows AE to used freely, Kotlin doesn't support them. - Main option is to write simple, mildly redundant code. (Or use std lib functionality such as forEachLine, etc.) Details: https://blog.kotlin-academy.com/kotlin-programmer-dictionary-statement-vs-expression-e6743ba1aaa0 https://discuss.kotlinlang.org/t/assignment-not-allow-in-while-expression/339/ https://stackoverflow.com/questions/41537638/assignment-not-allowed-in-while-expression Rust, 2010: Design goal: Safe replacement for C/C++, etc. - Assignment is a statement, not an expression. - C/C++ allow AE - Rust doesn't, but allows an assignment clause in if/while. if let Some(3) = some_u8_value { ? } while let Some(byte) = input.next() { ? } Details: https://doc.rust-lang.org/book/second-edition/ch06-03-if-let.html https://doc.rust-lang.org/stable/rust-by-example/flow_control/while_let.html Swift, 2014: Design goal: Modern replacement for ObjectiveC with high interop. - Assignment returns Void - ObjC allows AE - Swift doesn't, but allows an assignment clause in if/while statements: if let NAME = ? { ? } if var NAME = ? { ? } while let line = aStreamReader.nextLine() { ? } Details: https://stackoverflow.com/questions/34173084/what-was-the-reason-for-swift-assignment-evaluation-to-void https://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/Statements.html#//apple_ref/swift/grammar/while-statement Conclusions ------------ It appears assignment expressions are no longer the favored solution for the "assign and compare" use case. Not one of these five newer languages supports them fully, as the language generations from C to C# did. Of those that have recognized the use case to be large enough?the solution has been rather more limited to the "if" and "while" statements only. Several folks here have expressed the same desire to confine AE there. Since Python's design goals are similar?to be safe and maintainable I'd recommend a similar strategy, with the addition of the list comprehension case. Going back to the C-style solution seems like the wrong direction. Since that would mean this special assignment functionality is not allowed to be used everywhere, it alleviates the pressure to make it fit into with/import/except statements. Furthermore, that frees up the keyword "as" again, which is Pythonic, short, memorable and has a history of being used for various assignment purposes, not to mention a prominent feature of SQL. In short, extend the "if/elif", "while", and comprehension to: if pattern.search(data) as match: ? while read_next_item() as value: ? May be best to disallow multiple assignment/conditions for now, but up for discussion. That leaves comprehensions, which could support a EXPR as NAME target also: filtered_data = [f(x) as y, x/y for x in data] or perhaps reuse of the if?as statement to keep it simpler: filtered_data = [y, x/y for x in data if f(x) as y] That variant might need an extra test if f(x) returns a falsey value, perhaps "is not None" on the end. Thoughts? -Mike From rosuav at gmail.com Fri May 18 21:10:30 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 19 May 2018 11:10:30 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: Message-ID: On Sat, May 19, 2018 at 10:54 AM, Mike Miller wrote: > In short, extend the "if/elif", "while", and comprehension to: > > if pattern.search(data) as match: > ? > > while read_next_item() as value: > ? > > May be best to disallow multiple assignment/conditions for now, but up for > discussion. That leaves comprehensions, which could support a EXPR as NAME > target also: > > filtered_data = [f(x) as y, x/y for x in data] > > or perhaps reuse of the if?as statement to keep it simpler: > > filtered_data = [y, x/y for x in data if f(x) as y] > > That variant might need an extra test if f(x) returns a falsey value, > perhaps > "is not None" on the end. The bit that you tag on as an afterthought is actually critically important here. You have two options: 1) The 'as' is part of the syntax of the 'if' and 'while' statements; or 2) The 'as' is part of the definition of an expression. The first case would be grammar like this: if_stmt: 'if' test ['as' NAME] ':' suite ('elif' test ':' suite)* ['else' ':' suite] The second is permitting 'as' name-bindings in arbitrary expressions, but then saying "except that they're only allowed in 'if' statements". As you've noted, the first one isn't sufficient. You can't use the restricted syntax for more than a small handful of conditions (including re.match(), but not including anything that might return None and might return other falsey values). So if this is valid: if (f(x) as y) is not None: then why isn't this valid: print(f(x) as y) or this: f(x) as y # identical to "y = f(x)" or even this: with (f(x) as y): ? If you say they should all be valid, have a read of the PEP - the consequences are pretty dangerous, especially the last one. If not, then where do you draw the line, and which consequences do you want to take? The 'as' syntax has been hammered out in great detail and is no longer recommended due to its negative interactions with existing constructs. ChrisA From apalala at gmail.com Fri May 18 21:39:13 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Fri, 18 May 2018 21:39:13 -0400 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: Message-ID: Conclusions > ------------ > > It appears assignment expressions are no longer the favored solution for > the > "assign and compare" use case. Not one of these five newer languages > supports > them fully, as the language generations from C to C# did. > > Of those that have recognized the use case to be large enough?the solution > has > been rather more limited to the "if" and "while" statements only. Several > folks here have expressed the same desire to confine AE there. Since > Python's > design goals are similar?to be safe and maintainable I'd recommend a > similar > strategy, with the addition of the list comprehension case. Going back to > the > C-style solution seems like the wrong direction. > > Since that would mean this special assignment functionality is not allowed > to > be used everywhere, it alleviates the pressure to make it fit into > with/import/except statements. Furthermore, that frees up the keyword "as" > again, which is Pythonic, short, memorable and has a history of being used > for > various assignment purposes, not to mention a prominent feature of SQL. > > In short, extend the "if/elif", "while", and comprehension to: > > if pattern.search(data) as match: > ? > > while read_next_item() as value: > ? > > May be best to disallow multiple assignment/conditions for now, but up for > discussion. That leaves comprehensions, which could support a EXPR as NAME > target also: > > filtered_data = [f(x) as y, x/y for x in data] > > or perhaps reuse of the if?as statement to keep it simpler: > > filtered_data = [y, x/y for x in data if f(x) as y] > > That variant might need an extra test if f(x) returns a falsey value, > perhaps > "is not None" on the end. > > Thoughts? > -Mike > Thanks for the research! Which is the niche, and how influential can it be, to have Python divert from the well learned "assignments must be statements". Cheers! -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri May 18 21:41:01 2018 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 18 May 2018 18:41:01 -0700 Subject: [Python-ideas] Async Lambda syntax In-Reply-To: <7ED94226-9104-4DFC-84E1-2CB3FAB3A422@gmail.com> References: <24B64AF1-A3B6-41C4-8BAE-B11102F55E3A@gmail.com> <7ED94226-9104-4DFC-84E1-2CB3FAB3A422@gmail.com> Message-ID: On Fri, May 18, 2018 at 1:53 PM, Noah Simon wrote: > Hello all, > I was developing a script using an asyncio-based API, when I came across the > need to define an asynchronous lambda. I found this syntax does not > currently exist. Obviously I could have (and did) just write a regular > coroutine, but for simple one-line functions and such, I think an > asynchronous lambda syntax would be useful. I do not have the experience to > write a PEP or implement this, so I was wondering what you all think of the > idea. > > What I was thinking: > > foo = async lambda a, b: b + await bar(a) This is the obvious syntax: it's unambiguous and consistent with the rest of the async/await syntax. Probably it's the only syntax worth considering. The question will be: does the need actually come up often enough in real life to be worth adding things to the language? Do you have any examples? Can you show what you were actually trying to do? Real examples are much more convincing than foo/bar examples. -n -- Nathaniel J. Smith -- https://vorpus.org From greg.ewing at canterbury.ac.nz Fri May 18 22:11:38 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 19 May 2018 14:11:38 +1200 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: Message-ID: <5AFF87DA.4030706@canterbury.ac.nz> Chris Angelico wrote: > The 'as' syntax has been hammered out in great detail and is no longer > recommended due to its negative interactions with existing constructs. Allowing it in arbitrary expressions has been ruled out on the grounds that the difference between "with x as y:" and "with (x as y):" would be too subtle. But that argument doesn't apply if "as" becomes part of the syntax of "if" and "while". Do we think that's a bad idea as well? From the survey of other modern languages that was just posted, it seems we'd be in good company if adopted something like that. -- Greg From rosuav at gmail.com Fri May 18 22:14:04 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 19 May 2018 12:14:04 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <5AFF87DA.4030706@canterbury.ac.nz> References: <5AFF87DA.4030706@canterbury.ac.nz> Message-ID: On Sat, May 19, 2018 at 12:11 PM, Greg Ewing wrote: > Chris Angelico wrote: >> >> The 'as' syntax has been hammered out in great detail and is no longer >> recommended due to its negative interactions with existing constructs. > > > Allowing it in arbitrary expressions has been ruled out on > the grounds that the difference between "with x as y:" and > "with (x as y):" would be too subtle. > > But that argument doesn't apply if "as" becomes part of the > syntax of "if" and "while". Do we think that's a bad idea > as well? Yes, largely because it's insufficient for all but a small handful of situations. I mentioned that in the previous email. ChrisA From carl.input at gmail.com Fri May 18 22:23:28 2018 From: carl.input at gmail.com (Carl Smith) Date: Sat, 19 May 2018 03:23:28 +0100 Subject: [Python-ideas] Verbatim names (allowing keywords as names) In-Reply-To: References: <20180518013426.GM12683@ando.pearwood.info> <20180518064729.GN12683@ando.pearwood.info> <20180518144827.GU12683@ando.pearwood.info> Message-ID: I was asked earlier to summarise the the proposal I've been advocating for, but have already gone over the central points a few times. I'll try and find time to write a clear explanation soon. -- Carl Smith carl.input at gmail.com On 18 May 2018 at 19:54, Neil Girdhar wrote: > > > On Fri, May 18, 2018 at 10:49 AM Steven D'Aprano > wrote: > >> On Fri, May 18, 2018 at 08:31:36AM -0400, Neil Girdhar wrote: >> [...] >> > > > I don't like multiple ways of doing the same thing. >> > > >> > > Ah, like when people want to use "class" as an identifier, and since >> > > they can't, they write: >> > > >> > > klass cls Class >> [...] >> > > Remind me again what the "one (obvious) way to do it" is? >> > > >> > >> > In most cases: cls >> > >> > https://www.python.org/dev/peps/pep-0008/#function-and-method-arguments >> >> The immediate next sentence goes on to say: >> >> If a function argument's name clashes with a reserved keyword, >> it is generally better to append a single trailing underscore >> rather than use an abbreviation or spelling corruption. Thus >> class_ is better than clss. (Perhaps better is to avoid such >> clashes by using a synonym.) >> >> So PEP 8 already torpedoes your preference for a single way to spell >> words that clash with keywords: >> >> - use a synonym; >> >> - or a trailing underscore >> >> - except for the first argument of class methods, where we >> use the misspelling (or abbreviation) "cls". >> >> If you object to this proposed verbatim names on the basis of disliking >> multiple ways to spell identifies that clash with keywords, that ship >> sailed long ago. There has always been multiple ways. >> > > Right. I think it's pretty clear that there is one way to avoid naming > conflict and keep the same name: use a trailing underscore except when it's > the first argument to a class method. > >> >> But I like to think that verbatim names could become the one OBVIOUS way >> to do it in the future. >> > > That's what I don't want. > >> >> >> [...] >> > All of your arguments would have applied to a keyword escaping proposal >> had >> > it been proposed before "given" was even considered. >> >> Every new idea has to be thought of for the first time. Otherwise it >> would have been in the language from day zero and it wouldn't be a new >> idea. If it wasn't "given", it could have been for "async" and "await", >> if not for them it could have been for "with", if not for "with" it >> might have been "yield", and so on. >> >> There had to be a first time for any idea. I would have >> suggested this twenty years ago if I thought of it twenty years ago, but >> I didn't, so I didn't. Dismissing the idea because I didn't suggest it >> earlier when other keywords were suggested is illogical. > > > > >> >> > The only reason we're >> > even considered considering escaping is to keep code that uses "given" >> as >> > an identifier working. >> >> That might be the only reason YOU are considing it, but it definitely >> isn't the only reason for me. And probably not for others. >> >> In fact, since I strongly dislike the "given" syntax, and really want >> that idea to be rejected, anything that increases its chances are a >> negative to me. >> >> Nevertheless, identifiers clashing with keywords isn't something brand >> new that only occurs thanks to "given". It has been a pain point >> forever. A small one, true, but still a nuisance. >> >> Verbatim names has the possibility to cut that nuisance value even >> further. But not if we short-sightedly limit it to a single case. > > > > >> >> > That's why I prefer the most modest solution of >> > only being able to escape given. >> >> Limiting a general method of mitigating the keyword clashing problem to >> a single keyword is as sensible as limiting the pathlib library to only >> work with a single hard-coded pathname. > > >> >> >> -- >> Steve >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/ >> topic/python-ideas/r1kFC8mYEKk/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Fri May 18 23:44:29 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Fri, 18 May 2018 20:44:29 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: Message-ID: <5AFF9D9D.90804@brenbarn.net> On 2018-05-18 17:54, Mike Miller wrote: > Background: > > While the previous discussions about assignment-expressions (PEP 572) > (abbreviated AE below) have been raging one thing that was noticeable is that > folks have been looking back to C for a solution. > > But how are newer languages solving the problem today? Believe Ryan brought > this up first on the list, but it had been in the back of my mind as well. > Finally have compiled my research, corrections welcome. In alphabetical order: Thanks, it is really great to see that comparison. That definitely pushes me even more towards disliking the assignment-expression proposals and preferring a more specific change to the syntax of particular constructs (if, while, maybe comprehensions) --- or no change at all. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From kenlhilton at gmail.com Fri May 18 23:58:28 2018 From: kenlhilton at gmail.com (Ken Hilton) Date: Sat, 19 May 2018 11:58:28 +0800 Subject: [Python-ideas] String and bytes bitwise operations Message-ID: On Fri, 18 May 2018 13:49:58 -0300, Facundo Batista wrote: > Once you think as the whole sequence of bytes as a sequence of bits (eight times longer, of course), all questions are easly answered, see below... I had never considered this before, but it seems very interesting. Essentially that would make bytes1 ^ bytes2 be equivalent to (int.from_bytes(bytes1, sys.byteorder) ^ int.from_bytes(bytes2, sys.byteorder)).to_bytes(max(len(bytes1), len(bytes2)), sys.byteorder) rather than bytes(a ^ b for a, b in zip(bytes1, bytes2)) I like that idea more than my original elementwise idea. ?Thinking, Ken Hilton;? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat May 19 01:20:30 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 May 2018 15:20:30 +1000 Subject: [Python-ideas] Make keywords KEYwords only in places they would have syntactical meaning In-Reply-To: References: <200b7139-4025-4d9e-d3d7-64475eab59ab@mrabarnett.plus.com> Message-ID: On 19 May 2018 at 04:34, Chris Angelico wrote: > "yield" would have to be a keyword in any context where an expression > is valid. Which, in turn, makes it utterly useless as a function name, > or any other identifier. > Right, I spent a fair bit of time thinking about this in the context of using "given" to introduce postfix assignment expressions, and while it's reasonably straightforward to make it so that keywords that can't start an expression or statement (currently "as", "in", "is", "and", "or") can also be used as names, we don't have that kind of freedom for keywords that can *start* an expression or statement ("async"/"await" relied on some parser tricks that relied on the fact that "async" always needed to be paired with "def" to introduce a coroutine, and "await EXPR" was only permitted inside coroutine definitions). We also run into the problem that even when the compiler can tell the difference, *humans* are still likely to be confused by the potential ambiguity (for the same reason that shadowing builtins is generally considered poor style). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From gadgetsteve at live.co.uk Fri May 18 05:34:43 2018 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Fri, 18 May 2018 09:34:43 +0000 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: <5AFE6DAD.4000203@canterbury.ac.nz> References: <20180517122043.GF12683@ando.pearwood.info> <20180517131321.GH12683@ando.pearwood.info> <5AFE6DAD.4000203@canterbury.ac.nz> Message-ID: On 18/05/2018 07:07, Greg Ewing wrote: > Steven D'Aprano wrote: >> But XORing bytes seems perfectly reasonable. Bytes are numbers, even >> if we display them as ASCII characters. > > Yep. Implement it for bytes, then the user can decode/encode > as appropriate for the application. > Personally I would suggest that the bitwise operations would only make sense between pairs of characters, (or bytes), of equal size. (It is possible to make a rule that the returned value is the same size as the larger of the two - i.e. the smaller is promoted and left padded with 0s). But such operations don't make much sense in most cases. However the bitwise operators could sensibly be used for set-wise operations on strings, i.e.: - "123xyz" | "abc" = "123abcxyz", - "Spam" & "Eggs" = "", "Spam" & "Scam" = "Sam" - "Spam" ^ "Scam" = "cp" This __might__ have some practical use? -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From ncoghlan at gmail.com Sat May 19 01:26:16 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 May 2018 15:26:16 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: Message-ID: On 19 May 2018 at 10:54, Mike Miller wrote: > Background: > > While the previous discussions about assignment-expressions (PEP 572) > (abbreviated AE below) have been raging one thing that was noticeable is > that > folks have been looking back to C for a solution. > > But how are newer languages solving the problem today? Believe Ryan > brought > this up first on the list, but it had been in the back of my mind as well. > Finally have compiled my research, corrections welcome. Thank you for compiling this! It's a very useful point of reference :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-ideas at shalmirane.com Sat May 19 01:56:54 2018 From: python-ideas at shalmirane.com (Ken Kundert) Date: Fri, 18 May 2018 22:56:54 -0700 Subject: [Python-ideas] Escaped braces in format specifiers In-Reply-To: References: <20180515064136.GA29222@kundert.designers-guide.com> Message-ID: <20180519055654.GA32380@kundert.designers-guide.com> On Tue, May 15, 2018 at 04:23:26PM -0400, Eric V. Smith wrote: > I'm busy at the sprints, so I don't have a lot of time to think about this. > > However, let me just say that recursive format specs are supported, > to a depth of 1. > > >>> width=10 > >>> f'{"test":{width}}' > 'test ' > > So first the string is basically expanded to: > f'{"test":10}' > Then the string is formatted again to produce the final result. > > That is why the braces must match: they're being used for recursive > format specs. There's no mechanism for having braces that aren't > inspected by the f-string machinery. > > Eric Eric, You say that recursive format specs are supported to a depth of 1, but it does not seem like true recursion to me. If it were true recursion, wouldn't the escaped braces be interpreted properly inside the format spec, just as they are in the literal portion of the f-string? Is there some reason why escaped braces are not supported in the format specs? Consider the following example: >>> class myclass: ... def __format__(self, fmt_spec): ... return 'fmt_spec: ' + fmt_spec >>> d = myclass() >>> x = 3 >>> print('1: ext={{x}} {d:int={{x}}}'.format(d=d)) 1: ext={x} fmt_spec: int={x} >>> print(f'2: ext={{x}} {d:int={{x}}}') 2: ext={x} fmt_spec: int={3} >>> print(f'3: ext={set([x])} {d:int={set([x])}}') 3: ext={3} fmt_spec: int={3} In each case, the same substring is found in the literal portion (ext) of the format string and in the format spec (int). In case 1, the format string is passed to a format method and the substring is {{x}}. In both cases, the double braces are interpreted as escaped braces, and both are converted to '{x}' in the final printed result. In case 2, the format string is an f string. The substring is still {{x}}. The first instance of {{x}} is found in the literal portion of the format string and the double braces are interpreted as escaped braces. It expands to {x} like in the first case. However, the second instance is found in the format spec, and this case the outer braces are stripped off and what is left over is treated as a Python expression. Here {x} is treated as a set literal containing one value, x. Since x is 3, the value of int is {3}. Case 3 confirms the interpretation of case 2 by simply replacing {x} with an alternate way of specifying a set literal that contains a single value. In this case there are no double braces and so both instances are treated the same, both expand to {3}. So the format method supports escaped braces in both the literal part of the format string and the format spec. f-strings supports escaped braces in the literal part of the format string, but in the format spec the outer level of braces are stripped off and what remains is interpreted as a Python expression. Is there some benefit to this difference? -Ken From paul-python at svensson.org Sat May 19 01:49:55 2018 From: paul-python at svensson.org (Paul Svensson) Date: Sat, 19 May 2018 01:49:55 -0400 (EDT) Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <5AFF87DA.4030706@canterbury.ac.nz> References: <5AFF87DA.4030706@canterbury.ac.nz> Message-ID: On Sat, 19 May 2018, Greg Ewing wrote: > Chris Angelico wrote: >> The 'as' syntax has been hammered out in great detail and is no longer >> recommended due to its negative interactions with existing constructs. > > Allowing it in arbitrary expressions has been ruled out on > the grounds that the difference between "with x as y:" and > "with (x as y):" would be too subtle. I don't quite get what's so subtle about it, am I missing something? The "with" keyword calls "__enter__", and "as" gives it a name. Just like "-x + y" is different from "-(x + y)", each part does it's thing, and you control the order with parens. /Paul From greg.ewing at canterbury.ac.nz Sat May 19 02:12:34 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 19 May 2018 18:12:34 +1200 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <5AFF87DA.4030706@canterbury.ac.nz> Message-ID: <5AFFC052.1020907@canterbury.ac.nz> Paul Svensson wrote: > I don't quite get what's so subtle about it, am I missing something? > > The "with" keyword calls "__enter__", and "as" gives it a name. > > Just like "-x + y" is different from "-(x + y)", I think the difference is that mentally one already tends to think of "with x as y" being grouped "with (x as y)" rather than "(with x) as y". So, if "x as y" becomes a legal expression all by itself, it will *seem* as though the meaning of "as" is changed simply by adding explicit parentheses around an expression that was already implicitly parenthesised. Whether this is too subtle or not is a matter of opinion, but it was raised as one of the objections to using "as", so some people obviously thought so. -- Greg From rosuav at gmail.com Sat May 19 02:39:35 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 19 May 2018 16:39:35 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <5AFFC052.1020907@canterbury.ac.nz> References: <5AFF87DA.4030706@canterbury.ac.nz> <5AFFC052.1020907@canterbury.ac.nz> Message-ID: On Sat, May 19, 2018 at 4:12 PM, Greg Ewing wrote: > Paul Svensson wrote: > >> I don't quite get what's so subtle about it, am I missing something? >> >> The "with" keyword calls "__enter__", and "as" gives it a name. >> >> Just like "-x + y" is different from "-(x + y)", > > > I think the difference is that mentally one already tends to > think of "with x as y" being grouped "with (x as y)" rather > than "(with x) as y". So, if "x as y" becomes a legal expression > all by itself, it will *seem* as though the meaning of "as" is > changed simply by adding explicit parentheses around an > expression that was already implicitly parenthesised. > > Whether this is too subtle or not is a matter of opinion, but > it was raised as one of the objections to using "as", so some > people obviously thought so. The reason it's dangerously subtle is that this will work: with (open("filename") as f): And this will work: with ( open("filename") as f ): And this will fail: with (threading.Lock() as lock): Do you know why? Do you want to try to debug that? I can guarantee you that a lot of people will go "hey, cool, now I can use parentheses in 'with' statements to wrap them across lines", and will do so very happily for a long time, wrapping their file-open blocks without an issue. But the semantics have changed *and they will not even know*. The current meaning of "as" is never "bind the expression on the left to the target on the right". Oh and here's another one: Should the target be required to be a name, or should it be allowed to be more complicated (eg "blah blah as self.spam")? Because Python isn't consistent on that point. Do you know which uses of "as" allow other targets than names and which don't? Again, do you want to have to debug situations where parentheses don't seem to matter for simple names, but with another kind of target, they're mandatory / forbidden? (Depending on which way the semantics are defined.) Using "as" in this way will NOT be consistent with the rest of Python, and it will introduce many subtleties for both the compiler and for humans. ChrisA From steve at pearwood.info Sat May 19 03:49:18 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 19 May 2018 17:49:18 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <5AFF87DA.4030706@canterbury.ac.nz> References: <5AFF87DA.4030706@canterbury.ac.nz> Message-ID: <20180519074917.GV12683@ando.pearwood.info> On Sat, May 19, 2018 at 02:11:38PM +1200, Greg Ewing wrote: > Chris Angelico wrote: > >The 'as' syntax has been hammered out in great detail and is no longer > >recommended due to its negative interactions with existing constructs. > > Allowing it in arbitrary expressions has been ruled out on > the grounds that the difference between "with x as y:" and > "with (x as y):" would be too subtle. > > But that argument doesn't apply if "as" becomes part of the > syntax of "if" and "while". Do we think that's a bad idea > as well? Yes. What is so special about "if" and "while" that they and they alone can make use of assignment expressions? The `and` and `or` short-cut operators are only useful in `if` and `while` statements, so we ought to prohibit them outside of `if` and `while` statements. Do you agree? Either assignment expressions are useful, and they should be allowed anywhere an expression is allowed, or they aren't, and they shouldn't be allowed at all. Breaking the rules for a special case is rarely the right decision. > From the survey of other modern languages that was just > posted, it seems we'd be in good company if adopted something > like that. /s/good/bad/ Compared to the Zen of Python, Go seems to break all the rules. Or at least most of them -- the beauty of the Zen is that it is open to interpetation :-) Go doesn't have exceptions. It requires the use of the tedious old anti-pattern of flag, result = function() if flag: do what you want else: # Now how the hell do I handle this? as a fundamental design principle, apparently because ugly and tedious boilerplate is a good thing, giving us all the verbosity of EAFP at its worst without any of the benefits. Similarly, its 2018 and the designers of Go turned their back on about 4/5th of the world, insisting that strings are ASCII and Unicode is an afterthought. I'm sure Rob Pike has his reasons for prohibiting exceptions, and network engineers probably think text is just another data format. Go might be an awesome language for what it is designed to do, but Forth is an awesome language for what it does best too, and we don't copy Forth's way of doing things. Before we copy Go, whether we copy a feature or a lack of feature, we ought to carefully consider how well that fits Python. (Sorry to just pick on Go, but it is the one I'm most familiar with.) -- Steve From eric at trueblade.com Sat May 19 07:06:41 2018 From: eric at trueblade.com (Eric V. Smith) Date: Sat, 19 May 2018 07:06:41 -0400 Subject: [Python-ideas] Escaped braces in format specifiers In-Reply-To: <20180519055654.GA32380@kundert.designers-guide.com> References: <20180515064136.GA29222@kundert.designers-guide.com> <20180519055654.GA32380@kundert.designers-guide.com> Message-ID: On 5/19/2018 1:56 AM, Ken Kundert wrote: > On Tue, May 15, 2018 at 04:23:26PM -0400, Eric V. Smith wrote: >> I'm busy at the sprints, so I don't have a lot of time to think about this. >> >> However, let me just say that recursive format specs are supported, >> to a depth of 1. >> >>>>> width=10 >>>>> f'{"test":{width}}' >> 'test' >> >> So first the string is basically expanded to: >> f'{"test":10}' >> Then the string is formatted again to produce the final result. >> >> That is why the braces must match: they're being used for recursive >> format specs. There's no mechanism for having braces that aren't >> inspected by the f-string machinery. >> >> Eric > > > Eric, > You say that recursive format specs are supported to a depth of 1, but it > does not seem like true recursion to me. If it were true recursion, wouldn't the > escaped braces be interpreted properly inside the format spec, just as they are > in the literal portion of the f-string? > > Is there some reason why escaped braces are not supported in the format specs? > > Consider the following example: > > >>> class myclass: > ... def __format__(self, fmt_spec): > ... return 'fmt_spec: ' + fmt_spec > > >>> d = myclass() > >>> x = 3 > > >>> print('1: ext={{x}} {d:int={{x}}}'.format(d=d)) > 1: ext={x} fmt_spec: int={x} > > >>> print(f'2: ext={{x}} {d:int={{x}}}') > 2: ext={x} fmt_spec: int={3} > > >>> print(f'3: ext={set([x])} {d:int={set([x])}}') > 3: ext={3} fmt_spec: int={3} > > In each case, the same substring is found in the literal portion (ext) of the > format string and in the format spec (int). > > In case 1, the format string is passed to a format method and the substring is > {{x}}. In both cases, the double braces are interpreted as escaped braces, and > both are converted to '{x}' in the final printed result. > > In case 2, the format string is an f string. The substring is still {{x}}. The > first instance of {{x}} is found in the literal portion of the format string and > the double braces are interpreted as escaped braces. It expands to {x} like in > the first case. However, the second instance is found in the format spec, and > this case the outer braces are stripped off and what is left over is treated as > a Python expression. Here {x} is treated as a set literal containing one value, > x. Since x is 3, the value of int is {3}. > > Case 3 confirms the interpretation of case 2 by simply replacing {x} with an > alternate way of specifying a set literal that contains a single value. In this > case there are no double braces and so both instances are treated the same, both > expand to {3}. > > So the format method supports escaped braces in both the literal part of the > format string and the format spec. f-strings supports escaped braces in the > literal part of the format string, but in the format spec the outer level of > braces are stripped off and what remains is interpreted as a Python expression. > > Is there some benefit to this difference? Is the difference you're seeing just in how f-strings and str.format() differ in evaluating expressions? str.format() isn't evaluating "int={x}", which makes sense to me. It never evaluates expressions, by design, it just treats strings as literals. If it did evaluate expressions, there's too great a chance of code being evaluated from user-defined strings. Eric From steve at pearwood.info Sat May 19 09:41:07 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 19 May 2018 23:41:07 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: Message-ID: <20180519134105.GW12683@ando.pearwood.info> Thanks for writing up the summary, but you have picked a very narrow subset of new languages. One might even say a biased subset. How about these new languages? Elixir, Elm, TypeScript, Hack, Julia, Perl6, Ring, LiveScript, Ballerina, Crystal, Opa, Red, Ceylon TL;DR At least 10 out of these 13 include some form of assignment expressions. If we include the five you selected, 13 out of 18, or 72% of the sample, include some form of assignment expression. Details follow below. Even in the limited set of five languages you chose, 3 out of the 5 allow some form of assignment expressions: - Go allows assignment expressions in "if" statements, but not "while"; - Rust allows assignment expressions in both "if" and "while"; - as does Swift; - Swift also allows assignment in arbitrary expressions, but it returns Void (equivalent to returning None). As for the other languages I found: Elixir (2011): - everything is an expression, including assignment if current_user = Plug.Conn.get_session(conn, :current_user) do Logger.info "User #{current_user.id} logged out" end https://stackoverflow.com/questions/42682961/assign-variable-and-test-in-if-statement-possible Elm (2012) - has "let expressions" http://elm-lang.org/docs/syntax#let-expressions TypeScript (2012) - a strict superset of Javascript, including assignment expressions Hack (2014) - a (not quite full) superset of PHP, including assignment expressions Julia (2012) - includes assignment expressions julia> x = 1 1 julia> y = (x = 2) + 1 3 julia> x 2 Perl 6 a.k.a. Rakudo (2015) - assignment is an expression: steve at orac ~ $ perl6 -e "say (my \$x = 32)+100; say \$x" 132 32 (the backslashes are escaping the dollar signs from the shell, they're not part of the Perl syntax) - also allows any(?) statement to be used as an expression with the `do` prefix Ring (2016) Documentation seems fairly poor, more concerned with singing the praises of the language than explaining the behaviour in detail, but maybe that's just me. But I *think* assignment is not an expression. Using the Ring interpreter here: http://ring-lang.net/ I tried running this: x = 55 x = (x=99) see x and got the output 0. My interpretation of this is that the x=99 expression is being interpreted as equals, not assignment. LiveScript (2011) - assignment is an expression which returns the value assigned http://livescript.net/#assignment Ballerina (2017) - as far as I can tell, assignment is purely a statement https://ballerina.io/res/Ballerina-Language-Specification-WD-2015-05-01.pdf Crystal (2014) - assignment is an expression which returns the value assigned I don't know if this link will work: https://play.crystal-lang.org/#/r/43fq but if not, try running this: x = 99; y = (x = 33) + 1; print x; print y; and the output ought to be 3334. Opa (2011) - assignment is an expression; the following two declarations are equivalent: two = { one = 1 // semicolon and newline are equivalent one + one } two = { one = 1; one + one // the exact same thing as above } https://github.com/MLstate/opalang/wiki/The-core-language Red (2011) (not to be confused with the US DOD "RED" language from 1979). I can't get the documentation to Red to display in a readable form in my browser, but it claims to be nearly identical to Rebol. Wikipedia describes Rebol (and presumably Red) as not having either expressions or statements in usual sense. Based on my reading, it is kinda-sorta like Forth except without the stack or the postfix syntax. The creator of Red describes it: "About the syntax and semantics, in a nutshell, it's a Lisp without parentheses and with infix operator support." which suggests that assignment could be an expression. There's also a "set" function which can assign values, but frankly the Rebol programming model confuses me and so I'll count this as a "Possibly, but I can't tell for sure so let's say No" for the question of assignment expressions. Ceylon (2011) - Ceylon has "let" assignment expressions. https://ceylon-lang.org/documentation/1.3/tour/attributes-control-structures/#tip_let_expressions - while loops support assignment https://ceylon-lang.org/documentation/1.3/tour/attributes-control-structures/#while_loops - and switch statements also support a named assignment https://ceylon-lang.org/documentation/1.3/tour/attributes-control-structures/#switch_conditionals -- Steve From steve at pearwood.info Sat May 19 09:52:29 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 19 May 2018 23:52:29 +1000 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: References: <20180517122043.GF12683@ando.pearwood.info> <20180517131321.GH12683@ando.pearwood.info> <5AFE6DAD.4000203@canterbury.ac.nz> Message-ID: <20180519135229.GX12683@ando.pearwood.info> On Fri, May 18, 2018 at 06:25:19PM -0400, Chris Barker - NOAA Federal via Python-ideas wrote: > > I suppose you could argue that a "byte" is a patch of > > storage capable of holding a number from 0 to 255, as opposed to being > > the number itself, but that's getting rather existential :) > > No, I?m making the distinction that an eight bit byte is, well, eight > bits, that CAN represent a number from 0 to 255, or it can represent > any other data type ? like one eighth of the bits in a float, for > instance. Or a bit field, or 1/2 a 16 bit int. That might be the case if we were talking about chunks of memory in RAM, but we're talking about *bytes objects in Python* where the individual items in the object most certainly represent ints between 0 and 255: py> b"abcde"[3] 100 Philosophical arguments about the nature of computer memory aside, byte objects in Python are collections of ints. If you want those ints to represent something else, you're responsible for handling that (say, using the struct module). -- Steve From python-ideas at mgmiller.net Sat May 19 12:40:59 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Sat, 19 May 2018 09:40:59 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180519134105.GW12683@ando.pearwood.info> References: <20180519134105.GW12683@ando.pearwood.info> Message-ID: <4d0caabe-eb36-166f-73de-6b71e5ca1f90@mgmiller.net> On 2018-05-19 06:41, Steven D'Aprano wrote: > Thanks for writing up the summary, but you have picked a very narrow > subset of new languages. One might even say a biased subset. How about > these new languages? Certainly. I chose basically on whether it was well used (popular), I'd heard about it, and it was easy to find documentation/discussion on. Then I ran out of energy around number six, haha. But I think this is representative of industry trends. -Mike From brenbarn at brenbarn.net Sat May 19 13:31:41 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Sat, 19 May 2018 10:31:41 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180519134105.GW12683@ando.pearwood.info> References: <20180519134105.GW12683@ando.pearwood.info> Message-ID: <5B005F7D.1050305@brenbarn.net> On 2018-05-19 06:41, Steven D'Aprano wrote: > TypeScript (2012) > - a strict superset of Javascript, including assignment expressions > > > Hack (2014) > - a (not quite full) superset of PHP, including assignment expressions Overall your list is fair, and you're right that more languages could be considered. However, you also argued elsewhere in this thread that Go had an unpythonic design so we shouldn't put too much weight on what it does. Given that, I think it's fair to add that, in my opinion, JavaScript and PHP are abysmally designed languages, so we should put little weight on any precedent they set. What would be nice is to know how much people like these various languages, and how productive people are with these languages, and then correlate that with their various features. However, as discussed on another thread, getting hard data about such matters is not easy. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From python-ideas at shalmirane.com Sat May 19 13:49:17 2018 From: python-ideas at shalmirane.com (Ken Kundert) Date: Sat, 19 May 2018 10:49:17 -0700 Subject: [Python-ideas] Escaped braces in format specifiers In-Reply-To: References: <20180515064136.GA29222@kundert.designers-guide.com> <20180519055654.GA32380@kundert.designers-guide.com> Message-ID: <20180519174917.GA6713@kundert.designers-guide.com> On Sat, May 19, 2018 at 07:06:41AM -0400, Eric V. Smith wrote: > On 5/19/2018 1:56 AM, Ken Kundert wrote: > >On Tue, May 15, 2018 at 04:23:26PM -0400, Eric V. Smith wrote: > >>I'm busy at the sprints, so I don't have a lot of time to think about this. > >> > >>However, let me just say that recursive format specs are supported, > >>to a depth of 1. > >> > >>>>>width=10 > >>>>>f'{"test":{width}}' > >>'test' > >> > >>So first the string is basically expanded to: > >>f'{"test":10}' > >>Then the string is formatted again to produce the final result. > >> > >>That is why the braces must match: they're being used for recursive > >>format specs. There's no mechanism for having braces that aren't > >>inspected by the f-string machinery. > >> > >>Eric > > > > > >Eric, > > You say that recursive format specs are supported to a depth of 1, but it > >does not seem like true recursion to me. If it were true recursion, wouldn't the > >escaped braces be interpreted properly inside the format spec, just as they are > >in the literal portion of the f-string? > > > >Is there some reason why escaped braces are not supported in the format specs? > > > >Consider the following example: > > > > >>> class myclass: > > ... def __format__(self, fmt_spec): > > ... return 'fmt_spec: ' + fmt_spec > > > > >>> d = myclass() > > >>> x = 3 > > > > >>> print('1: ext={{x}} {d:int={{x}}}'.format(d=d)) > > 1: ext={x} fmt_spec: int={x} > > > > >>> print(f'2: ext={{x}} {d:int={{x}}}') > > 2: ext={x} fmt_spec: int={3} > > > > >>> print(f'3: ext={set([x])} {d:int={set([x])}}') > > 3: ext={3} fmt_spec: int={3} > > > >In each case, the same substring is found in the literal portion (ext) of the > >format string and in the format spec (int). > > > >In case 1, the format string is passed to a format method and the substring is > >{{x}}. In both cases, the double braces are interpreted as escaped braces, and > >both are converted to '{x}' in the final printed result. > > > >In case 2, the format string is an f string. The substring is still {{x}}. The > >first instance of {{x}} is found in the literal portion of the format string and > >the double braces are interpreted as escaped braces. It expands to {x} like in > >the first case. However, the second instance is found in the format spec, and > >this case the outer braces are stripped off and what is left over is treated as > >a Python expression. Here {x} is treated as a set literal containing one value, > >x. Since x is 3, the value of int is {3}. > > > >Case 3 confirms the interpretation of case 2 by simply replacing {x} with an > >alternate way of specifying a set literal that contains a single value. In this > >case there are no double braces and so both instances are treated the same, both > >expand to {3}. > > > >So the format method supports escaped braces in both the literal part of the > >format string and the format spec. f-strings supports escaped braces in the > >literal part of the format string, but in the format spec the outer level of > >braces are stripped off and what remains is interpreted as a Python expression. > > > >Is there some benefit to this difference? > > Is the difference you're seeing just in how f-strings and > str.format() differ in evaluating expressions? str.format() isn't > evaluating "int={x}", which makes sense to me. It never evaluates > expressions, by design, it just treats strings as literals. If it > did evaluate expressions, there's too great a chance of code being > evaluated from user-defined strings. > > Eric Eric, My example does highlight the fact that f-strings do evaluate the embedded expressions while str.format() does not, but that is not the point I was trying to make. The substring 'int={{x}}' has two possible interpretations, it is either a string that contains escaped braces and when the escaping is processed becomes 'int={x}', or the outer braces signal an embedded expression, the expression being {x}, a set literal, and when interpolated the substring becomes 'int={3}'. My point was that str.format() uses the first interpretation for both ext and int, but f-strings use the first interpretation for ext and the second interpretation for int. Using the second interpretation for int in the f-string seems inconsistent. Specifically, int and ext are interpreted differently in the f-string, and int is interpreted differently by str.format() and the f-string. This difference does not seem to provide any benefit, so why does it exist? In other words, why aren't the escaped braces supported in the format_spec of f-strings? -Ken From kirillbalunov at gmail.com Sat May 19 14:07:03 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Sat, 19 May 2018 21:07:03 +0300 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: Message-ID: 2018-05-19 3:54 GMT+03:00 Mike Miller : > Background: > > While the previous discussions about assignment-expressions (PEP 572) > (abbreviated AE below) have been raging one thing that was noticeable is > that > folks have been looking back to C for a solution. > > But how are newer languages solving the problem today? Believe Ryan > brought > this up first on the list, but it had been in the back of my mind as well. > Finally have compiled my research, corrections welcome. In alphabetical > order: > > Thank you Mike this is a very useful analysis. It would be also great to know when they introduced this feature - from the beginning or after a while. Your results are completely consistent with all the examples from the numerous threads - that this feature should be added only in `while` and `if` statements. [Chris] The bit that you tag on as an afterthought is actually critically > important here. You have two options: > 1) The 'as' is part of the syntax of the 'if' and 'while' statements; or > 2) The 'as' is part of the definition of an expression. > The first case would be grammar like this: > if_stmt: 'if' test ['as' NAME] ':' suite ('elif' test ':' suite)* > ['else' ':' suite] > The second is permitting 'as' name-bindings in arbitrary expressions, > but then saying "except that they're only allowed in 'if' statements". > As you've noted, the first one isn't sufficient. You can't use the > restricted syntax for more than a small handful of conditions > (including re.match(), but not including anything that might return > None and might return other falsey values). > So if this is valid: > if (f(x) as y) is not None: > then why isn't this valid: > print(f(x) as y) > or this: > f(x) as y # identical to "y = f(x)" > or even this: > with (f(x) as y): I think I have a very strong argument "why are not others valid" - Because already three months have passed and among 1300+ messages there was not a single real example where assignment expression would be convenient or useful outside `while` and `if` statements. If you have a counterargument (with an example), I would be glad to see. This thread is about the design choices which were made in other modern languages. And from Mike's analysis it is evident that other languages have chosen to allow the assignment as an expression only where it is practical - in `while` and `if` statements, or not to allow it at all. I find this at least absolutely not constructive to push `print(f(x) as y)` or `print(y:=f(x))` in every thread. Is this really the only compelling example outside `while` and `if`? Also if we consider only the usage in `while` and `if` statements, what is the point of discussing differences between `with (f(x) as y)` and `with f(x) as y`? The first one will be a ` SyntaxError`, period. Can you also explain why the assignment expression should be valid outside `while` and `if` statements? - for dummy consistency? The `as` variant was _hammered_ only in the general case, which in itself does not make sense (which is confirmed by 1300+ messages without examples). Some time ago Guido said that he is not _impressed_ with `as` variant. But I think he does not like this form because it does not play well in general case. There are several reasons why. The main one is if `as` will be accepted it will prevent the use of this form in theoretically future statements (for example match statement). Another point that it looks ugly for general case. As it seems to me, the intersection with `with`, `except` is also important but can be overcomed. The use of any system requires some study and understanding. BUT I also think that Guido will not so strongly object if only `while` and `if` statements were under consideration, because there is no other choice. Examples from other languages are very important and their choice should be taken into account. Those languages are also designed by smart guys and I'm sure that they were thinking about where the assignement expression should be allowed and 1300+ messages only confirm their choice. Therefore, it may be worthwhile to focus only on actual use cases - in `while` and `if` statements??? and stop to flood every thread with dummy examples. p.s.: I deliberately did not mention the pseudo _usefulness_ of assignment expression in generators/comprehensions because the benefits are very questionable. Therefore, to begin with, we should start with `while` and `if` statements, and possibly in the future add it to comprehensions/generators. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat May 19 15:52:59 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 20 May 2018 05:52:59 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: Message-ID: On Sun, May 20, 2018 at 4:07 AM, Kirill Balunov wrote: > [Chris] > >> The bit that you tag on as an afterthought is actually critically >> important here. You have two options: >> 1) The 'as' is part of the syntax of the 'if' and 'while' statements; or >> 2) The 'as' is part of the definition of an expression. >> The first case would be grammar like this: >> if_stmt: 'if' test ['as' NAME] ':' suite ('elif' test ':' suite)* >> ['else' ':' suite] >> The second is permitting 'as' name-bindings in arbitrary expressions, >> but then saying "except that they're only allowed in 'if' statements". >> As you've noted, the first one isn't sufficient. You can't use the >> restricted syntax for more than a small handful of conditions >> (including re.match(), but not including anything that might return >> None and might return other falsey values). >> So if this is valid: >> if (f(x) as y) is not None: > > > I think I have a very strong argument "why are not others valid" - Because > already three months have passed and among 1300+ messages there was not a > single real example where assignment expression would be convenient or > useful outside `while` and `if` statements. If you have a counterargument > (with an example), I would be glad to see. Okay. Let me put it to you this way, then: Write up a simple and easy-to-explain set of rules for exactly what is valid and what is not. Make an actual proposal here. Be sure to demonstrate that the new construct is useful, with examples (like you're asking me for), and be sure that you're thorough enough in your definitions that this could actually be coded. I think you'll find that 1300+ messages aren't enough to adequately define what you're asking for. Or, alternatively: Give up on the 'as' syntax, because it was dropped from the PEP for a reason. ChrisA From carl.input at gmail.com Sat May 19 15:57:08 2018 From: carl.input at gmail.com (Carl Smith) Date: Sat, 19 May 2018 20:57:08 +0100 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: Message-ID: > JavaScript and PHP are abysmally designed languages, so we should > put little weight on any precedent they set. JavaScript has issues, due to its unique history, but to dismiss the entire language as too poorly designed to take seriously... Many clever people prefer JS to Python. -- Carl Smith carl.input at gmail.com On 19 May 2018 at 19:07, Kirill Balunov wrote: > > > 2018-05-19 3:54 GMT+03:00 Mike Miller : > >> Background: >> >> While the previous discussions about assignment-expressions (PEP 572) >> (abbreviated AE below) have been raging one thing that was noticeable is >> that >> folks have been looking back to C for a solution. >> >> But how are newer languages solving the problem today? Believe Ryan >> brought >> this up first on the list, but it had been in the back of my mind as well. >> Finally have compiled my research, corrections welcome. In alphabetical >> order: >> >> > Thank you Mike this is a very useful analysis. It would be also great to > know when they introduced this feature - from the beginning or after a > while. Your results are completely consistent with all the examples from > the numerous threads - that this feature should be added only in `while` > and `if` statements. > > > [Chris] > > The bit that you tag on as an afterthought is actually critically >> important here. You have two options: >> 1) The 'as' is part of the syntax of the 'if' and 'while' statements; or >> 2) The 'as' is part of the definition of an expression. >> The first case would be grammar like this: >> if_stmt: 'if' test ['as' NAME] ':' suite ('elif' test ':' suite)* >> ['else' ':' suite] >> The second is permitting 'as' name-bindings in arbitrary expressions, >> but then saying "except that they're only allowed in 'if' statements". >> As you've noted, the first one isn't sufficient. You can't use the >> restricted syntax for more than a small handful of conditions >> (including re.match(), but not including anything that might return >> None and might return other falsey values). >> So if this is valid: >> if (f(x) as y) is not None: >> then why isn't this valid: >> print(f(x) as y) >> or this: >> f(x) as y # identical to "y = f(x)" >> or even this: >> with (f(x) as y): > > > I think I have a very strong argument "why are not others valid" - Because > already three months have passed and among 1300+ messages there was not a > single real example where assignment expression would be convenient or > useful outside `while` and `if` statements. If you have a counterargument > (with an example), I would be glad to see. > > This thread is about the design choices which were made in other modern > languages. And from Mike's analysis it is evident that other languages have > chosen to allow the assignment as an expression only where it is practical > - in `while` and `if` statements, or not to allow it at all. > > I find this at least absolutely not constructive to push `print(f(x) as y)` > or `print(y:=f(x))` in every thread. Is this really the only compelling > example outside `while` and `if`? Also if we consider only the usage in > `while` and `if` statements, what is the point of discussing differences > between `with (f(x) as y)` and `with f(x) as y`? The first one will be a > `SyntaxError`, period. Can you also explain why the assignment expression > should be valid outside `while` and `if` statements? - for dummy > consistency? The `as` variant was _hammered_ only in the general case, > which in itself does not make sense (which is confirmed by 1300+ messages > without examples). > > Some time ago Guido said that he is not _impressed_ with `as` variant. But > I think he does not like this form because it does not play well in general > case. There are several reasons why. The main one is if `as` will be > accepted it will prevent the use of this form in theoretically future > statements (for example match statement). Another point that it looks ugly > for general case. As it seems to me, the intersection with `with`, `except` > is also important but can be overcomed. The use of any system requires some > study and understanding. BUT I also think that Guido will not so strongly > object if only `while` and `if` statements were under consideration, > because there is no other choice. > > Examples from other languages are very important and their choice should > be taken into account. Those languages are also designed by smart guys and > I'm sure that they were thinking about where the assignement expression > should be allowed and 1300+ messages only confirm their choice. Therefore, > it may be worthwhile to focus only on actual use cases - in `while` and > `if` statements??? and stop to flood every thread with dummy examples. > > > p.s.: I deliberately did not mention the pseudo _usefulness_ of assignment > expression in generators/comprehensions because the benefits are very > questionable. Therefore, to begin with, we should start with `while` and > `if` statements, and possibly in the future add it to > comprehensions/generators. > > With kind regards, > -gdg > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Sat May 19 16:59:18 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sat, 19 May 2018 21:59:18 +0100 Subject: [Python-ideas] Make keywords KEYwords only in places they would have syntactical meaning In-Reply-To: References: Message-ID: <3336a1ee-8733-65c7-ad44-f6abf06d1e56@btinternet.com> On 18/05/2018 12:22, Ken Hilton wrote: > Hi all, > > Yes, this is another idea for avoiding breaking existing code when > introducing new keywords. I'm not sure if this is too similar to > Guido's previous "allow keywords in certain places" idea, but here goes: > > Only treat keywords as having any special meaning when they are in > places with syntactical significance. > So, currently, let's say someone set the variable "and_" to some > value. The following lines are both SyntaxErrors: > > ? True and_ False > ? obj.and = value > > And the following are both correct: > > ? True and False > ? obj.and_ = value > > My idea is to only treat keywords as having special meaning when > they're in the right place. So the following would all be legal: > > ? >>> from operator import and > ? >>> var = and(True, False) > ? >>> var > ? False > ? >>> var = True and False > ? >>> var > ? False > ? >>> def except(exc, def): > ? ...? ? ?try: > ? ...? ? ? ? ?return def() > ? ...? ? ?except exc as e: > ? ...? ? ? ? ?return e > ? ... > ? >>> except(ZeroDivisionError, lambda: 1/0) > ? ZeroDivisionError('division by zero',) > ? >>> except(ZeroDivisionError, lambda: 0/1) > ? 0.0 > ? >>> import asyncio as await #this is already currently legal, but > will not be in the __future__ > ? >>> async def async(def): > ? ...? ? ?return await await.get_event_loop().run_in_executor(None, def) > ? ... > ? >>> > > And so on. > > What are your thoughts? > > ?Sharing, > > ?Ken Hilton > ; > Great idea, but why restrict it to keywords?? Why not also allow operators to be variable names: >>> + = 23 >>> + + + + + 69 >>> - = 21 >>> - - >>> -21 There would be some minor ambiguities, e.g. would ??? - - - - evaluate to ??? (- -) - (-) # -42 or ??? (-) - (- -) # +42 or ??? - ( - ( - ( - ))) # -21 or ??? - (- - -) # 0 and indeed ??? + + + + + could *in theory* evaluate to ??? + ( + ( + ( + ( +)))) # 23 or ??? (++) + (++) # 46 but I'm sure we could formulate some sensible rules to always get the intended meaning. :-) Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-ideas at mgmiller.net Sat May 19 18:39:42 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Sat, 19 May 2018 15:39:42 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <4d0caabe-eb36-166f-73de-6b71e5ca1f90@mgmiller.net> References: <20180519134105.GW12683@ando.pearwood.info> <4d0caabe-eb36-166f-73de-6b71e5ca1f90@mgmiller.net> Message-ID: Also meant to mention, I didn't pick them specifically in advance to match a goal. I was mildly surprised at their similar design on that front. -Mike From python-ideas at mgmiller.net Sat May 19 18:44:12 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Sat, 19 May 2018 15:44:12 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: Message-ID: <3932b2f0-6a9e-3b52-468f-28ade9aef268@mgmiller.net> On 2018-05-18 18:10, Chris Angelico wrote: > The bit that you tag on as an afterthought is actually critically > important here. You have two options: > > 1) The 'as' is part of the syntax of the 'if' and 'while' statements; or This first choice appears the more conservative compromise that several have chosen (substituting "as" with "let"), only as part of the these two statements. -Mike From guido at python.org Sat May 19 18:56:16 2018 From: guido at python.org (Guido van Rossum) Date: Sat, 19 May 2018 15:56:16 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <4d0caabe-eb36-166f-73de-6b71e5ca1f90@mgmiller.net> References: <20180519134105.GW12683@ando.pearwood.info> <4d0caabe-eb36-166f-73de-6b71e5ca1f90@mgmiller.net> Message-ID: On Sat, May 19, 2018 at 9:40 AM, Mike Miller wrote: > > On 2018-05-19 06:41, Steven D'Aprano wrote: > >> Thanks for writing up the summary, but you have picked a very narrow >> subset of new languages. One might even say a biased subset. How about >> these new languages? >> > > Certainly. I chose basically on whether it was well used (popular), I'd > heard about it, and it was easy to find documentation/discussion on. Then > I ran out of energy around number six, haha. > > But I think this is representative of industry trends. Maybe Python can set a new trend. It's happened before. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-ideas at mgmiller.net Sat May 19 18:56:52 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Sat, 19 May 2018 15:56:52 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <5AFF87DA.4030706@canterbury.ac.nz> Message-ID: <960f1042-06ee-7eaa-6968-9a058418dfe7@mgmiller.net> On 2018-05-18 19:14, Chris Angelico wrote: > Yes, largely because it's insufficient for all but a small handful of > situations. I mentioned that in the previous email. I'd argue that they are the bulk of occurrences, that's why they've been chosen as the compromise in those languages. Once there is an assignment and two compares, the time to break up the statement is approaching quickly, no? -Mike From rosuav at gmail.com Sat May 19 19:00:40 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 20 May 2018 09:00:40 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <960f1042-06ee-7eaa-6968-9a058418dfe7@mgmiller.net> References: <5AFF87DA.4030706@canterbury.ac.nz> <960f1042-06ee-7eaa-6968-9a058418dfe7@mgmiller.net> Message-ID: On Sun, May 20, 2018 at 8:56 AM, Mike Miller wrote: > > On 2018-05-18 19:14, Chris Angelico wrote: >> >> Yes, largely because it's insufficient for all but a small handful of >> situations. I mentioned that in the previous email. > > > I'd argue that they are the bulk of occurrences, that's why they've been > chosen as the compromise in those languages. > > Once there is an assignment and two compares, the time to break up the > statement is approaching quickly, no? > But you can't put a comparison after the assignment, if it's part of the syntax of the 'if' statement. That's not how grammar works. So you have two options: either the ONLY thing you can capture is the condition value (which you already know to be truthy), or it's part of a more general expression, and is independent of if/while. You said yourself that the first one isn't enough. So it HAS to be an expression feature, not a statement feature. ChrisA From python-ideas at mgmiller.net Sat May 19 19:28:10 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Sat, 19 May 2018 16:28:10 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180519134105.GW12683@ando.pearwood.info> References: <20180519134105.GW12683@ando.pearwood.info> Message-ID: On 2018-05-19 06:41, Steven D'Aprano wrote: > Details follow below. Thanks for this, had some more time to read it more closely. Correct me if I'm probably wrong, but most of these are not used by many. Except perhaps: - Typescript, which is constrained by a compatibility goal with JavaScript. - Julia, - Possibly Elixir, have heard good things about it. My focus was on "industry standard" languages however, not pushing the envelope too far but rather on humdrum bug reduction and maintainability, an area Python aspires to, I believe. There was a stack overflow survey recently, should probably dig that up. -Mike From python-ideas at mgmiller.net Sat May 19 19:34:47 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Sat, 19 May 2018 16:34:47 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <5AFF87DA.4030706@canterbury.ac.nz> <960f1042-06ee-7eaa-6968-9a058418dfe7@mgmiller.net> Message-ID: On 2018-05-19 16:00, Chris Angelico wrote: > But you can't put a comparison after the assignment, if it's part of > the syntax of the 'if' statement. That's not how grammar works. So you > have two options: either the ONLY thing you can capture is the > condition value (which you already know to be truthy), or it's part of > a more general expression, and is independent of if/while. You said > yourself that the first one isn't enough. So it HAS to be an > expression feature, not a statement feature. Sorry, may have not written clearly. But the gist of the thread is several languages found the simple form (one assignment and compare) adequate. Believe the examples we've looked at in the various threads have been the simple form. So I argue it is enough in most cases. -Mike From steve at pearwood.info Sat May 19 21:43:34 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 20 May 2018 11:43:34 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> Message-ID: <20180520014333.GZ12683@ando.pearwood.info> On Sat, May 19, 2018 at 04:28:10PM -0700, Mike Miller wrote: > > On 2018-05-19 06:41, Steven D'Aprano wrote: > >Details follow below. > > Thanks for this, had some more time to read it more closely. Correct me if > I'm probably wrong, but most of these are not used by many. Except perhaps: According to the latest survey on TIOBE (May 2018), *none* of the languages either of us surveyed are used by more than a small niche. I realise that the TIOBE ranking is not the only available ranking, nor is their methodology necessarily the best, but if other people want to look at other language surveys they are free to do so. According to TIOBE, the rankings of your five languages are: Go #14 (up two places) 0.970% Swift #19 (down six places) 0.907% Dart #26 0.859% Kotlin #49 0.292% Rust #51-100 (percentages too small to differentiate) So even the most popular of your languages are still very niche. Go and Swift have a lot of industry "buzz" about them, but that's not translating to jobs or Google searches yet. The thirteen languages I surveyed are in a similar position: the highest ranked of them is Julia at #46. (I don't think TIOBE distinguishes between Perl at #18 and Perl 6.) Almost by definition, any new language is going to only be used by a small subset of programmers. [...] > My focus was on "industry standard" languages however, I'm sorry, but no it wasn't. If we want *industry standard* languages, none of them are going to be *new*, and we need to look at those at the top of the TIOBE rankings: Java C C++ Python C# VB .Net PHP Javascript SQL Ruby R Delphi/Object Pascal Go and Swift are backed by large comporations and may some day be "industry standard", but they aren't yet. [Aside: it is sobering to realise that according to at least one objective ranking, "ancient and obsolete" Pascal is still more popular than "cool" new languages like Go and Swift.] I've somewhat arbitrarily cut the list off at "languages ranked above 1% on TIOBE", but we have to cut the list of somewhere. And of course in certain specific industries the standard languages may be very different, e.g. there are still tens of millions of lines of COBOL code being maintained in the banking industry. Out of those industry standard languages (as ranked by TIOBE, other methodology may result in other rankings) we find: 8/12 have some form of assignment expressions; (Java, C, C++, C#, PHP, Javascript, Ruby, R) 4/12 do not (Python, VB .Net, SQL, Delphi). -- Steve From rosuav at gmail.com Sat May 19 22:18:18 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 20 May 2018 12:18:18 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180520014333.GZ12683@ando.pearwood.info> References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> Message-ID: On Sun, May 20, 2018 at 11:43 AM, Steven D'Aprano wrote: > Out of those industry standard languages (as ranked by TIOBE, other > methodology may result in other rankings) we find: > > 8/12 have some form of assignment expressions; > (Java, C, C++, C#, PHP, Javascript, Ruby, R) > > 4/12 do not (Python, VB .Net, SQL, Delphi). > SQL isn't really comparable here. To figure out whether Python is in better company *with* or *without* assignment expressions, we need to compare with languages that have a concept of "assignment" and "expression". SQL itself most certainly has expressions, but it doesn't have assignment per se. There are named sub-expressions in SELECT statements, but that's more of a temporary view on a table than anything like a Python variable / name binding; I don't think standard SQL has any way to fetch up a scalar value and then reuse it, other than burying it in a table and selecting twice from that table. Some database engines have an SQL-based procedural language, but it's non-standard. FWIW, PostgreSQL's "PL/pgSQL" has assignment, but it is a statement and not an expression, probably because its expression evaluator is defined in terms of SQL's SELECT statement. So there are really eleven: eight that do, two that don't, and one that currently doesn't, but its BDFL is discussing the possibility of adding it. ChrisA From marcidy at gmail.com Sun May 20 01:19:12 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Sat, 19 May 2018 22:19:12 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: Message-ID: On Sat, May 19, 2018, 11:07 Kirill Balunov wrote: > > > I think I have a very strong argument "why are not others valid" - Because > already three months have passed and among 1300+ messages there was not a > single real example where assignment expression would be convenient or > useful outside `while` and `if` statements. If you have a counterargument > (with an example), I would be glad to see. > comprehensions. I don't think you read any of the messages. about 1000 were solely about comprehensions. How about every example brought up in 1300+ messages that assignment expressions could handle that otherwise required far more code? Some including imports? Did you miss where _every_ counter example was either doing what assignment expressions do with more code, or where outright misunderstandings of how they will work? Did you read Chris' responses where he clearly refused each example? Really hard to see how you got that take away if you read those messages. Basically you are not backed up at all by citing that. assignment expressions handle many cases with ONE syntax. They are the "one way to do it" where "it" covers every example brought up so far. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Sun May 20 01:21:16 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 20 May 2018 00:21:16 -0500 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180520014333.GZ12683@ando.pearwood.info> References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> Message-ID: [Steven D'Aprano ] > ... > Red (2011) > ... > ... claims to be nearly identical to Rebol. > > Wikipedia describes Rebol (and presumably Red) as not having either > expressions or statements in usual sense. Based on my reading, it is > kinda-sorta like Forth except without the stack or the postfix syntax. > The creator of Red describes it: > > "About the syntax and semantics, in a nutshell, it's a Lisp without > parentheses and with infix operator support." > > which suggests that assignment could be an expression. There's also a > "set" function which can assign values, but frankly the Rebol > programming model confuses me and so I'll count this as a "Possibly, but > I can't tell for sure so let's say No" for the question of assignment > expressions. I was an early REBOL user, and my head still hurts ;-) It was ... different, for sure. The syntax is in some sense extremely simple, hoping to make it easy to define problem-specific layers of syntax on top of it. But I gave up before they got that far. In any case, everything is "an expression" there, including assignment. That's spelled: WORD ":" WHITESPACE+ EXPRESSION although that's lying a bit. Here from a Red shell: >> i: 1 == 1 >> j: k: 2 == 2 >> print [i j k] 1 2 2 >> i: 12 j: 13 == 13 >> print [i j] 12 13 >> x: 12 print add y: 88 x 100 >> y == 88 That last one shows one of the challenges for people coming from "normal" languages: this does the same: >> (x: 12) (print (add (y: 88) x)) 100 showing that it really is a lot like "Lisp without parentheses". Without the parens, it's impossible to tell where function calls begin and end without knowing which names denote functions and how many arguments each function requires! That's "a feature", they insist ;-) To be fair, you really do get used to it quickly for the heavily used builtin functions. One more: >> i1+2=3*88: 6 == 6 >> i1+2=3*88 == 6 Yup! "i1+2=3*88" is a variable name there :-) Short course: definitely "yes" on assignment expressions for Red. But I'm going to out on a limb and guess that the typical Python programmer wouldn't find "but Red does it!" persuasive ;-) From steve at pearwood.info Sun May 20 01:35:50 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 20 May 2018 15:35:50 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> Message-ID: <20180520053549.GD12683@ando.pearwood.info> On Sun, May 20, 2018 at 12:21:16AM -0500, Tim Peters wrote: > I was an early REBOL user, and my head still hurts ;-) It was ... > different, for sure. [...] Yeah, to me it looks more like a prefix version of Forth than Lisp. Complete with "anything can be a name": > Yup! "i1+2=3*88" is a variable name there :-) but maybe when languages are weird enough they all look the same :-) > Short course: definitely "yes" on assignment expressions for Red. > But I'm going to out on a limb and guess that the typical Python > programmer wouldn't find "but Red does it!" persuasive ;-) True :-) Thanks for doing the testing. -- Steve From greg.ewing at canterbury.ac.nz Sun May 20 05:07:44 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 20 May 2018 21:07:44 +1200 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180520053549.GD12683@ando.pearwood.info> References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520053549.GD12683@ando.pearwood.info> Message-ID: <5B013AE0.3070702@canterbury.ac.nz> Steven D'Aprano wrote: > Yeah, to me it looks more like a prefix version of Forth than Lisp. Throf? -- Greg From apalala at gmail.com Sun May 20 11:19:37 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Sun, 20 May 2018 11:19:37 -0400 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180520053549.GD12683@ando.pearwood.info> References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520053549.GD12683@ando.pearwood.info> Message-ID: On Sun, May 20, 2018 at 1:35 AM, Steven D'Aprano wrote: > > but maybe when languages are weird enough they all look the same :-) > > https://www.dreamsongs.com/WorseIsBetter.html Look at what happened with PERL... IMPORTANT NOTE: Enabling "as" in "if" and "while" doesn't have to be at the expense of some form of assignment expression. We've been having this discussion as if has to be either one or the other, and there's no reason we can't have both (as mentioned before, "reduce()" is live and well in the libraries, for people who need to use it). I in particular would not like it at all if something like ":=" was at the expense of "as", and I think that others on the side of expanding "if" and "while" may feel the same way. As a reminder, some of the arguments in favor of "as" are around the patterns exemplified by the "re" module; patterns which seem correct and useful, and that are used by other standard and 3rd-party modules. if os.fork() as child_pid: parent(child_pid) else: child() Cheers! -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun May 20 13:46:02 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 21 May 2018 03:46:02 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520053549.GD12683@ando.pearwood.info> Message-ID: On Mon, May 21, 2018 at 1:19 AM, Juancarlo A?ez wrote: > IMPORTANT NOTE: > > Enabling "as" in "if" and "while" doesn't have to be at the expense of some > form of assignment expression. We've been having this discussion as if has > to be either one or the other, and there's no reason we can't have both (as > mentioned before, "reduce()" is live and well in the libraries, for people > who need to use it). > > I in particular would not like it at all if something like ":=" was at the > expense of "as", and I think that others on the side of expanding "if" and > "while" may feel the same way. Let's suppose that the := syntax already existed in Python - that you could write this: if child_pid := os.fork(): parent(child_pid) else: child() What would be the benefit of adding the "as child_pid" syntax? The full assignment expression syntax completely covers all the jobs that you can do with a capturing if statement. ChrisA From python-ideas at mgmiller.net Sun May 20 13:57:39 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Sun, 20 May 2018 10:57:39 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180520014333.GZ12683@ando.pearwood.info> References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> Message-ID: You seem determined to throw out requirements when it suits. One of which was for newer languages. While this thread's original post was not so specific on that, in a previous post (I believe you read) defined that as created in the "last decade or so". The ones created to address shortcomings in the previous generation. The point being to find momentum in design of newer languages. Your list of older languages is therefore not pertinent. -Mike On 2018-05-19 18:43, Steven D'Aprano wrote: > Java > C > C++ > Python > C# > VB .Net > PHP > Javascript > SQL > Ruby > R > Delphi/Object Pascal > From python-ideas at mgmiller.net Sun May 20 14:04:43 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Sun, 20 May 2018 11:04:43 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180520014333.GZ12683@ando.pearwood.info> References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> Message-ID: For more background, this is the thread that inspired this one: https://mail.python.org/pipermail/python-dev/2018-April/153071.html -Mike From apalala at gmail.com Sun May 20 14:53:46 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Sun, 20 May 2018 14:53:46 -0400 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520053549.GD12683@ando.pearwood.info> Message-ID: > What would be the benefit of adding the "as child_pid" syntax? The > full assignment expression syntax completely covers all the jobs that > you can do with a capturing if statement. > > The "as" syntax over "if" and "while" may go in with all the restrictions and caveats already discussed. The discussion about the semantics of how ":=" should affect the enclosing context when used in generators, etc. may continue for as long as it must. And there are some of us who like the consistency of "us", and would shy away from ":=". For example, Should this be valid? if child_pid := os.fork(): parent(child_pid) else: child() print(child_pid) This shouldn't be: if os.fork() as child_pid: parent(child_pid) else: child() print(child_pid) # child_pid is undefined -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcidy at gmail.com Sun May 20 14:55:47 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Sun, 20 May 2018 11:55:47 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> Message-ID: Anyone can trivially construct a scope that limits applicable cases to support a specific point. This thread is pointless without full context. On Sun, May 20, 2018, 11:05 Mike Miller wrote: > For more background, this is the thread that inspired this one: > > https://mail.python.org/pipermail/python-dev/2018-April/153071.html > > > -Mike > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Sun May 20 15:01:19 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 20 May 2018 14:01:19 -0500 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180520053549.GD12683@ando.pearwood.info> References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520053549.GD12683@ando.pearwood.info> Message-ID: [Tim] >> I was an early REBOL user, and my head still hurts ;-) It was ... >> different, for sure. [Steven D'Aprano ] > Yeah, to me it looks more like a prefix version of Forth than Lisp. > Complete with "anything can be a name": The example I gave just strung "words" together, but just as fundamental is the notion of "block": a series of words and/or blocks enclosed in square brackets. That's akin to Lisp's "S expressions", but Rebol/Red _don't_ "evaluate" blocks by default. Blocks can hold data or code, or a mix of both, and the syntax doesn't care. That's the real reason "(almost) anything can be a name (word)". Here I'll build to a simple example where _some_ Pythoneers will experience envy rather than revulsion ;-) >> [x y] == [x y] The block isn't evaluated. Depending on how it's _used_, it may be treated as data (perhaps you want to think of it as being a sequence of two symbols, or strings). If you evaluate it, it blows up (because I don't happen to have any variables with those names defined): >> do [x y] *** Script Error: x has no value Give the names some values, and _then_ it can be evaluated; and evaluating a block returns the value of the last expression in the block: >> x: 12 y: 13 == 13 >> [x y] == [x y] >> do [x y] == 13 If you want a block with all the expressions' values, use `reduce` instead: >> reduce [x y] == [12 13] Code? Data? No difference. Here's the part where some Pythoneers will get jealous: >> sum: func [x y] [x + y] == func [x y][x + y] >> sum 8 2 == 10 `func` is just another function that happens to build an anonymous function. It takes two blocks as arguments: a block containing the formal argument names, and a block with the code to execute. Both blocks are essentially "data" to `func`. It doesn't look much like Forth anymore ;-) Note that the following does exactly the same: >> arglist: [x y] == [x y] >> body: [x + y] == [x + y] >> sum2: func arglist body == func [x y][x + y] >> sum2 8 2 == 10 In practice, slinging Rebol/Red most feels like working in a functional language. One of their real goals, though, is to enable writing functions that can be called like so: schedule "Change furnace filter" [ starting today then every 3 months ] There's an elaborate `parse` function built in that supports many ways of applying user-supplied rules to parse blocks like the one in that example. Of course you can emulate much the same in Python by, e.g., passing triple-quoted strings instead. In that specific example. Rebol/Red provide a minimum of syntax shared by all such applications, and when the sub-language gets elaborate enough that _nesting_ blocks makes good sense, nesting triple-quoted strings sucks ;-) All languages have some good things going for them. But I'm not sure I've ever seen Rebol/Red code that _used_ the value of an assignment expression; that it has them at all seems much more to follow from that everything is an expression. As in most functional languages, if you want initialized local variables, you're more likely to invoke an anonymous spelled-inline function instead. Which is what, e.g., Haskell's "let PILE_OF_BINDINGS in EXPRESSION" is syntactic sugar for doing. From greg.ewing at canterbury.ac.nz Sun May 20 19:10:23 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 21 May 2018 11:10:23 +1200 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520053549.GD12683@ando.pearwood.info> Message-ID: <5B02005F.8010003@canterbury.ac.nz> Tim Peters wrote: >>>x: 12 y: 13 > == 13 >>>[x y] > == [x y] >>>do [x y] > == 13 How does scoping work? If you pass a block to a function which evaluates it, how are names in the block resolved? -- Greg From steve at pearwood.info Sun May 20 19:11:00 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 21 May 2018 09:11:00 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520053549.GD12683@ando.pearwood.info> Message-ID: <20180520231059.GE12683@ando.pearwood.info> On Sun, May 20, 2018 at 02:53:46PM -0400, Juancarlo A?ez wrote: > This shouldn't be: > > if os.fork() as child_pid: > parent(child_pid) > else: > child() > > print(child_pid) # child_pid is undefined Why on earth would it be undefined? Anyway, if you want to propose an alternative to PEP 572, you ought to write your own competing PEP. -- Steve From steve at pearwood.info Sun May 20 19:48:41 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 21 May 2018 09:48:41 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> Message-ID: <20180520234840.GF12683@ando.pearwood.info> On Sun, May 20, 2018 at 10:57:39AM -0700, Mike Miller wrote: > You seem determined to throw out requirements when it suits. One of which > was for newer languages. YOU are the one who changed the requirements: first you said "newer languages", then you changed it to industry standard. I showed that a broad sample of new languages by far favours assignment expressions (14 out of 18 including Red, which I wrongly guessed didn't) and even the narrow subset that you wrote up included some form of assignment expressions in 3 out of the 5 cases. So you shifted the goal posts from "newer languages" to "industry standard". That was YOUR choice, not mine. Based on the TIOBE language popularity rankings, there is no overlap between "industry standard" and "newer" languages. Not even Go or Swift. Delphi/Pascal is more popular than Go, unless you happen to work for Google. If you disagree about using TIOBE, feel free to choose your own objective and neutral ranking system, so long as it is better than "I've heard of these five languages, so they're industry standard". In 2016 Liefehacker came up with this list, based on combining rankings from TIOBE, PYPL and various job sites: Java, C, Python, C++, JavaScript, C#, PHP, Swift, Objective-C, R https://www.lifehacker.com.au/2016/12/the-most-popular-programming-languages-based-on-jobs-search-engines-and-more/ only one of which is in your list of new languages, and that one (Swift) is one of those which includes assignment as an expression. Looking at job rankings: https://www.codingdojo.com/blog/9-most-in-demand-programming-languages-of-2017/ https://www.codingdojo.com/blog/7-most-in-demand-programming-languages-of-2018/ and you won't find any new languages. [...] > The point being to find momentum in design of newer languages. Your list > of older languages is therefore not pertinent. Then you shouldn't have raised "industry standard" as relevant. If we limit the list to new AND "industry buzz", then we get a sample space of *two*: Swift and Go, both of which have some form of assignment expressions: - Swift has assignment expressions; - Go allows assignment in if statements, but not while statements. That's not a big sample to go by, and it is subject to Rob Pike's idiosyncratic design (not necessary bad, but bucking language trends). -- Steve From tim.peters at gmail.com Sun May 20 19:49:22 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 20 May 2018 18:49:22 -0500 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <5B02005F.8010003@canterbury.ac.nz> References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520053549.GD12683@ando.pearwood.info> <5B02005F.8010003@canterbury.ac.nz> Message-ID: [Tim, on Rebol/Red] >> >> x: 12 y: 13 >> == 13 >> >> [x y] >> == [x y] >> >> >> >> do [x y] >> == 13 [Greg Ewing ] > How does scoping work? If you pass a block to a function > which evaluates it, how are names in the block resolved? Too involved, but basically a form of lexical scoping. Rebol struggled with this, and details vary by version. Here are the last docs for version 2: http://www.rebol.com/r3/docs/concepts/funcs-scope.html Details changed again for version 3. Best I can tell, Red hasn't written up its own rules yet. The model did _not_ support closures naturally (in the sense of local bindings persisting beyond a function's return). Version 3 added an alternative to `func`, named `closure`, which could be used when you wanted a closure: http://www.rebol.com/r3/docs/datatypes/closure.html But I noticed just now that Red doesn't have `closure` built in, and its funcs don't support closures naturally either: >> adder: func [x] [func [y] [x + y]] == func [x][func [y] [x + y]] >> add3: adder 3 == func [y][x + y] >> add3 12 *** Script Error: x is not in the specified context *** Where: add3 *** Stack: add3 That's not dynamic scoping, either - it's still unhappy if `x` is defined at top level: >> x: 18 == 18 >> add3 12 *** Script Error: x is not in the specified context *** Where: add3 *** Stack: add3 From apalala at gmail.com Sun May 20 20:55:20 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Sun, 20 May 2018 20:55:20 -0400 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180520231059.GE12683@ando.pearwood.info> References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520053549.GD12683@ando.pearwood.info> <20180520231059.GE12683@ando.pearwood.info> Message-ID: > > print(child_pid) # child_pid is undefined > > Why on earth would it be undefined? > Indeed, users would expect the new uses of "as" to behave as the previous ones. The problem is that "with" and "except" do things differently: In [*1*]: *import* *os* In [*2*]: *with* open(os.path.expanduser('~/tmp/xx')) *as* f: ...: *pass* ...: In [*3*]: print(f) <_io.TextIOWrapper name='/Users/apalala/tmp/xx' mode='r' encoding='UTF-8'> In [*4*]: *try*: ...: print('ok') ...: *raise* *Exception*() ...: *except* *Exception* *as* e: ...: *pass* ...: In [*5*]: print(e) --------------------------------------------------------------------------- NameError Traceback (most recent call last) in () ----> 1 print(e) NameError: name 'e' is not defined > Anyway, if you want to propose an alternative to PEP 572, you ought to > write your own competing PEP. > I don't take that as a challenge. It would be good to have different, maybe somewhat competing PEPs. It's been done before. -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhodri at kynesim.co.uk Mon May 21 06:58:28 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Mon, 21 May 2018 11:58:28 +0100 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: Message-ID: <5e2b83d4-45ae-d31b-509f-683dd24f0d22@kynesim.co.uk> On 20/05/18 06:19, Matt Arcidy wrote: > On Sat, May 19, 2018, 11:07 Kirill Balunov wrote: > >> >> >> I think I have a very strong argument "why are not others valid" - Because >> already three months have passed and among 1300+ messages there was not a >> single real example where assignment expression would be convenient or >> useful outside `while` and `if` statements. If you have a counterargument >> (with an example), I would be glad to see. >> > > comprehensions. I don't think you read any of the messages. about 1000 > were solely about comprehensions. I'm afraid I haven't seen a convincing example of an useful assignment expression in a comprehension that wasn't also a convincing example of things that shouldn't have been comprehensions in the first place. -- Rhodri James *-* Kynesim Ltd From rhodri at kynesim.co.uk Mon May 21 07:05:40 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Mon, 21 May 2018 12:05:40 +0100 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: Message-ID: <2412f419-2506-a53d-02e2-cba4c62495a7@kynesim.co.uk> On 19/05/18 01:54, Mike Miller wrote: > In short, extend the "if/elif", "while", and comprehension to: > > ??? if pattern.search(data) as match: > ??????? ? > > ??? while read_next_item() as value: > ??????? ? Thanks for the analysis, but I'm afraid I must disagree with your recommendation. It was the thought I first had when Chris came out with his first draft of the PEP several months ago, but it's not enough to cope with my usual use cases. What I normally want is the Python equivalent of: while ((v = get_something()) != INCONVENIENT_SENTINEL) do_something(v); The condition expression itself is not what I want to capture; I need a subexpression, which the "as" syntax won't give me. -- Rhodri James *-* Kynesim Ltd From dmoisset at machinalis.com Mon May 21 07:29:36 2018 From: dmoisset at machinalis.com (Daniel Moisset) Date: Mon, 21 May 2018 12:29:36 +0100 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <2412f419-2506-a53d-02e2-cba4c62495a7@kynesim.co.uk> References: <2412f419-2506-a53d-02e2-cba4c62495a7@kynesim.co.uk> Message-ID: On 21 May 2018 at 12:05, Rhodri James wrote: > > Thanks for the analysis, but I'm afraid I must disagree with your > recommendation. It was the thought I first had when Chris came out with > his first draft of the PEP several months ago, but it's not enough to cope > with my usual use cases. What I normally want is the Python equivalent of: > > while ((v = get_something()) != INCONVENIENT_SENTINEL) > do_something(v); > > The condition expression itself is not what I want to capture; I need a > subexpression, which the "as" syntax won't give me. > > That use case should be covered by for v in iter(get_something, INCOVENIENT_SENTINEL): do_something(v) -- Daniel Moisset UK COUNTRY MANAGER A: 1 Fore Street, EC2Y 9DT London P: +44 7398 827139 <+44+7398+827139> M: dmoisset at machinalis.com | S: dmoisset Machinalis Limited is a company registered in England and Wales. Registered number: 10574987. -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcidy at gmail.com Mon May 21 07:37:50 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Mon, 21 May 2018 04:37:50 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <5e2b83d4-45ae-d31b-509f-683dd24f0d22@kynesim.co.uk> References: <5e2b83d4-45ae-d31b-509f-683dd24f0d22@kynesim.co.uk> Message-ID: On Mon, May 21, 2018, 03:58 Rhodri James wrote: > On 20/05/18 06:19, Matt Arcidy wrote: > > On Sat, May 19, 2018, 11:07 Kirill Balunov > wrote: > > > >> > >> > >> I think I have a very strong argument "why are not others valid" - > Because > >> already three months have passed and among 1300+ messages there was not > a > >> single real example where assignment expression would be convenient or > >> useful outside `while` and `if` statements. If you have a > counterargument > >> (with an example), I would be glad to see. > >> > > > > comprehensions. I don't think you read any of the messages. about 1000 > > were solely about comprehensions. > > I'm afraid I haven't seen a convincing example of an useful assignment > expression in a comprehension that wasn't also a convincing example of > things that shouldn't have been comprehensions in the first place. > I haven't seen a convincing counter argument that doesn't axiomatically depend on personal opinion. Can you define your "shouldn't" in a way that isn't just your personal taste? I can't know from your statement here. > -- > Rhodri James *-* Kynesim Ltd > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Mon May 21 08:22:46 2018 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Mon, 21 May 2018 08:22:46 -0400 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <2412f419-2506-a53d-02e2-cba4c62495a7@kynesim.co.uk> References: <2412f419-2506-a53d-02e2-cba4c62495a7@kynesim.co.uk> Message-ID: > while ((v = get_something()) != INCONVENIENT_SENTINEL) > do_something(v); > The current pattern in Python would be something like: v = get_something() while v != INCONVENIENT_SENTINEL: do_something(v) v = get_something() With "as" allowed in "while", they pattern might be: while get_something() as v: if v == INCONVENIENT_SENTINEL: break do_something(v) The discussion isn't over, so it could also be: while (get_something() as v) != INCONVENIENT_SENTINEL: do_something(v) Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.input at gmail.com Mon May 21 08:37:36 2018 From: carl.input at gmail.com (Carl Smith) Date: Mon, 21 May 2018 13:37:36 +0100 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <2412f419-2506-a53d-02e2-cba4c62495a7@kynesim.co.uk> Message-ID: v = get_something() while v != INCONVENIENT_SENTINEL: do_something(v) v = get_something() I'd personally go with: while True: v = get_something() if v != INCONVENIENT_SENTINEL: break do_something(v) But it's not much different. I'd really like to be able to use jump statements in ternary expressions, like: do_something(v) But that's another story. -- Carl Smith carl.input at gmail.com On 21 May 2018 at 13:22, Juancarlo A?ez wrote: > > while ((v = get_something()) != INCONVENIENT_SENTINEL) >> do_something(v); >> > > > The current pattern in Python would be something like: > > v = get_something() > > while v != INCONVENIENT_SENTINEL: > > do_something(v) > > v = get_something() > > With "as" allowed in "while", they pattern might be: > > while get_something() as v: > > if v == INCONVENIENT_SENTINEL: > > break > > do_something(v) > > > The discussion isn't over, so it could also be: > > while (get_something() as v) != INCONVENIENT_SENTINEL: > > do_something(v) > > > Cheers, > > -- > Juancarlo *A?ez* > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.input at gmail.com Mon May 21 08:38:34 2018 From: carl.input at gmail.com (Carl Smith) Date: Mon, 21 May 2018 13:38:34 +0100 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <2412f419-2506-a53d-02e2-cba4c62495a7@kynesim.co.uk> Message-ID: Sorry, hit send by accident. I meant to say: do_something(v) *if* v != INCONVENIENT_SENTINEL *else break* -- Carl Smith carl.input at gmail.com On 21 May 2018 at 13:37, Carl Smith wrote: > v = get_something() > > while v != INCONVENIENT_SENTINEL: > > do_something(v) > > v = get_something() > > > I'd personally go with: > > while True: > v = get_something() > if v != INCONVENIENT_SENTINEL: break > do_something(v) > > But it's not much different. I'd really like to be able to use jump > statements > in ternary expressions, like: > > do_something(v) > > But that's another story. > > -- Carl Smith > carl.input at gmail.com > > On 21 May 2018 at 13:22, Juancarlo A?ez wrote: > >> >> while ((v = get_something()) != INCONVENIENT_SENTINEL) >>> do_something(v); >>> >> >> >> The current pattern in Python would be something like: >> >> v = get_something() >> >> while v != INCONVENIENT_SENTINEL: >> >> do_something(v) >> >> v = get_something() >> >> With "as" allowed in "while", they pattern might be: >> >> while get_something() as v: >> >> if v == INCONVENIENT_SENTINEL: >> >> break >> >> do_something(v) >> >> >> The discussion isn't over, so it could also be: >> >> while (get_something() as v) != INCONVENIENT_SENTINEL: >> >> do_something(v) >> >> >> Cheers, >> >> -- >> Juancarlo *A?ez* >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhodri at kynesim.co.uk Mon May 21 09:14:32 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Mon, 21 May 2018 14:14:32 +0100 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <2412f419-2506-a53d-02e2-cba4c62495a7@kynesim.co.uk> Message-ID: On 21/05/18 12:29, Daniel Moisset wrote: > On 21 May 2018 at 12:05, Rhodri James wrote: > >> >> Thanks for the analysis, but I'm afraid I must disagree with your >> recommendation. It was the thought I first had when Chris came out with >> his first draft of the PEP several months ago, but it's not enough to cope >> with my usual use cases. What I normally want is the Python equivalent of: >> >> while ((v = get_something()) != INCONVENIENT_SENTINEL) >> do_something(v); >> >> The condition expression itself is not what I want to capture; I need a >> subexpression, which the "as" syntax won't give me. >> >> > That use case should be covered by > > for v in iter(get_something, INCOVENIENT_SENTINEL): > do_something(v) There are many ways round my use case, all of them inelegant. That has to be one of the less comprehensible alternatives. -- Rhodri James *-* Kynesim Ltd From rhodri at kynesim.co.uk Mon May 21 09:19:08 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Mon, 21 May 2018 14:19:08 +0100 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <2412f419-2506-a53d-02e2-cba4c62495a7@kynesim.co.uk> Message-ID: On 21/05/18 13:22, Juancarlo A?ez wrote: >> while ((v = get_something()) != INCONVENIENT_SENTINEL) >> do_something(v); >> > > > The current pattern in Python would be something like: > > v = get_something() > > while v != INCONVENIENT_SENTINEL: > > do_something(v) > > v = get_something() Actually more usually while True: v = get_something() if v == INCONVENIENT_SENTINEL: break do_something(v) Inelegant and, as has been pointed out, frankly misleading about the nature of the loop, but at least you can tell what's going on fairly straightforwardly. > With "as" allowed in "while", they pattern might be: > > while get_something() as v: > > if v == INCONVENIENT_SENTINEL: > > break > > do_something(v) > > > The discussion isn't over, so it could also be: > > while (get_something() as v) != INCONVENIENT_SENTINEL: > > do_something(v) These two are somewhat different things. -- Rhodri James *-* Kynesim Ltd From carl.input at gmail.com Mon May 21 09:21:58 2018 From: carl.input at gmail.com (Carl Smith) Date: Mon, 21 May 2018 14:21:58 +0100 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <2412f419-2506-a53d-02e2-cba4c62495a7@kynesim.co.uk> Message-ID: > > for v in iter(get_something, INCOVENIENT_SENTINEL): > do_something(v) > > There are many ways round my use case, all of them inelegant. That has to > be one of the less comprehensible alternatives. In for-loops (because they include an assignment already) we can improve this with more indicatively named functions in practice. -- Carl Smith carl.input at gmail.com On 21 May 2018 at 14:14, Rhodri James wrote: > On 21/05/18 12:29, Daniel Moisset wrote: > >> On 21 May 2018 at 12:05, Rhodri James wrote: >> >> >>> Thanks for the analysis, but I'm afraid I must disagree with your >>> recommendation. It was the thought I first had when Chris came out with >>> his first draft of the PEP several months ago, but it's not enough to >>> cope >>> with my usual use cases. What I normally want is the Python equivalent >>> of: >>> >>> while ((v = get_something()) != INCONVENIENT_SENTINEL) >>> do_something(v); >>> >>> The condition expression itself is not what I want to capture; I need a >>> subexpression, which the "as" syntax won't give me. >>> >>> >>> That use case should be covered by >> >> for v in iter(get_something, INCOVENIENT_SENTINEL): >> do_something(v) >> > > There are many ways round my use case, all of them inelegant. That has to > be one of the less comprehensible alternatives. > > > -- > Rhodri James *-* Kynesim Ltd > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-ideas at mgmiller.net Mon May 21 12:43:45 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Mon, 21 May 2018 09:43:45 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180520234840.GF12683@ando.pearwood.info> References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520234840.GF12683@ando.pearwood.info> Message-ID: To clarify there were three main criteria, and one minor. Newer, popular/becoming industry standard, and designed to address shortcomings in previous generations. Finally, the limit of my energy when already working on a project. I also should have provided the link to the previous discussion. When writing the first message I indeed thought a wiki might be better, because I'd undoubtedly forget to mention, or inaccurately state something. Meta discussion: While I do prefer the "as" version of assignment, I didn't enter into the inquiry to prove it better, but rather to find industry momentum, and had only some minor exposure to go/kotlin via their tutorials beforehand. I did a similar thing when "f-string" design was debated. At first I wasn't in favor of them handling expressions, but after doing a survey of other modern/popular languages found it was working well elsewhere and changed my opinion. So it goes both ways, and I'm open to being convinced against my first preference and not heavily invested in it. -Mike From python-ideas at mgmiller.net Mon May 21 13:00:01 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Mon, 21 May 2018 10:00:01 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <2412f419-2506-a53d-02e2-cba4c62495a7@kynesim.co.uk> Message-ID: <18f36640-c743-1fd2-0ab3-d15eb2cbc550@mgmiller.net> On 2018-05-21 05:22, Juancarlo A?ez wrote: > This is a good summary of the choices. I think the second one is a good compromise. More elegant, yet avoiding the problems with assignment-expressions available everywhere. It is true that := handles more (perhaps less-common) use cases, but subjectively it is not as "Pythonic." Also doesn't appear to be the direction the surveyed languages are going. YMMV, -Mike From tjreedy at udel.edu Mon May 21 13:00:51 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 21 May 2018 13:00:51 -0400 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <2412f419-2506-a53d-02e2-cba4c62495a7@kynesim.co.uk> Message-ID: On 5/21/2018 9:14 AM, Rhodri James wrote: > On 21/05/18 12:29, Daniel Moisset wrote: >> On 21 May 2018 at 12:05, Rhodri James >> wrote: >>> with my usual use cases.? What I normally want is the Python >>> equivalent of: >>> >>> ?? while ((v = get_something()) != INCONVENIENT_SENTINEL) >>> ???? do_something(v); >> That use case should be covered by >> >> for v in iter(get_something, INCOVENIENT_SENTINEL): >> ???? do_something(v) > > There are many ways round my use case, all of them inelegant.? That has > to be one of the less comprehensible alternatives. The following might be slightly clearer. item_iterator = iter(get_something, INCONVENIENT_SENTINEL) for item in item_iterator: do_something(item) I think iter is the right thing to use here. I think your contrary response is a least partly due to unfamiliarity with the two-argument form of iter and how it abbreviates and encapsulates the alternatives you already know. The essential idea of 'iter' is to produce iterators from Python objects. When the input is a function, sentinel pair, iter returns an instance of a internal callable_iterator class, equivalent to class callable_iterator def __init__(self, function, sentinel) self.function = function self.sentinel = sentinel def __iter__(self): return self def __next__(self): item = self.function() if item == self.sentinel: raise StopIteration return item If iter were actually written in Python, it might instead return the generator returned by a generator function encapsulating one of well-know while loop idioms. # Double call form. def function_sentinel(function, sentinel) item = function() while item != sentinel yield item item = function() # Loop and a half form; the loop body is essentially the same # as the body of callable_iterator.__next__, above def function_sentinel(function, sentinel): while True: item = function() # No 'self.' if item == sentinel: # No 'self.' break # Instead of 'raise StopIteration yield item # Instead of 'return item' The essential idea of for-loops is to cleanly separate sequential production of items to be processed, in the header, from processing of each item, in the body. It always calls iter(iterable) for you. If you need to pass two arguments to iter, you must do it explicitly. The while alternatives for this case intermix getting and processing items in the body. -- Terry Jan Reedy From chris.barker at noaa.gov Mon May 21 14:22:17 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 21 May 2018 11:22:17 -0700 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: <20180519135229.GX12683@ando.pearwood.info> References: <20180517122043.GF12683@ando.pearwood.info> <20180517131321.GH12683@ando.pearwood.info> <5AFE6DAD.4000203@canterbury.ac.nz> <20180519135229.GX12683@ando.pearwood.info> Message-ID: On Sat, May 19, 2018 at 6:52 AM, Steven D'Aprano wrote: > Philosophical arguments about the nature of computer memory aside, byte > objects in Python are collections of ints. > not when you start talking about bit-wise operations :-) If a "byte" in python was an integer, then we'd use b**2 rather than b << 1 (yes, I know bit shifting can be more efficient, but I don't think that's why it's there in python) The entire point of bitwise operators is so that the bits themselves can be accessed and manipulated. > If you want those ints to represent something else, you're responsible > for handling that (say, using the struct module). yup -- with struct, and, hmm, maybe bitwise operators? Anyway, as you say, this is a Philosophical (or semantic) point -- I don't think it effects the discussion at hand. However, when you talk about bit-shifting a bytes object, you do need to decide if each byte is handled individually, or if they are one big collection. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon May 21 15:11:57 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 22 May 2018 05:11:57 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520234840.GF12683@ando.pearwood.info> Message-ID: On Tue, May 22, 2018 at 2:43 AM, Mike Miller wrote: > To clarify there were three main criteria, and one minor. Newer, > popular/becoming industry standard, and designed to address shortcomings in > previous generations. Finally, the limit of my energy when already working > on a project. Note how not one of your criteria says that the language has to have made the right choice - only that it's made a DIFFERENT choice to languages in a previous generation. So you're heavily selecting in favour of languages that say "hey look, we know better than everyone else does", without actually proving that it's better. There's no shortage of language designers who say "oh look how terrible C is", and when a language designer has a lot of clout behind him (like "apps for this new mobile phone should be written using this language"), it can get extremely popular among people who don't actually have much choice of language - and certainly don't have the option to use C. Much more useful would be to look at languages that (a) work in a field where programmers have ample freedom to choose between languages, and (b) have been around long enough to actually demonstrate that people want to use them. Look through the Stack Overflow Developer Survey's report on languages: https://insights.stackoverflow.com/survey/2018/#most-loved-dreaded-and-wanted A "Wanted" language is one that many developers say "I don't currently use, but I would like to". (It may also be a language that has murdered semicolons. I believe the bounty on JavaScript's head is quite high now.) Go through that list and you'll get an idea of what people wish they could use; then toss out anything that hasn't been around for at least 10 years, because there's a tendency for new technologies to be over-represented in a "Wanted" listing (partly because fewer programmers already know them, and partly because people want to try the latest toys). That may give you a better list of languages to compare against. ChrisA From carl.input at gmail.com Mon May 21 17:37:39 2018 From: carl.input at gmail.com (Carl Smith) Date: Mon, 21 May 2018 22:37:39 +0100 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520234840.GF12683@ando.pearwood.info> Message-ID: Chris makes a lot of good points regarding *which* languages to look at, but it seems like that line of enquiry is unlikely to suggest anything more than it has so far, especially if we're limiting it to languages everyone has heard of. They either use a keyword, an operator, don't support the feature or they're wacky. If anything, the survey says we need to think outside the box. -- Carl Smith carl.input at gmail.com On 21 May 2018 at 20:11, Chris Angelico wrote: > On Tue, May 22, 2018 at 2:43 AM, Mike Miller > wrote: > > To clarify there were three main criteria, and one minor. Newer, > > popular/becoming industry standard, and designed to address shortcomings > in > > previous generations. Finally, the limit of my energy when already > working > > on a project. > > Note how not one of your criteria says that the language has to have > made the right choice - only that it's made a DIFFERENT choice to > languages in a previous generation. So you're heavily selecting in > favour of languages that say "hey look, we know better than everyone > else does", without actually proving that it's better. There's no > shortage of language designers who say "oh look how terrible C is", > and when a language designer has a lot of clout behind him (like "apps > for this new mobile phone should be written using this language"), it > can get extremely popular among people who don't actually have much > choice of language - and certainly don't have the option to use C. > > Much more useful would be to look at languages that (a) work in a > field where programmers have ample freedom to choose between > languages, and (b) have been around long enough to actually > demonstrate that people want to use them. Look through the Stack > Overflow Developer Survey's report on languages: > > https://insights.stackoverflow.com/survey/2018/ > #most-loved-dreaded-and-wanted > > A "Wanted" language is one that many developers say "I don't currently > use, but I would like to". (It may also be a language that has > murdered semicolons. I believe the bounty on JavaScript's head is > quite high now.) Go through that list and you'll get an idea of what > people wish they could use; then toss out anything that hasn't been > around for at least 10 years, because there's a tendency for new > technologies to be over-represented in a "Wanted" listing (partly > because fewer programmers already know them, and partly because people > want to try the latest toys). That may give you a better list of > languages to compare against. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon May 21 19:02:57 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 22 May 2018 09:02:57 +1000 Subject: [Python-ideas] String and bytes bitwise operations In-Reply-To: References: <20180517122043.GF12683@ando.pearwood.info> <20180517131321.GH12683@ando.pearwood.info> <5AFE6DAD.4000203@canterbury.ac.nz> <20180519135229.GX12683@ando.pearwood.info> Message-ID: <20180521230257.GG12683@ando.pearwood.info> On Mon, May 21, 2018 at 11:22:17AM -0700, Chris Barker wrote: > On Sat, May 19, 2018 at 6:52 AM, Steven D'Aprano > wrote: > > > Philosophical arguments about the nature of computer memory aside, byte > > objects in Python are collections of ints. > > > > not when you start talking about bit-wise operations :-) > > If a "byte" in python was an integer, then we'd use b**2 rather than b << 1 Why? Ints support bit-shift operations: py> 45 << 1 90 as well as other bitwise operations. (And b<<1 isn't the same as b**2.) I trust that you aren't going to argue that 45 isn't an int. > (yes, I know bit shifting can be more efficient, but I don't think that's > why it's there in python) > > The entire point of bitwise operators is so that the bits themselves can be > accessed and manipulated. Then where are the primitives for accessing and manipulating individual bits? > > If you want those ints to represent something else, you're responsible > > for handling that (say, using the struct module). > > > yup -- with struct, and, hmm, maybe bitwise operators? Right. Which is why we want to add support for bitwise operators to bytes. > Anyway, as you say, this is a Philosophical (or semantic) point -- I don't > think it effects the discussion at hand. > > However, when you talk about bit-shifting a bytes object, you do need to > decide if each byte is handled individually, or if they are one big > collection. True, we have to decide that. Is there a good reason to handle each byte individually? -- Steve From carl.input at gmail.com Mon May 21 19:22:23 2018 From: carl.input at gmail.com (Carl Smith) Date: Tue, 22 May 2018 00:22:23 +0100 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520234840.GF12683@ando.pearwood.info> Message-ID: If the ability to capture a subexpression is given up on, then we are are down to just capturing the value of the predicate in if statements. In loops, it is only the predicate and iteration index. If you mashup `if` and `def`, you end up with this mess: if get_something() def (value): do_something(value) And `while` and `def` is no prettier: while get_something() def (value, index): do_something(value) Both ugly, and totally misleading, but being able to accept the values as params is nice, and the parenthesis make things clearer and more Pythonic IMO, so maybe just use `as` but require the names be parenthesised: if predicate as (value): use(value) while predicate as (value, index): use(value) The next options may be a bit too functional-looking, when the suite is not a function body, but I thought they read better: while predicate pass (value, index): use(value) if predicate -> (value): use(value) while predicate yield (value, index): use(value) The main idea is to reuse comma separated params in parens to make the assignment clearer and to look more like the `def` and `class` grammar. -- Carl Smith carl.input at gmail.com On 21 May 2018 at 22:37, Carl Smith wrote: > Chris makes a lot of good points regarding *which* languages to look at, > but > it seems like that line of enquiry is unlikely to suggest anything more > than it > has so far, especially if we're limiting it to languages everyone has > heard of. > They either use a keyword, an operator, don't support the feature or > they're > wacky. > > If anything, the survey says we need to think outside the box. > > > -- Carl Smith > carl.input at gmail.com > > On 21 May 2018 at 20:11, Chris Angelico wrote: > >> On Tue, May 22, 2018 at 2:43 AM, Mike Miller >> wrote: >> > To clarify there were three main criteria, and one minor. Newer, >> > popular/becoming industry standard, and designed to address >> shortcomings in >> > previous generations. Finally, the limit of my energy when already >> working >> > on a project. >> >> Note how not one of your criteria says that the language has to have >> made the right choice - only that it's made a DIFFERENT choice to >> languages in a previous generation. So you're heavily selecting in >> favour of languages that say "hey look, we know better than everyone >> else does", without actually proving that it's better. There's no >> shortage of language designers who say "oh look how terrible C is", >> and when a language designer has a lot of clout behind him (like "apps >> for this new mobile phone should be written using this language"), it >> can get extremely popular among people who don't actually have much >> choice of language - and certainly don't have the option to use C. >> >> Much more useful would be to look at languages that (a) work in a >> field where programmers have ample freedom to choose between >> languages, and (b) have been around long enough to actually >> demonstrate that people want to use them. Look through the Stack >> Overflow Developer Survey's report on languages: >> >> https://insights.stackoverflow.com/survey/2018/#most-loved- >> dreaded-and-wanted >> >> A "Wanted" language is one that many developers say "I don't currently >> use, but I would like to". (It may also be a language that has >> murdered semicolons. I believe the bounty on JavaScript's head is >> quite high now.) Go through that list and you'll get an idea of what >> people wish they could use; then toss out anything that hasn't been >> around for at least 10 years, because there's a tendency for new >> technologies to be over-represented in a "Wanted" listing (partly >> because fewer programmers already know them, and partly because people >> want to try the latest toys). That may give you a better list of >> languages to compare against. >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon May 21 19:40:03 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 22 May 2018 09:40:03 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520234840.GF12683@ando.pearwood.info> Message-ID: <20180521234003.GH12683@ando.pearwood.info> On Mon, May 21, 2018 at 09:43:45AM -0700, Mike Miller wrote: > To clarify there were three main criteria, and one minor. Newer, > popular/becoming industry standard, and designed to address shortcomings in > previous generations. Finally, the limit of my energy when already working > on a project. To take your criteria in reverse order: 1. Why should the limit on *your* energy be a deciding factor? I was willing to spend a few hours doing a more complete sample of new languages. Should we ignore those because you ran out of energy? If anyone else wants to extend it even further, and survey more languages, we should welcome an even more extensive survey. 2. By definition, EVERY new language is designed to "address shortcomings in previous generations" of languages. If the designer thought previous languages were ideal, they wouldn't invent a new language. 3. None of the languages you surveyed are "popular/becoming industry standard" according to the closest thing we have to an objective measure of popularity: rankings like those provided by TIOBE. If you don't like TIOBE's methodology, feel free to propose a different, neutral, ranking. If you want a subjective opinion based on "industry buzz", then I would say that out of the five languages you listed, only two (Go and Swift) are anything close to "becoming industry standard", a tiny sample indeed, but nevertheless one where both languages have some form of assignment expressions (Go only in "if" statements, Swift everywhere). 4. What was your criteria for "newer"? I must admit I assumed it was languages invented since 2010, but I see Go was invented in 2009. If your cutoff is 2009, then we ought to include Coffeescript, which also has assignment expressions. -- Steve From steve at pearwood.info Mon May 21 19:50:55 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 22 May 2018 09:50:55 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <18f36640-c743-1fd2-0ab3-d15eb2cbc550@mgmiller.net> References: <2412f419-2506-a53d-02e2-cba4c62495a7@kynesim.co.uk> <18f36640-c743-1fd2-0ab3-d15eb2cbc550@mgmiller.net> Message-ID: <20180521235055.GI12683@ando.pearwood.info> On Mon, May 21, 2018 at 10:00:01AM -0700, Mike Miller wrote: > It is true that := handles more (perhaps less-common) use cases, but > subjectively it is not as "Pythonic." Also doesn't appear to be the > direction the surveyed languages are going. Your conclusion does not follow even from the limited sample of languages you looked at. Out of the five languages you surveyed, three include some form of assignment expressions. It completely collapses if we look at a more extensive sample of languages dating back to 2009. If we limit ourselves to the three clear "winners" since 2009 (languages which have significant, if still niche, popularity/relevance according to TIOBE), we find all three: Coffeescript, Go, Swift have some form of assignment expressions. Go's is very limited: only in "if" statements. Coffeescript and Swift allow assignment expressions anywhere. -- Steve From python-ideas at mgmiller.net Mon May 21 20:31:43 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Mon, 21 May 2018 17:31:43 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180521235055.GI12683@ando.pearwood.info> References: <2412f419-2506-a53d-02e2-cba4c62495a7@kynesim.co.uk> <18f36640-c743-1fd2-0ab3-d15eb2cbc550@mgmiller.net> <20180521235055.GI12683@ando.pearwood.info> Message-ID: <17074357-93b1-ee32-5808-3cb03ea2c738@mgmiller.net> On 2018-05-21 16:50, Steven D'Aprano wrote: > Your conclusion does not follow even from the limited sample of They're not going in the direction of assignment-expressions everywhere, but rather one built in to the if/while statement. Coffeescript could certainly be a good candidate, though falling out of favor to typescript. Swift may allow them but are not useful when returning Void. -Mike From python-ideas at mgmiller.net Mon May 21 20:45:34 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Mon, 21 May 2018 17:45:34 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180521234003.GH12683@ando.pearwood.info> References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520234840.GF12683@ando.pearwood.info> <20180521234003.GH12683@ando.pearwood.info> Message-ID: On 2018-05-21 16:40, Steven D'Aprano wrote: Consider the link Chris sent above: https://insights.stackoverflow.com/survey/2018/#most-loved-dreaded-and-wanted The top six coincide with my list, plus TypeScript (superset of JS) and Python. I'm pretty happy with those chosen, considering. -Mike From rosuav at gmail.com Mon May 21 20:48:11 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 22 May 2018 10:48:11 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520234840.GF12683@ando.pearwood.info> <20180521234003.GH12683@ando.pearwood.info> Message-ID: On Tue, May 22, 2018 at 10:45 AM, Mike Miller wrote: > > On 2018-05-21 16:40, Steven D'Aprano wrote: > > Consider the link Chris sent above: > > https://insights.stackoverflow.com/survey/2018/#most-loved-dreaded-and-wanted > > The top six coincide with my list, plus TypeScript (superset of JS) and > Python. I'm pretty happy with those chosen, considering. > But as I mentioned, the "wanted" list over-represents the new languages. For it to be useful to this discussion, you have to *reject* anything less than about ten years old. ChrisA From leewangzhong+python at gmail.com Mon May 21 21:09:37 2018 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Mon, 21 May 2018 21:09:37 -0400 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180521234003.GH12683@ando.pearwood.info> References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520234840.GF12683@ando.pearwood.info> <20180521234003.GH12683@ando.pearwood.info> Message-ID: On Mon, May 21, 2018 at 7:40 PM, Steven D'Aprano wrote: > On Mon, May 21, 2018 at 09:43:45AM -0700, Mike Miller wrote: >> To clarify there were three main criteria, and one minor. Newer, >> popular/becoming industry standard, and designed to address shortcomings in >> previous generations. Finally, the limit of my energy when already working >> on a project. > > To take your criteria in reverse order: What acknowledgement do you want him to make? That your list is better? I'm not sure what's the point of continuing this line of questioning. Personally, I don't think we should restrict the survey to newer languages. New changes to older languages are also relevant. C++17 is adding initialization syntax to `if`. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0305r0.html Why they're doing that, when C++ has assignment expressions, may or may not be relevant to this discussion. > 1. Why should the limit on *your* energy be a deciding factor? I was > willing to spend a few hours doing a more complete sample of new > languages. Should we ignore those because you ran out of energy? If > anyone else wants to extend it even further, and survey more languages, > we should welcome an even more extensive survey. It should not. But it is. He's simply admitting it. > 2. By definition, EVERY new language is designed to "address > shortcomings in previous generations" of languages. If the designer > thought previous languages were ideal, they wouldn't invent a new > language. Perhaps Mark wants to say that these languages were meant to replace particular languages. Each of his examples listed languages that they were meant to replace, and those replaced languages aren't exactly esoteric, academic, or whatever Perl is. They're languages by industry giants. > 3. None of the languages you surveyed are "popular/becoming industry > standard" according to the closest thing we have to an objective > measure of popularity: rankings like those provided by TIOBE. If you > don't like TIOBE's methodology, feel free to propose a different, > neutral, ranking. > > If you want a subjective opinion based on "industry buzz", then I would > say that out of the five languages you listed, only two (Go and Swift) > are anything close to "becoming industry standard", a tiny sample > indeed, but nevertheless one where both languages have some form of > assignment expressions (Go only in "if" statements, Swift everywhere). "Industry _standard_" may be the wrong term. Except for maybe Kotlin, the languages are created and backed by major players in the industry: Apple, Mozilla, and Google. > 4. What was your criteria for "newer"? I must admit I assumed it > was languages invented since 2010, but I see Go was invented in > 2009. Current top 50 programming languages, filtered by year >= 2000. (Years are from quick Google and Wikipedia lookups.) 6 Visual Basic .NET 2001 14 Go 2009 19 Swift 2014 20 Scala 2004 21 Apex 2006 26 Dart 2011 31 D 2001 33 Scratch 2002 (2007) 37 Clojure 2007 41 OpenCL 2009 46 Julia 2012 49 Kotlin 2011 Possible removals: - Some might argue that Visual Basic .NET is just a rebranding of Visual Basic, an older language. - Scratch looks like it's a language for teaching, not using, and I'm not sure why it's on the list. - I don't know whether to remove Apex, since it's specific to one platform and it sounds like it's almost Java. - Similarly, OpenCL sounds like it's just a fork of C/C++ for GPUs. With those languages removed, and reverse-sorted by year: 19 Swift 2014 46 Julia 2012 26 Dart 2011 49 Kotlin 2011 14 Go 2009 37 Clojure 2007 20 Scala 2004 31 D 2001 Mark's list captures four of the first five, while Rust is nowhere to be seen. Without Julia, "publicized in the last ten years" describes the split pretty well. Clojure and Scala may be outside of Mark's experience. They are functional languages, with at least a lean toward pure functions. Clojure, being a Lisp, probably uses `let` for name-binding, and I assume you'd make a macro for assign-and-compare if you want it, so it might not be informative for Python's decision. In Scala, pretty much everything is an expression (as it is in Lisp), so it might also be uninformative . D is much older than the others, but given its design goals, it might be interesting to look at anyway. That leaves Julia. > If your cutoff is 2009, then we ought to include Coffeescript, > which also has assignment expressions. CoffeeScript, like Scala, favors making things expression, so its decision might not be informative. CoffeeScript's grammar notes say, "[E]verything that can be an expression is one." http://coffeescript.org/v2/annotated-source/grammar.html From guido at python.org Mon May 21 21:45:36 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 21 May 2018 18:45:36 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520234840.GF12683@ando.pearwood.info> <20180521234003.GH12683@ando.pearwood.info> Message-ID: I don't know what to do with this thread. I enjoyed reading Mike's survey of what other languages do. I also enjoyed Chris's survey of what some other languages do. Then the thread veered off into several unproductive directions at once: a mini-tutorial for Rebol (or Red?), and endless bickering about which list of languages to allow in the survey. But that's not how this stuff gets decided. It's not the case that if we find that 90% of languages surveyed don't have inline assignment that means Python should not have it either. So changing the criteria for inclusion in the survey is entirely unproductive (except in that it may create a strong bonding experience between participants in the discussion -- though in this case perhaps the forces involved may be better described as repulsion than attraction :-). *Perhaps* it would help if we could ask those languages' designers for their rationale (most of them are still alive :-). But nobody proposed that. And it probably wouldn't help either, because the answers of actual language designers for such questions are typically a mixture of strongly held opinion on issues that aren't decidable based on hard facts alone, and hard but incredibly subtle facts related to the rest of the language's design and implementation (including the state of tooling and the structure of the compiler used as a reference implementation). So let's just end this thread. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Mon May 21 21:45:58 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Mon, 21 May 2018 18:45:58 -0700 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520234840.GF12683@ando.pearwood.info> Message-ID: <5B037656.1070700@brenbarn.net> On 2018-05-21 12:11, Chris Angelico wrote: > Much more useful would be to look at languages that (a) work in a > field where programmers have ample freedom to choose between > languages, and (b) have been around long enough to actually > demonstrate that people want to use them. Look through the Stack > Overflow Developer Survey's report on languages: > > https://insights.stackoverflow.com/survey/2018/#most-loved-dreaded-and-wanted > > A "Wanted" language is one that many developers say "I don't currently > use, but I would like to". (It may also be a language that has > murdered semicolons. I believe the bounty on JavaScript's head is > quite high now.) Go through that list and you'll get an idea of what > people wish they could use; then toss out anything that hasn't been > around for at least 10 years, because there's a tendency for new > technologies to be over-represented in a "Wanted" listing (partly > because fewer programmers already know them, and partly because people > want to try the latest toys). That may give you a better list of > languages to compare against. I'd say that also has limited usefulness. The problem is that people may "want" to learn a language for many reasons, and "the language made good design choices" is only one such reason. A lot of people may want to use JavaScript because it's hip or in demand or because they can (or think they can) make money with it. But I'm not so interested in that. What interests me is: what are the languages that people specifically believe are superior to other languages *in design*? (Even better would be what are the languages that actually ARE superior, in some reasonably nonsubjective, definable, way, but we have even less data on that.) -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From rosuav at gmail.com Mon May 21 21:58:41 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 22 May 2018 11:58:41 +1000 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <5B037656.1070700@brenbarn.net> References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520234840.GF12683@ando.pearwood.info> <5B037656.1070700@brenbarn.net> Message-ID: On Tue, May 22, 2018 at 11:45 AM, Brendan Barnwell wrote: > On 2018-05-21 12:11, Chris Angelico wrote: >> >> Much more useful would be to look at languages that (a) work in a >> field where programmers have ample freedom to choose between >> languages, and (b) have been around long enough to actually >> demonstrate that people want to use them. Look through the Stack >> Overflow Developer Survey's report on languages: >> >> >> https://insights.stackoverflow.com/survey/2018/#most-loved-dreaded-and-wanted >> >> A "Wanted" language is one that many developers say "I don't currently >> use, but I would like to". (It may also be a language that has >> murdered semicolons. I believe the bounty on JavaScript's head is >> quite high now.) Go through that list and you'll get an idea of what >> people wish they could use; then toss out anything that hasn't been >> around for at least 10 years, because there's a tendency for new >> technologies to be over-represented in a "Wanted" listing (partly >> because fewer programmers already know them, and partly because people >> want to try the latest toys). That may give you a better list of >> languages to compare against. > > > I'd say that also has limited usefulness. The problem is that > people may "want" to learn a language for many reasons, and "the language > made good design choices" is only one such reason. A lot of people may want > to use JavaScript because it's hip or in demand or because they can (or > think they can) make money with it. But I'm not so interested in that. > What interests me is: what are the languages that people specifically > believe are superior to other languages *in design*? (Even better would be > what are the languages that actually ARE superior, in some reasonably > nonsubjective, definable, way, but we have even less data on that.) > What you want is virtually impossible to obtain, so we go for whatever approximations we can actually get hold of. There are a few reasons someone might want to use a language. One is "I hate this language but it'll earn me money", yes, but the way the Stack Overflow survey is worded, those ones won't come up. IIRC the wording is "which languages do you desire to be using next year", after removing the ones for which you also said that you're using them now. So there are a good few reasons that you might wish you could be using a language: 1) You believe it's a good language, worth using, but just have never gotten around to starting with it 2) You think it's really fun and awesome, and wish your employer would let you use that instead of what you currently use 3) It represents a particular coding arena that you want to get into (eg you want to get into iOS development, so you pick "Swift") 4) It's an absolutely awesome language and you have plans to learn it, but haven't executed on them yet 5) It's a new language, and you want the shinies 6) Etc, etc, etc. Generally speaking, for a language to show up in the "Most Wanted", a lot of developers have to think it's something worth knowing. After cutting out the youngest languages (which I defined as "released within the last ten years") to remove their overrepresentation, you're left with languages that, in the opinions of people who don't yet program in them, are worth learning. That's far from an exact answer to the question we really want to ask, but it's reasonably concrete (to the extent that surveys ever are). But as Guido says, this is not a popular vote among languages. It's interesting to notice what other languages are doing, but harder to pin down what's good or bad about what they're doing. ChrisA From carl.input at gmail.com Tue May 22 04:04:56 2018 From: carl.input at gmail.com (Carl Smith) Date: Tue, 22 May 2018 09:04:56 +0100 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> <20180520234840.GF12683@ando.pearwood.info> <5B037656.1070700@brenbarn.net> Message-ID: I thought this thread did a good job of establishing that looking at other languages is not going to help with introducing assignment expressions into Python. It was still interesting to read. If we can't copy from other languages (or even agree on *which* languages to copy), Python will have to do something novel or give up on this. I thought, if the thread's dead, it'd be nice to do a bit of bike-shedding on the end, but there was no appetite for that. I'm not the most sensitive guy, and don't really have a sense of how my posts are being received, but would appreciate being told if my contribution isn't especially welcome. Best, -- Carl Smith carl.input at gmail.com On 22 May 2018 at 02:58, Chris Angelico wrote: > On Tue, May 22, 2018 at 11:45 AM, Brendan Barnwell > wrote: > > On 2018-05-21 12:11, Chris Angelico wrote: > >> > >> Much more useful would be to look at languages that (a) work in a > >> field where programmers have ample freedom to choose between > >> languages, and (b) have been around long enough to actually > >> demonstrate that people want to use them. Look through the Stack > >> Overflow Developer Survey's report on languages: > >> > >> > >> https://insights.stackoverflow.com/survey/2018/ > #most-loved-dreaded-and-wanted > >> > >> A "Wanted" language is one that many developers say "I don't currently > >> use, but I would like to". (It may also be a language that has > >> murdered semicolons. I believe the bounty on JavaScript's head is > >> quite high now.) Go through that list and you'll get an idea of what > >> people wish they could use; then toss out anything that hasn't been > >> around for at least 10 years, because there's a tendency for new > >> technologies to be over-represented in a "Wanted" listing (partly > >> because fewer programmers already know them, and partly because people > >> want to try the latest toys). That may give you a better list of > >> languages to compare against. > > > > > > I'd say that also has limited usefulness. The problem is that > > people may "want" to learn a language for many reasons, and "the language > > made good design choices" is only one such reason. A lot of people may > want > > to use JavaScript because it's hip or in demand or because they can (or > > think they can) make money with it. But I'm not so interested in that. > > What interests me is: what are the languages that people specifically > > believe are superior to other languages *in design*? (Even better would > be > > what are the languages that actually ARE superior, in some reasonably > > nonsubjective, definable, way, but we have even less data on that.) > > > > What you want is virtually impossible to obtain, so we go for whatever > approximations we can actually get hold of. > > There are a few reasons someone might want to use a language. One is > "I hate this language but it'll earn me money", yes, but the way the > Stack Overflow survey is worded, those ones won't come up. IIRC the > wording is "which languages do you desire to be using next year", > after removing the ones for which you also said that you're using them > now. So there are a good few reasons that you might wish you could be > using a language: > > 1) You believe it's a good language, worth using, but just have never > gotten around to starting with it > 2) You think it's really fun and awesome, and wish your employer would > let you use that instead of what you currently use > 3) It represents a particular coding arena that you want to get into > (eg you want to get into iOS development, so you pick "Swift") > 4) It's an absolutely awesome language and you have plans to learn it, > but haven't executed on them yet > 5) It's a new language, and you want the shinies > 6) Etc, etc, etc. > > Generally speaking, for a language to show up in the "Most Wanted", a > lot of developers have to think it's something worth knowing. After > cutting out the youngest languages (which I defined as "released > within the last ten years") to remove their overrepresentation, you're > left with languages that, in the opinions of people who don't yet > program in them, are worth learning. That's far from an exact answer > to the question we really want to ask, but it's reasonably concrete > (to the extent that surveys ever are). > > But as Guido says, this is not a popular vote among languages. It's > interesting to notice what other languages are doing, but harder to > pin down what's good or bad about what they're doing. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenlhilton at gmail.com Tue May 22 05:21:17 2018 From: kenlhilton at gmail.com (Ken Hilton) Date: Tue, 22 May 2018 17:21:17 +0800 Subject: [Python-ideas] Make asyncio.get_event_loop a builtin Message-ID: Hi all, Just a simple idea I wanted to bring forth. Although I know that you get a lot more asyncio control by importing the asyncio module itself, I'd like to see a way to make simple asynchronous applications without ever importing asyncio itself. To that end, I propose making asyncio.get_event_loop() a builtin. This would enable simple things like this (example copied from websockets.readthedocs.io with slight modifications): import websockets async def hello(): async with websockets.connect('wss://echo.websocket.org') as ws: amsg = 'a message' print(f'> {amsg}') await ws.send(amsg) ret = await ws.recv() print(f'< {ret}') get_event_loop().run_until_complete(hello()) See how that never imported asyncio? I just think it would be more convenient this way. But there may be major problems that I hadn't anticipated, so as always: Thoughts? Sincerely, Ken H ilton ; -------------- next part -------------- An HTML attachment was scrubbed... URL: From julian.demille at demilletech.net Tue May 22 12:46:40 2018 From: julian.demille at demilletech.net (Julian DeMille) Date: Tue, 22 May 2018 12:46:40 -0400 Subject: [Python-ideas] Allow multiple imports from a package while preserving its namespace In-Reply-To: <5AE51FA1.1010907@canterbury.ac.nz> References: <5AE51FA1.1010907@canterbury.ac.nz> Message-ID: The fact of explicit dependency noting is why I suggested something that explicitly defines multiple imports in one line On Sat, Apr 28, 2018 at 9:28 PM Greg Ewing wrote: > Nick Coghlan wrote: > > I find the imports at the top of the file to be a nice > > catalog of external dependencies. > > Not only is it useful for human readers, it's also useful > for packaging tools such as py2exe that need to know which > modules are being used. > > I experimented once with auto-importing in PyGUI, but in > the end I dropped it, partly because of this consideration. > > There were other problems with it as well. I don't recall > all the details, but I think one issue is that any errors > resulting from an import triggered by an attribute access > get masked and turned into an AttributeError, making them > very confusing to diagnose. > > Also, importing requires acquisition of the import lock, > which could cause problems in a multithreaded environment > if it happens at unpredictable times. > > For these reasons I'm inclined to regard auto-importing > as an anti-pattern -- it seems like it should be a good > idea, but it leads to more problems than it solves. > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Thanks, Julian DeMille CEO, demilleTech, LLC This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue May 22 14:08:35 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 22 May 2018 14:08:35 -0400 Subject: [Python-ideas] Make asyncio.get_event_loop a builtin In-Reply-To: References: Message-ID: On 5/22/2018 5:21 AM, Ken Hilton wrote: > Hi all, > > Just a simple idea I wanted to bring forth. Although I know that you get > a lot more asyncio control by importing the asyncio module itself, I'd > like to see a way to make simple asynchronous applications without ever > importing asyncio itself. To that end, I propose making > asyncio.get_event_loop() a builtin. This would enable simple things like > this (example copied from websockets.readthedocs.io > with slight modifications): > > ? ? import websockets > ? ? async def hello(): > ? ? ? ? async with websockets.connect('wss://echo.websocket.org > ') as ws: > ? ? ? ? ? ? amsg = 'a message' > ? ? ? ? ? ? print(f'> {amsg}') > ? ? ? ? ? ? await ws.send(amsg) > ? ? ? ? ? ? ret = await ws.recv() > ? ? ? ? ? ? print(f'< {ret}') > ? ? get_event_loop().run_until_complete(hello()) (Should that really be 'hello()' rather than 'hello'? I don't remember.) I like the idea of making coroutines easier and use. It would make more sense to me to expose an eventloop class as a builtin, so that one would write eventloop().run_until_complete(hello) eventloop would not necessarily have to be exactly the same as the default returned by get_event_loop. Would all the asyncio eventloop methods be needed? For running coroutines, it would be even nicer to write eventloop().run(hello) Eventloop could have an .__init__ method, or be a factory function, with a 'loop' parameter. The value specifies which eventloop implementation adaptor to use. The default might be 'asyncio', with alternatives such as 'uvloop', 'tkloop' (partly prototyped), 'twisted', and others. The adaptors should all have the same api: .run method that exits on completion, or a .run (forever) method paired with .stop. -- Terry Jan Reedy From yselivanov.ml at gmail.com Tue May 22 14:22:06 2018 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 22 May 2018 14:22:06 -0400 Subject: [Python-ideas] Make asyncio.get_event_loop a builtin In-Reply-To: References: Message-ID: On Tue, May 22, 2018 at 2:09 PM Terry Reedy wrote: [..] > I like the idea of making coroutines easier and use. It would make more > sense to me to expose an eventloop class as a builtin, so that one would > write > eventloop().run_until_complete(hello) > eventloop would not necessarily have to be exactly the same as the > default returned by get_event_loop. Would all the asyncio eventloop > methods be needed? For running coroutines, it would be even nicer to write > eventloop().run(hello) We have asyncio.run() in 3.7: https://docs.python.org/3.7/library/asyncio-task.html#asyncio.run (event loop can be customized via policy). Yury From listes at salort.eu Tue May 22 15:22:43 2018 From: listes at salort.eu (Julien Salort) Date: Tue, 22 May 2018 21:22:43 +0200 Subject: [Python-ideas] Modern language design survey for "assign and compare" statements In-Reply-To: <20180520014333.GZ12683@ando.pearwood.info> References: <20180519134105.GW12683@ando.pearwood.info> <20180520014333.GZ12683@ando.pearwood.info> Message-ID: Le 20/05/2018 ? 03:43, Steven D'Aprano a ?crit?: > I've somewhat arbitrarily cut the list off at "languages ranked above 1% > on TIOBE", but we have to cut the list of somewhere. And of course in > certain specific industries the standard languages may be very > different, e.g. there are still tens of millions of lines of COBOL code > being maintained in the banking industry. And also many lines of Matlab/Octave code in experimental science. Surprisingly enough, it seems that Matlab does not support AE, but GNU Octave does. I don't know if that was ever discussed when GNU Octave was designed. octave:1> y=(x=2)+1 y =? 3 octave:2> x x =? 2 versus Matlab R2018b: >> y=(x=2)+1 ?y=(x=2)+1 ???? ? Error: Incorrect use of '=' operator. To assign a value to a variable, use '='. To compare values for equality, use '=='. Julien From J.Demeyer at UGent.be Tue May 22 17:28:02 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Tue, 22 May 2018 23:28:02 +0200 Subject: [Python-ideas] Meta-PEP about built-in functions Message-ID: <5B048B62.4080805@UGent.be> Hello, Both PEP 573 and PEP 575 deal with built-in functions. Additionally, some people (Stefan Behnel, Robert Bradshaw, Jim Pivarski and me) are currently brainstorming about a yet-to-be-written PEP to allow calling the underlying C function of a built-in function using native types (for example, a C long instead of a Python int). Think of it as analogous to the buffer protocol: the buffer protocol exposes C *data* while this would expose C *callables*. Since all these PEPs would overlap somewhat, I'm starting to wonder about the best way to organize this. Ideally, I would like some kind of "meta-PEP" where we discuss the future of built-in functions in general terms without too much details. This would be followed by several PEPs each going in detail about one specific aspect. Is there a precedent for this? What do the seasoned Python developers think? Jeroen. From kirillbalunov at gmail.com Tue May 22 17:32:36 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Wed, 23 May 2018 00:32:36 +0300 Subject: [Python-ideas] "Assignment expression" with function call-alike syntax Message-ID: Just one more variation on "assignment exression" syntax to make the list more complete :) Sorry, if something similar has already been suggested. The idea is to use function's call-like syntax in the from: `this( name = expr )`. I'm not sure that such idea will find its supporters and whether it is possible to implement it in a general way, but nevertheless. Below is a half- and slow-working prototype and some silly examples just for the sake of feeling the idea: import sys import ctypes def this(**assign): assert len(assign) == 1 [(name, value)] = assign.items() frame = sys._getframe(1) frame.f_locals[name] = value ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(frame), ctypes.c_int(0)) return value Loop-and-a-half example with `dummy` function[1]: # in global scope everything works ok since locals is globals >>> while len( this( val = dummy() ) ) >= 0: ... print(val) [0, 1] [0, 1, 2] [0, 1, 2, 3] ... # needs to set the same local name to work inside function def func(): val = ... while len( this( val = dummy() ) ) >= 0: print(val) >>> # relies on the implicit underlying function's local `x` >>> [[this(x = (x+10)), x//10, x*10] for x in [100, 200, 300]] [[110, 11, 1100], [210, 21, 2100], [310, 31, 3100]] In this way it is somewhat possible to make an assignment in `while` and `if ` headers right now, which covers 90% cases, but it is damn slow. Maybe be it is worth to make `this` magic call-alike object work fast...on the other hand does anyone like `this`? With kind regards, -gdg [1]: def dummy(ls = [0]): ls.append(ls[-1]+1) return ls -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue May 22 22:08:40 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 22 May 2018 22:08:40 -0400 Subject: [Python-ideas] Make asyncio.get_event_loop a builtin In-Reply-To: References: Message-ID: >>To that end, I propose making asyncio.get_event_loop() a builtin. I like the idea of making coroutines easier and use. I do too, but ... > > Eventloop could have an .__init__ method, or be a factory function, with a > 'loop' parameter. The value specifies which eventloop implementation > adaptor to use. The default might be 'asyncio', with alternatives such as > 'uvloop', 'tkloop' (partly prototyped), 'twisted', and others. this is a key point -- while asyncio is in the standard library, it is not intended to be THE async event loop implementation -- there are others already, and hopefully that will continue (trio looks pretty cool, for instance...) so yes to making async easier, but no to putting asycio in builtins. even the idea of a builtin EventLoop that other implementations could register with seems kinda pointless -- why not import the one you want? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From gvanrossum at gmail.com Tue May 22 23:33:05 2018 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue, 22 May 2018 20:33:05 -0700 Subject: [Python-ideas] Meta-PEP about built-in functions In-Reply-To: <5B048B62.4080805@UGent.be> References: <5B048B62.4080805@UGent.be> Message-ID: This sounds fine to me. For type hint s we did a similar thing with PEPs 482, 483 and 484. You probably want to make everyone involved a co-author. On Tue, May 22, 2018, 14:29 Jeroen Demeyer wrote: > Hello, > > Both PEP 573 and PEP 575 deal with built-in functions. Additionally, > some people (Stefan Behnel, Robert Bradshaw, Jim Pivarski and me) are > currently brainstorming about a yet-to-be-written PEP to allow calling > the underlying C function of a built-in function using native types (for > example, a C long instead of a Python int). Think of it as analogous to > the buffer protocol: the buffer protocol exposes C *data* while this > would expose C *callables*. > > Since all these PEPs would overlap somewhat, I'm starting to wonder > about the best way to organize this. Ideally, I would like some kind of > "meta-PEP" where we discuss the future of built-in functions in general > terms without too much details. This would be followed by several PEPs > each going in detail about one specific aspect. > > Is there a precedent for this? What do the seasoned Python developers > think? > > > Jeroen. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed May 23 02:05:57 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 23 May 2018 02:05:57 -0400 Subject: [Python-ideas] "Assignment expression" with function call-alike syntax In-Reply-To: References: Message-ID: On 5/22/2018 5:32 PM, Kirill Balunov wrote: > Just one more variation on "assignment exression" syntax to make the > list more complete :) Sorry, if something similar has already been > suggested. The idea is to use function's call-like syntax in the from: > `this( name = expr )`. Functions names should be verbs. For this one, 'bind' or even better, 'let' as in 'let(name=expr)' ?I'm not sure that such idea will find its > supporters and whether it is possible to implement it in a general way, > but nevertheless.?Below is a half- and slow-working prototype and some > silly examples just for the sake of feeling the idea: > > import sys > import ctypes > > def this(**assign): > ? ? assert len(assign) == 1 > ? ? [(name, value)] = assign.items() > ? ? frame = sys._getframe(1) > ? ? frame.f_locals[name] = value > ? ? ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(frame), > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ctypes.c_int(0)) > ? ? return value > > Loop-and-a-half example with `dummy` function[1]: > > # in global scope everything works ok since locals is globals > >>> while len( this( val = dummy() ) ) >= 0: > ...? ? ?print(val) > [0, 1] > [0, 1, 2] > [0, 1, 2, 3] > ? ? ... > > > # needs to set the same local name to work inside function > def func(): > ? ? val = ... > while len( this( val = dummy() ) ) >= 0: > ? ? ? ? print(val) > > > >>> # relies on the implicit underlying function's local `x` > >>> [[this(x = (x+10)), x//10, x*10] for x in [100, 200, 300]] > [[110, 11, 1100], [210, 21, 2100], [310, 31, 3100]] > > > In this way?it is somewhat possible to make an assignment in `while` and > `if`headers right now, which covers 90% cases, but it is?damn > slow.?Maybe be it is worth to make `this` magic call-alike object work > fast...on the other hand does anyone like `this`? > > > With kind regards, > -gdg > > [1]: > > def dummy(ls = [0]): > ? ? ls.append(ls[-1]+1) > ? ? return ls > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Terry Jan Reedy From ma3yuki.8mamo10 at gmail.com Wed May 23 07:19:35 2018 From: ma3yuki.8mamo10 at gmail.com (Masayuki YAMAMOTO) Date: Wed, 23 May 2018 20:19:35 +0900 Subject: [Python-ideas] "Assignment expression" with function call-alike syntax In-Reply-To: References: Message-ID: 2018-05-23 6:32 GMT+09:00 Kirill Balunov : > Just one more variation on "assignment exression" syntax to make the list > more complete :) Sorry, if something similar has already been suggested. > The idea is to use function's call-like syntax in the from: `this( name = > expr )`. I'm not sure that such idea will find its supporters and whether > it is possible to implement it in a general way, but nevertheless. Below is > a half- and slow-working prototype and some silly examples just for the > sake of feeling the idea: > > import sys > import ctypes > > def this(**assign): > assert len(assign) == 1 > [(name, value)] = assign.items() > frame = sys._getframe(1) > frame.f_locals[name] = value > ctypes.pythonapi.PyFrame_LocalsToFast(ctypes.py_object(frame), > ctypes.c_int(0)) > return value > > FYI, I found a package [*] which binds a value by function keyword (I'm not the author). Note that the package writes a value in the global (or raise an error if the name already exists in local) unlike your idea. [*] https://pypi.org/project/let3/ -- Masayuki -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed May 23 09:46:22 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 23 May 2018 23:46:22 +1000 Subject: [Python-ideas] Meta-PEP about built-in functions In-Reply-To: <5B048B62.4080805@UGent.be> References: <5B048B62.4080805@UGent.be> Message-ID: On 23 May 2018 at 07:28, Jeroen Demeyer wrote: > Hello, > > Both PEP 573 and PEP 575 deal with built-in functions. Additionally, some > people (Stefan Behnel, Robert Bradshaw, Jim Pivarski and me) are currently > brainstorming about a yet-to-be-written PEP to allow calling the underlying > C function of a built-in function using native types (for example, a C long > instead of a Python int). Think of it as analogous to the buffer protocol: > the buffer protocol exposes C *data* while this would expose C *callables*. > > Since all these PEPs would overlap somewhat, I'm starting to wonder about > the best way to organize this. Ideally, I would like some kind of > "meta-PEP" where we discuss the future of built-in functions in general > terms without too much details. This would be followed by several PEPs each > going in detail about one specific aspect. > > Is there a precedent for this? What do the seasoned Python developers > think? > Probably the closest recent precedent would be PEPs 482 and 483, which laid out some background material and concepts so that PEP 484 could reference them, without needing to include them directly. I think doing something like that for the C level callable API to describe the status quo and the challenges it raises (both internally in CPython and for third party projects like Cython) could be a very good way to go, as that way the actual change proposals can focus on what they're proposing to change and why, without each needing to include all the background details regarding the specifics of the current situation. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-ideas at mgmiller.net Wed May 23 10:54:28 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Wed, 23 May 2018 07:54:28 -0700 Subject: [Python-ideas] "Assignment expression" with function call-alike syntax In-Reply-To: References: Message-ID: <77893ab9-5329-f3e2-242c-9cc8a535663e@mgmiller.net> On 2018-05-22 14:32, Kirill Balunov wrote: > # in global scope everything works ok since locals is globals > >>> while len( this( val = dummy() ) ) >= 0: > ...? ? ?print(val) > [0, 1] > [0, 1, 2] > [0, 1, 2, 3] Interesting! Although the example with a len() and mutable default arguments obscured the utility, I thought. Also, why limit it to one assignment? Liked Terry's suggestions for name. The name that jumped into my brain as I read was a reuse of the locals() or globals() callable with key words, that would change their behavior to what you suggest. -Mike From kirillbalunov at gmail.com Wed May 23 13:59:20 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Wed, 23 May 2018 20:59:20 +0300 Subject: [Python-ideas] "Assignment expression" with function call-alike syntax In-Reply-To: References: Message-ID: 2018-05-23 9:05 GMT+03:00 Terry Reedy : > On 5/22/2018 5:32 PM, Kirill Balunov wrote: > >> Just one more variation on "assignment exression" syntax to make the list >> more complete :) Sorry, if something similar has already been suggested. >> The idea is to use function's call-like syntax in the from: `this( name = >> expr )`. >> > > Functions names should be verbs. For this one, 'bind' or even better, > 'let' as in 'let(name=expr)' It only looks like a function call, but in fact it should be an _magic object_ that can on the call bind a name to an expression in the current local scope. I chose `this` because in my opinion it is easily perceived: `while this( name = expr ) > 0` can be readed as "While this name assigned to an expression is greater than zero do..." the same interpretation for `if` statement. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirillbalunov at gmail.com Wed May 23 14:13:59 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Wed, 23 May 2018 21:13:59 +0300 Subject: [Python-ideas] "Assignment expression" with function call-alike syntax In-Reply-To: References: Message-ID: 2018-05-23 14:19 GMT+03:00 Masayuki YAMAMOTO : > FYI, I found a package [*] which binds a value by function keyword (I'm > not the author). Note that the package writes a value in the global (or > raise an error if the name already exists in local) unlike your idea. > > [*] https://pypi.org/project/let3/ > > -- > Masayuki > Thank you! Writing to a globals is an even more stripped-down version from the inteded behaviour, for example it would not work in comprehensions at all and would not work if function has the same local name. In fact comprehension/generator cases are the least interesting for me. I just wanted to minimize side-effects. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirillbalunov at gmail.com Wed May 23 14:48:07 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Wed, 23 May 2018 21:48:07 +0300 Subject: [Python-ideas] "Assignment expression" with function call-alike syntax In-Reply-To: <77893ab9-5329-f3e2-242c-9cc8a535663e@mgmiller.net> References: <77893ab9-5329-f3e2-242c-9cc8a535663e@mgmiller.net> Message-ID: 2018-05-23 17:54 GMT+03:00 Mike Miller : > > On 2018-05-22 14:32, Kirill Balunov wrote: > >> # in global scope everything works ok since locals is globals >> >>> while len( this( val = dummy() ) ) >= 0: >> ... print(val) >> [0, 1] >> [0, 1, 2] >> [0, 1, 2, 3] >> > > Interesting! Although the example with a len() and mutable default > arguments obscured the utility, I thought. Also, why limit it to one > assignment? > > I just want some dummy example which can not be trivially handled with `for name in iter(func, value)`. But the latter is also only partly true. You can make something like: class P: __slots__ = ('func') def __init__(self, func): self.func = func def __eq__(self, other): return not self.func(other) for val in iter(dummy, P(lambda x: len(x) < 5)): print(val) [0, 1] [0, 1, 2] [0, 1, 2, 3] The only restriction is that you can not pass arguments to a `func`, but this, to some extent, can also be handled with lambda. I limit it to _one assignment_ firstly to make it simple, and secondly to get rid of some edge cases. > Liked Terry's suggestions for name. The name that jumped into my brain as > I read was a reuse of the locals() or globals() callable with key words, > that would change their behavior to what you suggest. I like `this` because it is easy to follow in my opinion: " While this name assigned to an expression is greater than zero do... " But of course I'm open to any other spelling. I'm just wondering, if someone like this form as an alternative for assignment expression (`:=`). With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.input at gmail.com Wed May 23 16:43:14 2018 From: carl.input at gmail.com (Carl Smith) Date: Wed, 23 May 2018 21:43:14 +0100 Subject: [Python-ideas] "Assignment expression" with function call-alike syntax In-Reply-To: References: <77893ab9-5329-f3e2-242c-9cc8a535663e@mgmiller.net> Message-ID: The name `this` is problematic as it has a well established, different meaning in lots of other languages. It's basically `self` for languages that assign to it automatically. Full stack Python users will have `this` meaning two different things at each end, and they cannot alias it in either of them. I'm not sure on the feature, but thought `let` was a perfect name for it. -- Carl Smith carl.input at gmail.com On 23 May 2018 at 19:48, Kirill Balunov wrote: > > > 2018-05-23 17:54 GMT+03:00 Mike Miller : > >> >> On 2018-05-22 14:32, Kirill Balunov wrote: >> >>> # in global scope everything works ok since locals is globals >>> >>> while len( this( val = dummy() ) ) >= 0: >>> ... print(val) >>> [0, 1] >>> [0, 1, 2] >>> [0, 1, 2, 3] >>> >> >> Interesting! Although the example with a len() and mutable default >> arguments obscured the utility, I thought. Also, why limit it to one >> assignment? >> >> > I just want some dummy example which can not be trivially handled with `for > name in iter(func, value)`. But the latter is also only partly true. You > can make something like: > > class P: > __slots__ = ('func') > def __init__(self, func): > self.func = func > def __eq__(self, other): > return not self.func(other) > > for val in iter(dummy, P(lambda x: len(x) < 5)): > print(val) > > [0, 1] > [0, 1, 2] > [0, 1, 2, 3] > > The only restriction is that you can not pass arguments to a `func`, but > this, to some extent, can also be handled with lambda. I limit it to _one > assignment_ firstly to make it simple, and secondly to get rid of some edge > cases. > > >> Liked Terry's suggestions for name. The name that jumped into my brain >> as I read was a reuse of the locals() or globals() callable with key words, >> that would change their behavior to what you suggest. > > > I like `this` because it is easy to follow in my opinion: " While this > name assigned to an expression is greater than zero do... " But of > course I'm open to any other spelling. I'm just wondering, if someone like > this form as an alternative for assignment expression (`:=`). > > With kind regards, > -gdg > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bradleyp81+python-ideas at gmail.com Wed May 23 20:52:30 2018 From: bradleyp81+python-ideas at gmail.com (Bradley Phillips) Date: Thu, 24 May 2018 08:52:30 +0800 Subject: [Python-ideas] multiprocessing, freeze support by default for Pool Message-ID: Hi All There are about 347 results for the following query on the google search engine: "python multiprocessing pyinstaller crash site:stackoverflow.com" The windows fork bomb is a known frequent occurrence. I know that freeze_support is supposed to be called but this issue still catches me out most times (and many others). I propose adding to line 117 of multiprocessing/context.py if (sys.platform == 'win32'): self.freeze_support() Thanks Regards Brad -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed May 23 21:21:04 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 24 May 2018 11:21:04 +1000 Subject: [Python-ideas] "Assignment expression" with function call-alike syntax In-Reply-To: References: Message-ID: <20180524012104.GM12683@ando.pearwood.info> On Wed, May 23, 2018 at 12:32:36AM +0300, Kirill Balunov wrote: > Just one more variation on "assignment exression" syntax to make the list > more complete :) Sorry, if something similar has already been suggested. > The idea is to use function's call-like syntax in the from: `this( name = > expr )`. [...] > In this way it is somewhat possible to make an assignment in `while` and `if > ` headers right now, which covers 90% cases, but it is damn slow. Maybe be > it is worth to make `this` magic call-alike object work fast...on the other > hand does anyone like `this`? Apart from needing extra typing, what does the function-call syntax gain us? It looks like a function you could call from anywhere, but you want to limit it to just "while" and "if", I expect that will just give us a flood of questions on Stackoverflow and other forums, "why can't I use this() outside of if and while loops?" A good question. Why shouldn't we use assignment outside of if and while? Since this is special syntax, not a function, the parens don't give us any benefit: this name = expression The word "this" seems inappropriate. Surely "let" or "set" would be better: let name = expression which at least has the benefit of matching syntax chosen in other languages. But if the only reason we include "let" is to avoid the equals/assignment error, that seems needlessly verbose. We can say the same thing more compactly: name := expression The difference between "let name = expression" and "name := expression" is just spelling. -- Steve From mistersheik at gmail.com Thu May 24 04:38:50 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 24 May 2018 01:38:50 -0700 (PDT) Subject: [Python-ideas] Thoughts on "extended mapping unpacking" Message-ID: <6eb35bd1-5888-4a23-af23-ddd16a563772@googlegroups.com> I was previously constructing an object like this: tb = TemporalBehavior(**kwargs, **parameters) where various subclasses were doing things like def __init__(self, some_kwarg, some_other_kwargs, some_parameter, some_other_parameter): Then I realized that I want to pass the paramters as a dictionary so that I can store it. I changed the code to this: def __init__(self, some_kwarg, some_other_kwargs, parameters): but I still need "some_parameter", so I did some_parmeter = parameters['some_parameter'] some_other_parmeter = parameters['some_other_parameter'] Great, but now I have to check that exactly the list of parameters that I need is being sent in, so I need to do something like if set(parameters) != ('some_parameter', 'some_other_parameter'): raise ValueError It might be nice to do instead {'some_parameter': p, 'some_other_parameter': q} = parameters I'm just throwing this suggestion out there. I realize that this is pretty niche, but who knows where Python will be in ten years. I also know that this is possible (and fairly easy) to implement from when I worked on PEP 448. This is similar to unpacking iterables like this: a, b = range(2) a, b, *c = range(5) It's the mapping version of it: {'a': a, 'b': b} = some_dict {'a': a, 'b': b, **c} = some_dict Best, Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu May 24 07:58:54 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 24 May 2018 14:58:54 +0300 Subject: [Python-ideas] Thoughts on "extended mapping unpacking" In-Reply-To: <6eb35bd1-5888-4a23-af23-ddd16a563772@googlegroups.com> References: <6eb35bd1-5888-4a23-af23-ddd16a563772@googlegroups.com> Message-ID: 24.05.18 11:38, Neil Girdhar ????: > I was previously constructing an object like this: > > tb = TemporalBehavior(**kwargs, **parameters) > > where various subclasses were doing things like > > def __init__(self, some_kwarg, some_other_kwargs, some_parameter, > some_other_parameter): > > Then I realized that I want to pass the paramters as a dictionary so > that I can store it..? I changed the code to this: > > def __init__(self, some_kwarg, some_other_kwargs, parameters): > > but I still need "some_parameter", so I did > > some_parmeter = parameters['some_parameter'] > some_other_parmeter = parameters['some_other_parameter'] > > Great, but now I have to check that exactly the list of parameters that > I need is being sent in, so I need to do something like > > if set(parameters) != ('some_parameter', 'some_other_parameter'): > ? ? raise ValueError > > It might be nice to do instead > > {'some_parameter': p, 'some_other_parameter': q} = parameters p = parameters.pop('some_parameter') q = parameters.pop('some_other_parameter') if parameters: raise ValueError or p, q = (lambda some_parameter, some_other_parameter: some_parameter, some_other_parameter)(**parameters) From peter.ed.oconnor at gmail.com Thu May 24 08:06:03 2018 From: peter.ed.oconnor at gmail.com (Peter O'Connor) Date: Thu, 24 May 2018 14:06:03 +0200 Subject: [Python-ideas] Proposal: A Reduce-Map Comprehension and a "last" builtin In-Reply-To: References: Message-ID: To give this old horse a kick: The "given" syntax in the recent thread could give a nice solution for the problem that started this thread. Instead of my proposal of: smooth_signal = [average := (1-decay)*average + decay*x for x in signal from average=0.] We could use given for both the in-loop variable update and the variable initialization: smooth_signal = [average given average=(1-decay)*average + decay*x for x in signal] given average=0. This especially makes sense for the extended syntax, where my proposal of: (z, y := f(z, x) -> y for x in iter_x from z=initial_z) Becomes: (y given z, y = f(z, x) for x in iter_x) given z=initial_z So in stead of adding 2 symbols and a keyword, we just need to add the one "given" keyword. It's worth noting, as Serhiy pointed out, that this is already supported in python, albeit with a very clunky syntax: smooth_signal = [average for average in [0] for x in signal for average in [(1-decay)*average + decay*x]] (y for z in [initial_z] for x in iter_x for z, y in [f(z, x)]) On Tue, Apr 17, 2018 at 12:02 AM, Danilo J. S. Bellini < danilo.bellini at gmail.com> wrote: > On 16 April 2018 at 10:49, Peter O'Connor > wrote: > >> Are you able to show how you'd implement the moving average example with >> your package? >> > > Sure! The single pole IIR filter you've shown is implemented here: > https://github.com/danilobellini/pyscanprev/blob/ > master/examples/iir-filter.rst > > I tried: >> >> @enable_scan("average") >> def exponential_moving_average_pyscan(signal, decay, initial=0): >> yield from ((1-decay)*(average or initial) + decay*x for x in >> signal) >> >> >> smooth_signal_9 = list(exponential_moving_average_pyscan(signal, >> decay=decay))[1:] >> >> Which almost gave the right result, but seemed to get the initial >> conditions wrong. >> > > I'm not sure what you were expecting. A sentinel as the first "average" > value? > > Before the loop begins, this scan-generator just echoes the first input, > like itertools.accumulate. > That is, the first value this generator yields is the first "signal" > value, which is then the first "average" value. > > To put an initial memory state, you should do something like this (I've > removed the floating point trailing noise): > > >>> from pyscanprev import enable_scan, prepend > >>> > >>> @enable_scan("y") > >>> def iir_filter(signal, decay, memory=0): > ... return ((1 - decay) * y + decay * x for x in prepend(memory, > signal)) > ... > >>> list(iir_filter([1, 2, 3, 2, 1, -1, -2], decay=.1, memory=5)) > [5, 4.6, 4.34, 4.206, 3.9854, 3.68686, 3.218174, 2.6963566] > > In that example, "y" is the "previous result" (a.k.a. accumulator, or > what had been called "average" here). > > -- > Danilo J. S. Bellini > --------------- > "*It is not our business to set up prohibitions, but to arrive at > conventions.*" (R. Carnap) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu May 24 08:49:01 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 24 May 2018 22:49:01 +1000 Subject: [Python-ideas] Proposal: A Reduce-Map Comprehension and a "last" builtin In-Reply-To: References: Message-ID: <20180524124900.GP12683@ando.pearwood.info> On Thu, May 24, 2018 at 02:06:03PM +0200, Peter O'Connor wrote: > To give this old horse a kick: The "given" syntax in the recent thread > could give a nice solution for the problem that started this thread. Your use-case is one of the motivating examples for PEP 572. Unless I'm confused, your use-case is intentionally left out of Nick's "given" proposal. He doesn't want to support your example. (Nick, please correct me if I'm mistaken.) > Instead of my proposal of: > smooth_signal = [average := (1-decay)*average + decay*x for x in signal > from average=0.] This would become: average = 0 smooth_signal = [average := (1-decay)*average + decay*x for x in signal] under PEP 572. If you insist on a one-liner (say, to win a bet), you could abuse the "or" operator: smooth_signal = (average := 0) or [average := (1-decay)*average + decay*x for x in signal] but I think that's the sort of example that people who dislike this proposal are worried about so please don't do that in serious code :-) > We could use given for both the in-loop variable update and the variable > initialization: > smooth_signal = [average given average=(1-decay)*average + decay*x for > x in signal] given average=0. I don't think that will work under Nick's proposal, as Nick does not want assignments inside the comprehension to be local to the surrounding scope. (Nick, please correct me if I'm wrong.) So in your example, the OUTER "given" creates a local variable in the current scope, average=0, but the INNER "given" inside the comprehension exists inside a separate, sub-local comprehension scope, where you will get an UnboundLocalError when it tries to evaluate (1-decay)*average the first time. [...] > So in stead of adding 2 symbols and a keyword, we just need to add the one > "given" keyword. PEP 572 will not only have the semantics you desire, but it requires only a single new symbol. If Nick writes up "given" as a PEP, I expect that it won't help your use-case. -- Steve From mistersheik at gmail.com Thu May 24 11:46:29 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 24 May 2018 11:46:29 -0400 Subject: [Python-ideas] Thoughts on "extended mapping unpacking" In-Reply-To: References: <6eb35bd1-5888-4a23-af23-ddd16a563772@googlegroups.com> Message-ID: On Thu, May 24, 2018 at 8:00 AM Serhiy Storchaka wrote: > 24.05.18 11:38, Neil Girdhar ????: > > I was previously constructing an object like this: > > > > tb = TemporalBehavior(**kwargs, **parameters) > > > > where various subclasses were doing things like > > > > def __init__(self, some_kwarg, some_other_kwargs, some_parameter, > > some_other_parameter): > > > > Then I realized that I want to pass the paramters as a dictionary so > > that I can store it.. I changed the code to this: > > > > def __init__(self, some_kwarg, some_other_kwargs, parameters): > > > > but I still need "some_parameter", so I did > > > > some_parmeter = parameters['some_parameter'] > > some_other_parmeter = parameters['some_other_parameter'] > > > > Great, but now I have to check that exactly the list of parameters that > > I need is being sent in, so I need to do something like > > > > if set(parameters) != ('some_parameter', 'some_other_parameter'): > > raise ValueError > > > > It might be nice to do instead > > > > {'some_parameter': p, 'some_other_parameter': q} = parameters > > p = parameters.pop('some_parameter') > q = parameters.pop('some_other_parameter') > if parameters: > raise ValueError > > parameters is a Mapping subclass and I don't want to destroy it > or > > p, q = (lambda some_parameter, some_other_parameter: some_parameter, > some_other_parameter)(**parameters) > > that works > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/rupoMkwAhi0/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu May 24 11:54:09 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 24 May 2018 11:54:09 -0400 Subject: [Python-ideas] Reuse "for" to express "given" Message-ID: I have read most of the PEP 572 related threads, but I don't think I've seen this idea. As many other people mentioned, Python already allows a trick to introduce local bindings in generator expressions and list comprehensions. This can be achieved by adding a "for var in []" clause. I propose to make "for var = " have the same effect as "for var in []". Note that in the case of filters, the order will be different from that in the original "given" proposal: instead of [(x, y, x/y) for x in data if y given y = f(x)] we will write [(x, y, x/y) for x in data for y = f(x) if y] The use of "for" in place of "given" in the if statements does not look too bad: if m for m = pattern.search(data): ... elif m for m = other_pattern.search(data)): ... else: ... I have to admit that while m for m = pattern.search(remaining_data): ... looks strange at first because both while and for are strongly associated with loops, but if you read "for var = " as "for var equals expression", the difference between that and "for var in (iterable) expression" becomes apparent. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertve92 at gmail.com Thu May 24 12:04:34 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Thu, 24 May 2018 18:04:34 +0200 Subject: [Python-ideas] Reuse "for" to express "given" In-Reply-To: References: Message-ID: This idea was mentioned (by me) at a time yes, but wasn't written in the document. I think one of the thing was that it would make the grammar non LL1 because when seeing the token "for" in a list comprehension it wouldn't know in advance if it's the loop or the assignment. And also, it might confuse people because 'for' is for iteration. Le jeu. 24 mai 2018 ? 17:54, Alexander Belopolsky < alexander.belopolsky at gmail.com> a ?crit : > I have read most of the PEP 572 related threads, but I don't think I've > seen this idea. As many other people mentioned, Python already allows a > trick to introduce local bindings in generator expressions and list > comprehensions. This can be achieved by adding a "for var in []" > clause. I propose to make "for var = " have the same effect as "for > var in []". > > Note that in the case of filters, the order will be different from that in > the original "given" proposal: instead of > > [(x, y, x/y) for x in data if y given y = f(x)] > > we will write > > [(x, y, x/y) for x in data for y = f(x) if y] > > The use of "for" in place of "given" in the if statements does not look > too bad: > > if m for m = pattern.search(data): > ... > elif m for m = other_pattern.search(data)): > ... > else: > ... > > I have to admit that > > while m for m = pattern.search(remaining_data): > ... > > looks strange at first because both while and for are strongly associated > with loops, but if you read "for var = " as "for var equals > expression", the difference between that and "for var in (iterable) > expression" becomes apparent. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu May 24 12:22:09 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 24 May 2018 12:22:09 -0400 Subject: [Python-ideas] Reuse "for" to express "given" In-Reply-To: References: Message-ID: On Thu, May 24, 2018 at 12:04 PM Robert Vanden Eynde wrote: > This idea was mentioned (by me) at a time yes, but wasn't written in the document. Can you point me to a specific post? There were so may that I must have missed that one. > I think one of the thing was that it would make the grammar non LL1 because when seeing the token "for" in a list comprehension it wouldn't know in advance if it's the loop or the assignment. I don't see how that can be a problem. From the grammar point of view, "for" in "for var = " may still be seen as introducing a "loop", but when "=" is seen in place of "in", the compiler will resize that the "loop" is one a single value and emit efficient code for it. At the AST level, "for var = " will look exactly the same as "for var in " only with an "=" instead of "in". > And also, it might confuse people because 'for' is for iteration. I think I addressed this in my previous post. Yes, for people with a C/C++ background, "for" may be too strongly associated with loops, but in mathematical sense, it seems clear that "for var in a set" means iteration over a set, while "for var = expression" means binding to a single value. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertve92 at gmail.com Thu May 24 12:25:40 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Thu, 24 May 2018 18:25:40 +0200 Subject: [Python-ideas] Reuse "for" to express "given" In-Reply-To: References: Message-ID: It was a long time ago I couldn't easily find the post but that's alright, you refreshed the idea :) Let's see what others think of for x = I also remembered some languages (like lua) use for x = range (5) interchangeably with for x in range (5) and guido said it will never make such a thing, for .. in being the iteration. Le jeu. 24 mai 2018 ? 18:22, Alexander Belopolsky < alexander.belopolsky at gmail.com> a ?crit : > > > On Thu, May 24, 2018 at 12:04 PM Robert Vanden Eynde > wrote: > > > This idea was mentioned (by me) at a time yes, but wasn't written in the > document. > > Can you point me to a specific post? There were so may that I must have > missed that one. > > > I think one of the thing was that it would make the grammar non LL1 > because when seeing the token "for" in a list comprehension it wouldn't > know in advance if it's the loop or the assignment. > > I don't see how that can be a problem. From the grammar point of view, > "for" in "for var = " may still be seen as introducing a "loop", but > when "=" is seen in place of "in", the compiler will resize that the "loop" > is one a single value and emit efficient code for it. At the AST level, > "for var = " will look exactly the same as "for var in " > only with an "=" instead of "in". > > > And also, it might confuse people because 'for' is for iteration. > > I think I addressed this in my previous post. Yes, for people with a > C/C++ background, "for" may be too strongly associated with loops, but in > mathematical sense, it seems clear that "for var in a set" means iteration > over a set, while "for var = expression" means binding to a single value. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu May 24 12:36:29 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 24 May 2018 19:36:29 +0300 Subject: [Python-ideas] Thoughts on "extended mapping unpacking" In-Reply-To: References: <6eb35bd1-5888-4a23-af23-ddd16a563772@googlegroups.com> Message-ID: 24.05.18 18:46, Neil Girdhar ????: > p = parameters.pop('some_parameter') > q = parameters.pop('some_other_parameter') > if parameters: > ? ? ?raise ValueError > > parameters is a Mapping subclass and I don't want to destroy it Oh, right. It works if parameters is a var-keyword parameter. def __init__(self, some_kwarg, some_other_kwargs, **parameters): From storchaka at gmail.com Thu May 24 12:43:20 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 24 May 2018 19:43:20 +0300 Subject: [Python-ideas] Reuse "for" to express "given" In-Reply-To: References: Message-ID: 24.05.18 18:54, Alexander Belopolsky ????: > I have read most of the PEP 572 related threads, but I don't think I've > seen this idea.? As many other people mentioned, Python already allows a > trick to introduce local bindings in generator expressions and list > comprehensions.? This can be achieved by adding a "for var in []" > clause.? I propose to make "for var = " have the same effect > as?"for var in []". It was discussed before (at least in the context of comprehensions), and was rejected because in some other popular languages `for var = ` is the same as `for var in ` in Python. From waksman at gmail.com Thu May 24 12:53:50 2018 From: waksman at gmail.com (George Leslie-Waksman) Date: Thu, 24 May 2018 09:53:50 -0700 Subject: [Python-ideas] Reuse "for" to express "given" In-Reply-To: References: Message-ID: I worry about the use of "for" because it will come up in contexts where "for" already has other meanings. In the case of the example list comprehension, the word "for" is being used to mean two entirely different things in a single expression, that seems rather precarious to me. I prefer "with" if we're looking for keywords to reuse. This feels fairly clean to me: [(x, y, x/y) for x in data with y = f(x) if y] if m with m = pattern.search(data): ... while m with m = pattern.search(remaining_data): ... Of course, "with" is not without problems and I'm not so happy about being able to: with open(a) as b with a = filename: ... --George On Thu, May 24, 2018 at 9:33 AM Robert Vanden Eynde wrote: > It was a long time ago I couldn't easily find the post but that's alright, > you refreshed the idea :) > > Let's see what others think of for x = > > I also remembered some languages (like lua) use for x = range (5) > interchangeably with for x in range (5) and guido said it will never make > such a thing, for .. in being the iteration. > > Le jeu. 24 mai 2018 ? 18:22, Alexander Belopolsky < > alexander.belopolsky at gmail.com> a ?crit : > >> >> >> On Thu, May 24, 2018 at 12:04 PM Robert Vanden Eynde < >> robertve92 at gmail.com> wrote: >> >> > This idea was mentioned (by me) at a time yes, but wasn't written in >> the document. >> >> Can you point me to a specific post? There were so may that I must have >> missed that one. >> >> > I think one of the thing was that it would make the grammar non LL1 >> because when seeing the token "for" in a list comprehension it wouldn't >> know in advance if it's the loop or the assignment. >> >> I don't see how that can be a problem. From the grammar point of view, >> "for" in "for var = " may still be seen as introducing a "loop", but >> when "=" is seen in place of "in", the compiler will resize that the "loop" >> is one a single value and emit efficient code for it. At the AST level, >> "for var = " will look exactly the same as "for var in " >> only with an "=" instead of "in". >> >> > And also, it might confuse people because 'for' is for iteration. >> >> I think I addressed this in my previous post. Yes, for people with a >> C/C++ background, "for" may be too strongly associated with loops, but in >> mathematical sense, it seems clear that "for var in a set" means iteration >> over a set, while "for var = expression" means binding to a single value. >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From waksman at gmail.com Thu May 24 13:08:28 2018 From: waksman at gmail.com (George Leslie-Waksman) Date: Thu, 24 May 2018 10:08:28 -0700 Subject: [Python-ideas] Thoughts on "extended mapping unpacking" In-Reply-To: References: <6eb35bd1-5888-4a23-af23-ddd16a563772@googlegroups.com> Message-ID: I have had plenty of instances where destructuring a mapping would have be convenient. Relating to iterable destructuring, I would expect the syntax to be of the form "variable: key". I also think the curly-braces make it harder to visually parse what's going on. So I might suggest something a little like: objkey = object() mydict = {'a': 1, 'b': 2, 'c': 3, 4: 5, None: 6, objkey: 7} var1: 'a', var2: 4, var3: None, var4: objkey, **rest = mydict assert var1 == 1 assert var2 == 5 assert var3 == 6 assert var4 == 7 assert rest == {'b': 2, 'c': 3} On Thu, May 24, 2018 at 9:37 AM Serhiy Storchaka wrote: > 24.05.18 18:46, Neil Girdhar ????: > > p = parameters.pop('some_parameter') > > q = parameters.pop('some_other_parameter') > > if parameters: > > raise ValueError > > > > parameters is a Mapping subclass and I don't want to destroy it > > Oh, right. It works if parameters is a var-keyword parameter. > > def __init__(self, some_kwarg, some_other_kwargs, **parameters): > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Thu May 24 13:28:14 2018 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 24 May 2018 18:28:14 +0100 Subject: [Python-ideas] Thoughts on "extended mapping unpacking" In-Reply-To: References: <6eb35bd1-5888-4a23-af23-ddd16a563772@googlegroups.com> Message-ID: On 2018-05-24 18:08, George Leslie-Waksman wrote: > I have had plenty of instances where destructuring a mapping would have > be convenient. Relating to iterable destructuring, I would expect the > syntax to be of the form "variable: key". I also think the curly-braces > make it harder to visually parse what's going on. So I might suggest > something a little like: > > objkey = object() > mydict = {'a': 1, 'b': 2, 'c': 3, 4: 5, None: 6, objkey: 7} > var1: 'a', var2: 4, var3: None, var4: objkey, **rest = mydict The problem there is that the key and the value are now the wrong way round... > assert var1 == 1 > assert var2 == 5 > assert var3 == 6 > assert var4 == 7 > assert rest == {'b': 2, 'c': 3} > > On Thu, May 24, 2018 at 9:37 AM Serhiy Storchaka > wrote: > > 24.05.18 18:46, Neil Girdhar ????: > >? ? ?p = parameters.pop('some_parameter') > >? ? ?q = parameters.pop('some_other_parameter') > >? ? ?if parameters: > >? ? ? ? ? ?raise ValueError > > > > parameters is a Mapping subclass and I don't want to destroy it > > Oh, right. It works if parameters is a var-keyword parameter. > > ? ? ?def __init__(self, some_kwarg, some_other_kwargs, **parameters): > From steve at pearwood.info Thu May 24 14:10:34 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 25 May 2018 04:10:34 +1000 Subject: [Python-ideas] Reuse "for" to express "given" In-Reply-To: References: Message-ID: <20180524181032.GR12683@ando.pearwood.info> On Thu, May 24, 2018 at 11:54:09AM -0400, Alexander Belopolsky wrote: > I have read most of the PEP 572 related threads, but I don't think I've > seen this idea. As many other people mentioned, Python already allows a > trick to introduce local bindings in generator expressions and list > comprehensions. This can be achieved by adding a "for var in []" > clause. I propose to make "for var = " have the same effect as "for > var in []". C is well known for encouraging the "assignment instead of equals" error when people mistakenly use = when they mean ==. Instead of copying this design flaw, we'll have our own infamous design flaw: "assignment instead of looping" errors, when we mistakenly type "for var = thing" instead of "for var in thing" (or vice versa). Who says Python never innovates? :-) [...] > [(x, y, x/y) for x in data for y = f(x) if y] [(x, y, x/y) for x in data if y := f(x)] In another post, you wrote: Yes, for people with a C/C++ background, "for" may be too strongly associated with loops, but in mathematical sense, it seems clear that "for var in a set" means iteration over a set, while "for var = expression" means binding to a single value. I don't come from a C/C++ background. I think "for" is strongly associated with loops in a large number of programming languages, including Python. http://rosettacode.org/wiki/Loops/For But I do have a mathematics background, and I don't remember ever seeing "for x = value" used in the sense you mean. I certainly wouldn't write that myself. I would expect to either: Given x = expression, then equation including x or the reverse order: equation including x where x = expression Normally I would only use "for" either in the context of sums, or the "for all" ? quantifier. (If that symbols doesn't show up, it is an upside down A.) -- Steve From carl.input at gmail.com Thu May 24 14:24:39 2018 From: carl.input at gmail.com (Carl Smith) Date: Thu, 24 May 2018 19:24:39 +0100 Subject: [Python-ideas] Mashup the existing statement grammars to capture predicates Message-ID: This is another suggestion for new syntax for assigning a name to the value of the predicate in an if, elif or while statement. It still uses `as` for its keyword, but with (more flexible) params instead of a direct assignment. It mashes up the if/while, def/class and for-in grammars, so it still looks like Python, and boils down to this: if|elif|while as (): If the params contain one simple name (the required minimum), the value of the predicate is assigned to that name. In any other case, the value must be a sequence, which gets unpacked: while input('$ ').split() as (command, *args): if run(command, parse(args)) as (result): render(result) else: sys.exit() -- Carl Smith carl.input at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu May 24 14:47:30 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 24 May 2018 14:47:30 -0400 Subject: [Python-ideas] Reuse "for" to express "given" In-Reply-To: <20180524181032.GR12683@ando.pearwood.info> References: <20180524181032.GR12683@ando.pearwood.info> Message-ID: > But I do have a mathematics background, and I don't remember ever seeing > "for x = value" used in the sense you mean. That's so because in mathematics, "for" is spelled ":" as in {2*a* : *a*?*Z*} If you can read the above, you should not have trouble reading {2*a* + *b* : *a*?*Z *: *b = *1} -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcidy at gmail.com Thu May 24 16:59:02 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Thu, 24 May 2018 13:59:02 -0700 Subject: [Python-ideas] Reuse "for" to express "given" In-Reply-To: References: <20180524181032.GR12683@ando.pearwood.info> Message-ID: On Thu, May 24, 2018, 11:47 Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > But I do have a mathematics background, and I don't remember ever seeing > > "for x = value" used in the sense you mean. > > That's so because in mathematics, "for" is spelled ":" as in > > {2*a* : *a*?*Z*} > > If you can read the above, you should not have trouble reading > > {2*a* + *b* : *a*?*Z *: *b = *1} > Inverted "A" is "for all", and colon means "such that". It may be acceptable somewhere to use a colon as you do, but the lone colon unambiguously is "such that" when constructing sets. I never once saw colon used this way, and would be interested to see it in a proper math paper to correct my world view. "for" only reads as "such that" as: "A such that x = 1" "A, for x = 1" which sounds a bit Shakespearian and is definitely not common. standard math: "2a plus b for all integers such that b is 1." or "2a plus b for each integer such that b is 1" using "for" as ":": "2a plus b for a in integers for b in 1" or "2a plus b for a eauals integers for b eauals 1" or "2a plus b for all a in integers for all b in 1" Understandable, but I would get zero point on my homework for those, they are correct. In the context of math in general, ":=" makes more sense, and is actual math notation for assignment: http://mathworld.wolfram.com/Colon.html ignoring math: I'm kind of fine with nudging the meaning of "for" as you describe to be assignment, it's close enough, such that it's not a long leap. I think this point is the best case for "for", but I don't think being the only language to do this is great, and I usually don't care about language similarities. I lack breadth, however, so please correct me here. However, I cannot say this is superior to ":=" based on the same logic, as ":=" achieves the same ends with less semantic confusion, typing, and uses earlier indication of assignment. And please note that making "for" ambiguous when there is already the ambiguity in post fix notation makes the spelling less desirable to me. In this spelling, for one assignment, there are two ambiguities that must be resolved: the assigned name and the meaning of "for", though the latter is cleared up quickly. This small ambiguity is still there and I don't see a reason to introduce it. Thanks > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu May 24 17:47:59 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 24 May 2018 17:47:59 -0400 Subject: [Python-ideas] Reuse "for" to express "given" In-Reply-To: References: <20180524181032.GR12683@ando.pearwood.info> Message-ID: On Thu, May 24, 2018 at 4:59 PM Matt Arcidy wrote: > > > On Thu, May 24, 2018, 11:47 Alexander Belopolsky < > alexander.belopolsky at gmail.com> wrote: > >> > But I do have a mathematics background, and I don't remember ever >> seeing >> > "for x = value" used in the sense you mean. >> >> That's so because in mathematics, "for" is spelled ":" as in >> >> {2*a* : *a*?*Z*} >> >> If you can read the above, you should not have trouble reading >> >> {2*a* + *b* : *a*?*Z *: *b = *1} >> > > Inverted "A" is "for all", and colon means "such that". It may be > acceptable somewhere to use a colon as you do, > See . Also, "[list comprehensions] is Python's way of implementing a well-known notation for sets as used by mathematicians." < https://www.python-course.eu/list_comprehension.php>. Although, the latter uses "|" instead of ":". -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenlhilton at gmail.com Thu May 24 20:42:16 2018 From: kenlhilton at gmail.com (Ken Hilton) Date: Fri, 25 May 2018 08:42:16 +0800 Subject: [Python-ideas] Make asyncio.get_event_loop a builtin Message-ID: On Tue May 22 22:08:40 (-0400), Chris Barker wrote: > while asyncio is in the standard library, it is not intended to be THE async event loop implementation I'm surprised this is true - with dedicated syntax like async def/await, it's still not THE async event loop implementation? As far as I know, "async def" is a shorthand for @asyncio.coroutine def and "await" is short for "yield from". Sincerely, Ken Hilton; -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcidy at gmail.com Thu May 24 23:02:33 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Thu, 24 May 2018 20:02:33 -0700 Subject: [Python-ideas] Reuse "for" to express "given" In-Reply-To: References: <20180524181032.GR12683@ando.pearwood.info> Message-ID: On Thu, May 24, 2018, 14:48 Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > > On Thu, May 24, 2018 at 4:59 PM Matt Arcidy wrote: > >> >> >> On Thu, May 24, 2018, 11:47 Alexander Belopolsky < >> alexander.belopolsky at gmail.com> wrote: >> >>> > But I do have a mathematics background, and I don't remember ever >>> seeing >>> > "for x = value" used in the sense you mean. >>> >>> That's so because in mathematics, "for" is spelled ":" as in >>> >>> {2*a* : *a*?*Z*} >>> >>> If you can read the above, you should not have trouble reading >>> >>> {2*a* + *b* : *a*?*Z *: *b = *1} >>> >> >> Inverted "A" is "for all", and colon means "such that". It may be >> acceptable somewhere to use a colon as you do, >> > > See . Also, "[list > comprehensions] is Python's way of implementing a well-known notation for > sets as used by mathematicians." < > https://www.python-course.eu/list_comprehension.php>. Although, the > latter uses "|" instead of ":". > My point was more along the lines of math doesn't use ":" for "for". "for" exists in math as a different symbol. Even in set builder notation ":" isn't interpretted as "for", it's "such that." Maybe the math discussion is totally tangential. I'm not clear why I'm making these points really, apologies if I took this off course, I'm happy to concede the math and just keep my other points. I don't think "for" is bad compared to some other alternatives, but I don't see it better than ":=" in the contexts you've raised. Even in the implementation of set building, "for" is only used to build the sets element wise, not as a partitioning, and is not a property if the set itself. Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmcs at jsantos.eu Fri May 25 02:57:30 2018 From: jmcs at jsantos.eu (=?UTF-8?B?Sm/Do28gU2FudG9z?=) Date: Fri, 25 May 2018 08:57:30 +0200 Subject: [Python-ideas] Make asyncio.get_event_loop a builtin In-Reply-To: References: Message-ID: Hi, You can use uvloop , for example, without using asyncio: import uvloop loop = uvloop.new_event_loop() loop.run_forever() Best regards, Jo?o Santos On Fri, 25 May 2018 at 02:42, Ken Hilton wrote: > On Tue May 22 22:08:40 (-0400), Chris Barker wrote: > > while asyncio is in the standard library, it is not intended to be THE > async event loop implementation > > I'm surprised this is true - with dedicated syntax like async def/await, > it's still not THE async event loop implementation? As far as I know, > "async def" is a shorthand for > > @asyncio.coroutine > def > > and "await" is short for "yield from". > > Sincerely, > Ken Hilton; > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.van.dorp at deonet.nl Fri May 25 06:06:42 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Fri, 25 May 2018 12:06:42 +0200 Subject: [Python-ideas] ternary without else Message-ID: I would like to carefully suggest a half form of the ternary expression. Currently, you can write code like: >>> if cond: >>> do_something However, especially if the condition and action are both really simple, taking two lines feels like a bit of a waste. So I sometimes write: >>> if cond: do_something However, this isn't PEP8 compliant, and every linter complains about it. They'd be right if the condition and action were a bit more complicated. I would very much like to write: >>> do_something if cond and be done with it. Like a ternary expression but without the else clause. I know this is probably not gonna make it due to "not strong enough pro's, just swallow the linters complaints and roll with the 'if cond: do_something' if you care so much about that 1 line", but I still wanted to make my case. ===Example Usecases=== (inside loop) >>> continue if x > 50 (logging) >>> logger.info(f"{var}") if DEBUG (Logging module ofc already does this, and in a much better way. However, if var is expensive to get it could save a bit of time.) (singleton creation) >>> cls.inst = super().__new__(cls, *args, **kwargs) if cls.inst is None (cache) >>> cache.warm() if cache is None From storchaka at gmail.com Fri May 25 06:33:05 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 25 May 2018 13:33:05 +0300 Subject: [Python-ideas] ternary without else In-Reply-To: References: Message-ID: 25.05.18 13:06, Jacco van Dorp ????: > I would like to carefully suggest a half form of the ternary expression. > > Currently, you can write code like: > >>>> if cond: >>>> do_something > > However, especially if the condition and action are both really > simple, taking two lines feels like a bit of a waste. So I sometimes > write: > >>>> if cond: do_something > > However, this isn't PEP8 compliant, and every linter complains about > it. They'd be right if the condition and action were a bit more > complicated. > > I would very much like to write: > >>>> do_something if cond > > and be done with it. Like a ternary expression but without the else clause. This isn't PEP8 compliant either. I suppose that if this syntax be accepted by the compiler, it will be explicitly disallowed by PEP8 for the same reason as "if cond: do_something". It is easier to pass an option to linter that will silence this warning than introduce a new ambiguous syntax. For example try to interpret "[a for b if c]" if "b if c" be a valid expression. From flying-sheep at web.de Fri May 25 06:45:00 2018 From: flying-sheep at web.de (Philipp A.) Date: Fri, 25 May 2018 12:45:00 +0200 Subject: [Python-ideas] ternary without else In-Reply-To: References: Message-ID: in jinja, you can do ?{{ 'foo' if bar }}?. it evaluates to ?'foo'? or an empty string (differently to python?s formatting, ?None? expands to an empty string in jinja) similarly I often do ?thing = 'foo' if bar else None? and it would be nice if i could shorten that by making ?else None? implicit. idk how often other people do that though Serhiy Storchaka schrieb am Fr., 25. Mai 2018 um 12:34 Uhr: > 25.05.18 13:06, Jacco van Dorp ????: > > I would like to carefully suggest a half form of the ternary expression. > > > > Currently, you can write code like: > > > >>>> if cond: > >>>> do_something > > > > However, especially if the condition and action are both really > > simple, taking two lines feels like a bit of a waste. So I sometimes > > write: > > > >>>> if cond: do_something > > > > However, this isn't PEP8 compliant, and every linter complains about > > it. They'd be right if the condition and action were a bit more > > complicated. > > > > I would very much like to write: > > > >>>> do_something if cond > > > > and be done with it. Like a ternary expression but without the else > clause. > > This isn't PEP8 compliant either. > > I suppose that if this syntax be accepted by the compiler, it will be > explicitly disallowed by PEP8 for the same reason as "if cond: > do_something". > > It is easier to pass an option to linter that will silence this warning > than introduce a new ambiguous syntax. > > For example try to interpret "[a for b if c]" if "b if c" be a valid > expression. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirillbalunov at gmail.com Fri May 25 08:26:06 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Fri, 25 May 2018 15:26:06 +0300 Subject: [Python-ideas] ternary without else In-Reply-To: References: Message-ID: 2018-05-25 13:06 GMT+03:00 Jacco van Dorp : > [...] > > I would very much like to write: > > >>> do_something if cond > > and be done with it. Like a ternary expression but without the else clause. > > If it is an expression, what should `do_something if cond` return on failure? If you don't care you can already use `cond and do_something`. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.van.dorp at deonet.nl Fri May 25 08:39:33 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Fri, 25 May 2018 14:39:33 +0200 Subject: [Python-ideas] ternary without else In-Reply-To: References: Message-ID: 2018-05-25 14:26 GMT+02:00 Kirill Balunov : > If it is an expression, what should `do_something if cond` return on > failure? If you don't care you can already use `cond and do_something`. Duh, forgot to mention. I wouldn't have it return anything. Ternary returns something because you have two options and picks one. This is conditional execution of a statement. I guess it would be a statement, like normal if. I guess cond and do_something would be equivalent, but that's not really the intention of and/or, no matter how much it's used for this sort of thing. Disclaimer: I don't really expect this to be accepted. It's more of a minor gripe i've occasionally had. This afternoon, I wrote: if not article["art_wt"]: article["art_wt"] = 0 if not article["px_pchs"]: article["px_pchs"] = 0 if not article["px_calc"]: article["px_calc"] = 0 if not article["px_sell"]: article["px_sell"] = 0 while preparing a dict for import in accountview. I try not to make a habit of dismissing syntax warnings. I consider the code above abundantly clear (no, I dont know what those keys mean. Since I have no power over that, I dont consider it detrimental to clarity.) From elazarg at gmail.com Fri May 25 08:57:43 2018 From: elazarg at gmail.com (Elazar) Date: Fri, 25 May 2018 05:57:43 -0700 Subject: [Python-ideas] ternary without else In-Reply-To: References: Message-ID: The title is misleading - this has nothing to do with the conditional operator, except small syntactic similarity. Elazar ?????? ??? ??, 25 ???? 2018, 05:40, ??? Jacco van Dorp ?< j.van.dorp at deonet.nl>: > 2018-05-25 14:26 GMT+02:00 Kirill Balunov : > > If it is an expression, what should `do_something if cond` return on > > failure? If you don't care you can already use `cond and do_something`. > > Duh, forgot to mention. > I wouldn't have it return anything. Ternary returns something because > you have two options and picks one. This is conditional execution of a > statement. I guess it would be a statement, like normal if. > > I guess cond and do_something would be equivalent, but that's not > really the intention of and/or, no matter how much it's used for this > sort of thing. > > Disclaimer: > I don't really expect this to be accepted. It's more of a minor gripe > i've occasionally had. This afternoon, I wrote: > > if not article["art_wt"]: article["art_wt"] = 0 > if not article["px_pchs"]: article["px_pchs"] = 0 > if not article["px_calc"]: article["px_calc"] = 0 > if not article["px_sell"]: article["px_sell"] = 0 > > while preparing a dict for import in accountview. I try not to make a > habit of dismissing syntax warnings. I consider the code above > abundantly clear (no, I dont know what those keys mean. Since I have > no power over that, I dont consider it detrimental to clarity.) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri May 25 07:38:24 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 25 May 2018 21:38:24 +1000 Subject: [Python-ideas] ternary without else In-Reply-To: References: Message-ID: <20180525113823.GV12683@ando.pearwood.info> On Fri, May 25, 2018 at 12:06:42PM +0200, Jacco van Dorp wrote: > I would like to carefully suggest a half form of the ternary expression. [...] > However, this isn't PEP8 compliant PEP 8 is not holy writ. Ignore it when your code is better for ignoring it. > I would very much like to write: > > >>> do_something if cond > > and be done with it. Like a ternary expression but without the else clause. I believe there are a number of languages out there which have clauses like this. I'm not going to do a language survey, but two popular choices are: something if condition something unless condition and similar variations. For example, Ruby: http://www.railstips.org/blog/archives/2008/12/01/unless-the-abused-ruby-conditional/ > ===Example Usecases=== > (inside loop) > >>> continue if x > 50 That's not going to work if this is an *expression*. You can't say: continue if x > 50 else "don't continue" The problem with an expression is, what does this return if the clause is false? So the comparison to the ternary if expression is invalid. This would have to be a statement. > > (logging) > >>> logger.info(f"{var}") if DEBUG > (Logging module ofc already does this, and in a much better way. > However, if var is expensive to get it could save a bit of time.) If the logging module already solves this problem in "a much better way", why are you including a much worse way? > (singleton creation) > >>> cls.inst = super().__new__(cls, *args, **kwargs) if cls.inst is None And this is a problem: that is ambiguous with the if...else ternary expression. Presumably you want that to look like a statement: cls.inst = super().__new__(cls, *args, **kwargs) guarded by an if statement: if cls.inst is None but it can also be interpreted as an assignment: cls.inst = with an incomplete ternary expression: super().__new__(cls, *args, **kwargs) if cls.inst is None else what???? Even if we decide to make it a rule that one interpretation beats the other, we still have to expect people will be confused by it. So we ought to be cautious about syntax which is ambiguous in this way. > (cache) > >>> cache.warm() if cache is None There are about a zillion things that you can write like this: if condition: do something so the use-cases certainly exist. The only question is a matter of taste and familiarity, whether it nicer to write it as above or like: do something if condition -- Steve From j.van.dorp at deonet.nl Fri May 25 09:10:12 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Fri, 25 May 2018 15:10:12 +0200 Subject: [Python-ideas] ternary without else In-Reply-To: References: Message-ID: 2018-05-25 14:57 GMT+02:00 Elazar : > The title is misleading - this has nothing to do with the conditional > operator, except small syntactic similarity. On second look, yeah, you're right. From rob.cliffe at btinternet.com Fri May 25 10:06:25 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 25 May 2018 15:06:25 +0100 Subject: [Python-ideas] Reuse "for" to express "given" In-Reply-To: References: Message-ID: <22a6f6b4-e9cd-ae4f-c2de-6d87a3f82336@btinternet.com> On 24/05/2018 16:54, Alexander Belopolsky wrote: > I have read most of the PEP 572 related threads, but I don't think > I've seen this idea.? As many other people mentioned, Python already > allows a trick to introduce local bindings in generator expressions > and list comprehensions.? This can be achieved by adding a "for var in > []" clause. I propose to make "for var = " have the same > effect as?"for var in []". > > I made this exact same suggestion some time ago (not in the context of PEP 572).? Guido rejected it because it is easy (especially for newbies) to confuse ??? for var in expr ??? for var = expr I point out that the confusion is particularly acute in the cases such as ??? for x = SomeTuple Rob Cliffe From rob.cliffe at btinternet.com Fri May 25 10:21:56 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 25 May 2018 15:21:56 +0100 Subject: [Python-ideas] ternary without else In-Reply-To: <20180525113823.GV12683@ando.pearwood.info> References: <20180525113823.GV12683@ando.pearwood.info> Message-ID: <3bf167dd-d456-6582-f277-74345818e33f@btinternet.com> On 25/05/2018 12:38, Steven D'Aprano wrote: > > PEP 8 is not holy writ. Ignore it when your code is better for ignoring > it. > > +1.? Not infrequently I judge that my code is easier both to read and to maintain with more than 1 (short) statement on a line, often when it allows similar items to be aligned vertically.? A spur-of-the moment, totally invented example: ??? if MON <= day <= FRI:??? Rate = Normal ??? if day==SAT:???????????????????? Rate = DoubleTime ??? if day==SUN:??????????????????? Rate = TripleTime I also feel no inhibitions about using Jacco's original example: ??? if cond: do_something Rob Cliffe From mehaase at gmail.com Fri May 25 14:07:11 2018 From: mehaase at gmail.com (Mark E. Haase) Date: Fri, 25 May 2018 14:07:11 -0400 Subject: [Python-ideas] Make asyncio.get_event_loop a builtin In-Reply-To: References: Message-ID: asyncio.coroutine and async def are slightly different?not synonyms. The former defines a generator function and the latter defines a coroutine function. Generators and coroutines are very similar in Python (they share lots of code in the CPython implementation) but coroutines are preferred for async programming. Furthermore, yield from (suspend a generator) and await (suspend a coroutine) are also similar but not synonyms. It is surprising that Python's async syntax isn't tightly bound to its built-in implementation of asynchronous I/O. This was a shrewd decision that?as others have pointed out?permits a variety of event loop implementations. The downside?as you point out?is that there is no built-in function that can drive an async function. On Thu, May 24, 2018 at 8:42 PM, Ken Hilton wrote: > On Tue May 22 22:08:40 (-0400), Chris Barker wrote: > > while asyncio is in the standard library, it is not intended to be THE > async event loop implementation > > I'm surprised this is true - with dedicated syntax like async def/await, > it's still not THE async event loop implementation? As far as I know, > "async def" is a shorthand for > > @asyncio.coroutine > def > > and "await" is short for "yield from". > > Sincerely, > Ken Hilton; > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Fri May 25 14:14:04 2018 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 25 May 2018 13:14:04 -0500 Subject: [Python-ideas] Proposal: A Reduce-Map Comprehension and a "last" builtin In-Reply-To: <20180524124900.GP12683@ando.pearwood.info> References: <20180524124900.GP12683@ando.pearwood.info> Message-ID: [Peter O'Connor] >> ... >> We could use given for both the in-loop variable update and the variable >> initialization: >> smooth_signal = [average given average=(1-decay)*average + decay*x >> for x in signal] given average=0. [Steven D'Aprano ] > I don't think that will work under Nick's proposal, as Nick does not > want assignments inside the comprehension to be local to the surrounding > scope. (Nick, please correct me if I'm wrong.) Nick appears to have moved on from "given" to more-general augmented assignment expressions. See PEP 577, but note that it's still a work-in-progress: https://github.com/python/peps/pull/665 Under that PEP, average = 0 smooth_signal = [(average := (1-decay)*average + decay*x) for x in signal] Or, for the running sums example: total = 0 sums = [(total += x) for x in data] I'm not entirely clear on whether the "extra" parens are needed, so added 'em anyway to make grouping clear. From carl.input at gmail.com Fri May 25 19:52:08 2018 From: carl.input at gmail.com (Carl Smith) Date: Sat, 26 May 2018 00:52:08 +0100 Subject: [Python-ideas] ternary without else In-Reply-To: <3bf167dd-d456-6582-f277-74345818e33f@btinternet.com> References: <20180525113823.GV12683@ando.pearwood.info> <3bf167dd-d456-6582-f277-74345818e33f@btinternet.com> Message-ID: ?You cannot have `expression if expression` in a language that also supports `expression if expression else expression` (like Python).? Otherwise, you have the dangling else problem: https://en.wikipedia.org/wiki/Dangling_else -- Carl Smith carl.input at gmail.com On 25 May 2018 at 15:21, Rob Cliffe via Python-ideas < python-ideas at python.org> wrote: > > > On 25/05/2018 12:38, Steven D'Aprano wrote: > >> >> PEP 8 is not holy writ. Ignore it when your code is better for ignoring >> it. >> >> >> +1. Not infrequently I judge that my code is easier both to read and to > maintain with more than 1 (short) statement on a line, > often when it allows similar items to be aligned vertically. A > spur-of-the moment, totally invented example: > if MON <= day <= FRI: Rate = Normal > if day==SAT: Rate = DoubleTime > if day==SUN: Rate = TripleTime > I also feel no inhibitions about using Jacco's original example: > if cond: do_something > Rob Cliffe > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mial.lohmann+pythonml at gmail.com Fri May 25 20:06:40 2018 From: mial.lohmann+pythonml at gmail.com (Michael Lohmann) Date: Sat, 26 May 2018 02:06:40 +0200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super Message-ID: Hi! ***Disclaimer: I am relatively new to Python*** I propose to add some mechanism that can automatically collect everything what would normally be collected by **kwargs in the __init__ method and directly pass it through to the super().__init__ call without being accessible in the __init__ itself. This way the autocompletion/docstring generation could safely ignore this argument which before would have showed up as **kwargs. I believe that this could greatly enhance the readability of automated documentation and especially the autocomplete of an IDE. I would like something like this to work: class A: @pass_through_kwargs # like this? Or maybe __init__(self, a_value, ***pass_through_kwargs)? def __init__(self, a_value): super().__init__() # <- realize that nothing is passed in here class B: def __init__(self, some_value): pass class MyClass(A, B): def __init__(self): super().__init__(a_value=1, some_value=2) Where the autocomplete of `A(` shows me just `A(a_value)`. The pass_through_kwargs can safely be ignored, since it is not "a real input" of A. It is automatically merged with the parameters of the `super(A, self).__init__()` call. How exactly this should be done would have to be discussed in more detail (I guess the actual values should probably override the passed through ones). With the current python version the code would have looked like class A: def __init__(self, a_value, **kwargs): super().__init__(**kwargs) But the user could have had the false impression that additional kwargs could be given when instantiating A. Obviously they are purely there for getting the init in the MRO working with multiple inheritance. One could also add something like pass_through_args as well but the usability of that is probably much more limited. Could this even work in theory? I guess it might be quite tricky since `super().__init__` doesn't even have to be called. If that would be the case: should it be called automatically if pass_through_kwargs were given after the init is done? By the way: Thank you all for this amazing language! Cheers, Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.input at gmail.com Fri May 25 21:58:36 2018 From: carl.input at gmail.com (Carl Smith) Date: Sat, 26 May 2018 02:58:36 +0100 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: References: Message-ID: > But the user could have had the false impression that additional kwargs > could be given when instantiating A. Obviously they are purely there for > getting the init in the MRO working with multiple inheritance. This just isn't true. You often do things with the kargs before passing them or other values to super. By using **kargs in the constructor and the call to `super`, you are indicating that the signature passes through, but there are many other things you could do instead. The docs *should* show that the derived method takes the same args as the inherited one it overrides. That is very useful information that belongs there. Best, -- Carl Smith carl.input at gmail.com On 26 May 2018 at 01:06, Michael Lohmann wrote: > Hi! > > ***Disclaimer: I am relatively new to Python*** > > I propose to add some mechanism that can automatically collect everything > what would normally be collected by **kwargs in the __init__ method and > directly pass it through to the super().__init__ call without being > accessible in the __init__ itself. This way the autocompletion/docstring > generation could safely ignore this argument which before would have showed > up as **kwargs. > > I believe that this could greatly enhance the readability of automated > documentation and especially the autocomplete of an IDE. > > I would like something like this to work: > > class A: > @pass_through_kwargs # like this? Or maybe __init__(self, > a_value, ***pass_through_kwargs)? > def __init__(self, a_value): > super().__init__() # <- realize that nothing is passed in here > > class B: > def __init__(self, some_value): > pass > > class MyClass(A, B): > def __init__(self): > super().__init__(a_value=1, some_value=2) > > Where the autocomplete of `A(` shows me just `A(a_value)`. The > pass_through_kwargs can safely be ignored, since it is not "a real input" > of A. It is automatically merged with the parameters of the `super(A, > self).__init__()` call. How exactly this should be done would have to be > discussed in more detail (I guess the actual values should probably > override the passed through ones). > > With the current python version the code would have looked like > > class A: > def __init__(self, a_value, **kwargs): > super().__init__(**kwargs) > > But the user could have had the false impression that additional kwargs > could be given when instantiating A. Obviously they are purely there for > getting the init in the MRO working with multiple inheritance. > > One could also add something like pass_through_args as well but the > usability of that is probably much more limited. > > Could this even work in theory? I guess it might be quite tricky since > `super().__init__` doesn't even have to be called. If that would be the > case: should it be called automatically if pass_through_kwargs were given > after the init is done? > > > By the way: Thank you all for this amazing language! > Cheers, Michael > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat May 26 00:16:49 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 26 May 2018 14:16:49 +1000 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: References: Message-ID: <20180526041649.GX12683@ando.pearwood.info> On Sat, May 26, 2018 at 02:06:40AM +0200, Michael Lohmann wrote: > I propose to add some mechanism that can automatically collect everything > what would normally be collected by **kwargs in the __init__ method and > directly pass it through to the super().__init__ call without being > accessible in the __init__ itself. This way the autocompletion/docstring > generation could safely ignore this argument which before would have showed > up as **kwargs. Why is this an advantage? Consider the following scenario: you are reading an unfamiliar code base and come across a class instantiation: obj = Aardvark(27, spam=3, eggs=5, cheese=True) So you look up help(Aardvark), and it tells you that the signature is Aardvark.__init__(self, foo) What the hell? If Aardvark.__init__ only takes a single argument (aside from self), why on earth isn't it an error to pass those keyword arguments? If you're a beginner, this is probably going to be a mind-melting WTF-I-don't-even moment. And even if you're *not* a beginner, it will probably cause a momentary sense of existential dread as, for a moment, everything you thought you understood about Python seems wrong. And then you remember that there's an obscure feature that tells the class to lie to you about what parameters it takes. It is bad enough when a method collects a bunch of parameters under **kwargs and I have to pour through the documentation for all its super classes to work out what they are. But at least the use of an explicit **kwargs gives me a clue that this is what is going on. Making **kwargs implicit and invisible reminds me of the old saying about coding for maintenance. To paraphrase: Design language features as if your language's users -- all of them -- are violent psychopaths who know where you live. [...] > With the current python version the code would have looked like > > class A: > def __init__(self, a_value, **kwargs): > super().__init__(**kwargs) > > But the user could have had the false impression that additional kwargs > could be given when instantiating A. But they *can* be given when instantiating A. What makes you think they can't be? > Obviously they are purely there for > getting the init in the MRO working with multiple inheritance. That's not always true. Even if it were true, it isn't obviously true. -- Steve From mial.lohmann at gmail.com Sat May 26 03:39:14 2018 From: mial.lohmann at gmail.com (Michael Lohmann) Date: Sat, 26 May 2018 09:39:14 +0200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super Message-ID: [Steven D'Aprano] > obj = Aardvark(27, spam=3, eggs=5, cheese=True) > > So you look up help(Aardvark), and it tells you that the signature is > > Aardvark.__init__(self, foo) > > What the hell? If Aardvark.__init__ only takes a single argument This is wrong! This would at some point down the line throw an error TypeError: __init__() got an unexpected keyword argument 'eggs? (or at some point **kwargs are being accepted and not passed on to super which would be terrible on its own). The whole point I was trying to make is: If it doesn?t make any sense to init a class with **kwargs: why write down that it would (or even **could**) accept them? Shouldn?t the init tell you something like 'If you instantiate this class then this is everything you can give me'? Well, right now in addition it says 'Just give me anything with keywords?. I would think that something like 'Oh, and if I am in the middle of an MRO: I will pass everything down the line? would be a much better description of it?s true intentions instead. [Carl Smith] > By using **kargs in the constructor and the call > to `super`, you are indicating that the signature passes through But can you be certain? Couldn?t someone have just used a `kargs.pop('eggs')`? Okay. You could argue that they probably wouldn?t have done so. But for making automated documentation it probably would be useful to make sure it didn?t happen. I think that (as Raymond Hettinger once said) 'super is super', but can?t you make it a bit smarter with telling it: 'Hey - If you don?t expect 'eggs', keep calm, it probably (or rather certainly) wasn?t meant for you so just pass it on to your super'. Best, Michael From rosuav at gmail.com Sat May 26 04:12:49 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 26 May 2018 18:12:49 +1000 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: References: Message-ID: On Sat, May 26, 2018 at 5:39 PM, Michael Lohmann wrote: > [Steven D'Aprano] >> obj = Aardvark(27, spam=3, eggs=5, cheese=True) >> >> So you look up help(Aardvark), and it tells you that the signature is >> >> Aardvark.__init__(self, foo) >> >> What the hell? If Aardvark.__init__ only takes a single argument > This is wrong! This would at some point down the line throw an error > TypeError: __init__() got an unexpected keyword argument 'eggs? > (or at some point **kwargs are being accepted and not passed on to super which would be terrible on its own). > > The whole point I was trying to make is: If it doesn?t make any sense to init a class with **kwargs: why write down that it would (or even **could**) accept them? Shouldn?t the init tell you something like 'If you instantiate this class then this is everything you can give me'? Well, right now in addition it says 'Just give me anything with keywords?. I would think that something like 'Oh, and if I am in the middle of an MRO: I will pass everything down the line? would be a much better description of it?s true intentions instead. > > > [Carl Smith] >> By using **kargs in the constructor and the call >> to `super`, you are indicating that the signature passes through > But can you be certain? Couldn?t someone have just used a `kargs.pop('eggs')`? Okay. You could argue that they probably wouldn?t have done so. But for making automated documentation it probably would be useful to make sure it didn?t happen. > > I think that (as Raymond Hettinger once said) 'super is super', but can?t you make it a bit smarter with telling it: 'Hey - If you don?t expect 'eggs', keep calm, it probably (or rather certainly) wasn?t meant for you so just pass it on to your super'. > Let's get some somewhat-more-plausible examples. class Pizza: def __init__(self, *, size, price): print("The price of this %s pizza is:", (size, price)) class HawaiianPizza(Pizza): def __init__(self, *, pineapple="chunked", **kw): print("This pizza has %s pineapple." % pineapple) super().__init__(**kw) class CheesyCrust(Pizza): """Mixin to alter the pizza's crust""" def __init__(self, *, crust_cheese="cheddar", surcharge=1.50): print("Surcharge %.2f for %s crust" % (surcharge, crust_cheese)) super().__init__(price=kw.pop("price") + surcharge, **kw) class BestPizza(HawaiianPizza, CheesyCrust): """Actually, the best pizza is pepperoni. Debate away!""" menu = [ BestPizza(size="large", price=12.50), BestPizza(size="personal", price=8.50), ] Okay. Now what are the possible keyword args for a BestPizza? The *entire set* of args for the hierarchy. You can set size, price, pineapple, crust_cheese, and surcharge. The fact that some of them (or even ALL of them, at the leaf node) are simply passed on up the chain doesn't change the fact that they're all valid. Does that make sense? ChrisA From steve at pearwood.info Sat May 26 04:29:04 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 26 May 2018 18:29:04 +1000 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: References: Message-ID: <20180526082903.GY12683@ando.pearwood.info> On Sat, May 26, 2018 at 09:39:14AM +0200, Michael Lohmann wrote: > [Steven D'Aprano] > > obj = Aardvark(27, spam=3, eggs=5, cheese=True) > > > > So you look up help(Aardvark), and it tells you that the signature is > > > > Aardvark.__init__(self, foo) > > > > What the hell? If Aardvark.__init__ only takes a single argument > > This is wrong! This would at some point down the line throw an error > TypeError: __init__() got an unexpected keyword argument 'eggs? What makes you say that? Since the kwargs are passed on to the super().__init__ call, the eggs argument is passed onto the super class of Aardvark. What makes you so certain that Aardvark's parent (or grandparent, or great-grandparent...) doesn't take a parameter "eggs"? Considering that *I* made this example up, it is a bit rich for you to tell me that "eggs" isn't used. Of course it is used, by one of the superclasses. That's why it was provided. > (or at some point **kwargs are being accepted and not passed on to > super which would be terrible on its own). No, it is a standard technique to collect keyword arguments, process those your class cares about, and pass the rest on to super(). Eventually the class second from the top (the class which inherits directly from object) MUST NOT pass those arguments to super, since object doesn't accept any arguments: you can't keep passing arguments up the MRO all the way to the top. At one point or another, each argument must be used and discarded. > The whole point I was trying to make is: If it doesn?t make any sense > to init a class with **kwargs: why write down that it would (or even > **could**) accept them? If your class doesn't accept **kwargs, then don't put **kwargs in the parameter list. > Shouldn?t the init tell you something like 'If > you instantiate this class then this is everything you can give me'? The way to do that is to write the __init__ of the class that only takes the parameters you want. > Well, right now in addition it says 'Just give me anything with > keywords?. Only if you intentionally add **kwargs to the parameter list. Why are you putting **kwargs in the parameter list if you don't want to accept them? > [Carl Smith] > > By using **kargs in the constructor and the call > > to `super`, you are indicating that the signature passes through > > But can you be certain? Couldn?t someone have just used a > `kargs.pop('eggs')`? "Somebody"? Who? It's my class. If "somebody" used kwargs.pop('eggs') it must have been me. If I did it, it means that my class wants to consume the eggs parameter and NOT pass it on. > You could argue that they probably wouldn?t > have done so. But for making automated documentation it probably would > be useful to make sure it didn?t happen. I don't understand this comment one bit. > I think that (as Raymond Hettinger once said) 'super is super', but > can?t you make it a bit smarter with telling it: 'Hey - If you don?t > expect 'eggs', keep calm, it probably (or rather certainly) wasn?t > meant for you so just pass it on to your super'. The way we do that is by explicitly collecting **kwargs and explicitly passing it on to the super call. I don't think I understand the scenario you have in mind. Here is what I have in mind: class Foo: def __init__(self, spam): self.spam = spam # this is a null-op, so it could be left out super().__init__() # calls object.__init__ which does nothing class Bar(Foo): def __init__(self, eggs, **kwargs): self.eggs = eggs super().__init__(**kwargs) class Baz(Bar): def __init__(self, cheese, **kwargs): self.cheese = cheese super().__init__(**kwargs) class Aardvark(Baz): def __init__(self, myarg, **kwargs): self.myarg = myarg super().__init__(**kwargs) obj = Aardvark(27, spam=3, eggs=5, cheese=True) What situation do you have in mind? -- Steve From kirillbalunov at gmail.com Sat May 26 05:00:45 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Sat, 26 May 2018 12:00:45 +0300 Subject: [Python-ideas] "Assignment expression" with function call-alike syntax In-Reply-To: <20180524012104.GM12683@ando.pearwood.info> References: <20180524012104.GM12683@ando.pearwood.info> Message-ID: 2018-05-24 4:21 GMT+03:00 Steven D'Aprano : > On Wed, May 23, 2018 at 12:32:36AM +0300, Kirill Balunov wrote: > > > Just one more variation on "assignment exression" syntax to make the list > > more complete :) Sorry, if something similar has already been suggested. > > The idea is to use function's call-like syntax in the from: `this( name = > > expr )`. > [...] > > In this way it is somewhat possible to make an assignment in `while` and > `if > > ` headers right now, which covers 90% cases, but it is damn slow. Maybe > be > > it is worth to make `this` magic call-alike object work fast...on the > other > > hand does anyone like `this`? > > Apart from needing extra typing, what does the function-call syntax gain > us? This will allow us to temporarily avoid the introduction of a new syntax. Since this option does not require a new syntax, you can provide a provisional module containing. This will allow to understand more clearly the actual use cases and the attitude of users towards the general idea. > It looks like a function you could call from anywhere, but you want > to limit it to just "while" and "if", I expect that will just give us a > flood of questions on Stackoverflow and other forums, "why can't I use > this() outside of if and while loops?" > I do not know how you decided that I propose to limit this call-alike variant only to `while` and `if`statements. I did not say that. I just said that personally I'm interested and see the advantages of using it only in `while` and `if` statements. Of course you are right that you can call it from anywhere. > > A good question. Why shouldn't we use assignment outside of if and while? > > Since this is special syntax, not a function, the parens don't give us > any benefit: > > this name = expression > > The word "this" seems inappropriate. Surely "let" or "set" would be > better: > > let name = expression > > which at least has the benefit of matching syntax chosen in other > languages. But if the only reason we include "let" is to avoid the > equals/assignment error, that seems needlessly verbose. We can say the > same thing more compactly: > I'm not much attached to `this`. Ok let it be `let`. > > name := expression > > > The difference between "let name = expression" and "name := expression" > is just spelling. The main point is to collect more information, since the idea of assignment expression will have a huge impact in all aspects of Python programming: how you structure your programm, how you write code, how you read code, how you parse code... Because at the moment the rule is simple that any name binding must occur only in statements. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat May 26 05:08:24 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 26 May 2018 19:08:24 +1000 Subject: [Python-ideas] "Assignment expression" with function call-alike syntax In-Reply-To: References: <20180524012104.GM12683@ando.pearwood.info> Message-ID: <20180526090824.GZ12683@ando.pearwood.info> On Sat, May 26, 2018 at 12:00:45PM +0300, Kirill Balunov wrote: > > It looks like a function you could call from anywhere, but you want > > to limit it to just "while" and "if", I expect that will just give us a > > flood of questions on Stackoverflow and other forums, "why can't I use > > this() outside of if and while loops?" > > I do not know how you decided that I propose to limit this call-alike > variant only to `while` and `if`statements. I did not say that. I just said > that personally I'm interested and see the advantages of using it only in > `while` and `if` statements. Of course you are right that you can call it > from anywhere. My mistake, sorry. -- Steve From antoine.rozo at gmail.com Sat May 26 05:21:03 2018 From: antoine.rozo at gmail.com (Antoine Rozo) Date: Sat, 26 May 2018 11:21:03 +0200 Subject: [Python-ideas] ternary without else In-Reply-To: References: <20180525113823.GV12683@ando.pearwood.info> <3bf167dd-d456-6582-f277-74345818e33f@btinternet.com> Message-ID: > if not article["art_wt"]: article["art_wt"] = 0 > if not article["px_pchs"]: article["px_pchs"] = 0 > if not article["px_calc"]: article["px_calc"] = 0 > if not article["px_sell"]: article["px_sell"] = 0 I think what you need is the setdefault method of dictionnaries, instead of a new syntax construct. 2018-05-26 1:52 GMT+02:00 Carl Smith : > ?You cannot have `expression if expression` in a language that also > supports > `expression if expression else expression` (like Python).? Otherwise, you > have > the dangling else problem: > > https://en.wikipedia.org/wiki/Dangling_else > > -- Carl Smith > carl.input at gmail.com > > On 25 May 2018 at 15:21, Rob Cliffe via Python-ideas < > python-ideas at python.org> wrote: > >> >> >> On 25/05/2018 12:38, Steven D'Aprano wrote: >> >>> >>> PEP 8 is not holy writ. Ignore it when your code is better for ignoring >>> it. >>> >>> >>> +1. Not infrequently I judge that my code is easier both to read and to >> maintain with more than 1 (short) statement on a line, >> often when it allows similar items to be aligned vertically. A >> spur-of-the moment, totally invented example: >> if MON <= day <= FRI: Rate = Normal >> if day==SAT: Rate = DoubleTime >> if day==SUN: Rate = TripleTime >> I also feel no inhibitions about using Jacco's original example: >> if cond: do_something >> Rob Cliffe >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Antoine Rozo -------------- next part -------------- An HTML attachment was scrubbed... URL: From mial.lohmann at gmail.com Sat May 26 05:22:39 2018 From: mial.lohmann at gmail.com (Michael Lohmann) Date: Sat, 26 May 2018 11:22:39 +0200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: References: Message-ID: Let me put it this way: class A(object): def __init__(self, a_value, **kwargs): print("This is a value:", a_value) super().__init__(**kwargs) Which parameters does `A` take when being initialized? Whenever you give any kwargs when directly instantiating `A` they will be passed down to super which in this case is `object`. And now to the follow-up question: Can you tell me which kwargs object takes as an input for it?s __init__? So does it EVER make ANY sense to specify them if you DIRECTLY create an instance of `A`? But now let?s say your MRO is `SuperClass, A, B`, then A should better be able to forward the kwargs, so currently you need to directly hand them to `A` and rely on the fact that it passes them on to `B`. Why should `A` even get those kwargs in the first place? They could just be "passed around it" as soon as the interpreter sees that they are in the what currently would be **kwargs and later reunited with the arguments that the super().__init__ call has. Of course this can?t be done by default. So why not tell the interpreter with something like "@pass_through_kwargs" that it should do this for you? [Chris Angelico] > Does that make sense? Well yes, of course it does. When instantiating a HawaiianPizza of course you want to set size and price. I don?t want to remove **kwargs and the current way of handeling this. But if you now add: class Lasagna: def __init__(self, *, number_of_layers=5): print("This Lasagna has %s layers", number_of_layers) class HaveYouEverTriedThis(Pizza, Lasagna): """Well, this is just bizarre""" Now suddenly `Pizza` would have to accept kwargs. Why????????? It didn?t "learn anything new". It can?t do anything with them on it?s own. Shouldn?t you expect from a function that in the brackets you are shown what is possible to actually call this function on? You could never actually call Pizza with kwargs, so 1) Why should you be able to do so in the first place? and 2) Why show the programmer that you could?. Just to make the MRO work. And I would suggest that to handle this it would be a lot more elegant to say: "If you are in the middle of an MRO: just pass everything unexpected down? [Steven D'Aprano] > Considering that *I* made this example up, it is a bit rich for you to > tell me that "eggs" isn't used. Of course it is used, by one of the > superclasses. That's why it was provided. Well, then your code should stay with the current version of accepting **kwargs and not do this. And of course you can only use one of **kwargs and this new proposal From antoine.rozo at gmail.com Sat May 26 05:24:42 2018 From: antoine.rozo at gmail.com (Antoine Rozo) Date: Sat, 26 May 2018 11:24:42 +0200 Subject: [Python-ideas] ternary without else In-Reply-To: References: <20180525113823.GV12683@ando.pearwood.info> <3bf167dd-d456-6582-f277-74345818e33f@btinternet.com> Message-ID: Dismiss my message, I have read `if "art_wt" not in article`. But in the same way, you could have a function to reset a value in your dict if the current value evaluates to False. 2018-05-26 11:21 GMT+02:00 Antoine Rozo : > > if not article["art_wt"]: article["art_wt"] = 0 > > if not article["px_pchs"]: article["px_pchs"] = 0 > > if not article["px_calc"]: article["px_calc"] = 0 > > if not article["px_sell"]: article["px_sell"] = 0 > > I think what you need is the setdefault method of dictionnaries, instead > of a new syntax construct. > > 2018-05-26 1:52 GMT+02:00 Carl Smith : > >> ?You cannot have `expression if expression` in a language that also >> supports >> `expression if expression else expression` (like Python).? Otherwise, you >> have >> the dangling else problem: >> >> https://en.wikipedia.org/wiki/Dangling_else >> >> -- Carl Smith >> carl.input at gmail.com >> >> On 25 May 2018 at 15:21, Rob Cliffe via Python-ideas < >> python-ideas at python.org> wrote: >> >>> >>> >>> On 25/05/2018 12:38, Steven D'Aprano wrote: >>> >>>> >>>> PEP 8 is not holy writ. Ignore it when your code is better for ignoring >>>> it. >>>> >>>> >>>> +1. Not infrequently I judge that my code is easier both to read and >>> to maintain with more than 1 (short) statement on a line, >>> often when it allows similar items to be aligned vertically. A >>> spur-of-the moment, totally invented example: >>> if MON <= day <= FRI: Rate = Normal >>> if day==SAT: Rate = DoubleTime >>> if day==SUN: Rate = TripleTime >>> I also feel no inhibitions about using Jacco's original example: >>> if cond: do_something >>> Rob Cliffe >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > > > -- > Antoine Rozo > -- Antoine Rozo -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat May 26 07:09:35 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 26 May 2018 21:09:35 +1000 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: References: Message-ID: On Sat, May 26, 2018 at 7:22 PM, Michael Lohmann wrote: > [Chris Angelico] >> Does that make sense? > Well yes, of course it does. When instantiating a HawaiianPizza of course you want to set size and price. I don?t want to remove **kwargs and the current way of handeling this. But if you now add: > class Lasagna: > def __init__(self, *, number_of_layers=5): > print("This Lasagna has %s layers", number_of_layers) > > class HaveYouEverTriedThis(Pizza, Lasagna): > """Well, this is just bizarre""" > > Now suddenly `Pizza` would have to accept kwargs. Why????????? It didn?t "learn anything new". It can?t do anything with them on it?s own. Shouldn?t you expect from a function that in the brackets you are shown what is possible to actually call this function on? You could never actually call Pizza with kwargs, so 1) Why should you be able to do so in the first place? and 2) Why show the programmer that you could?. Just to make the MRO work. And I would suggest that to handle this it would be a lot more elegant to say: "If you are in the middle of an MRO: just pass everything unexpected down? > Right, which means that Pizza and Lasagna are not compatible classes in that way. If you were to try to concoct some sort of, I don't know, layered conglomerate with meat, tomato, and pizza bases, you'd have to write an init method that does that for you. class Pizzagna(Pizza, Lasagna): def __init__(self, *, number_of_layers, **kw): Lasagna.__init__(self, number_of_layers=number_of_layers) Pizza.__init__(self, **kw) The Pizza class was deliberately designed to be an apex class - one that is the head of a hierarchy. The Lasagna class isn't part of that hierarchy, so merging it in takes some extra work - a pizza can't automatically subsume a lasagna. We had the family over here for dinner tonight, and we ate two pizzas. Now I'm wondering if a Pizzagna would have been a better choice... Anyhow. Further discussion about this should probably go onto python-list. ChrisA From mial.lohmann at gmail.com Sat May 26 09:09:50 2018 From: mial.lohmann at gmail.com (Michael Lohmann) Date: Sat, 26 May 2018 15:09:50 +0200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: References: Message-ID: > Right, which means that Pizza and Lasagna are not compatible classes > in that way. Okay, let me try it one final time with the original pizza example. Let?s assume that your restaurant has a special offer on all Hawaiian Pizzas where you can get all sizes for 10$. Now the only reasonable thing to pass into **kw is size, so let?s say that for the readability of the class we decide to replace it. class Pizza: def __init__(self, *, size, price): print("The price of this %s pizza is:", (size, price)) class HawaiianPizza(Pizza): def __init__(self, *, pineapple="chunked", size=8): # No **kw here since no longer needed print("This pizza has %s pineapple." % pineapple) super().__init__(price=10, size=size) class CheesyCrust(Pizza): """Mixin to alter the pizza's crust""" def __init__(self, *, crust_cheese="cheddar", surcharge=1.50): print("Surcharge %.2f for %s crust" % (surcharge, crust_cheese)) super().__init__(price=kw.pop("price") + surcharge, **kw) class BestPizza(HawaiianPizza, CheesyCrust): """Actually, the best pizza is pepperoni. Debate away!""" BestPizza(crust_cheese="gouda?) # Doesn?t work since Hawaii doesn?t bypass it But now the HawaiianPizza gets annoyed because it doesn?t know what to do with the crust_cheese. So just to make BestPizza work I will have to add **kw to HawaiianPizza again. But now let?s say we have a hungry programmer that just takes a short look and sees that HawaiianPizza is a subclass of Pizza and he thinks "Okay then, just get some HawaiianPizza(price=8)". In fact if he directly orders any HawaiianPizza there is nothing he can pass in into **kw that wouldn?t result in an error. I am sorry to annoy you all with this. Maybe this problem just isn?t as common as I thought it was? Michael From brenbarn at brenbarn.net Sat May 26 13:57:35 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Sat, 26 May 2018 10:57:35 -0700 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: References: Message-ID: <5B09A00F.9030806@brenbarn.net> On 2018-05-26 02:22, Michael Lohmann wrote: > Whenever you give any kwargs when directly instantiating `A` they > will be passed down to super which in this case is `object`. And now > to the follow-up question: Can you tell me which kwargs object takes > as an input for it?s __init__? So does it EVER make ANY sense to > specify them if you DIRECTLY create an instance of `A`? > > But now let?s say your MRO is `SuperClass, A, B`, then A should > better be able to forward the kwargs, so currently you need to > directly hand them to `A` and rely on the fact that it passes them on > to `B`. If I understand correctly, the essence of your argument seems to be that you want be able to write a class A, and you want to be able to use that class EITHER as the top of an inheritance chain (i.e., have it inherit directly from object) OR in the middle of an inheritance chain (i.e., inheriting from some other class, but not object). But you can't, because if you pass on extra **kwargs, that will fail if the class inherits directly from object, but if you don't pass on extra **kwargs, that will fail if the class doesn't inherit directly from object. Is that correct? I agree that it is somewhat somewhat awkward that "is this the top class in the hierarchy" is something that has to be known when writing a class. I think this would be ameliorated by having "object" accept and ignore extra arguments. (I seem to recall that was decided to be a bad idea at some time in the past, though.) But I don't really see how your solution of magically making kwargs appear and disappear is a good solution to that problem. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From mial.lohmann at gmail.com Sun May 27 04:42:08 2018 From: mial.lohmann at gmail.com (Michael Lohmann) Date: Sun, 27 May 2018 10:42:08 +0200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: <5B09A00F.9030806@brenbarn.net> References: <5B09A00F.9030806@brenbarn.net> Message-ID: <0C4710A9-DB39-43DA-8009-DB3F96CC66DC@googlemail.com> > If I understand correctly, the essence of your argument seems to be that you want be able to write a class A, and you want to be able to use that class EITHER as the top of an inheritance chain (i.e., have it inherit directly from object) OR in the middle of an inheritance chain (i.e., inheriting from some other class, but not object). Well, it does not necessarily inherit from object, but from any class, that does not accept any more kwargs. E.g. if you have a diamond structure as your class-hierachy then one branch could forward information to the second (if you understand what I mean?). > But I don't really see how your solution of magically making kwargs appear and disappear is a good solution to that problem. I intended the following text for the python-list mailing-list, but it I think I might have structured my ideas a bit better than in my previous messages and the summary in __Reason__ might tackle why this could be a nice idea (spoiler: make super even more super by doing things if it is NOT there in a clever manner). Let us start with a simple class: class Aardvark: def __init__(self, quantity): print("There is some quantity:", quantity) Well, that works nicely and we can go and send our Aardvark to survey some quantities. But if we now want to get the multi-inheritance to work properly we need to change it to: class Aardvark: def __init__(self, quantity, **kwargs): print("There is some quantity:", quantity) # I actually don?t care about **kwargs and just hand them on super().__init__(**kwargs) class Clever: def __init__(self, cleverness=1): print("You are %d clever? % cleverness) class Ethel(Aardvark, Clever): """Ethel is a very clever Aardvark""" def __init__(self): super().__init__(quantity="some spam", cleverness=1000) But if you now look at the declaration of the Aardvark .__init__, it seems like you could instantiate it with **kwargs. This in fact is not true. As soon as you create a direct instance of Aardvark, `object` as the super() doesn?t accept any kwargs. So basically I think that the parameters for the init should just say `quantity ` while still preserving the functionality. Now that obviously doesn?t work until now. But could you add something that lets this class tell the interpreter instead: "Hey, could you just forward anything that this init doesn?t need to super().__init__" ? I have something like this in mind: class Aardvark: @bypass_kwargs_to_super def __init__(self, *, quantity): print("There is some quantity:", quantity) super().__init__() This would collect everything "behind the scenes" that usually **kwargs would have collected and "append" it to the super call. Question: If Aardvark knows that he really does not need the kwargs himself: why give them to him in the first place? I mean, I trust him - he seems like a very nice guy, but he might have accidentally forgotten to forward them unintentionally. You would obviously still be able to use **kwargs the usual way but then you couldn?t use this technique and would need to take care of passing down all the information to super() yourself as usual. __Reason__: With something like this it is immediately obvious that Aardvark ONLY takes `quantity` as an input if you instantiate it directly but if subclassed it is able to hand information down the MRO to something from a different "Branch". And in addition to the better human readability: any form of automated docstring-genaration now can be certain that you don?t do anything with (or to) the **kwargs (since the init doesn?t get them in the first place). In addition it could make super even more super by doing something if it is NOT there as proposed in the second of the the following problems. Of course you need to be quite clever to do this properly, for example 1) What to do if there are collisions between the bypassed kwargs and the ones from the init call? - Probably keep the ones you bypassed since they come from the top of the MRO 2) What do you do if super().__init__ was not called? The most clever thing would be to go ?up and to the next branch? of the inheritance diagram. As in: if Aardvark is a Subclass of Animal, don?t call its init but directly Clevers - (you would have to look up the MRO of the super of Aardvark and skip them in the Ethel MRO before calling the next init automatically). In theory you could also add something that forwards *args as well but the usage of that is probably much more limited... On thing that might look a bit strange is that you actually could actually pass in additional kwargs despite the init saying otherwise: `Aardvark(quantity='some spam', something="Object will throw an error now unexpected kwarg")` and this would then throw an error that "object (instead of Aardvark) does not expect something". But I guess that would be okay since the init now is much better readable of what it actually expects. Michael From ncoghlan at gmail.com Sun May 27 11:02:26 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 28 May 2018 01:02:26 +1000 Subject: [Python-ideas] Proposal: A Reduce-Map Comprehension and a "last" builtin In-Reply-To: References: <20180524124900.GP12683@ando.pearwood.info> Message-ID: On 26 May 2018 at 04:14, Tim Peters wrote: > [Peter O'Connor] > >> ... > >> We could use given for both the in-loop variable update and the variable > >> initialization: > >> smooth_signal = [average given average=(1-decay)*average + decay*x > >> for x in signal] given average=0. > > [Steven D'Aprano ] > > I don't think that will work under Nick's proposal, as Nick does not > > want assignments inside the comprehension to be local to the surrounding > > scope. (Nick, please correct me if I'm wrong.) > > Nick appears to have moved on from "given" to more-general augmented > assignment expressions. Aye, while I still don't want comprehensions to implicitly create new locals in their parent scope, I've come around on the utility of letting inline assignment targets be implicitly nonlocal references to the nearest block scope. > See PEP 577, but note that it's still a > work-in-progress: > > https://github.com/python/peps/pull/665 > > > Under that PEP, > > average = 0 > smooth_signal = [(average := (1-decay)*average + decay*x) > for x in signal] > > Or, for the running sums example: > > total = 0 > sums = [(total += x) for x in data] > > I'm not entirely clear on whether the "extra" parens are needed, so > added 'em anyway to make grouping clear. > I think the parens would technically be optional (as in PEP 572), since "EXPR for" isn't legal syntax outside parentheses/brackets/braces, so the parser would terminate the assignment expression when it sees the "for" keyword. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun May 27 08:45:09 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 28 May 2018 00:45:09 +1200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: <5B09A00F.9030806@brenbarn.net> References: <5B09A00F.9030806@brenbarn.net> Message-ID: <5B0AA855.6090604@canterbury.ac.nz> Brendan Barnwell wrote: > If I understand correctly, the essence of your argument seems to be > that you want be able to write a class A, and you want to be able to use > that class EITHER as the top of an inheritance chain (i.e., have it > inherit directly from object) OR in the middle of an inheritance chain > (i.e., inheriting from some other class, but not object). This shouldn't be a problem if each method along the way absorbs all the arguments it knows about, and only passes on the ones it doesn't. E.g. class A: def __init__(self, alpha, **kwds): print("alpha =", alpha) super().__init__(**kwds) class B(A): def __init__(self, beta, **kwds): print("beta =", beta) super().__init__(**kwds) b = B(alpha = 17, beta = 42) Both A and B here pass on a **kwds argument, but by the time it get to object, there are no arguments left, so it's fine. -- Greg From mial.lohmann at gmail.com Sun May 27 13:19:56 2018 From: mial.lohmann at gmail.com (Michael Lohmann) Date: Sun, 27 May 2018 19:19:56 +0200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: <5B0AA855.6090604@canterbury.ac.nz> References: <5B09A00F.9030806@brenbarn.net> <5B0AA855.6090604@canterbury.ac.nz> Message-ID: <0ACB829E-954D-4801-B329-FA689C6042F0@googlemail.com> I realized that bypassing kwargs is probably the least important thing of this idea - so if implemented it definitely had to get a better name. Just look at the following example: class Magic: magic_number = 42 def __init__(self): A.magic_number = 0 # As soon as you look too deep into it all the Magic vanishes class Foo(Magic): def __init__(self): print("foo") # Let's say we want magic_number==42 in Foo # So you can?t call super().__init__ here if it is not in the middle of an MRO # Technically you could get it working by going to the first init in the MRO of self after Foo \ # that isn?t in the MRO of Foo but you can see that this gets quite ugly to write (and read!). # And then you would still have the problem of indicating that Foo seems to accepted kwargs as input class Bar: def __init__(self, bar): print("bar:", bar) class FooBar(Foo, Bar): def __init__(self): # There is no easy way right now to avoid writing Foo.__init__() Bar.__init__(bar="bar?) But if Foo adopted this protocol of "automated MRO handling"/"cooperativity" (or however you want to call it) Bar.__init__ could be called automatically. What do I mean by that? Well, basically do automatically what I described in the comment of Foo if no super().__init__ call was registered in any cooperative class. Okay, this example might be a bit far-fetched, but it shows, that you could easily get the MRO working as expected with a simple super().__init__(bar="bar") in FooBar. > This shouldn't be a problem if each method along the way > absorbs all the arguments it knows about, and only passes > on the ones it doesn?t. Two remarks: 1) Everything my idea has to offer really is just reasonable if you don?t have single inheritance only and 2) This wasn?t really my problem with the current status (in fact: the way I would tackle this problem would also only work if all kwargs get fully consumed). But: What bugs me is that in my example from yesterday ( class Aardvark: def __init__(self, quantity, **kwargs): print("There is some quantity:", quantity) # I actually don?t care about **kwargs and just hand them on super().__init__(**kwargs) class Clever: def __init__(self, cleverness=1): print("You are %d clever? % cleverness) class Ethel(Aardvark, Clever): """Ethel is a very clever Aardvark""" def __init__(self): super().__init__(quantity="some spam", cleverness=1000) ) if you want to instantiate an Aardvark directly there is NO WAY EVER that you could give him any kwargs. So why should the __init__ indicate something else? Well, just to make the MRO work. All I want is to make it as obvious as possible that an Aardvark ONLY takes `quantity` as input, but is fully "cooperative" with other classes if it is in the middle of the MRO (by which I mean that it will automatically call the __init__ and hand on any kwargs it didn?t expect to a class from a different branch of the class hierarchy). From mial.lohmann at gmail.com Sun May 27 14:07:18 2018 From: mial.lohmann at gmail.com (Michael Lohmann) Date: Sun, 27 May 2018 20:07:18 +0200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: <0ACB829E-954D-4801-B329-FA689C6042F0@googlemail.com> References: <5B09A00F.9030806@brenbarn.net> <5B0AA855.6090604@canterbury.ac.nz> <0ACB829E-954D-4801-B329-FA689C6042F0@googlemail.com> Message-ID: > Everything my idea has to offer really is just reasonable if you don?t have single inheritance only I would like to correct myself immediately on that one: In the Pizza-example (from yesterday as well) it would be possible to overwrite the default price of the HawaiianPizza with the merging of bypassed kwargs with the ones from the call itself. Let us assume that the exemption of the 10$ price for all HawaiianPizzas would be a Family-sized pizza for 20$: class Pizza: def __init__(self, *, size, price): print("The price of this %s pizza is:", (size, price)) @cooperative class HawaiianPizza(Pizza): def __init__(self, *, pineapple="chunked", size=8): print("This pizza has %s pineapple." % pineapple) super().__init__(price=10, size=size) class FamilyHawaiianPizza(HawaiianPizza): def __init__(self, *, pineapple="chunked"): super().__init__(price=20, size=20, pineapple=pineapple) #This would overwrite the price of the HawaiianPizza But this would not be the real intention of the usage, but just a side effect of correctly handling a simple diamond-like hierarchy. The reason that bypassed kwargs should overwrite them is that they come from up the MRO and so obviously should have priority. Sorry to spam you all, Have a nice Sunday! Michael From greg.ewing at canterbury.ac.nz Sun May 27 20:17:35 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 28 May 2018 12:17:35 +1200 Subject: [Python-ideas] Proposal: A Reduce-Map Comprehension and a "last" builtin In-Reply-To: References: <20180524124900.GP12683@ando.pearwood.info> Message-ID: <5B0B4A9F.5020204@canterbury.ac.nz> Nick Coghlan wrote: > Aye, while I still don't want comprehensions to implicitly create new > locals in their parent scope, I've come around on the utility of letting > inline assignment targets be implicitly nonlocal references to the > nearest block scope. What if you're only intending to use it locally within the comprehension? Would you have to put a dummy assignment in the surrounding scope to avoid a NameError? That doesn't sound very nice. -- Greg From greg.ewing at canterbury.ac.nz Sun May 27 20:34:28 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 28 May 2018 12:34:28 +1200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: <0ACB829E-954D-4801-B329-FA689C6042F0@googlemail.com> References: <5B09A00F.9030806@brenbarn.net> <5B0AA855.6090604@canterbury.ac.nz> <0ACB829E-954D-4801-B329-FA689C6042F0@googlemail.com> Message-ID: <5B0B4E94.5020706@canterbury.ac.nz> Michael Lohmann wrote: > class Magic: > magic_number = 42 > def __init__(self): > A.magic_number = 0 # As soon as you look too deep into it all the Magic vanishes What is A here? Did you mean something else? -- Greg From j.van.dorp at deonet.nl Mon May 28 02:37:47 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Mon, 28 May 2018 08:37:47 +0200 Subject: [Python-ideas] "Assignment expression" with function call-alike syntax In-Reply-To: References: <20180524012104.GM12683@ando.pearwood.info> Message-ID: 2018-05-26 11:00 GMT+02:00 Kirill Balunov : > The main point is to collect more information, since the idea of assignment > expression will have a huge impact in all aspects of Python programming: how > you structure your programm, how you write code, how you read code, how you > parse code... Because at the moment the rule is simple that any name binding > must occur only in statements. > > With kind regards, > -gdg Downside here is, if you first implement it like this, everyone will be used to that implementation. If after that a special syntax gets introduced, there will be people confusing it and having to use multiple libraries which use the two different ways. And most people will stick to the old way because it'll have more compatible python versions, no matter how much better a new syntax could be. Unless you want a deprecation cycle planning for a feature not even implemented yet ? I think that if implemented, it needs to be the final implementation right away. From j.van.dorp at deonet.nl Mon May 28 02:47:11 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Mon, 28 May 2018 08:47:11 +0200 Subject: [Python-ideas] ternary without else In-Reply-To: References: <20180525113823.GV12683@ando.pearwood.info> <3bf167dd-d456-6582-f277-74345818e33f@btinternet.com> Message-ID: 2018-05-26 11:24 GMT+02:00 Antoine Rozo : > Dismiss my message, I have read `if "art_wt" not in article`. But in the > same way, you could have a function to reset a value in your dict if the > current value evaluates to False. That won't work, since at other places, I do the same with bools and strings in the same dict. (I read out an online database, which only yields data in json format. I dont want an extra parsing step to transform it to another data structure than dicts. ). Also, empty fields are set to None, no matter actual field type(actually, it doesn't pass empty fields for some godawful reason, so I have a __missing__ that returns None if and only if it's in a list of expected fields. ). As for the dangling else problem, noting a good rule (left to right ?) and a few parens in other cases can't fix. That said, because of the rest of this thread, it seems to me like a statement would be the proper implementation. So the thread title has become misleading. This would remove the dangling else problem unless there's an if/else in the condition, which I can't figure out why you'd need that. From mial.lohmann at gmail.com Mon May 28 03:27:24 2018 From: mial.lohmann at gmail.com (Michael Lohmann) Date: Mon, 28 May 2018 09:27:24 +0200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: <5B0B4E94.5020706@canterbury.ac.nz> References: <5B09A00F.9030806@brenbarn.net> <5B0AA855.6090604@canterbury.ac.nz> <0ACB829E-954D-4801-B329-FA689C6042F0@googlemail.com> <5B0B4E94.5020706@canterbury.ac.nz> Message-ID: <1D0C43C8-F674-4D0D-9D5B-471DE8BEEBD0@googlemail.com> >> class Magic: >> magic_number = 42 >> def __init__(self): >> A.magic_number = 0 # As soon as you look too deep into it all the Magic vanishes > > What is A here? Did you mean something else? Sorry for that. Yes, it should have been Magic (I renamed the class after writing it and didn?t pay attention). I just wanted to override the class-variable `magic_number` to give a reason why I don?t ever want to call Magic.__init__ in Foo. If you want, you can have this class instead: class Magic: def __init__(self): raise RuntimeError("Do not initialize this class") but I figured that this might look a bit artificial... From j.van.dorp at deonet.nl Mon May 28 03:31:54 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Mon, 28 May 2018 09:31:54 +0200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: <1D0C43C8-F674-4D0D-9D5B-471DE8BEEBD0@googlemail.com> References: <5B09A00F.9030806@brenbarn.net> <5B0AA855.6090604@canterbury.ac.nz> <0ACB829E-954D-4801-B329-FA689C6042F0@googlemail.com> <5B0B4E94.5020706@canterbury.ac.nz> <1D0C43C8-F674-4D0D-9D5B-471DE8BEEBD0@googlemail.com> Message-ID: I'd say NOT wanting to call an __init__ method of a superclass is a rather uncommon occurence. It's generally a huge error. So I think it's worth not accomodating that. 2018-05-28 9:27 GMT+02:00 Michael Lohmann : > >>> class Magic: >>> magic_number = 42 >>> def __init__(self): >>> A.magic_number = 0 # As soon as you look too deep into it all the Magic vanishes >> >> What is A here? Did you mean something else? > > Sorry for that. Yes, it should have been Magic (I renamed the class after writing it and didn?t pay attention). I just wanted to override the class-variable `magic_number` to give a reason why I don?t ever want to call Magic.__init__ in Foo. If you want, you can have this class instead: > > class Magic: > def __init__(self): > raise RuntimeError("Do not initialize this class") > > but I figured that this might look a bit artificial... > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From mial.lohmann at gmail.com Mon May 28 03:44:48 2018 From: mial.lohmann at gmail.com (Michael Lohmann) Date: Mon, 28 May 2018 09:44:48 +0200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: References: <5B09A00F.9030806@brenbarn.net> <5B0AA855.6090604@canterbury.ac.nz> <0ACB829E-954D-4801-B329-FA689C6042F0@googlemail.com> <5B0B4E94.5020706@canterbury.ac.nz> <1D0C43C8-F674-4D0D-9D5B-471DE8BEEBD0@googlemail.com> Message-ID: > I'd say NOT wanting to call an __init__ method of a superclass is a > rather uncommon occurence. It's generally a huge error. So I think > it's worth not accomodating that. I will give you an example then where I am absolutely fine with calling super().__init__ in all classes and describe why I am not really satisfied with the current status. (It is from my email from yesterday 17:19 GMT): > What bugs me is that in my example from yesterday ( > class Aardvark: > def __init__(self, quantity, **kwargs): > print("There is some quantity:", quantity) > # I actually don?t care about **kwargs and just hand them on > super().__init__(**kwargs) > > class Clever: > def __init__(self, cleverness=1): > print("You are %d clever? % cleverness) > > class Ethel(Aardvark, Clever): > """Ethel is a very clever Aardvark""" > def __init__(self): > super().__init__(quantity="some spam", cleverness=1000) > ) if you want to instantiate an Aardvark directly there is NO WAY EVER that you could give him ANY kwargs. So why should the __init__ indicate something else? Well, just to make the MRO work. All I want is to make it as obvious as possible that an Aardvark only takes `quantity` as input, but is fully "cooperative" with other classes if it is in the middle of the MRO (by which I mean that it will automatically call the __init__ and hand on any kwargs it didn?t expect to a class from a different branch of the class hierarchy). From j.van.dorp at deonet.nl Mon May 28 04:24:08 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Mon, 28 May 2018 10:24:08 +0200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: References: <5B09A00F.9030806@brenbarn.net> <5B0AA855.6090604@canterbury.ac.nz> <0ACB829E-954D-4801-B329-FA689C6042F0@googlemail.com> <5B0B4E94.5020706@canterbury.ac.nz> <1D0C43C8-F674-4D0D-9D5B-471DE8BEEBD0@googlemail.com> Message-ID: 2018-05-28 9:44 GMT+02:00 Michael Lohmann : > >> I'd say NOT wanting to call an __init__ method of a superclass is a >> rather uncommon occurence. It's generally a huge error. So I think >> it's worth not accomodating that. > > I will give you an example then where I am absolutely fine with calling super().__init__ in all classes and describe why I am not really satisfied with the current status. (It is from my email from yesterday 17:19 GMT): > >> What bugs me is that in my example from yesterday ( >> class Aardvark: >> def __init__(self, quantity, **kwargs): >> print("There is some quantity:", quantity) >> # I actually don?t care about **kwargs and just hand them on >> super().__init__(**kwargs) >> >> class Clever: >> def __init__(self, cleverness=1): >> print("You are %d clever? % cleverness) >> >> class Ethel(Aardvark, Clever): >> """Ethel is a very clever Aardvark""" >> def __init__(self): >> super().__init__(quantity="some spam", cleverness=1000) >> ) if you want to instantiate an Aardvark directly there is NO WAY EVER that you could give him ANY kwargs. So why should the __init__ indicate something else? Well, just to make the MRO work. All I want is to make it as obvious as possible that an Aardvark only takes `quantity` as input, but is fully "cooperative" with other classes if it is in the middle of the MRO (by which I mean that it will automatically call the __init__ and hand on any kwargs it didn?t expect to a class from a different branch of the class hierarchy). This is more an issue with your Ethel class. In *normal* subclassing, it's init would look like: class Ethel(Aardvark, Clever): def __init__(self, **kwargs): super().__init__(**kwargs) and you'd instantiate it like: e = Ethel(quantity="some spam", cleverness=1000) or even (because keyword argument order doesn't matter): e = Ethel(cleverness=1000, quantity="some spam") Because don't forget: assert isinstance(e, Ethel) assert isinstance(e, Aardvark) assert isinstance(e, Clever) will all pass perfectly fine. It's not just an Ethel or an Aardvark, it's also a Clever. (which doesn't sound like it makes sense...perhaps clever should be an attribute on an Aardvark/Ethel instead ?). That all aside, for a moment. You actually CAN call object.__init__(**kwargs) no problem - as long as kwargs is empty. I'd have written your classes like this: class Aardvark: def __init__(self, quantity, **kwargs): print("There is some quantity:", quantity) # I actually don?t care about **kwargs and just hand them on super().__init__(**kwargs) class Clever: def __init__(self, cleverness=1, **kwargs): print("You are %d clever" % cleverness) super().__init__(**kwargs) class Ethel(Aardvark, Clever): """Ethel is a very clever Aardvark""" def __init__(self, **kwargs): super().__init__(**kwargs) Just try it - it works perfectly fine. Each constructor will consume the keywords meant for it, therefore, the last super() call will call the object constructor without keyword arguments. **kwargs is the price we have to pay for good multiple inheritance - and IMO, it's a low one. If we're talking about signalling what the arguments mean, just ensure you have a good docstring. class Aardvark: def __init__(self, quantity, **kwargs): """ Aardvarks are gentle creatures, and therefore cooperate with multiple inheritance. :param quantity: The quantity of Aardvark(s). """ From j.van.dorp at deonet.nl Mon May 28 05:46:46 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Mon, 28 May 2018 11:46:46 +0200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: <9A22B236-369B-4359-8CCC-63BDFD3620A8@googlemail.com> References: <5B09A00F.9030806@brenbarn.net> <5B0AA855.6090604@canterbury.ac.nz> <0ACB829E-954D-4801-B329-FA689C6042F0@googlemail.com> <5B0B4E94.5020706@canterbury.ac.nz> <1D0C43C8-F674-4D0D-9D5B-471DE8BEEBD0@googlemail.com> <9A22B236-369B-4359-8CCC-63BDFD3620A8@googlemail.com> Message-ID: 2018-05-28 11:07 GMT+02:00 Michael Lohmann : > But maybe it is just me who thinks that you should make it as obvious as possible what a class itself really can get as an input and what is done just to get the multiple inheritance to work... So I think if no one else agrees with me, we don?t need to further spam everybody. I think the largest difference is that most people have a different opinion about what is more explicit/obvious. It does look like you could build something like this for yourself with a decorator: from inspect import signature, Parameter def forwards_kwargs_to_super(classname): def deco(method): sig = signature(method) numargs = len([param for param in sig.parameters.values() if param.kind is Parameter.POSITIONAL_ONLY]) keykwargs = {param.name for param in sig.parameters.values() if param.kind in (Parameter.POSITIONAL_OR_KEYWORD, Parameter.KEYWORD_ONLY)} def newinit(self, *args, **kwargs): print(f"Call looks like: {classname}: {args[:numargs]}, { {key: val for key, val in kwargs.items() if key in keykwargs} }") method(self, *args[:numargs], **{key: val for key, val in kwargs.items() if key in keykwargs}) super(globals()[classname], self).__init__(*args[numargs:], **{key: val for key, val in kwargs.items() if key not in keykwargs}) return newinit return deco And then use it like this: class Aardvark: @forwards_kwargs_to_super("Aardvark") def __init__(self, quantity): print("There is some quantity:", quantity) # I actually don?t care about **kwargs and just hand them on class Clever: @forwards_kwargs_to_super("Clever") def __init__(self, cleverness=1): print("You are %d clever" % cleverness) class Ethel(Aardvark, Clever): """Ethel is a very clever Aardvark""" @forwards_kwargs_to_super("Ethel") def __init__(self): pass e = Ethel(quantity="some spam", cleverness=100) Disclaimer: - Whenever you import the inspect module, you should ask yourself whether it's really a good idea. - I'm assuming that decorated functions have no *args or **kwargs in their signature that you DO want to accept, and that all arguments with a default value are given as keyword arguments. - If you are passing new positional arguments with this function, the subclass positional arguments need to come before the superclass positional arguments (only important if both have positional arguments.) - If you use multiple decorators, this should be the innermost decorator, or it'll probably make your code barf at runtime. - If you try do this in my company, i'll do whatever I can to get you fired. For the love of Python, don't use this code at places where anybody except you has to do maintenance. And if you come back in half a year and are amazed by your own code, I promise you'll understand why this might not be a real good solution. - Yes, passing your class name and then getting it from globals is an ugly hack. The alternative is grabbing it from the stack. If anybody else has a better idea, feel free, but AFAIK it's impossible to do this better. (I once wrote an overload decorator as a personal exercise, using the stack hack.) - I haven't tested this with actual positional arguments, but I -think- it will work. From j.van.dorp at deonet.nl Mon May 28 05:49:50 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Mon, 28 May 2018 11:49:50 +0200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: References: <5B09A00F.9030806@brenbarn.net> <5B0AA855.6090604@canterbury.ac.nz> <0ACB829E-954D-4801-B329-FA689C6042F0@googlemail.com> <5B0B4E94.5020706@canterbury.ac.nz> <1D0C43C8-F674-4D0D-9D5B-471DE8BEEBD0@googlemail.com> <9A22B236-369B-4359-8CCC-63BDFD3620A8@googlemail.com> Message-ID: Bright idea the moment after sending that mail: You could remove the globals() hack if you make it a class decorator instead. From ncoghlan at gmail.com Mon May 28 09:20:09 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 28 May 2018 23:20:09 +1000 Subject: [Python-ideas] Proposal: A Reduce-Map Comprehension and a "last" builtin In-Reply-To: <5B0B4A9F.5020204@canterbury.ac.nz> References: <20180524124900.GP12683@ando.pearwood.info> <5B0B4A9F.5020204@canterbury.ac.nz> Message-ID: On 28 May 2018 at 10:17, Greg Ewing wrote: > Nick Coghlan wrote: > >> Aye, while I still don't want comprehensions to implicitly create new >> locals in their parent scope, I've come around on the utility of letting >> inline assignment targets be implicitly nonlocal references to the nearest >> block scope. >> > > What if you're only intending to use it locally within the > comprehension? Would you have to put a dummy assignment in > the surrounding scope to avoid a NameError? That doesn't > sound very nice. > The draft PEP discusses that - it isn't saying "Always have them raise TargetNameError, now and forever", it's saying "Have them raise TargetNameError in the first released iteration of the capability, so we can separate the discussion of binding semantics in scoped expressions from the discussion of declaration semantics". I still want to leave the door open to giving comprehensions and lambdas a way to declare and bind truly local variables, and that gets more difficult if we go straight to having the binding expressions they contain *implicitly* declare new variables in the parent scope (rather than only binding previously declared ones). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon May 28 12:16:04 2018 From: brett at python.org (Brett Cannon) Date: Mon, 28 May 2018 09:16:04 -0700 Subject: [Python-ideas] multiprocessing, freeze support by default for Pool In-Reply-To: References: Message-ID: I would just open an issues on bugs.python.org for this. On Wed, 23 May 2018 at 17:54 Bradley Phillips < bradleyp81+python-ideas at gmail.com> wrote: > Hi All > > There are about 347 results for the following query on the google search > engine: > "python multiprocessing pyinstaller crash site:stackoverflow.com" > > The windows fork bomb is a known frequent occurrence. I know that > freeze_support is supposed to be called but this issue still catches me out > most times (and many others). > > I propose adding to line 117 of multiprocessing/context.py > if (sys.platform == 'win32'): > self.freeze_support() > > Thanks > Regards > Brad > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Mon May 28 16:23:03 2018 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Mon, 28 May 2018 22:23:03 +0200 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) Message-ID: ...as in (not tested): def _rchown(dir, user, group): for root, dirs, files in os.walk(dir, topdown=False): for name in files: chown(os.path.join(root, name), user, group) def chown(path, user=None, group=None, recursive=False): if recursive and os.path.isdir(path): _rchown(dir, user, group) ... It appears like a common enough use case to me ("chown -R path"). Thoughts? -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Mon May 28 17:21:47 2018 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Mon, 28 May 2018 23:21:47 +0200 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: <25FFAB81-B7C9-45A8-A667-4DDABC07E7F0@barrys-emacs.org> References: <25FFAB81-B7C9-45A8-A667-4DDABC07E7F0@barrys-emacs.org> Message-ID: On Mon, May 28, 2018 at 11:13 PM, Barry wrote: > > On 28 May 2018, at 21:23, Giampaolo Rodola' wrote: > > ...as in (not tested): > > def _rchown(dir, user, group): > for root, dirs, files in os.walk(dir, topdown=False): > for name in files: > chown(os.path.join(root, name), user, group) > > def chown(path, user=None, group=None, recursive=False): > if recursive and os.path.isdir(path): > _rchown(dir, user, group) > ... > > It appears like a common enough use case to me ("chown -R path"). > Thoughts? > > > I wonder if it is very common. > Don?t you have to be root or use sudo chown? > In which case it is only python code running as root that could use this. > > Barry > You're right, I didn't think about that. I remember myself doing "chown -R dir" every once in a while but didn't recall I prepended "sudo". =) -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at barrys-emacs.org Mon May 28 17:13:47 2018 From: barry at barrys-emacs.org (Barry) Date: Mon, 28 May 2018 22:13:47 +0100 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: References: Message-ID: <25FFAB81-B7C9-45A8-A667-4DDABC07E7F0@barrys-emacs.org> > On 28 May 2018, at 21:23, Giampaolo Rodola' wrote: > > ...as in (not tested): > > def _rchown(dir, user, group): > for root, dirs, files in os.walk(dir, topdown=False): > for name in files: > chown(os.path.join(root, name), user, group) > > def chown(path, user=None, group=None, recursive=False): > if recursive and os.path.isdir(path): > _rchown(dir, user, group) > ... > > It appears like a common enough use case to me ("chown -R path"). > Thoughts? I wonder if it is very common. Don?t you have to be root or use sudo chown? In which case it is only python code running as root that could use this. Barry > > -- > Giampaolo - http://grodola.blogspot.com > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Mon May 28 19:07:29 2018 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 28 May 2018 19:07:29 -0400 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: References: <25FFAB81-B7C9-45A8-A667-4DDABC07E7F0@barrys-emacs.org> Message-ID: path.py Path.choen supports names in addition to the uid/gid numbers which os.chown supports: https://pathpy.readthedocs.io/en/stable/api.html#path.Path.chown https://github.com/jaraco/path.py/blob/master/path.py#L1176 https://pathpy.readthedocs.io/en/stable/api.html#path.Path.walk On Monday, May 28, 2018, Giampaolo Rodola' wrote: > > > On Mon, May 28, 2018 at 11:13 PM, Barry wrote: > >> >> On 28 May 2018, at 21:23, Giampaolo Rodola' wrote: >> >> ...as in (not tested): >> >> def _rchown(dir, user, group): >> for root, dirs, files in os.walk(dir, topdown=False): >> for name in files: >> chown(os.path.join(root, name), user, group) >> >> def chown(path, user=None, group=None, recursive=False): >> if recursive and os.path.isdir(path): >> _rchown(dir, user, group) >> ... >> >> It appears like a common enough use case to me ("chown -R path"). >> Thoughts? >> >> >> I wonder if it is very common. >> Don?t you have to be root or use sudo chown? >> In which case it is only python code running as root that could use this. >> >> Barry >> > > You're right, I didn't think about that. I remember myself doing "chown -R > dir" every once in a while but didn't recall I prepended "sudo". =) > > -- > Giampaolo - http://grodola.blogspot.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon May 28 19:47:42 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 29 May 2018 09:47:42 +1000 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: <25FFAB81-B7C9-45A8-A667-4DDABC07E7F0@barrys-emacs.org> References: <25FFAB81-B7C9-45A8-A667-4DDABC07E7F0@barrys-emacs.org> Message-ID: <20180528234741.GC12683@ando.pearwood.info> On Mon, May 28, 2018 at 10:13:47PM +0100, Barry wrote: > > > On 28 May 2018, at 21:23, Giampaolo Rodola' wrote: [...] > > It appears like a common enough use case to me ("chown -R path"). > > Thoughts? > > I wonder if it is very common. > Don?t you have to be root or use sudo chown? > In which case it is only python code running as root that could use this. Certainly not. You only have to be root to change permissions on files that you otherwise wouldn't be able to change permissions on. chmod -R works fine for regular users changing their own files. Why wouldn't it? [steve at ando ~]$ ls -lR test test: total 12 -rw-rw-r-- 1 steve steve 5 Feb 4 2017 eggs.py drwxrwxr-x 2 steve steve 4096 May 29 09:41 package -rw-rw-r-- 1 steve steve 40 Feb 4 2017 spam.py test/package: total 0 -rw-rw-r-- 1 steve steve 0 May 29 09:41 __init__.py -rw-rw-r-- 1 steve steve 0 May 29 09:41 spam.py [steve at ando ~]$ chmod -R a-w test [steve at ando ~]$ ls -lR test test: total 12 -r--r--r-- 1 steve steve 5 Feb 4 2017 eggs.py dr-xr-xr-x 2 steve steve 4096 May 29 09:41 package -r--r--r-- 1 steve steve 40 Feb 4 2017 spam.py test/package: total 0 -r--r--r-- 1 steve steve 0 May 29 09:41 __init__.py -r--r--r-- 1 steve steve 0 May 29 09:41 spam.py -- Steve From greg.ewing at canterbury.ac.nz Mon May 28 19:52:04 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 29 May 2018 11:52:04 +1200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: References: <5B09A00F.9030806@brenbarn.net> <5B0AA855.6090604@canterbury.ac.nz> <0ACB829E-954D-4801-B329-FA689C6042F0@googlemail.com> <5B0B4E94.5020706@canterbury.ac.nz> <1D0C43C8-F674-4D0D-9D5B-471DE8BEEBD0@googlemail.com> Message-ID: <5B0C9624.5080304@canterbury.ac.nz> Michael Lohmann wrote: >> class Ethel(Aardvark, Clever): >> """Ethel is a very clever Aardvark""" >> def __init__(self): >> super().__init__(quantity="some spam", cleverness=1000)) >> >> if you want to instantiate an Aardvark directly there is NO WAY EVER that >> you could give him ANY kwargs. So why should the __init__ indicate >> something else? You seem to want the signature of Ethel.__init__ to indicate exactly what arguments it can accept. But even with your proposed feature, that isn't going to happen. It will only show arguments that appear explicitly in Ethel.__init__'s argument list. If it didn't override the default value for cleverness, you would never know that it was a possible parameter. In situations like this there's no substitute for hand-written documentation, and that can include explaining what is allowed for **kwds. -- Greg From greg.ewing at canterbury.ac.nz Mon May 28 19:53:49 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 29 May 2018 11:53:49 +1200 Subject: [Python-ideas] Keyword for direct pass through of kwargs to super In-Reply-To: <1D0C43C8-F674-4D0D-9D5B-471DE8BEEBD0@googlemail.com> References: <5B09A00F.9030806@brenbarn.net> <5B0AA855.6090604@canterbury.ac.nz> <0ACB829E-954D-4801-B329-FA689C6042F0@googlemail.com> <5B0B4E94.5020706@canterbury.ac.nz> <1D0C43C8-F674-4D0D-9D5B-471DE8BEEBD0@googlemail.com> Message-ID: <5B0C968D.5050708@canterbury.ac.nz> Michael Lohmann wrote: > I just wanted to override the > class-variable `magic_number` to give a reason why I don?t ever want to call > Magic.__init__ in Foo. If you want, you can have this class instead: > > class Magic: > def __init__(self): raise RuntimeError("Do not initialize this > class") > > but I figured that this might look a bit artificial... But your original example looks just as artificial. Skipping the initialisation of a class you're inheriting from is an extremely weird thing to do, and in any real-life situation there's almost certainly a better design. In any case, I don't see how this has anything to do with invisible passing of **kwds. In short, I don't understand what you're saying at all. -- Greg From rosuav at gmail.com Mon May 28 20:11:22 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 29 May 2018 10:11:22 +1000 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: <20180528234741.GC12683@ando.pearwood.info> References: <25FFAB81-B7C9-45A8-A667-4DDABC07E7F0@barrys-emacs.org> <20180528234741.GC12683@ando.pearwood.info> Message-ID: On Tue, May 29, 2018 at 9:47 AM, Steven D'Aprano wrote: > On Mon, May 28, 2018 at 10:13:47PM +0100, Barry wrote: >> >> > On 28 May 2018, at 21:23, Giampaolo Rodola' wrote: > [...] >> > It appears like a common enough use case to me ("chown -R path"). >> > Thoughts? >> >> I wonder if it is very common. >> Don?t you have to be root or use sudo chown? >> In which case it is only python code running as root that could use this. > > Certainly not. You only have to be root to change permissions on files > that you otherwise wouldn't be able to change permissions on. chmod -R > works fine for regular users changing their own files. Why wouldn't it? That's chmod. The OP asked about chown. ChrisA From steve at pearwood.info Mon May 28 20:49:25 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 29 May 2018 10:49:25 +1000 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: References: <25FFAB81-B7C9-45A8-A667-4DDABC07E7F0@barrys-emacs.org> <20180528234741.GC12683@ando.pearwood.info> Message-ID: <20180529004925.GD12683@ando.pearwood.info> On Tue, May 29, 2018 at 10:11:22AM +1000, Chris Angelico wrote: > On Tue, May 29, 2018 at 9:47 AM, Steven D'Aprano wrote: > > Certainly not. You only have to be root to change permissions on files > > that you otherwise wouldn't be able to change permissions on. chmod -R > > works fine for regular users changing their own files. Why wouldn't it? > > That's chmod. The OP asked about chown. /face-palm Indeed he did. But it doesn't matter: regular users can call chown -R: [steve at ando ~]$ chown -R steve.users test [steve at ando ~]$ ls -lR test test: total 12 -rw-rw-rw- 1 steve users 5 Feb 4 2017 eggs.py drwxrwxrwx 2 steve users 4096 May 29 09:41 package -rw-rw-rw- 1 steve users 40 Feb 4 2017 spam.py test/package: total 0 -rw-rw-rw- 1 steve users 0 May 29 09:41 __init__.py -rw-rw-rw- 1 steve users 0 May 29 09:41 spam.py The limitations on calling chown apply equally to the recursive and non-recursive case. -- Steve From greg.ewing at canterbury.ac.nz Tue May 29 02:29:50 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 29 May 2018 18:29:50 +1200 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: <20180529004925.GD12683@ando.pearwood.info> References: <25FFAB81-B7C9-45A8-A667-4DDABC07E7F0@barrys-emacs.org> <20180528234741.GC12683@ando.pearwood.info> <20180529004925.GD12683@ando.pearwood.info> Message-ID: <5B0CF35E.7090704@canterbury.ac.nz> Steven D'Aprano wrote: > But it doesn't matter: regular users can call chown -R: Only if you're not actually telling it to change anything. % ls -l foo.txt -rw-r--r-- 1 greg users 1 29 May 18:19 foo.txt % chown greg foo.txt % chown fred foo.txt chown: foo.txt: Operation not permitted So you have to be root in order to do anything *useful* with it. -- Greg From steve at pearwood.info Tue May 29 03:15:37 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 29 May 2018 17:15:37 +1000 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: <5B0CF35E.7090704@canterbury.ac.nz> References: <25FFAB81-B7C9-45A8-A667-4DDABC07E7F0@barrys-emacs.org> <20180528234741.GC12683@ando.pearwood.info> <20180529004925.GD12683@ando.pearwood.info> <5B0CF35E.7090704@canterbury.ac.nz> Message-ID: <20180529071537.GE12683@ando.pearwood.info> On Tue, May 29, 2018 at 06:29:50PM +1200, Greg Ewing wrote: > Steven D'Aprano wrote: > >But it doesn't matter: regular users can call chown -R: > > Only if you're not actually telling it to change anything. That's not correct. Look closely at my example: the file ownership recursively changed from steve.steve to steve.users. > % ls -l foo.txt > -rw-r--r-- 1 greg users 1 29 May 18:19 foo.txt > % chown greg foo.txt > % chown fred foo.txt > chown: foo.txt: Operation not permitted And yet Python provides a chown function despite this alleged uselessness. Maybe it's not quite so useless as you think? Here's a thought... maybe sometimes people actually do run Python scripts as root? And as the comments to this Stackoverflow post explain: https://unix.stackexchange.com/questions/119229/can-not-chown-a-file-from-my-user-to-another-user the ability for unprivileged users to change ownership to another user is configurable under POSIX. > So you have to be root in order to do anything *useful* > with it. Fortunately that is not correct, but even if it were, that's no reason to not allow chown to apply recursively. -- Steve From g.rodola at gmail.com Tue May 29 04:56:16 2018 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Tue, 29 May 2018 10:56:16 +0200 Subject: [Python-ideas] shutil zero-copy and exotic filesystems Message-ID: Hello, I've been working on a patch which speeds up shutil.copy* operations for all 3 major platforms (Linux, Windows, OSX): https://bugs.python.org/issue33671 Since the speedup is quite consistent I'd love to see this merged in, but considering shutil.copy* is quite crucial I wanted to hear other folk's opinion first. Attached patch attempts to use platform-specific zero-copy syscalls [1] by default and fallbacks on using plain read() / write() variant in case of immediate failure. In theory this should work fine, in practice I haven't tested it on exotic (e.g. network) filesystems. In order to diminish risks of breakage I think if it would make sense to: - add a global shutil.NO_ZEROCOPY variable defaulting to False - add a "no_zerocopy" argument to all functions involving a copy (copyfile(). copy(), copy2(), copytree(), move()) Thoughts? [1] sendfile() (Linux), fcopyfile() (OSX), CopyFileW (Windows) since the matter is a bit sensitive in terms of potential breakage on exotic / untested (e.g. network) filesystems I want to raise some attention about: https://bugs.python.org/issue33671 Attached patch attempts to use platform-specific zero-copy syscalls by default and fallbacks on using plain read() / write() copy in case of immediate failure. In order to diminish risks I think it would make sense to: - add a global shutil.NO_ZEROCOPY variable defaulting to False - add a "no_zerocopy" argument to all functions involving a copy (copyfile(). copy(), copy2(), copytree(), move()) Thoughts? -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Tue May 29 04:57:43 2018 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Tue, 29 May 2018 10:57:43 +0200 Subject: [Python-ideas] shutil zero-copy and exotic filesystems In-Reply-To: References: Message-ID: Whops, I hit "send" too soon. Sorry about the messed up message. On Tue, May 29, 2018 at 10:56 AM, Giampaolo Rodola' wrote: > Hello, > I've been working on a patch which speeds up shutil.copy* operations for > all 3 major platforms (Linux, Windows, OSX): > https://bugs.python.org/issue33671 > Since the speedup is quite consistent I'd love to see this merged in, but > considering shutil.copy* is quite crucial I wanted to hear other folk's > opinion first. Attached patch attempts to use platform-specific zero-copy > syscalls [1] by default and fallbacks on using plain read() / write() > variant in case of immediate failure. In theory this should work fine, in > practice I haven't tested it on exotic (e.g. network) filesystems. In order > to diminish risks of breakage I think if it would make sense to: > - add a global shutil.NO_ZEROCOPY variable defaulting to False > - add a "no_zerocopy" argument to all functions involving a copy > (copyfile(). copy(), copy2(), copytree(), move()) > > Thoughts? > > [1] sendfile() (Linux), fcopyfile() (OSX), CopyFileW (Windows) > > > > > since the matter is a bit sensitive in terms of potential breakage on > exotic / untested (e.g. network) filesystems I want to raise some attention > about: > https://bugs.python.org/issue33671 > Attached patch attempts to use platform-specific zero-copy syscalls by > default and fallbacks on using plain read() / write() copy in case of > immediate failure. In order to diminish risks I think it would make sense > to: > - add a global shutil.NO_ZEROCOPY variable defaulting to False > - add a "no_zerocopy" argument to all functions involving a copy > (copyfile(). copy(), copy2(), copytree(), move()) > > Thoughts? > > -- > Giampaolo - http://grodola.blogspot.com > > > -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From encukou at gmail.com Tue May 29 06:06:42 2018 From: encukou at gmail.com (Petr Viktorin) Date: Tue, 29 May 2018 12:06:42 +0200 Subject: [Python-ideas] Was `os.errno` undocumented? Message-ID: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> Hello, Python 3.7 removes the undocumented internal import `os.errno`. We consider that a an implementation detail, which can be changed *without notice* even in a bugfix release. Projects that depend on it are incorrect and should be fixed. On bpo-33666, there's a debate on whether the removal should be mentioned in release notes, on the grounds that it broke some projects, is used in quire a few tutorials/books/examples, and it's been working since Python 2.5 or so. But here's the thing: the more I think about this, the less I consider `os.errno` as "undocumented". Here's what I consider a reasonable path a programmer might go through: # Where do I find errno values? # Maybe it's in `os`, like all other basic platform bindings? >>> import os >>> os.err os.errno os.error( >>> help(os.errno) Help on built-in module errno: ... # Yup, There it is! Even more serious: # Hmm, this old example on some website tells me to use `os.errno`. # It's the first time I'm hear about that, so I'll be extra careful; >>> import os >>> os.errno # Yup, it's there! Let's double-check the docs. >>> help(os.errno) Help on built-in module errno: ... # Yup, looks quite alright! Let's use it! As you can see, the built-in documentation does not contain *any* warnings against using `os.errno`. You might think the fact that it's called "errno" and not "os.errno" is a red flag, but it's not, really -- "os.path" is (on my system) named "posixpath", yet "os.path" is the name to use. While most people might prefer searching docs.python org to `help()`, editors are getting better and better to presenting introspection and the built-in docs, so more and more people are preferring `pydoc`-ish docs to going online. I don't think we can reasonably expect people who used built-in help as above to go back and check that Python's official docs *do not* contain `os.errno`. Effectively, while `os.errno` is not very *discoverable* using official docs alone, I don't think calling it *undocumented* is fair. So, removing it without notice is not very friendly to our users. Is that reasoning sound? Should our policy on removing internal imports take that into account? From songofacandy at gmail.com Tue May 29 06:24:32 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 29 May 2018 19:24:32 +0900 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> Message-ID: > I don't think we can reasonably expect people who used built-in help as > above to go back and check that Python's official docs *do not* contain > `os.errno`. Agree, but... > Effectively, while `os.errno` is not very *discoverable* > using official docs alone, I don't think calling it *undocumented* is fair. `pydoc os` shows: This exports: - all functions from posix or nt, e.g. unlink, stat, etc. - os.path is either posixpath or ntpath - os.name is either 'posix' or 'nt' - os.curdir is a string representing the current directory (always '.') - os.pardir is a string representing the parent directory (always '..') - os.sep is the (or a most common) pathname separator ('/' or '\\') - os.extsep is the extension separator (always '.') - os.altsep is the alternate pathname separator (None or '/') - os.pathsep is the component separator used in $PATH etc - os.linesep is the line separator in text files ('\r' or '\n' or '\r\n') - os.defpath is the default search path for executables - os.devnull is the file path of the null device ('/dev/null', etc.) So I think calling `os.errno` as *undocumented* is fair. > So, removing it without notice is not very friendly to our users. > Is that reasoning sound? Depending on such subimports is common pitfall all experienced Python programmer knows. Of course, it's not very friendly, but a pitfall. But I don't think it's enough reason to call it *documented*. > Should our policy on removing internal imports take that into account? Maybe, we should use `_` prefix for all new private subimports. I don't have any idea having better "maintainability : user friendly" ratio than it. Third party linter may be able to notice warning for using private subimports in user code. Regards, -- INADA Naoki From rosuav at gmail.com Tue May 29 06:45:41 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 29 May 2018 20:45:41 +1000 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> Message-ID: On Tue, May 29, 2018 at 8:06 PM, Petr Viktorin wrote: > I don't think we can reasonably expect people who used built-in help as > above to go back and check that Python's official docs *do not* contain > `os.errno`. Effectively, while `os.errno` is not very *discoverable* using > official docs alone, I don't think calling it *undocumented* is fair. > So, removing it without notice is not very friendly to our users. > At no point did you show that it's *documented*. What you've shown is that it's an unadorned name. There are a lot of those around: >>> antigravity.webbrowser >>> base64.struct >>> dis.types If people are attempting "from os import errno", they need to learn to type "import errno" instead. While it's nice to hide those modules with "import errno as _errno", failure to do so is a bug, not an API feature. The change might break code, but ANY change might break code: https://xkcd.com/1172/ ChrisA From steve at pearwood.info Tue May 29 07:57:57 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 29 May 2018 21:57:57 +1000 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> Message-ID: <20180529115756.GG12683@ando.pearwood.info> On Tue, May 29, 2018 at 12:06:42PM +0200, Petr Viktorin wrote: > Hello, > Python 3.7 removes the undocumented internal import `os.errno`. > We consider that a an implementation detail, which can be changed > *without notice* even in a bugfix release. Projects that depend on it > are incorrect and should be fixed. PEP 8 makes it clear that imports are implementation details unless explicitly documented otherwise: Imported names should always be considered an implementation detail. Other modules must not rely on indirect access to such imported names unless they are an explicitly documented part of the containing module's API, such as os.path or a package's __init__ module that exposes functionality from submodules. This decision dates back to 2013: https://mail.python.org/pipermail/python-dev/2013-July/127284.html so it has been around for a while, long enough that linters ought to be enforcing it, and people ought to know about it. If not, that's a failure of the linter and/or the coder. > On bpo-33666, there's a debate on whether the removal should be > mentioned in release notes, on the grounds that it broke some projects, > is used in quire a few tutorials/books/examples, and it's been working > since Python 2.5 or so. I don't see why there should be a debate about mentioning it in the release notes. There's no harm in adding a few lines: "os.errno is a non-public implementation detail, as described in PEP 8, and has been removed. Import the errno module instead." Being an implementation detail, we aren't *required* to do so, but given that (so you say) a few tutorials etc use it, I think it is the kind thing to do. > But here's the thing: the more I think about this, the less I consider > `os.errno` as "undocumented". Here's what I consider a reasonable path a > programmer might go through: [...] All of this is reasonable, and I'm sympathetic, *except* that it is officially documented policy that imports are implementation details unless otherwise stated. If os.errno was gone and there was no easy fix, I think we could be merciful and rethink the decision. But there is an easy fix: import errno directly instead. And maybe this will encourage linters to flag this risky usage, if they aren't already doing so. > As you can see, the built-in documentation does not contain *any* > warnings against using `os.errno`. It doesn't need to. And of course, help(os.errno) *cannot* warn about os.errno, since it only receives the errno module as argument, not the expression you used to pass it. -- Steve From encukou at gmail.com Tue May 29 08:17:26 2018 From: encukou at gmail.com (Petr Viktorin) Date: Tue, 29 May 2018 14:17:26 +0200 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: <20180529115756.GG12683@ando.pearwood.info> References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> <20180529115756.GG12683@ando.pearwood.info> Message-ID: <1e6417b1-4f3f-ee72-8b00-f661091e7ce9@gmail.com> On 05/29/18 13:57, Steven D'Aprano wrote: > On Tue, May 29, 2018 at 12:06:42PM +0200, Petr Viktorin wrote: >> Hello, >> Python 3.7 removes the undocumented internal import `os.errno`. >> We consider that a an implementation detail, which can be changed >> *without notice* even in a bugfix release. Projects that depend on it >> are incorrect and should be fixed. > > PEP 8 makes it clear that imports are implementation details unless > explicitly documented otherwise: > > Imported names should always be considered an implementation detail. > Other modules must not rely on indirect access to such imported names > unless they are an explicitly documented part of the containing > module's API, such as os.path or a package's __init__ module that > exposes functionality from submodules. > > > This decision dates back to 2013: > > https://mail.python.org/pipermail/python-dev/2013-July/127284.html > > so it has been around for a while, long enough that linters ought to be > enforcing it, and people ought to know about it. If not, that's a > failure of the linter and/or the coder. >> On bpo-33666, there's a debate on whether the removal should be >> mentioned in release notes, on the grounds that it broke some projects, >> is used in quire a few tutorials/books/examples, and it's been working >> since Python 2.5 or so. > > I don't see why there should be a debate about mentioning it in the > release notes. There's no harm in adding a few lines: > > "os.errno is a non-public implementation detail, as described in PEP 8, > and has been removed. Import the errno module instead." > > Being an implementation detail, we aren't *required* to do so, but given > that (so you say) a few tutorials etc use it, I think it is the kind > thing to do. > > > >> But here's the thing: the more I think about this, the less I consider >> `os.errno` as "undocumented". Here's what I consider a reasonable path a >> programmer might go through: > [...] > > All of this is reasonable, and I'm sympathetic, *except* that it is > officially documented policy that imports are implementation details > unless otherwise stated. > > If os.errno was gone and there was no easy fix, I think we could be > merciful and rethink the decision. But there is an easy fix: import > errno directly instead. > > And maybe this will encourage linters to flag this risky usage, if they > aren't already doing so. > > >> As you can see, the built-in documentation does not contain *any* >> warnings against using `os.errno`. > > It doesn't need to. > > And of course, help(os.errno) *cannot* warn about os.errno, since it > only receives the errno module as argument, not the expression you used > to pass it. > > From encukou at gmail.com Tue May 29 08:30:55 2018 From: encukou at gmail.com (Petr Viktorin) Date: Tue, 29 May 2018 14:30:55 +0200 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: <20180529115756.GG12683@ando.pearwood.info> References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> <20180529115756.GG12683@ando.pearwood.info> Message-ID: (Apologies if you received an empty/unfinished e-mail here earlier; I hit Send by mistake.) On 05/29/18 13:57, Steven D'Aprano wrote: > On Tue, May 29, 2018 at 12:06:42PM +0200, Petr Viktorin wrote: >> Hello, >> Python 3.7 removes the undocumented internal import `os.errno`. >> We consider that a an implementation detail, which can be changed >> *without notice* even in a bugfix release. Projects that depend on it >> are incorrect and should be fixed. > > PEP 8 makes it clear that imports are implementation details unless > explicitly documented otherwise: > > Imported names should always be considered an implementation detail. > Other modules must not rely on indirect access to such imported names > unless they are an explicitly documented part of the containing > module's API, such as os.path or a package's __init__ module that > exposes functionality from submodules. > > > This decision dates back to 2013: > > https://mail.python.org/pipermail/python-dev/2013-July/127284.html > > so it has been around for a while, long enough that linters ought to be > enforcing it, and people ought to know about it. If not, that's a > failure of the linter and/or the coder. > > >> On bpo-33666, there's a debate on whether the removal should be >> mentioned in release notes, on the grounds that it broke some projects, >> is used in quire a few tutorials/books/examples, and it's been working >> since Python 2.5 or so. > > I don't see why there should be a debate about mentioning it in the > release notes. There's no harm in adding a few lines: > > "os.errno is a non-public implementation detail, as described in PEP 8, > and has been removed. Import the errno module instead." > > Being an implementation detail, we aren't *required* to do so, but given > that (so you say) a few tutorials etc use it, I think it is the kind > thing to do. > > > >> But here's the thing: the more I think about this, the less I consider >> `os.errno` as "undocumented". Here's what I consider a reasonable path a >> programmer might go through: > [...] > > All of this is reasonable, and I'm sympathetic, *except* that it is > officially documented policy that imports are implementation details > unless otherwise stated. I agree that it's technically well within our rights to remove it without notice. But ... PEP8? A style guide defines what is a CPython implementation detail? That's not a place to point to while saying "told you so!" -- perhaps "sorry for the inconvenience" would be more appropriate :) > If os.errno was gone and there was no easy fix, I think we could be > merciful and rethink the decision. But there is an easy fix: import > errno directly instead. > > And maybe this will encourage linters to flag this risky usage, if they > aren't already doing so. How do linters find out what's an internal import, and what's correct usage (like os.path)? >> As you can see, the built-in documentation does not contain *any* >> warnings against using `os.errno`. > > It doesn't need to. > > And of course, help(os.errno) *cannot* warn about os.errno, since it > only receives the errno module as argument, not the expression you used > to pass it. Indeed. That's unfortunate, but it is a reason for us to be careful, or perhaps find/document a better policy for handling these. I'm not looking for evidence to justify the changes; I'm looking for ways to be more friendly to our users -- most of whom have not read all of the docs. From songofacandy at gmail.com Tue May 29 08:42:10 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 29 May 2018 21:42:10 +0900 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> <20180529115756.GG12683@ando.pearwood.info> Message-ID: > I agree that it's technically well within our rights to remove it > without notice. > But ... PEP8? A style guide defines what is a CPython implementation > detail? That's not a place to point to while saying "told you so!" -- > perhaps "sorry for the inconvenience" would be more appropriate :) https://www.python.org/dev/peps/pep-0008/#public-and-internal-interfaces "Any backwards compatibility guarantees apply only to public interfaces." "All undocumented interfaces should be assumed to be internal." "Imported names should always be considered an implementation detail. Other modules must not rely on indirect access to such imported names unless they are an explicitly documented part of the containing module's API, such as os.path or a package's __init__ module that exposes functionality from submodules." > > And maybe this will encourage linters to flag this risky usage, if they > > aren't already doing so. > How do linters find out what's an internal import, and what's correct > usage (like os.path)? It is `correct usage` only when it is in __all__. >>> "path" in os.__all__ True >>> "errno" in os.__all__ False Regards, -- INADA Naoki From ncoghlan at gmail.com Tue May 29 08:57:57 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 29 May 2018 22:57:57 +1000 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: References: Message-ID: On 29 May 2018 at 06:23, Giampaolo Rodola' wrote: > ...as in (not tested): > > def _rchown(dir, user, group): > for root, dirs, files in os.walk(dir, topdown=False): > for name in files: > chown(os.path.join(root, name), user, group) > > def chown(path, user=None, group=None, recursive=False): > if recursive and os.path.isdir(path): > _rchown(dir, user, group) > ... > > It appears like a common enough use case to me ("chown -R path"). > Thoughts? > https://bugs.python.org/issue13033 is a long-open RFE for this, proposing to add it as "shutil.chowntree" (naming inspired by "shutil.rmtree" and "shutil.copytree"). The "walkdir" project I mention on that PR has been on hiatus for a few years now (aside from a bit of activity to get a new release out in 2016 with several contributed fixes), but the main point of the comment where I mentioned it still stands: the hard part of designing recursive state modification APIs is deciding what to do when an operation fails after you've already made changes to the state of the disk. shutil.rmtree fortunately provides some good precedent there, but it does mean this feature would need to be implemented as its own API, rather than as an option on shutil.chown. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue May 29 09:07:36 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 29 May 2018 23:07:36 +1000 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> Message-ID: On 29 May 2018 at 20:06, Petr Viktorin wrote: > Is that reasoning sound? > Should our policy on removing internal imports take that into account? > As Steven noted, the normative answer to this is in PEP 8: https://www.python.org/dev/peps/pep-0008/#public-and-internal-interfaces Since `os.errno` is a transitive import, is not included in `os.__all__`, and isn't documented in the library reference, it's considered an implementation detail that can be removed without a deprecation period. That said, it should still be mentioned in the "Porting to Python 3.7" section of the What's New guide, with the fix being to switch to "import errno" in any affected code. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue May 29 10:37:06 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 29 May 2018 17:37:06 +0300 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> Message-ID: 29.05.18 13:06, Petr Viktorin ????: > As you can see, the built-in documentation does not contain *any* > warnings against using `os.errno`. > You might think the fact that it's called "errno" and not "os.errno" is > a red flag, but it's not, really -- "os.path" is (on my system) named > "posixpath", yet "os.path" is the name to use. Unlike to os.path, which is explicitly documented and included in __all__, os.abc, os.errno, os.sys and os.st are not documented and not included in __all__. They were always an implementations detail. os.errno was added as a side effect of issue1755179. [1] > While most people might prefer searching docs.python org to `help()`, > editors are getting better and better to presenting introspection and > the built-in docs, so more and more people are preferring `pydoc`-ish > docs to going online. > > I don't think we can reasonably expect people who used built-in help as > above to go back and check that Python's official docs *do not* contain > `os.errno`. Effectively, while `os.errno` is not very *discoverable* > using official docs alone, I don't think calling it *undocumented* is fair. > So, removing it without notice is not very friendly to our users. We have removed over 100 module attributes in 3.6 [2] and over 200 module attributes in 3.7 [3]. Some of these removals broke third-part projects [4], even removals of underscored names. If we will document the removal of os.errno, we should to document the removal of at least other names for which there are known examples of breaking third-party projects, e.g. re._pattern_type, uuid._uuid_generate_time and several typing attributes. [1] https://bugs.python.org/issue1755179 [2] https://bugs.python.org/msg317881 [3] https://bugs.python.org/msg317876 [4] https://bugs.python.org/msg317998 From steve at pearwood.info Tue May 29 10:55:47 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 30 May 2018 00:55:47 +1000 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> Message-ID: <20180529145547.GH12683@ando.pearwood.info> On Tue, May 29, 2018 at 05:37:06PM +0300, Serhiy Storchaka wrote: > We have removed over 100 module attributes in 3.6 [2] and over 200 > module attributes in 3.7 [3]. > > Some of these removals broke third-part projects [4], even removals of > underscored names. Yes, people will rely on undocumented features. Even when you tell them not to. > If we will document the removal of os.errno, we > should to document the removal of at least other names for which there > are known examples of breaking third-party projects, e.g. > re._pattern_type, uuid._uuid_generate_time and several typing attributes. I disagree. I think that _leading underscore names can be removed without mentioning the removal. If it breaks code, too bad. Anyone using an explicitly named _private name has only themselves to blame. But non-public names *without* a leading underscore are in a grey area, since they *look* like public names. I think we can afford to be kind by documenting such removals -- especially since it may help to educate people that imported modules without a leading underscore are still considered implementation details. -- Steve From ericfahlgren at gmail.com Tue May 29 11:42:13 2018 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Tue, 29 May 2018 08:42:13 -0700 Subject: [Python-ideas] shutil zero-copy and exotic filesystems In-Reply-To: References: Message-ID: I have been doing speed testing on our network frequently enough over the last five year that I have a good feel for these changes. I have been benchmarking various means for copying files of 1.0-500 MB between workstations on our 1 Gbps LAN to local servers, and on our WAN which was just upgraded two months ago from a 7x T1 (11.5 Mbps) to 100 Mbps fiber. This is an all Windows network, workstations are Win7/10 and servers are Win Server 2003 (!!!), 2008, 2012 and 2016. The difference between open(dst).write(open(src).read()) and CopyFileW can be pretty signficant, especially for older Win server versions. I see maybe 4x speedup using CopyFileW to those machines with larger files, so it's a big win. On newer Windows Server versions it become sort of a wash, with both ways working better than anything else (much faster than command line utilities like DOS "copy" and Cygwin "cp", or Windows file explorer drag and drop, and very much faster than a C-code write(read()) loop). Here's an example I just ran for a Win Server 2003 copy. 2018-05-29T07:45:09 PDT ---Using CopyFileW Copy 'xmas.mpg' to 'p:/build/' (28,872,704 bytes) 1/3 - 0.27 s ( 905.2 Mbps) 2/3 - 0.25 s ( 983.9 Mbps) 3/3 - 0.26 s ( 968.6 Mbps) Theoretical: 0.25 s at 1000.0 Mbps Best time : 0.25 s at 983.9 Mbps ---Using write(read()) Copy 'xmas.mpg' to 'p:/build/' (28,872,704 bytes) 1/3 - 1.47 s ( 169.9 Mbps) 2/3 - 1.21 s ( 205.0 Mbps) 3/3 - 1.22 s ( 203.4 Mbps) Theoretical: 0.25 s at 1000.0 Mbps Best time : 1.21 s at 205.0 Mbps On our WAN, which has a VPN endpoint 3000 miles from our office, routing back to a test server another 2000 miles inside the network (tracert shows 12-15 hops, 200 ms latency, arrrg), copying is slow no matter what: we are lucky to see 40 Mbps on a connection that has the slowest link section at 100 Mbps. The biggest improvement we saw on WAN copies was when Ben Hoyt built scandir on top of Windows primitives. We were doing a copytree using os.walk that would take almost 20 (twenty) minutes just to build the tree before it copied the first file. When scandir was first released, I rewrote my code with fingers crossed that it would help a bit. On my first test, it took something like 15-20 seconds to build the tree and start copying. I was sure I had screwed up and broken it, but was quite shocked when I saw that it was in fact working correctly. Bottom line is +1 on switching to OS-specific copy mechanisms. Python is already the best* language for this sort of work (I've tried C/C++ and C# as alternatives), and this will make it even better. *Best for me in this particular case is "resulting code is fastest, ease of implementation is secondary." On Tue, May 29, 2018 at 1:59 AM Giampaolo Rodola' wrote: > Whops, I hit "send" too soon. Sorry about the messed up message. > > On Tue, May 29, 2018 at 10:56 AM, Giampaolo Rodola' > wrote: > >> Hello, >> I've been working on a patch which speeds up shutil.copy* operations for >> all 3 major platforms (Linux, Windows, OSX): >> https://bugs.python.org/issue33671 >> Since the speedup is quite consistent I'd love to see this merged in, but >> considering shutil.copy* is quite crucial I wanted to hear other folk's >> opinion first. Attached patch attempts to use platform-specific zero-copy >> syscalls [1] by default and fallbacks on using plain read() / write() >> variant in case of immediate failure. In theory this should work fine, in >> practice I haven't tested it on exotic (e.g. network) filesystems. In order >> to diminish risks of breakage I think if it would make sense to: >> - add a global shutil.NO_ZEROCOPY variable defaulting to False >> - add a "no_zerocopy" argument to all functions involving a copy >> (copyfile(). copy(), copy2(), copytree(), move()) >> >> Thoughts? >> >> [1] sendfile() (Linux), fcopyfile() (OSX), CopyFileW (Windows) >> >> >> >> >> since the matter is a bit sensitive in terms of potential breakage on >> exotic / untested (e.g. network) filesystems I want to raise some attention >> about: >> https://bugs.python.org/issue33671 >> Attached patch attempts to use platform-specific zero-copy syscalls by >> default and fallbacks on using plain read() / write() copy in case of >> immediate failure. In order to diminish risks I think it would make sense >> to: >> - add a global shutil.NO_ZEROCOPY variable defaulting to False >> - add a "no_zerocopy" argument to all functions involving a copy >> (copyfile(). copy(), copy2(), copytree(), move()) >> >> Thoughts? >> >> -- >> Giampaolo - http://grodola.blogspot.com >> >> >> > > > -- > Giampaolo - http://grodola.blogspot.com > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gvanrossum at gmail.com Tue May 29 12:09:45 2018 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue, 29 May 2018 09:09:45 -0700 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: <20180529145547.GH12683@ando.pearwood.info> References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> <20180529145547.GH12683@ando.pearwood.info> Message-ID: Yes. What Steve says at the end. Let's be kind and mention this one on the release notes. But not all the others. On Tue, May 29, 2018, 07:57 Steven D'Aprano wrote: > On Tue, May 29, 2018 at 05:37:06PM +0300, Serhiy Storchaka wrote: > > > We have removed over 100 module attributes in 3.6 [2] and over 200 > > module attributes in 3.7 [3]. > > > > Some of these removals broke third-part projects [4], even removals of > > underscored names. > > Yes, people will rely on undocumented features. Even when you tell them > not to. > > > If we will document the removal of os.errno, we > > should to document the removal of at least other names for which there > > are known examples of breaking third-party projects, e.g. > > re._pattern_type, uuid._uuid_generate_time and several typing attributes. > > I disagree. I think that _leading underscore names can be removed > without mentioning the removal. If it breaks code, too bad. Anyone using > an explicitly named _private name has only themselves to blame. > > But non-public names *without* a leading underscore are in a grey area, > since they *look* like public names. > > I think we can afford to be kind by documenting such removals -- > especially since it may help to educate people that imported modules > without a leading underscore are still considered implementation > details. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue May 29 12:38:48 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 29 May 2018 09:38:48 -0700 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: References: Message-ID: Honestly, despite the occasional use case(1), I'm not sure that this is a battery we need in the stdlib. Nobody seems too excited about writing the code, and when the operation is needed, shelling out to the system chown is not too onerous. (Ditto for chmod.) (1) Not even sure that a use case was shown -- it was just shown that the operation is not necessarily useless. On Tue, May 29, 2018 at 5:57 AM, Nick Coghlan wrote: > On 29 May 2018 at 06:23, Giampaolo Rodola' wrote: > >> ...as in (not tested): >> >> def _rchown(dir, user, group): >> for root, dirs, files in os.walk(dir, topdown=False): >> for name in files: >> chown(os.path.join(root, name), user, group) >> >> def chown(path, user=None, group=None, recursive=False): >> if recursive and os.path.isdir(path): >> _rchown(dir, user, group) >> ... >> >> It appears like a common enough use case to me ("chown -R path"). >> Thoughts? >> > > https://bugs.python.org/issue13033 is a long-open RFE for this, proposing > to add it as "shutil.chowntree" (naming inspired by "shutil.rmtree" and > "shutil.copytree"). > > The "walkdir" project I mention on that PR has been on hiatus for a few > years now (aside from a bit of activity to get a new release out in 2016 > with several contributed fixes), but the main point of the comment where I > mentioned it still stands: the hard part of designing recursive state > modification APIs is deciding what to do when an operation fails after > you've already made changes to the state of the disk. > > shutil.rmtree fortunately provides some good precedent there, but it does > mean this feature would need to be implemented as its own API, rather than > as an option on shutil.chown. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at barrys-emacs.org Tue May 29 13:17:46 2018 From: barry at barrys-emacs.org (Barry Scott) Date: Tue, 29 May 2018 18:17:46 +0100 Subject: [Python-ideas] shutil zero-copy and exotic filesystems In-Reply-To: References: Message-ID: <3617781.QGVUIP0ij3@varric.chelsea.private> On Tuesday, 29 May 2018 16:42:13 BST Eric Fahlgren wrote: snip... > On our WAN, which has a VPN endpoint 3000 miles from our office, routing > back to a test server another 2000 miles inside the network (tracert shows > 12-15 hops, 200 ms latency, arrrg), copying is slow no matter what: we are > lucky to see 40 Mbps on a connection that has the slowest link section at > 100 Mbps. > This is a guess on my part, I may be way off base. Maybe you are suffering from TCP windows scaling not work well enough? This page claims there are bugs in the windows implementations. Its also claimed, elsewhere, that some middle boxes mess up TCP windows scaling. http://web.archive.org/web/20120217135039/http://fasterdata.es.net:80/ fasterdata/host-tuning/ms-windows Wikipedia has a good description of TCP windows scalling. Barry From carl.input at gmail.com Tue May 29 14:48:40 2018 From: carl.input at gmail.com (Carl Smith) Date: Tue, 29 May 2018 19:48:40 +0100 Subject: [Python-ideas] Mashup the existing statement grammars to capture predicates In-Reply-To: References: Message-ID: Nah?? -- Carl Smith carl.input at gmail.com On 24 May 2018 at 19:24, Carl Smith wrote: > This is another suggestion for new syntax for assigning a name to the value > of the predicate in an if, elif or while statement. It still uses `as` for > its keyword, but with (more flexible) params instead of a direct > assignment. > > It mashes up the if/while, def/class and for-in grammars, so it still looks > like Python, and boils down to this: > > if|elif|while as (): > > If the params contain one simple name (the required minimum), the value of > the predicate is assigned to that name. In any other case, the value must > be a sequence, which gets unpacked: > > while input('$ ').split() as (command, *args): > > if run(command, parse(args)) as (result): render(result) > else: sys.exit() > > -- Carl Smith > carl.input at gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue May 29 14:52:05 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 30 May 2018 04:52:05 +1000 Subject: [Python-ideas] Mashup the existing statement grammars to capture predicates In-Reply-To: References: Message-ID: On Wed, May 30, 2018 at 4:48 AM, Carl Smith wrote: > Nah?? Nah. It's already been discussed at interminable length as part of PEP 572. Feel free to browse that document. ChrisA From ericfahlgren at gmail.com Tue May 29 15:07:54 2018 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Tue, 29 May 2018 12:07:54 -0700 Subject: [Python-ideas] shutil zero-copy and exotic filesystems In-Reply-To: <3617781.QGVUIP0ij3@varric.chelsea.private> References: <3617781.QGVUIP0ij3@varric.chelsea.private> Message-ID: On Tue, May 29, 2018 at 10:17 AM Barry Scott wrote: > On Tuesday, 29 May 2018 16:42:13 BST Eric Fahlgren wrote: > > Maybe you are suffering from TCP windows scaling not work well enough? > Thanks for the tip, I'll have to mention that to our IT infrastructure guys and see if they can check it out. -------------- next part -------------- An HTML attachment was scrubbed... URL: From carl.input at gmail.com Tue May 29 16:00:25 2018 From: carl.input at gmail.com (Carl Smith) Date: Tue, 29 May 2018 21:00:25 +0100 Subject: [Python-ideas] Mashup the existing statement grammars to capture predicates In-Reply-To: References: Message-ID: Ah. Nice one. I'll look through that. -- Carl Smith carl.input at gmail.com On 29 May 2018 at 19:52, Chris Angelico wrote: > On Wed, May 30, 2018 at 4:48 AM, Carl Smith wrote: > > Nah?? > > Nah. It's already been discussed at interminable length as part of PEP > 572. Feel free to browse that document. > > ChrisA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue May 29 18:53:18 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 29 May 2018 18:53:18 -0400 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> <20180529145547.GH12683@ando.pearwood.info> Message-ID: On 5/29/2018 12:09 PM, Guido van Rossum wrote: > Yes. What Steve says at the end. Let's be kind and mention this one on > the release notes. But not all the others. How about adding a general reminder. In Porting to 3.7: "Imports into a module are a private implementation detail unless otherwise noted. Private imports can be removed when not needed. (See PEP 8.) For example, the os module no longer needs the errno module, so 'import errno' was removed for 3.7. If you accessed 'errno' as 'os.errno', add 'import errno' to your code and remove the 'os.' prefix. The same applies to the 200 other unneeded imports removed in 3.7." Since this is a repeated problem, and a repeated drain on us core developers, a version of this should be repeated in every What's New. -- Terry Jan Reedy From guido at python.org Tue May 29 18:58:14 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 29 May 2018 15:58:14 -0700 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> <20180529145547.GH12683@ando.pearwood.info> Message-ID: Have we received complaints about other cases? Whenever this breaks code for which I am responsible I just blush and fix it, I don't complain. Repeating the same warning typically just causes warning fatigue rather than helping anyone. On Tue, May 29, 2018 at 3:53 PM, Terry Reedy wrote: > On 5/29/2018 12:09 PM, Guido van Rossum wrote: > >> Yes. What Steve says at the end. Let's be kind and mention this one on >> the release notes. But not all the others. >> > > How about adding a general reminder. In Porting to 3.7: > > "Imports into a module are a private implementation detail unless > otherwise noted. Private imports can be removed when not needed. (See PEP > 8.) For example, the os module no longer needs the errno module, so > 'import errno' was removed for 3.7. If you accessed 'errno' as 'os.errno', > add 'import errno' to your code and remove the 'os.' prefix. The same > applies to the 200 other unneeded imports removed in 3.7." > > Since this is a repeated problem, and a repeated drain on us core > developers, a version of this should be repeated in every What's New. > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue May 29 19:06:10 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 30 May 2018 11:06:10 +1200 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: <20180529071537.GE12683@ando.pearwood.info> References: <25FFAB81-B7C9-45A8-A667-4DDABC07E7F0@barrys-emacs.org> <20180528234741.GC12683@ando.pearwood.info> <20180529004925.GD12683@ando.pearwood.info> <5B0CF35E.7090704@canterbury.ac.nz> <20180529071537.GE12683@ando.pearwood.info> Message-ID: <5B0DDCE2.8000001@canterbury.ac.nz> Steven D'Aprano wrote: > Look closely at my example: the file ownership > recursively changed from steve.steve to steve.users. You're quite right. I hadn't realised that chown can be used to change the group as well as the user. It's permitted to change the group to one that the user is a member of. -- Greg From greg.ewing at canterbury.ac.nz Tue May 29 19:11:17 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 30 May 2018 11:11:17 +1200 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: <20180529071537.GE12683@ando.pearwood.info> References: <25FFAB81-B7C9-45A8-A667-4DDABC07E7F0@barrys-emacs.org> <20180528234741.GC12683@ando.pearwood.info> <20180529004925.GD12683@ando.pearwood.info> <5B0CF35E.7090704@canterbury.ac.nz> <20180529071537.GE12683@ando.pearwood.info> Message-ID: <5B0DDE15.6020402@canterbury.ac.nz> BTW, I wouldn't argue that Python shouldn't provide things that are only useful to root. While writing setuid utilities in Python is a bad idea for lots of reasons, I don't think there's anything wrong with becoming root by another means and then running a Python program that you know well enough to trust. -- Greg From tjreedy at udel.edu Tue May 29 19:24:01 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 29 May 2018 19:24:01 -0400 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> Message-ID: On 5/29/2018 6:06 AM, Petr Viktorin wrote: > Python 3.7 removes the undocumented internal import `os.errno`. Among a couple of hundred others, including some from idlelib. > We consider that a an implementation detail, which can be changed > *without notice* even in a bugfix release. We core developers occasionally run a linter on our code, or someone does it for us. Those of us who like clean code attend to warnings such as 'unused import'. I removed a couple of idlelib inports because of someone else running flake8. > Projects that depend on it are incorrect and should be fixed. I think we can agree that better warning of 'use of stdlib internal import' would be great. > On bpo-33666, there's a debate on whether the removal should be > mentioned in release notes, on the grounds that it broke some projects, > is used in quire a few tutorials/books/examples, and it's been working > since Python 2.5 or so. Since I agree that *this* removal may possibly have the greatest impact, and that the purpose of What's New is to help users, I am glad Guido spoke up to say 'do it'. See my reponse to his post for a suggested entry. > But here's the thing: the more I think about this, the less I consider > `os.errno` as "undocumented". Here's what I consider a reasonable path a > programmer might go through: > > # Where do I find errno values? > # Maybe it's in `os`, like all other basic platform bindings? > >>> import os > >>> os.err > os.errno?? os.error( > >>> help(os.errno) > Help on built-in module errno: > ... > # Yup, There it is! Suppose a naive beginner who confuses 'IDLE' with 'python' (they exist!) thinks "Where can I find re functions? How about 'pyshell' >>> import idlelib.pyshell as ps >>> ps.r >>> help(ps.re) Help on module re: ! > Is that reasoning sound? Do you claim that os.errno is more documented (by the 'official' docs and help) than idlelib.pyshell.re? -- Terry Jan Reedy From rosuav at gmail.com Tue May 29 19:44:09 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 30 May 2018 09:44:09 +1000 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: <5B0DDE15.6020402@canterbury.ac.nz> References: <25FFAB81-B7C9-45A8-A667-4DDABC07E7F0@barrys-emacs.org> <20180528234741.GC12683@ando.pearwood.info> <20180529004925.GD12683@ando.pearwood.info> <5B0CF35E.7090704@canterbury.ac.nz> <20180529071537.GE12683@ando.pearwood.info> <5B0DDE15.6020402@canterbury.ac.nz> Message-ID: On Wed, May 30, 2018 at 9:11 AM, Greg Ewing wrote: > BTW, I wouldn't argue that Python shouldn't provide things > that are only useful to root. While writing setuid utilities > in Python is a bad idea for lots of reasons, I don't think > there's anything wrong with becoming root by another means > and then running a Python program that you know well enough > to trust. I'd go further. Once a shell script gets longer than about a page or two of code, it often needs to be rewritten in a different language, and Python is well situated to be that language. That doesn't change when the script is to be run as root. I've written many Python scripts to do sysadminning jobs for me - usually one-shot scripts, but also some that stick around. Since I wrote the scripts myself, the trust issue doesn't come up; I trust the Python interpreter the same way that I trust /bin/bash. ChrisA From gadgetsteve at live.co.uk Wed May 30 00:18:51 2018 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Wed, 30 May 2018 04:18:51 +0000 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> <20180529145547.GH12683@ando.pearwood.info> Message-ID: On 29/05/2018 23:58, Guido van Rossum wrote: > Have we received complaints about other cases? Whenever this? breaks > code for which I am responsible I just blush and fix it, I don't > complain. Repeating the same warning typically just causes warning > fatigue rather than helping anyone. > Maybe what we need is to add a, possibly optional, or suppressible, warning whenever the import system encounters an implicit/indirect import? If an import that is working because the package we are importing it from has imported it from elsewhere, but it is not included in the __init__ for that package, then code authors will get some warning before the breakage and any blushes will really be due. If I was getting something along the lines of: Warning: Implicit Import in MyCode.py line 16 - import of os.errno is unsupported and may stop working on any update consider direct import. Then I would try to address it at the current version. I would be happy to try to implement this and think that the performance impact should be low. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From wes.turner at gmail.com Wed May 30 02:01:57 2018 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 30 May 2018 02:01:57 -0400 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: References: <25FFAB81-B7C9-45A8-A667-4DDABC07E7F0@barrys-emacs.org> <20180528234741.GC12683@ando.pearwood.info> <20180529004925.GD12683@ando.pearwood.info> <5B0CF35E.7090704@canterbury.ac.nz> <20180529071537.GE12683@ando.pearwood.info> <5B0DDE15.6020402@canterbury.ac.nz> Message-ID: Configuration management tools with far more code than this are regularly run with root privileges. OTOH, Salt and Ansible, for example, both support recursive chown and chmod; and report what actually changed. Yum/dnf probably do, too. Supporting recursive chmod/chown OOB may be useful. That it might be run as root is not the criteria, AFAIU. On Tuesday, May 29, 2018, Chris Angelico wrote: > On Wed, May 30, 2018 at 9:11 AM, Greg Ewing > wrote: > > BTW, I wouldn't argue that Python shouldn't provide things > > that are only useful to root. While writing setuid utilities > > in Python is a bad idea for lots of reasons, I don't think > > there's anything wrong with becoming root by another means > > and then running a Python program that you know well enough > > to trust. > > I'd go further. Once a shell script gets longer than about a page or > two of code, it often needs to be rewritten in a different language, > and Python is well situated to be that language. That doesn't change > when the script is to be run as root. I've written many Python scripts > to do sysadminning jobs for me - usually one-shot scripts, but also > some that stick around. Since I wrote the scripts myself, the trust > issue doesn't come up; I trust the Python interpreter the same way > that I trust /bin/bash. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Wed May 30 05:42:21 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 30 May 2018 02:42:21 -0700 (PDT) Subject: [Python-ideas] A real life example of "given" Message-ID: I thought I would share a recent use I had for "given": I have this comprehension: potential_updates = {y: command.create_potential_update(y) for x in need_initialization_nodes for y in [x, *x.synthetic_inputs()]} I want to filter out values that are None. I don't want to call the function call twice, so I have to resort to using a loop and appending or the for z in [y] trick. With "given", I can write: potential_updates = { y: potential_update for x in need_initialization_nodes for y in [x, *x.synthetic_inputs()] given potential_update = command.create_potential_update(y) if potential_update is not None} I also wanted to point out that code like this is self-commenting because the key and value of the comprehension can be given names. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed May 30 07:58:52 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 30 May 2018 21:58:52 +1000 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> <20180529145547.GH12683@ando.pearwood.info> Message-ID: <20180530115851.GI12683@ando.pearwood.info> On Wed, May 30, 2018 at 04:18:51AM +0000, Steve Barnes wrote: > Maybe what we need is to add a, possibly optional, or suppressible, > warning whenever the import system encounters an implicit/indirect > import? I don't think your terminology ("implicit/indirect") is very accurate. from os import errno is as explicit as you can get. > If an import that is working because the package we are > importing it from has imported it from elsewhere, but it is not included > in the __init__ for that package, I think you mean __all__ for the module. I'm not sure how that check would work. For a simple module, whenever you call "from module import name", the interpreter has to inspect the object it just imported, and if it is a module itself, check whether "name" is in the owning module's __all__. How would it work for packages? "from package import submodule" ought to work without a warning even if submodule isn't listed in __all__. Even for simple modules, it is prone to false positives: if "name" is documented as public, but not listed in __all__ then that would wrongly be detected as a non-public import. But most problematic, it does nothing about this case: import os os.errno I think this is best left for linters. -- Steve From steve at pearwood.info Wed May 30 08:22:19 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 30 May 2018 22:22:19 +1000 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: Message-ID: <20180530122218.GJ12683@ando.pearwood.info> On Wed, May 30, 2018 at 02:42:21AM -0700, Neil Girdhar wrote: > With "given", I can write: > > potential_updates = { > y: potential_update > for x in need_initialization_nodes > for y in [x, *x.synthetic_inputs()] > given potential_update = command.create_potential_update(y) > if potential_update is not None} I'm not sure if that would be legal for the "given" syntax. As I understand it, the "given" syntax is: expression given name = another_expression but you've got half of the comprehension stuffed in the gap between the leading expression and the "given" keyword: expression COMPREH- given name = another_expression -ENSION so I think that's going to be illegal. I think it wants to be written this way: potential_updates = { y: potential_update for x in need_initialization_nodes for y in [x, *x.synthetic_inputs()] if potential_update is not None given potential_update = command.create_potential_update(y) } Or maybe it should be this? potential_updates = { y: potential_update given potential_update = command.create_potential_update(y) for x in need_initialization_nodes for y in [x, *x.synthetic_inputs()] if potential_update is not None } I'm damned if I know which way is correct. Either of them? Neither? In comparison, I think that := is much simpler. There's only one place it can go: potential_updates = { y: potential_update for x in need_initialization_nodes for y in [x, *x.synthetic_inputs()] if ( potential_update := command.create_potential_update(y) ) is not None } -- Steve From ncoghlan at gmail.com Wed May 30 08:59:12 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 30 May 2018 22:59:12 +1000 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: References: Message-ID: On 30 May 2018 at 02:38, Guido van Rossum wrote: > Honestly, despite the occasional use case(1), I'm not sure that this is a > battery we need in the stdlib. Nobody seems too excited about writing the > code, and when the operation is needed, shelling out to the system chown is > not too onerous. (Ditto for chmod.) > > (1) Not even sure that a use case was shown -- it was just shown that the > operation is not necessarily useless. > My main use cases have been in installers and test suites, but those cases have also been for Linux-specific code where shelling out to "chown -R" and "chmod -R" was an entirely acceptable alternative. I think one of the other key points here is that "chown" and "chmod" inherently don't map at all well to the Windows filesystem access control model [1], so there's no new portability challenges arising from expecting the chown and chmod commands to be available. Cheers, Nick. [1] os.chown and shutil.chown don't exist at all there, and os.chmod only supports setting a file to read-only - there isn't any access to user or group permissions. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed May 30 11:12:15 2018 From: guido at python.org (Guido van Rossum) Date: Wed, 30 May 2018 08:12:15 -0700 Subject: [Python-ideas] Was `os.errno` undocumented? In-Reply-To: <20180530115851.GI12683@ando.pearwood.info> References: <203e5845-47d5-1f42-c2a5-ef70e9ba2585@gmail.com> <20180529145547.GH12683@ando.pearwood.info> <20180530115851.GI12683@ando.pearwood.info> Message-ID: Mypy would totally catch this. PEP 484 has specific words for whether imports are re-exported (they aren't unless you use 'as'). On Wed, May 30, 2018 at 4:58 AM, Steven D'Aprano wrote: > On Wed, May 30, 2018 at 04:18:51AM +0000, Steve Barnes wrote: > > > Maybe what we need is to add a, possibly optional, or suppressible, > > warning whenever the import system encounters an implicit/indirect > > import? > > I don't think your terminology ("implicit/indirect") is very accurate. > > from os import errno > > is as explicit as you can get. > > > > If an import that is working because the package we are > > importing it from has imported it from elsewhere, but it is not included > > in the __init__ for that package, > > I think you mean __all__ for the module. > > I'm not sure how that check would work. For a simple module, whenever > you call "from module import name", the interpreter has to inspect the > object it just imported, and if it is a module itself, check whether > "name" is in the owning module's __all__. > > How would it work for packages? "from package import submodule" ought to > work without a warning even if submodule isn't listed in __all__. > > Even for simple modules, it is prone to false positives: if "name" is > documented as public, but not listed in __all__ then that would wrongly > be detected as a non-public import. > > But most problematic, it does nothing about this case: > > import os > os.errno > > I think this is best left for linters. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.ed.oconnor at gmail.com Wed May 30 11:23:21 2018 From: peter.ed.oconnor at gmail.com (Peter O'Connor) Date: Wed, 30 May 2018 17:23:21 +0200 Subject: [Python-ideas] A real life example of "given" In-Reply-To: <20180530122218.GJ12683@ando.pearwood.info> References: <20180530122218.GJ12683@ando.pearwood.info> Message-ID: > > In comparison, I think that := is much simpler. In this case that's true, but a small modification: updates = { y: do_something_to(potential_update) for x in need_initialization_nodes for y in [x, *x.synthetic_inputs()] if potential_update is not None given potential_update = command.create_potential_update(y) } Shows the flexibility of this given syntax vs ":=" If we think of "given" as just inserting a line with variable-definitions before the preceding statement, it seems clear that: updates = { y: potential_update given potential_update = command.create_potential_update(y) for x in need_initialization_nodes for y in [x, *x.synthetic_inputs()] if potential_update is not None } Should raise a NameError: name 'potential_update' is not defined, and updates = { y: potential_update for x in need_initialization_nodes for y in [x, *x.synthetic_inputs()] given potential_update = command.create_potential_update(y) if potential_update is not None } Should raise a NameError: name 'y' is not defined. For safety it seems reasonable that if a variable is "given" in a comprehension, trying to refer to it (even if it defined in the enclosing scope) before the inner-definition will result in a NameError. On Wed, May 30, 2018 at 2:22 PM, Steven D'Aprano wrote: > On Wed, May 30, 2018 at 02:42:21AM -0700, Neil Girdhar wrote: > > > With "given", I can write: > > > > potential_updates = { > > y: potential_update > > for x in need_initialization_nodes > > for y in [x, *x.synthetic_inputs()] > > given potential_update = command.create_potential_update(y) > > if potential_update is not None} > > I'm not sure if that would be legal for the "given" syntax. As I > understand it, the "given" syntax is: > > expression given name = another_expression > > but you've got half of the comprehension stuffed in the gap between the > leading expression and the "given" keyword: > > expression COMPREH- given name = another_expression -ENSION > > so I think that's going to be illegal. > > > I think it wants to be written this way: > > potential_updates = { > y: potential_update > for x in need_initialization_nodes > for y in [x, *x.synthetic_inputs()] > if potential_update is not None > given potential_update = command.create_potential_update(y) > } > > > Or maybe it should be this? > > potential_updates = { > y: potential_update > given potential_update = command.create_potential_update(y) > for x in need_initialization_nodes > for y in [x, *x.synthetic_inputs()] > if potential_update is not None > } > > > I'm damned if I know which way is correct. Either of them? Neither? > > In comparison, I think that := is much simpler. There's only one place > it can go: > > potential_updates = { > y: potential_update > for x in need_initialization_nodes > for y in [x, *x.synthetic_inputs()] > if ( > potential_update := command.create_potential_update(y) > ) is not None > } > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.ed.oconnor at gmail.com Wed May 30 12:05:33 2018 From: peter.ed.oconnor at gmail.com (Peter O'Connor) Date: Wed, 30 May 2018 18:05:33 +0200 Subject: [Python-ideas] Proposal: A Reduce-Map Comprehension and a "last" builtin In-Reply-To: <20180524124900.GP12683@ando.pearwood.info> References: <20180524124900.GP12683@ando.pearwood.info> Message-ID: On Thu, May 24, 2018 at 2:49 PM, Steven D'Aprano wrote: > On Thu, May 24, 2018 at 02:06:03PM +0200, Peter O'Connor wrote: > > We could use given for both the in-loop variable update and the variable > > initialization: > > smooth_signal = [average given average=(1-decay)*average + decay*x > for > > x in signal] given average=0. > > So in your example, the OUTER "given" creates a local variable in the > current scope, average=0, but the INNER "given" inside the comprehension > exists inside a separate, sub-local comprehension scope, where you will > get an UnboundLocalError when it tries to evaluate (1-decay)*average the > first time. You're right, having re-thought it, it seems that the correct way to write it would be to define both of them in the scope of the comprehension: smooth_signal = [average given average=(1-decay)*average + decay*x for x in signal given average=0.] This makes sense and follows a simple rule: "B given A" just causes A to be executed before B - that holds true whether B is a variable or a loop declaration like "for x in x_gen". So a_gen = (g(a) given a=f(a, x) for x in x_gen given a=0) would be a compact form of: def a_gen_func(x_gen): a=0 for x in x_gen: a = f(a, x) yield g(a) a_gen = a_gen_func() -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Wed May 30 12:19:34 2018 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 30 May 2018 12:19:34 -0400 Subject: [Python-ideas] Add shutil.chown(..., recursive=False) In-Reply-To: References: Message-ID: On Wednesday, May 30, 2018, Nick Coghlan wrote: > On 30 May 2018 at 02:38, Guido van Rossum wrote: > >> Honestly, despite the occasional use case(1), I'm not sure that this is a >> battery we need in the stdlib. Nobody seems too excited about writing the >> code, and when the operation is needed, shelling out to the system chown is >> not too onerous. (Ditto for chmod.) >> >> (1) Not even sure that a use case was shown -- it was just shown that the >> operation is not necessarily useless. >> > > My main use cases have been in installers and test suites, but those cases > have also been for Linux-specific code where shelling out to "chown -R" and > "chmod -R" was an entirely acceptable alternative. > > I think one of the other key points here is that "chown" and "chmod" > inherently don't map at all well to the Windows filesystem access control > model [1], so there's no new portability challenges arising from expecting > the chown and chmod commands to be available. > > Cheers, > Nick. > > [1] os.chown and shutil.chown don't exist at all there, and os.chmod only > supports setting a file to read-only - there isn't any access to user or > group permissions. > Notably, Ansible and Salt don't haven't even tried to abstract Linux/BSD and Windows filesystem ACLs into a common API: https://github.com/ansible/ansible/blob/devel/lib/ansible/ modules/files/acl.py https://github.com/ansible/ansible/blob/devel/lib/ansible/ modules/windows/win_acl.py https://github.com/ansible/ansible/blob/devel/lib/ ansible/modules/windows/win_acl.ps1 https://github.com/saltstack/salt/blob/develop/salt/states/file.py https://github.com/saltstack/salt/blob/develop/salt/states/win_dacl.py https://github.com/saltstack/salt/blob/develop/salt/modules/win_dacl.py https://github.com/saltstack/salt/blob/develop/salt/utils/win_dacl.py https://github.com/saltstack/salt/blob/develop/salt/states/linux_acl.py https://github.com/saltstack/salt/blob/develop/salt/modules/linux_acl.py ... Looked these up and thought I'd share. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Wed May 30 13:50:24 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 30 May 2018 13:50:24 -0400 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> Message-ID: On Wed, May 30, 2018 at 11:32 AM Peter O'Connor wrote: > In comparison, I think that := is much simpler. > > > In this case that's true, but a small modification: > > updates = { > y: do_something_to(potential_update) > for x in need_initialization_nodes > for y in [x, *x.synthetic_inputs()] > if potential_update is not None > given potential_update = command.create_potential_update(y) > } > > Shows the flexibility of this given syntax vs ":=" > > If we think of "given" as just inserting a line with variable-definitions > before the preceding statement, it seems clear that: > > updates = { > y: potential_update > given potential_update = command.create_potential_update(y) > for x in need_initialization_nodes > for y in [x, *x.synthetic_inputs()] > if potential_update is not None > } > > Should raise a NameError: name 'potential_update' is not defined, and > > updates = { > y: potential_update > for x in need_initialization_nodes > for y in [x, *x.synthetic_inputs()] > given potential_update = command.create_potential_update(y) > if potential_update is not None > } > > > Should raise a NameError: name 'y' is not defined. > The reason I want it like that for comprehensions is that I think of it as equivalent to: updates = {} for x in need_initialization_nodes: for y in [x, *x.synthetic_inputs()]: potential_update = command.create_potential_update(y) if potential_update is not None: updates[y] = potential_update But you're right that this would be a second addition to the grammar. One addition would be to "test" for something like test: bool_test [comp_given] bool_test: or_test ['if' or_test 'else' test] | lambdef comp_given: 'given' testlist_star_expr annassign The second would permit the usage in comprehensions: comp_iter: comp_for | comp_if | comp_given Best, Neil For safety it seems reasonable that if a variable is "given" in a > comprehension, trying to refer to it (even if it defined in the enclosing > scope) before the inner-definition will result in a NameError. > > > On Wed, May 30, 2018 at 2:22 PM, Steven D'Aprano > wrote: > >> On Wed, May 30, 2018 at 02:42:21AM -0700, Neil Girdhar wrote: >> >> > With "given", I can write: >> > >> > potential_updates = { >> > y: potential_update >> > for x in need_initialization_nodes >> > for y in [x, *x.synthetic_inputs()] >> > given potential_update = command.create_potential_update(y) >> > if potential_update is not None} >> >> I'm not sure if that would be legal for the "given" syntax. As I >> understand it, the "given" syntax is: >> >> expression given name = another_expression >> >> but you've got half of the comprehension stuffed in the gap between the >> leading expression and the "given" keyword: >> >> expression COMPREH- given name = another_expression -ENSION >> >> so I think that's going to be illegal. >> >> >> I think it wants to be written this way: >> >> potential_updates = { >> y: potential_update >> for x in need_initialization_nodes >> for y in [x, *x.synthetic_inputs()] >> if potential_update is not None >> given potential_update = command.create_potential_update(y) >> } >> >> >> Or maybe it should be this? >> >> potential_updates = { >> y: potential_update >> given potential_update = command.create_potential_update(y) >> for x in need_initialization_nodes >> for y in [x, *x.synthetic_inputs()] >> if potential_update is not None >> } >> >> >> I'm damned if I know which way is correct. Either of them? Neither? >> >> In comparison, I think that := is much simpler. There's only one place >> it can go: >> >> potential_updates = { >> y: potential_update >> for x in need_initialization_nodes >> for y in [x, *x.synthetic_inputs()] >> if ( >> potential_update := command.create_potential_update(y) >> ) is not None >> } >> >> >> -- >> Steve >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/keaR3FudcwQ/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/keaR3FudcwQ/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed May 30 13:50:48 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 31 May 2018 03:50:48 +1000 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> Message-ID: On Thu, May 31, 2018 at 1:23 AM, Peter O'Connor wrote: >> In comparison, I think that := is much simpler. > > > In this case that's true, but a small modification: > > updates = { > y: do_something_to(potential_update) > for x in need_initialization_nodes > for y in [x, *x.synthetic_inputs()] > if potential_update is not None > given potential_update = command.create_potential_update(y) > } > > Shows the flexibility of this given syntax vs ":=" I don't understand what you're showcasing here. With :=, you give a name to something at the exact point that it happens: updates = { y: do_something_to(potential_update) for x in need_initialization_nodes for y in [x, *x.synthetic_inputs()] if (potential_update := command.create_potential_update(y)) is not None } Personally, I'd use a shorter name for something that's used in such a small scope (same as you use one-letter "x" and "y"). But that's the only way that the 'given' syntax looks at all better - by encouraging you to use yet another line, it conceals some of its immense verbosity. (Note how the name "potential_update" is used twice with :=, once to set and one to retrieve; but with given, it's used three times - retrieve, retrieve, and set.) How does this show that 'given' is more flexible? ChrisA From mistersheik at gmail.com Wed May 30 13:56:08 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 30 May 2018 13:56:08 -0400 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> Message-ID: On Wed, May 30, 2018 at 1:52 PM Chris Angelico wrote: > On Thu, May 31, 2018 at 1:23 AM, Peter O'Connor > wrote: > >> In comparison, I think that := is much simpler. > > > > > > In this case that's true, but a small modification: > > > > updates = { > > y: do_something_to(potential_update) > > for x in need_initialization_nodes > > for y in [x, *x.synthetic_inputs()] > > if potential_update is not None > > given potential_update = command.create_potential_update(y) > > } > > > > Shows the flexibility of this given syntax vs ":=" > > I don't understand what you're showcasing here. With :=, you give a > name to something at the exact point that it happens: > > updates = { > y: do_something_to(potential_update) > for x in need_initialization_nodes > for y in [x, *x.synthetic_inputs()] > if (potential_update := > command.create_potential_update(y)) is not None > } > > Personally, I'd use a shorter name for something that's used in such a > small scope (same as you use one-letter "x" and "y"). But that's the > only way that the 'given' syntax looks at all better - by encouraging > you to use yet another line, it conceals some of its immense > verbosity. (Note how the name "potential_update" is used twice with > :=, once to set and one to retrieve; but with given, it's used three > times - retrieve, retrieve, and set.) > > How does this show that 'given' is more flexible? > > Oh yeah, good point, I forgot that I could use := within the condition itself. It does show that this feature is useful, but not that given is more flexible than :=. > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/keaR3FudcwQ/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Wed May 30 13:59:37 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 30 May 2018 13:59:37 -0400 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> Message-ID: On Wed, May 30, 2018 at 1:56 PM Neil Girdhar wrote: > On Wed, May 30, 2018 at 1:52 PM Chris Angelico wrote: > >> On Thu, May 31, 2018 at 1:23 AM, Peter O'Connor >> wrote: >> >> In comparison, I think that := is much simpler. >> > >> > >> > In this case that's true, but a small modification: >> > >> > updates = { >> > y: do_something_to(potential_update) >> > for x in need_initialization_nodes >> > for y in [x, *x.synthetic_inputs()] >> > if potential_update is not None >> > given potential_update = command.create_potential_update(y) >> > } >> > >> > Shows the flexibility of this given syntax vs ":=" >> >> I don't understand what you're showcasing here. With :=, you give a >> name to something at the exact point that it happens: >> >> updates = { >> y: do_something_to(potential_update) >> for x in need_initialization_nodes >> for y in [x, *x.synthetic_inputs()] >> if (potential_update := >> command.create_potential_update(y)) is not None >> } >> >> Personally, I'd use a shorter name for something that's used in such a >> small scope (same as you use one-letter "x" and "y"). But that's the >> only way that the 'given' syntax looks at all better - by encouraging >> you to use yet another line, it conceals some of its immense >> verbosity. (Note how the name "potential_update" is used twice with >> :=, once to set and one to retrieve; but with given, it's used three >> times - retrieve, retrieve, and set.) >> >> How does this show that 'given' is more flexible? >> >> > Oh yeah, good point, I forgot that I could use := within the condition > itself. It does show that this feature is useful, but not that given is > more flexible than :=. > This example shows additional flexibility: z = {a: transformed_b for b in bs given transformed_b = transform(b) for a in as_} There is no nice, equivalent := version as far as I can tell. > > >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/python-ideas/keaR3FudcwQ/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.ed.oconnor at gmail.com Wed May 30 14:09:32 2018 From: peter.ed.oconnor at gmail.com (Peter O'Connor) Date: Wed, 30 May 2018 20:09:32 +0200 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> Message-ID: On Wed, May 30, 2018 at 7:59 PM, Neil Girdhar wrote: > > z = {a: transformed_b > for b in bs > given transformed_b = transform(b) > for a in as_} > > There is no nice, equivalent := version as far as I can tell. > Well you could just do: z = {a: b for b in (transform(bi) for bi in bs) for a in as_} -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed May 30 14:06:51 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 31 May 2018 04:06:51 +1000 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> Message-ID: On Thu, May 31, 2018 at 3:59 AM, Neil Girdhar wrote: > This example shows additional flexibility: > > z = {a: transformed_b > for b in bs > given transformed_b = transform(b) > for a in as_} > > There is no nice, equivalent := version as far as I can tell. True. However, it took me several readings to understand what you were doing here. I think I actually prefer "for transformed_b in [transform(b)]" to this syntax, which is saying something. ChrisA From mistersheik at gmail.com Wed May 30 14:34:01 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 30 May 2018 14:34:01 -0400 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> Message-ID: Peter wrote: > Well you could just do: z = {a: b for b in (transform(bi) for bi in bs) for a in as_} That works, but I prefer the implicit nesting of a sequence of "comp_for" expressions to a the nested generator. On Wed, May 30, 2018 at 2:16 PM Chris Angelico wrote: > On Thu, May 31, 2018 at 3:59 AM, Neil Girdhar > wrote: > > This example shows additional flexibility: > > > > z = {a: transformed_b > > for b in bs > > given transformed_b = transform(b) > > for a in as_} > > > > There is no nice, equivalent := version as far as I can tell. > > True. However, it took me several readings to understand what you were > doing here. I think I actually prefer "for transformed_b in > [transform(b)]" to this syntax, which is saying something. > I feel you. I think of "given" as an assignment that is in front of the expression, just like "for" (in comp_for) is a for loop that is in front, and "if" (in comp_if) is a condition that is in front. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/keaR3FudcwQ/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From drekin at gmail.com Wed May 30 18:36:45 2018 From: drekin at gmail.com (=?UTF-8?B?QWRhbSBCYXJ0b8Wh?=) Date: Thu, 31 May 2018 00:36:45 +0200 Subject: [Python-ideas] A real life example of "given" Message-ID: A side comment: > potential_updates = { > y: potential_update > for x in need_initialization_nodes > for y in [x, *x.synthetic_inputs()] > given potential_update = command.create_potential_update(y) > if potential_update is not None} Probably, I would write @make_dict def potential_updates(): for x in need_initialization_nodes: for y in [x, *x.synthetic_inputs()]: potential_update = command.create_potential_update(y) if potential_update is not None: yield y, potential_update If such pattern covered a lot of examples, even some syntax sugar could be added. For example: foo[0].potential_updates = dict(_) from: for x in need_initialization_nodes: for y in [x, *x.synthetic_inputs()]: potential_update = command.create_potential_update(y) if potential_update is not None: yield y, potential_update where from: would be equivalent to def _(): _ = _() Best regards, Adam Barto? -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed May 30 19:53:35 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 31 May 2018 09:53:35 +1000 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> Message-ID: <20180530235334.GK12683@ando.pearwood.info> On Wed, May 30, 2018 at 01:59:37PM -0400, Neil Girdhar wrote: > This example shows additional flexibility: > > z = {a: transformed_b > for b in bs > given transformed_b = transform(b) > for a in as_} Is that even legal? Again, you're putting half of the comprehension in the middle of the given expression. I believe that "given" expression syntax is: expression given name = another_expression it's not a syntactic form that we can split across arbitrary chunks of code: # surely this won't be legal? def method(self, arg, x=spam): body given spam = expression Comprehension syntax in this case is: {key:expr for b in it1 for a in it2} (of course comprehensions can also include more loops and if clauses, but this example doesn't use those). So you've interleaved part of the given expression and part of the comprehension: {key: expression COMPRE- given name = another_expression -HENSION} That's the second time you've done that. Neil, if my analysis is correct, I think you have done us a great service: showing that the "given" expression syntax really encourages people to generate syntax errors in their comprehensions. > There is no nice, equivalent := version as far as I can tell. Given (pun intended) the fact that you only use transformed_b in a single place, I don't think it is necessary to use := at all. z = {a: transform(b) for b in bs for a in as_} But if you really insist: # Pointless use of := z = {a: (transformed_b := transform(b)) for b in bs for a in as_} -- Steve From rosuav at gmail.com Wed May 30 20:05:33 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 31 May 2018 10:05:33 +1000 Subject: [Python-ideas] A real life example of "given" In-Reply-To: <20180530235334.GK12683@ando.pearwood.info> References: <20180530122218.GJ12683@ando.pearwood.info> <20180530235334.GK12683@ando.pearwood.info> Message-ID: On Thu, May 31, 2018 at 9:53 AM, Steven D'Aprano wrote: >> There is no nice, equivalent := version as far as I can tell. > > Given (pun intended) the fact that you only use transformed_b in a > single place, I don't think it is necessary to use := at all. > > z = {a: transform(b) for b in bs for a in as_} > > But if you really insist: > > # Pointless use of := > z = {a: (transformed_b := transform(b)) for b in bs for a in as_} > That's the subtlety of the 'given' usage here. You fell for the same trap I did: thinking "it's only used once". Actually, what he has is equivalent to: z = {a: tb for b in bs for tb in [transform(b)] for a in as_} which means it evaluates transform(b) once regardless of the length of as_. But it's really REALLY not obvious. That's why I actually prefer the "interpolated 'for' loop" notation, despite it being distinctly distasteful in general. At least it's obvious that something weird is happening, so you don't instantly assume that you can inline the single usage. ChrisA From steve at pearwood.info Wed May 30 20:05:25 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 31 May 2018 10:05:25 +1000 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> Message-ID: <20180531000524.GL12683@ando.pearwood.info> On Thu, May 31, 2018 at 04:06:51AM +1000, Chris Angelico wrote: > On Thu, May 31, 2018 at 3:59 AM, Neil Girdhar wrote: > > This example shows additional flexibility: > > > > z = {a: transformed_b > > for b in bs > > given transformed_b = transform(b) > > for a in as_} > > > > There is no nice, equivalent := version as far as I can tell. > > True. However, it took me several readings to understand what you were > doing here. Possibly you shouldn't have tried reading at 4am. Either that or I shouldn't be reading before I've had a coffee :-) Have I missed something that you have seen? Even if the syntax were legal, that seems to be a pointless use of an assignment expression. Since the new name "transformed_b" is only used once, we can and should just use the transform(b) in place: z = {a: transform(b) for b in bs for a in as_} If we need to use it twice, we can do this: # assume "@" stands in for something useful z = {a: (transformed_b := transform(b)) @ transformed_b for b in bs for a in as_} I'm not seeing the advantage of given, or any extra flexibility here, unless the aim is to encourage people to make syntax errors :-) What have I missed? -- Steve From rosuav at gmail.com Wed May 30 20:14:39 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 31 May 2018 10:14:39 +1000 Subject: [Python-ideas] A real life example of "given" In-Reply-To: <20180531000524.GL12683@ando.pearwood.info> References: <20180530122218.GJ12683@ando.pearwood.info> <20180531000524.GL12683@ando.pearwood.info> Message-ID: On Thu, May 31, 2018 at 10:05 AM, Steven D'Aprano wrote: > On Thu, May 31, 2018 at 04:06:51AM +1000, Chris Angelico wrote: >> On Thu, May 31, 2018 at 3:59 AM, Neil Girdhar wrote: >> > This example shows additional flexibility: >> > >> > z = {a: transformed_b >> > for b in bs >> > given transformed_b = transform(b) >> > for a in as_} >> > >> > There is no nice, equivalent := version as far as I can tell. >> >> True. However, it took me several readings to understand what you were >> doing here. > > Possibly you shouldn't have tried reading at 4am. > > Either that or I shouldn't be reading before I've had a coffee :-) > > Have I missed something that you have seen? Yep, as mentioned in the other post. The fact that you talk like this about it - asserting that it's obvious what this does, while still considering it to be utterly useless - is proof, IMO, that this should be frowned upon in style guides. ChrisA From rob.cliffe at btinternet.com Wed May 30 20:45:20 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 31 May 2018 01:45:20 +0100 Subject: [Python-ideas] Proposal: A Reduce-Map Comprehension and a "last" builtin In-Reply-To: References: <20180524124900.GP12683@ando.pearwood.info> Message-ID: On 30/05/2018 17:05, Peter O'Connor wrote: > > On Thu, May 24, 2018 at 2:49 PM, Steven D'Aprano > wrote: > > On Thu, May 24, 2018 at 02:06:03PM +0200, Peter O'Connor wrote: > > We could use given for both the in-loop variable update and the > variable > > initialization: > >? ? smooth_signal =? [average given average=(1-decay)*average + > decay*x for > > x in signal] given average=0. > > So in your example, the OUTER "given" creates a local variable in the > current scope, average=0, but the INNER "given" inside the > comprehension > exists inside a separate, sub-local comprehension scope, where you > will > get an UnboundLocalError when it tries to evaluate > (1-decay)*average the > first time. > > > You're right, having re-thought it, it seems that the correct way to > write it would be to define both of them in the scope of the > comprehension: > > ? smooth_signal =? [average given average=(1-decay)*average + decay*x > for?x in signal given average=0.] > > This makes sense and follows a simple rule: "B given A" just causes A > to be executed before B - that holds true whether B is a variable or a > loop declaration like "for x in x_gen". > > So > > ? ? a_gen = (g(a) given a=f(a, x) for x in x_gen given a=0) > > would be a compact form of: > > ? ? def a_gen_func(x_gen): > ? ? ? ? a=0 > ? ? ? ? for x in x_gen: > ? ? ? ? ? ? a = f(a, x) > ? ? ? ? ? ? yield g(a) > ? ? a_gen = a_gen_func() [There is a typo here - a_gen_func is defined to take 1 argument but is called with none.] After - *I think* - understanding this, I would try to make the one-line clearer by parenthesizing it thus (whether or not the grammar required it): ??? a_gen = ( ((g(a) given a=f(a, x)) for x in x_gen) given a=0) Even then, it would make my head spin if I came across it.? I hope no-one would write code like that. I'm not keen on given, but I must admit that ISTM that this example shows something that can only be done with given: putting some initialisation, viz. "a=0", into a generator expression.? With :=, it would need a trick: ??? a_gen = (g( a:=f(a, x) ) for x in [x_gen, a:=0][0] ) or ??? a_gen = (g( a:=f(a, x) ) for a in [0] for x in x_gen] ) Of course, in the case of a list comprehension (as opposed to a genexp), the initialisation could be done separately: ??? a = 0 ??? a_list = [g( a:=f(a, x) ) for x in x_gen] Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed May 30 21:01:05 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 31 May 2018 11:01:05 +1000 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> <20180530235334.GK12683@ando.pearwood.info> Message-ID: <20180531010104.GM12683@ando.pearwood.info> On Thu, May 31, 2018 at 10:05:33AM +1000, Chris Angelico wrote: > On Thu, May 31, 2018 at 9:53 AM, Steven D'Aprano wrote: > >> There is no nice, equivalent := version as far as I can tell. > > > > Given (pun intended) the fact that you only use transformed_b in a > > single place, I don't think it is necessary to use := at all. > > > > z = {a: transform(b) for b in bs for a in as_} > > > > But if you really insist: > > > > # Pointless use of := > > z = {a: (transformed_b := transform(b)) for b in bs for a in as_} > > > > That's the subtlety of the 'given' usage here. You fell for the same > trap I did: thinking "it's only used once". But it is only used once. I meant once per loop. It isn't used in the "for a in as_" inner loop, there's no "if transformed_b" condition, and it only is used once in the key:value part of the comprehension. > Actually, what he has is equivalent to: > > z = {a: tb for b in bs for tb in [transform(b)] for a in as_} Which also uses tb only once, making it a Useless Use Of Assignment. (I assume we're not calling transform() for some side-effect, like logging a message, or erasing your hard drive.) > which means it evaluates transform(b) once regardless of the length of > as_. Ah yes, I see what you mean. Expanded to a loop: for b in bs: tb = transform(b) for a in as_: z[a] = tb It's a little ugly, but there's a trick I already use today: py> [x+y for x in "abc" if print(x) or True for y in "de"] a b c ['ad', 'ae', 'bd', 'be', 'cd', 'ce'] So we can adapt that to assignment instead of output: # Don't do this! z = {a: tb for b in bs if (tb := transform(b)) or True for a in as_} But I wouldn't do that. If I'm concerned about the call to transform (because it is super expensive, say) then I set up a pipeline: tbs = (transform(b) for b in bs) # or map(transform, bs) z = {a: tb for tb in tbs for a in as_} The first generator comprehension can be easily embedded in the other: z = {a: tb for tb in (transform(b) for b in bs) for a in as_} This makes it super-obvious that transform is called for each b, not for each (b, a) pair, it works today, and there's no assignment expression needed at all. Assignment expressions should not be about adding yet a third way to solve a problem that already has a perfectly good solution! ("Expand to a loop statement" is not a *perfectly* good solution.) To showcase assignment expressions, we should be solving problems that don't have a good solution now. I'm still not convinced that Neil's "given" example will even work (see below) but *if he is right* that it does, perhaps that's a good reason to prefer the simpler := assignment expression syntax, since we're less likely to use it in confusing ways. > But it's really REALLY not obvious. But is it even legal? As I understand it, "given" is an expression, not an addition to comprehension syntax. In that case, I don't think Neil's example will work at all, for reasons I've already stated. If that's not the case, then until somebody tells me what this new comprehension syntax means, and what it looks like, I have no idea what is intended. Which of these can we write, and what do they do? [expression given name=something for x in seq] [expression for x given name=something in seq] [expression for x in seq given name=something] [expression for x in seq if given name=something condition] [expression for x in seq if condition given name=something] -- Steve From mistersheik at gmail.com Wed May 30 22:22:10 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 30 May 2018 22:22:10 -0400 Subject: [Python-ideas] A real life example of "given" In-Reply-To: <20180531000524.GL12683@ando.pearwood.info> References: <20180530122218.GJ12683@ando.pearwood.info> <20180531000524.GL12683@ando.pearwood.info> Message-ID: On Wed, May 30, 2018 at 8:10 PM Steven D'Aprano wrote: > On Thu, May 31, 2018 at 04:06:51AM +1000, Chris Angelico wrote: > > On Thu, May 31, 2018 at 3:59 AM, Neil Girdhar > wrote: > > > This example shows additional flexibility: > > > > > > z = {a: transformed_b > > > for b in bs > > > given transformed_b = transform(b) > > > for a in as_} > > > > > > There is no nice, equivalent := version as far as I can tell. > > > > True. However, it took me several readings to understand what you were > > doing here. > > Possibly you shouldn't have tried reading at 4am. > > Either that or I shouldn't be reading before I've had a coffee :-) > > Have I missed something that you have seen? Even if the syntax were > legal, that seems to be a pointless use of an assignment expression. > Since the new name "transformed_b" is only used once, we can and should > just use the transform(b) in place: > > z = {a: transform(b) for b in bs for a in as_} > Chris just explained it to you. You're calling transform too often. > > If we need to use it twice, we can do this: > > # assume "@" stands in for something useful > z = {a: (transformed_b := transform(b)) @ transformed_b > for b in bs for a in as_} > > > I'm not seeing the advantage of given, or any extra flexibility here, > unless the aim is to encourage people to make syntax errors :-) > > The flexibility of "given" is in giving names to elements of expressions and comprehensions to avoid recalculation. > What have I missed? > > Like you say, := and given both work for expressions. "given" could theoretically also be used in comprehensions. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/keaR3FudcwQ/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Wed May 30 22:19:27 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 30 May 2018 22:19:27 -0400 Subject: [Python-ideas] A real life example of "given" In-Reply-To: <20180530235334.GK12683@ando.pearwood.info> References: <20180530122218.GJ12683@ando.pearwood.info> <20180530235334.GK12683@ando.pearwood.info> Message-ID: On Wed, May 30, 2018 at 7:54 PM Steven D'Aprano wrote: > On Wed, May 30, 2018 at 01:59:37PM -0400, Neil Girdhar wrote: > > > This example shows additional flexibility: > > > > z = {a: transformed_b > > for b in bs > > given transformed_b = transform(b) > > for a in as_} > > Is that even legal? > In case you missed my earlier reply to you: One addition to the grammar would be to "test" for something like test: bool_test [comp_given] bool_test: or_test ['if' or_test 'else' test] | lambdef comp_given: 'given' testlist_star_expr annassign The second would permit the usage in comprehensions: comp_iter: comp_for | comp_if | comp_given > > Again, you're putting half of the comprehension in the middle of > the given expression. I believe that "given" expression syntax is: > > expression given name = another_expression > > it's not a syntactic form that we can split across arbitrary chunks of > code: > > # surely this won't be legal? > def method(self, arg, x=spam): > body > given spam = expression > > > Comprehension syntax in this case is: > > {key:expr for b in it1 for a in it2} > > (of course comprehensions can also include more loops and if clauses, > but this example doesn't use those). So you've interleaved part of the > given expression and part of the comprehension: > > {key: expression COMPRE- given name = another_expression -HENSION} > > > That's the second time you've done that. Neil, if my analysis is > correct, I think you have done us a great service: showing that the > "given" expression syntax really encourages people to generate syntax > errors in their comprehensions. > > > There is no nice, equivalent := version as far as I can tell. > > Given (pun intended) the fact that you only use transformed_b in a > single place, I don't think it is necessary to use := at all. > > z = {a: transform(b) for b in bs for a in as_} > > But if you really insist: > > # Pointless use of := > z = {a: (transformed_b := transform(b)) for b in bs for a in as_} > > Those call transform for every a needlessly. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/keaR3FudcwQ/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Wed May 30 22:50:46 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 30 May 2018 22:50:46 -0400 Subject: [Python-ideas] A real life example of "given" In-Reply-To: <20180531010104.GM12683@ando.pearwood.info> References: <20180530122218.GJ12683@ando.pearwood.info> <20180530235334.GK12683@ando.pearwood.info> <20180531010104.GM12683@ando.pearwood.info> Message-ID: On Wed, May 30, 2018 at 9:02 PM Steven D'Aprano wrote: > On Thu, May 31, 2018 at 10:05:33AM +1000, Chris Angelico wrote: > > On Thu, May 31, 2018 at 9:53 AM, Steven D'Aprano > wrote: > > >> There is no nice, equivalent := version as far as I can tell. > > > > > > Given (pun intended) the fact that you only use transformed_b in a > > > single place, I don't think it is necessary to use := at all. > > > > > > z = {a: transform(b) for b in bs for a in as_} > > > > > > But if you really insist: > > > > > > # Pointless use of := > > > z = {a: (transformed_b := transform(b)) for b in bs for a in as_} > > > > > > > That's the subtlety of the 'given' usage here. You fell for the same > > trap I did: thinking "it's only used once". > > But it is only used once. I meant once per loop. > > It isn't used in the "for a in as_" inner loop, there's no "if > transformed_b" condition, and it only is used once in the key:value part > of the comprehension. > > > > Actually, what he has is equivalent to: > > > > z = {a: tb for b in bs for tb in [transform(b)] for a in as_} > > Which also uses tb only once, making it a Useless Use Of Assignment. > > (I assume we're not calling transform() for some side-effect, like > logging a message, or erasing your hard drive.) > > > > which means it evaluates transform(b) once regardless of the length of > > as_. > > Ah yes, I see what you mean. Expanded to a loop: > > for b in bs: > tb = transform(b) > for a in as_: > z[a] = tb > > > It's a little ugly, but there's a trick I already use today: > > py> [x+y for x in "abc" if print(x) or True for y in "de"] > a > b > c > ['ad', 'ae', 'bd', 'be', 'cd', 'ce'] > > So we can adapt that to assignment instead of output: > > # Don't do this! > z = {a: tb for b in bs if (tb := transform(b)) or True for a in as_} > > But I wouldn't do that. If I'm concerned about the call to transform > (because it is super expensive, say) then I set up a pipeline: > > tbs = (transform(b) for b in bs) # or map(transform, bs) > z = {a: tb for tb in tbs for a in as_} > > The first generator comprehension can be easily embedded in the other: > > z = {a: tb for tb in (transform(b) for b in bs) for a in as_} > > This makes it super-obvious that transform is called for each b, not for > each (b, a) pair, it works today, and there's no assignment expression > needed at all. > > Assignment expressions should not be about adding yet a third way to > solve a problem that already has a perfectly good solution! ("Expand to > a loop statement" is not a *perfectly* good solution.) To showcase > assignment expressions, we should be solving problems that don't have a > good solution now. > > I'm still not convinced that Neil's "given" example will even work (see > below) but *if he is right* that it does, perhaps that's a good reason > to prefer the simpler := assignment expression syntax, since we're > less likely to use it in confusing ways. > > > > But it's really REALLY not obvious. > > But is it even legal? > > As I understand it, "given" is an expression, not an addition to > comprehension syntax. In that case, I don't think Neil's example will > work at all, for reasons I've already stated. > > If that's not the case, then until somebody tells me what this new > comprehension syntax means, and what it looks like, I have no idea what > is intended. > > Which of these can we write, and what do they do? > Great question. The trick is to just write them as a sequence of statements without changing the order except to put the expression last. > > [expression given name=something for x in seq] > retval = [] name = something for x in seq: retval.append(expression) return retval > > [expression for x given name=something in seq] > this one doesn't make sense. [expression for x in seq given name=something] > > retval = [] for x in seq: name = something retval.append(expression) return retval > [expression for x in seq if given name=something condition] > > this one doesn't make sense. > [expression for x in seq if condition given name=something] > > retval = [] for x in seq: if condition: name = something retval.append(expression) return retval and of course, the original proposal expression given name=something means: name = something retval = expression return retval > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/keaR3FudcwQ/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danilo.bellini at gmail.com Thu May 31 00:47:17 2018 From: danilo.bellini at gmail.com (Danilo J. S. Bellini) Date: Thu, 31 May 2018 01:47:17 -0300 Subject: [Python-ideas] Let try-except check the exception instance Message-ID: Hi! I was working on handling some exceptions from external software (e.g. database constraint triggers) switching the handler based on the messages that had been sent. Today we can do something like (running on Python 3.6.5): >>> try: ... # [...] ... session.commit() # Here it raises! ... # [...] ... except DatabaseError as exc: ... msg = get_db_error_msg_from_exception(exc) ... if msg == "beyond_limit": ... # [...] ... elif msg == "no_funds": ... # [...] ... else: ... raise That works, but I'd like to do something like: >>> try: ... # [...] ... except BeyondLimit: ... # [...] ... except NoFunds: ... # [...] Creating classes to "match" the exception in their __instancecheck__. Well, I tried to do so. A simplified example would be: >>> class MsgCheckerMeta(type): ... def __instancecheck__(cls, instance): ... return str(instance) == cls.internal ... >>> class ExceptionHasMessage(Exception, metaclass=MsgCheckerMeta): ... internal = "message" Using these new classes, we would get this: >>> try: ... raise Exception("message") ... except ExceptionHasMessage: ... print("Yeah!") ... Traceback (most recent call last): File "", line 2, in Exception: message Yet, >>> isinstance(Exception("message"), ExceptionHasMessage) True >>> try: ... raise Exception("message") ... except Exception as exc: ... print(isinstance(exc, ExceptionHasMessage)) ... True The idea is to allow catching exceptions beyond checking their MRO, using a class that checks the exception instance by implementing a custom __instancecheck__. -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu May 31 01:41:48 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 31 May 2018 15:41:48 +1000 Subject: [Python-ideas] Let try-except check the exception instance In-Reply-To: References: Message-ID: <20180531054148.GN12683@ando.pearwood.info> On Thu, May 31, 2018 at 01:47:17AM -0300, Danilo J. S. Bellini wrote: > >>> try: > ... # [...] > ... session.commit() # Here it raises! > ... # [...] > ... except DatabaseError as exc: > ... msg = get_db_error_msg_from_exception(exc) > ... if msg == "beyond_limit": > ... # [...] > ... elif msg == "no_funds": > ... # [...] > ... else: > ... raise Since error messages are rarely part of the exception API, testing for them is fragile and prone to errors. For example, what if the message changes to "no money" or "out of funds" or "insufficient funds" or "keine Mittel"? Since the error message is not part of the API, that can happen at any time, without warning. If you want to make such a risky check in your own code, of course you can do so, but we shouldn't encourage it or add functionality to make it easier. You have the right to shoot yourself in the foot, but don't expect us to load the gun, hand it to you, point it at your foot, and place your finger on the trigger :-) > That works, but I'd like to do something like: > > > >>> try: > ... # [...] > ... except BeyondLimit: > ... # [...] > ... except NoFunds: > ... # [...] The obvious way to do that is to create BeyondLimit and NoFunds subclasses of DatabaseError. Why can't you do that? > Creating classes to "match" the exception in their __instancecheck__. > Well, I tried to do so. A simplified example would be: > > >>> class MsgCheckerMeta(type): > ... def __instancecheck__(cls, instance): > ... return str(instance) == cls.internal > ... > >>> class ExceptionHasMessage(Exception, metaclass=MsgCheckerMeta): > ... internal = "message" That seems like a backwards way to do it to me. If you're the author of the exception class, why not just subclass your exceptions the regular way and get a real subclass? If you have enough control of the source to change the exception from DatabaseError to ExceptionHasMessage, then you ought to have enough control to change to a specific subclass as needed. And if you *don't* have that control, then you're probably monkey- patching something you don't control and again, we shouldn't encourage that. Do you have a better example of checking __instancecheck__ that doesn't involve something that's a risky hack? -- Steve From leewangzhong+python at gmail.com Thu May 31 01:55:34 2018 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Thu, 31 May 2018 01:55:34 -0400 Subject: [Python-ideas] Let try-except check the exception instance In-Reply-To: References: Message-ID: Would guards (such as from the switch-case discussions) be a good fit? try: ... except RuntimeError as e if e.message == "tie": ... On Thu, May 31, 2018, 00:48 Danilo J. S. Bellini wrote: > Hi! > I was working on handling some exceptions from external software > (e.g. database constraint triggers) > switching the handler based on the messages that had been sent. > Today we can do something like (running on Python 3.6.5): > > > >>> try: > ... # [...] > ... session.commit() # Here it raises! > ... # [...] > ... except DatabaseError as exc: > ... msg = get_db_error_msg_from_exception(exc) > ... if msg == "beyond_limit": > ... # [...] > ... elif msg == "no_funds": > ... # [...] > ... else: > ... raise > > > That works, but I'd like to do something like: > > > >>> try: > ... # [...] > ... except BeyondLimit: > ... # [...] > ... except NoFunds: > ... # [...] > > > Creating classes to "match" the exception in their __instancecheck__. > Well, I tried to do so. A simplified example would be: > > > >>> class MsgCheckerMeta(type): > ... def __instancecheck__(cls, instance): > ... return str(instance) == cls.internal > ... > >>> class ExceptionHasMessage(Exception, metaclass=MsgCheckerMeta): > ... internal = "message" > > > Using these new classes, we would get this: > > > >>> try: > ... raise Exception("message") > ... except ExceptionHasMessage: > ... print("Yeah!") > ... > Traceback (most recent call last): > File "", line 2, in > Exception: message > > > Yet, > > > >>> isinstance(Exception("message"), ExceptionHasMessage) > True > >>> try: > ... raise Exception("message") > ... except Exception as exc: > ... print(isinstance(exc, ExceptionHasMessage)) > ... > True > > > The idea is to allow catching exceptions beyond checking their MRO, > using a class that checks the exception instance by implementing > a custom __instancecheck__. > > -- > Danilo J. S. Bellini > --------------- > "*It is not our business to set up prohibitions, but to arrive at > conventions.*" (R. Carnap) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu May 31 03:19:27 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 31 May 2018 03:19:27 -0400 Subject: [Python-ideas] Let try-except check the exception instance In-Reply-To: References: Message-ID: On 5/31/2018 12:47 AM, Danilo J. S. Bellini wrote: > Hi! > I was working on handling some exceptions from external software > (e.g. database constraint triggers) > switching the handler based on the messages that had been sent. > Today we can do something like (running on Python 3.6.5): > > >>>> try: > ... ??? # [...] > ... ??? session.commit() # Here it raises! > ... ??? # [...] > ... except DatabaseError as exc: > ... ??? msg = get_db_error_msg_from_exception(exc) > ... ??? if msg == "beyond_limit": > ... ??????? # [...] > ... ??? elif msg == "no_funds": > ... ??????? # [...] > ... ??? else: > ... ??????? raise > > > That works, Yes, it works perfectly well, AND it exposes the fact that your code depends on the message, which I think is a good thing. As Stephen said, messages are intentionally not part of the defined API. As a matter of curtesy, we usually restrict message changes to new versions and do not backport the change. An exception may be made if we decide that a message is sufficiently erroneous that is likely misleads people. In any case, message dependent code may be version dependent. -- Terry Jan Reedy From mistersheik at gmail.com Thu May 31 03:55:14 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 31 May 2018 03:55:14 -0400 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> <20180530235334.GK12683@ando.pearwood.info> <20180531010104.GM12683@ando.pearwood.info> Message-ID: Okay, I though about it some more, and I think I'm mistaken about the possibility of adding both rules to the grammar since in that case it is ambiguous whether given binds more tightly to a trailing expression or to the comp_iter. It's too bad. On Wed, May 30, 2018 at 10:50 PM Neil Girdhar wrote: > On Wed, May 30, 2018 at 9:02 PM Steven D'Aprano > wrote: > >> On Thu, May 31, 2018 at 10:05:33AM +1000, Chris Angelico wrote: >> > On Thu, May 31, 2018 at 9:53 AM, Steven D'Aprano >> wrote: >> > >> There is no nice, equivalent := version as far as I can tell. >> > > >> > > Given (pun intended) the fact that you only use transformed_b in a >> > > single place, I don't think it is necessary to use := at all. >> > > >> > > z = {a: transform(b) for b in bs for a in as_} >> > > >> > > But if you really insist: >> > > >> > > # Pointless use of := >> > > z = {a: (transformed_b := transform(b)) for b in bs for a in as_} >> > > >> > >> > That's the subtlety of the 'given' usage here. You fell for the same >> > trap I did: thinking "it's only used once". >> >> But it is only used once. I meant once per loop. >> >> It isn't used in the "for a in as_" inner loop, there's no "if >> transformed_b" condition, and it only is used once in the key:value part >> of the comprehension. >> >> >> > Actually, what he has is equivalent to: >> > >> > z = {a: tb for b in bs for tb in [transform(b)] for a in as_} >> >> Which also uses tb only once, making it a Useless Use Of Assignment. >> >> (I assume we're not calling transform() for some side-effect, like >> logging a message, or erasing your hard drive.) >> >> >> > which means it evaluates transform(b) once regardless of the length of >> > as_. >> >> Ah yes, I see what you mean. Expanded to a loop: >> >> for b in bs: >> tb = transform(b) >> for a in as_: >> z[a] = tb >> >> >> It's a little ugly, but there's a trick I already use today: >> >> py> [x+y for x in "abc" if print(x) or True for y in "de"] >> a >> b >> c >> ['ad', 'ae', 'bd', 'be', 'cd', 'ce'] >> >> So we can adapt that to assignment instead of output: >> >> # Don't do this! >> z = {a: tb for b in bs if (tb := transform(b)) or True for a in as_} >> >> But I wouldn't do that. If I'm concerned about the call to transform >> (because it is super expensive, say) then I set up a pipeline: >> >> tbs = (transform(b) for b in bs) # or map(transform, bs) >> z = {a: tb for tb in tbs for a in as_} >> >> The first generator comprehension can be easily embedded in the other: >> >> z = {a: tb for tb in (transform(b) for b in bs) for a in as_} >> >> This makes it super-obvious that transform is called for each b, not for >> each (b, a) pair, it works today, and there's no assignment expression >> needed at all. >> >> Assignment expressions should not be about adding yet a third way to >> solve a problem that already has a perfectly good solution! ("Expand to >> a loop statement" is not a *perfectly* good solution.) To showcase >> assignment expressions, we should be solving problems that don't have a >> good solution now. >> >> I'm still not convinced that Neil's "given" example will even work (see >> below) but *if he is right* that it does, perhaps that's a good reason >> to prefer the simpler := assignment expression syntax, since we're >> less likely to use it in confusing ways. >> >> >> > But it's really REALLY not obvious. >> >> But is it even legal? >> >> As I understand it, "given" is an expression, not an addition to >> comprehension syntax. In that case, I don't think Neil's example will >> work at all, for reasons I've already stated. >> >> If that's not the case, then until somebody tells me what this new >> comprehension syntax means, and what it looks like, I have no idea what >> is intended. >> >> Which of these can we write, and what do they do? >> > > Great question. The trick is to just write them as a sequence of > statements without changing the order except to put the expression last. > > >> >> [expression given name=something for x in seq] >> > > retval = [] > name = something > for x in seq: > retval.append(expression) > return retval > >> >> [expression for x given name=something in seq] >> > > this one doesn't make sense. > > [expression for x in seq given name=something] >> >> > retval = [] > for x in seq: > name = something > retval.append(expression) > return retval > > >> [expression for x in seq if given name=something condition] >> >> this one doesn't make sense. > > >> [expression for x in seq if condition given name=something] >> >> retval = [] > for x in seq: > if condition: > name = something > retval.append(expression) > return retval > > and of course, the original proposal > > expression given name=something > > means: > > name = something > retval = expression > return retval > > >> >> -- >> Steve >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/python-ideas/keaR3FudcwQ/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.ed.oconnor at gmail.com Thu May 31 04:32:51 2018 From: peter.ed.oconnor at gmail.com (Peter O'Connor) Date: Thu, 31 May 2018 10:32:51 +0200 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> <20180530235334.GK12683@ando.pearwood.info> <20180531010104.GM12683@ando.pearwood.info> Message-ID: On Thu, May 31, 2018 at 4:50 AM, Neil Girdhar wrote: > > >> [expression given name=something for x in seq] >> > > retval = [] > name = something > for x in seq: > retval.append(expression) > return retval > That's a little strange confusing then, because, given the way given is used outside of comprehensions, you would expect for x in range(3): y given y=2*x [y given y=2*x for x in range(3)] to return [0, 2, 4], but it would actually raise an error. -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.ed.oconnor at gmail.com Thu May 31 04:34:00 2018 From: peter.ed.oconnor at gmail.com (Peter O'Connor) Date: Thu, 31 May 2018 10:34:00 +0200 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> <20180530235334.GK12683@ando.pearwood.info> <20180531010104.GM12683@ando.pearwood.info> Message-ID: * Sorry, message sent too early: On Thu, May 31, 2018 at 4:50 AM, Neil Girdhar wrote: > > >> [expression given name=something for x in seq] >> > > retval = [] > name = something > for x in seq: > retval.append(expression) > return retval > That's a little confusing then, because, given the way given is used outside of comprehensions, you would expect [y given y=2*x for x in range(3)] to return [0, 2, 4], but it would actually raise an error. On Thu, May 31, 2018 at 10:32 AM, Peter O'Connor wrote: > > > On Thu, May 31, 2018 at 4:50 AM, Neil Girdhar > wrote: >> >> >>> [expression given name=something for x in seq] >>> >> >> retval = [] >> name = something >> for x in seq: >> retval.append(expression) >> return retval >> > > That's a little strange confusing then, because, given the way given is > used outside of comprehensions, you would expect > > for x in range(3): > y given y=2*x > > [y given y=2*x for x in range(3)] > > to return [0, 2, 4], but it would actually raise an error. > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Thu May 31 04:44:18 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 31 May 2018 04:44:18 -0400 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> <20180530235334.GK12683@ando.pearwood.info> <20180531010104.GM12683@ando.pearwood.info> Message-ID: Yes, you're right. That's the ambiguity I mentioned in my last message. It's too bad because I want given for expressions and given for comprehensions. But if you have both, there's ambiguity and you would at least need parentheses: [(y given y=2*x) for x in range(3)] That might be fine. On Thu, May 31, 2018 at 4:34 AM Peter O'Connor wrote: > * Sorry, message sent too early: > > On Thu, May 31, 2018 at 4:50 AM, Neil Girdhar > wrote: >> >> >>> [expression given name=something for x in seq] >>> >> >> retval = [] >> name = something >> for x in seq: >> retval.append(expression) >> return retval >> > > That's a little confusing then, because, given the way given is used > outside of comprehensions, you would expect > > [y given y=2*x for x in range(3)] > > to return [0, 2, 4], but it would actually raise an error. > > > On Thu, May 31, 2018 at 10:32 AM, Peter O'Connor < > peter.ed.oconnor at gmail.com> wrote: > >> >> >> On Thu, May 31, 2018 at 4:50 AM, Neil Girdhar >> wrote: >>> >>> >>>> [expression given name=something for x in seq] >>>> >>> >>> retval = [] >>> name = something >>> for x in seq: >>> retval.append(expression) >>> return retval >>> >> >> That's a little strange confusing then, because, given the way given is >> used outside of comprehensions, you would expect >> >> for x in range(3): >> y given y=2*x >> >> [y given y=2*x for x in range(3)] >> >> to return [0, 2, 4], but it would actually raise an error. >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Thu May 31 05:29:31 2018 From: stephanh42 at gmail.com (Stephan Houben) Date: Thu, 31 May 2018 11:29:31 +0200 Subject: [Python-ideas] Let try-except check the exception instance In-Reply-To: References: Message-ID: Current documentation says: "An object is compatible with an exception if it is the class or a base class of the exception object or a tuple containing an item compatible with the exception." https://docs.python.org/3/reference/compound_stmts.html#the-try-statement It is, in my opinion, not very clear from this that the __instancecheck__ mechanism is bypassed. Should the documentation perhaps be adapted to explain that the class needs to actually occur in the MRO and that virtual base classes are not considered for matching purposes? "An object is compatible with an exception if it is the class or a non-virtual base class of the exception object or a tuple containing an item compatible with the exception. The exception matching machinery ignores the __instancecheck__ mechanism." Stephan 2018-05-31 9:19 GMT+02:00 Terry Reedy : > On 5/31/2018 12:47 AM, Danilo J. S. Bellini wrote: > >> Hi! >> I was working on handling some exceptions from external software >> (e.g. database constraint triggers) >> switching the handler based on the messages that had been sent. >> Today we can do something like (running on Python 3.6.5): >> >> >> try: >>>>> >>>> ... # [...] >> ... session.commit() # Here it raises! >> ... # [...] >> ... except DatabaseError as exc: >> ... msg = get_db_error_msg_from_exception(exc) >> ... if msg == "beyond_limit": >> ... # [...] >> ... elif msg == "no_funds": >> ... # [...] >> ... else: >> ... raise >> >> >> That works, >> > > Yes, it works perfectly well, AND it exposes the fact that your code > depends on the message, which I think is a good thing. > > As Stephen said, messages are intentionally not part of the defined API. > As a matter of curtesy, we usually restrict message changes to new versions > and do not backport the change. An exception may be made if we decide that > a message is sufficiently erroneous that is likely misleads people. In any > case, message dependent code may be version dependent. > > -- > Terry Jan Reedy > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.ed.oconnor at gmail.com Thu May 31 05:38:57 2018 From: peter.ed.oconnor at gmail.com (Peter O'Connor) Date: Thu, 31 May 2018 11:38:57 +0200 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> <20180530235334.GK12683@ando.pearwood.info> <20180531010104.GM12683@ando.pearwood.info> Message-ID: Well, there need not be any ambiguity if you think of "B given A" as "execute A before B", and remember that "given" has a lower precedence than "for" (So [B given A for x in seq] is parsed as [(B given A) for x in seq] Then > > retval = [expr(name) given name=something(x) for x in seq] > Is: retval = [] for x in seq: name = something(x) retval.append(expr(name)) And retval = [expr(name, x) for x in seq given name=something] Is: retval = [] name = something for x in seq: retval.append(expr(name, x)) But this is probably not a great solution, as it forces you to mentally unwrap comprehensions in a strange order and remember a non-obvious precedence rule. On the plus-side, it lets you initialize generators with in-loop updates (which cannot as far as I see be done nicely with ":="): retval = [expr(name, x) given name=update(name, x) for x in seq given name=something] Is: retval = [] name = something for x in seq: name = update(name, x) retval.append(expr(name, x)) On Thu, May 31, 2018 at 10:44 AM, Neil Girdhar wrote: > Yes, you're right. That's the ambiguity I mentioned in my last message. > It's too bad because I want given for expressions and given for > comprehensions. But if you have both, there's ambiguity and you would at > least need parentheses: > > [(y given y=2*x) for x in range(3)] > > That might be fine. > > On Thu, May 31, 2018 at 4:34 AM Peter O'Connor > wrote: > >> * Sorry, message sent too early: >> >> On Thu, May 31, 2018 at 4:50 AM, Neil Girdhar >> wrote: >>> >>> >>>> [expression given name=something for x in seq] >>>> >>> >>> retval = [] >>> name = something >>> for x in seq: >>> retval.append(expression) >>> return retval >>> >> >> That's a little confusing then, because, given the way given is used >> outside of comprehensions, you would expect >> >> [y given y=2*x for x in range(3)] >> >> to return [0, 2, 4], but it would actually raise an error. >> >> >> On Thu, May 31, 2018 at 10:32 AM, Peter O'Connor < >> peter.ed.oconnor at gmail.com> wrote: >> >>> >>> >>> On Thu, May 31, 2018 at 4:50 AM, Neil Girdhar >>> wrote: >>>> >>>> >>>>> [expression given name=something for x in seq] >>>>> >>>> >>>> retval = [] >>>> name = something >>>> for x in seq: >>>> retval.append(expression) >>>> return retval >>>> >>> >>> That's a little strange confusing then, because, given the way given is >>> used outside of comprehensions, you would expect >>> >>> for x in range(3): >>> y given y=2*x >>> >>> [y given y=2*x for x in range(3)] >>> >>> to return [0, 2, 4], but it would actually raise an error. >>> >>> >>> >>> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Thu May 31 07:55:38 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 31 May 2018 07:55:38 -0400 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> <20180530235334.GK12683@ando.pearwood.info> <20180531010104.GM12683@ando.pearwood.info> Message-ID: On Thu, May 31, 2018 at 5:39 AM Peter O'Connor wrote: > Well, there need not be any ambiguity if you think of "B given A" as > "execute A before B", and remember that "given" has a lower precedence than > "for" (So [B given A for x in seq] is parsed as [(B given A) for x in seq] > > Then > >> >> retval = [expr(name) given name=something(x) for x in seq] >> > > Is: > > retval = [] > for x in seq: > name = something(x) > retval.append(expr(name)) > > And > > retval = [expr(name, x) for x in seq given name=something] > > Is: > retval = [] > name = something > for x in seq: > retval.append(expr(name, x)) > > > But this is probably not a great solution, as it forces you to mentally > unwrap comprehensions in a strange order and remember a non-obvious > precedence rule. > You hit the nail on the head. It forces you to unwarp comprehensions in a strange order. This is why I want the "given" to be interspersed with the "for" and "if" and for everything to be in the order you declare it. > > > On the plus-side, it lets you initialize generators with in-loop updates > (which cannot as far as I see be done nicely with ":="): > > retval = [expr(name, x) given name=update(name, x) for x in seq given > name=something] > > Is: > > retval = [] > name = something > for x in seq: > name = update(name, x) > retval.append(expr(name, x)) > Why wouldn't you want to just put the outer given outside the entire comprehension? retval = [expr(name, x) given name=update(name, x) for x in seq] given name=something The more I think about it, the more i want to keep "given" in comprehensions, and given in expressions using parentheses when given is supposed to bind to the expression first. > > > > On Thu, May 31, 2018 at 10:44 AM, Neil Girdhar > wrote: > >> Yes, you're right. That's the ambiguity I mentioned in my last message. >> It's too bad because I want given for expressions and given for >> comprehensions. But if you have both, there's ambiguity and you would at >> least need parentheses: >> >> [(y given y=2*x) for x in range(3)] >> >> That might be fine. >> >> On Thu, May 31, 2018 at 4:34 AM Peter O'Connor < >> peter.ed.oconnor at gmail.com> wrote: >> >>> * Sorry, message sent too early: >>> >>> On Thu, May 31, 2018 at 4:50 AM, Neil Girdhar >>> wrote: >>>> >>>> >>>>> [expression given name=something for x in seq] >>>>> >>>> >>>> retval = [] >>>> name = something >>>> for x in seq: >>>> retval.append(expression) >>>> return retval >>>> >>> >>> That's a little confusing then, because, given the way given is used >>> outside of comprehensions, you would expect >>> >>> [y given y=2*x for x in range(3)] >>> >>> to return [0, 2, 4], but it would actually raise an error. >>> >>> >>> On Thu, May 31, 2018 at 10:32 AM, Peter O'Connor < >>> peter.ed.oconnor at gmail.com> wrote: >>> >>>> >>>> >>>> On Thu, May 31, 2018 at 4:50 AM, Neil Girdhar >>>> wrote: >>>>> >>>>> >>>>>> [expression given name=something for x in seq] >>>>>> >>>>> >>>>> retval = [] >>>>> name = something >>>>> for x in seq: >>>>> retval.append(expression) >>>>> return retval >>>>> >>>> >>>> That's a little strange confusing then, because, given the way given is >>>> used outside of comprehensions, you would expect >>>> >>>> for x in range(3): >>>> y given y=2*x >>>> >>>> [y given y=2*x for x in range(3)] >>>> >>>> to return [0, 2, 4], but it would actually raise an error. >>>> >>>> >>>> >>>> >>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.ed.oconnor at gmail.com Thu May 31 08:22:21 2018 From: peter.ed.oconnor at gmail.com (Peter O'Connor) Date: Thu, 31 May 2018 14:22:21 +0200 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530122218.GJ12683@ando.pearwood.info> <20180530235334.GK12683@ando.pearwood.info> <20180531010104.GM12683@ando.pearwood.info> Message-ID: On Thu, May 31, 2018 at 1:55 PM, Neil Girdhar wrote: > Why wouldn't you want to just put the outer given outside the entire > comprehension? > retval = [expr(name, x) given name=update(name, x) for x in seq] > given name=something > There seems to be a lot of controversy about updating variables defined outside a comprehension within a comprehension. Seems like it could lead to a lot of bugs and unintended consequences, and that it's safer to not allow side-effects of comprehensions. > The more I think about it, the more i want to keep "given" in > comprehensions, and given in expressions using parentheses when given is > supposed to bind to the expression first. > I think the problem is that you given to be used in two ways: - You want "B given A" to mean "execute A before B" when B is a simple expression - "given A B" to mean "execute A before B" when B is a loop declaration. The more I think about it, the more I think comprehensions should be parsed in reverse order: "from right to left". In this alternate world, your initial example would have been: potential_updates = { (1) y: command.create_potential_update(y) (2) if potential_update is not None (3) given potential_update = command.create_potential_update(y) (4) for y in [x, *x.synthetic_inputs()] (5) for x in need_initialization_nodes } Which would translate to: potential_updates = {} (5) for x in need_initialization_nodes: (4) for y in [x, *x.synthetic_inputs()]: (3) potential_update = command.create_potential_update(y) (2) if potential_update is not None: (1) potential_updates[y] = command.create_potential_ update(y) And there would be no ambiguity about the use of given. Also, variables would tend to be used closer to their declarations. But it's way to late to make a change like that to Python. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu May 31 08:24:21 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 31 May 2018 22:24:21 +1000 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530235334.GK12683@ando.pearwood.info> <20180531010104.GM12683@ando.pearwood.info> Message-ID: <20180531122420.GO12683@ando.pearwood.info> On Thu, May 31, 2018 at 04:44:18AM -0400, Neil Girdhar wrote: > Yes, you're right. That's the ambiguity I mentioned in my last message. > It's too bad because I want given for expressions and given for > comprehensions. Why? So far you haven't given (heh, pun intended) any examples of something you can do better with "given for comprehensions" which isn't either already doable or will be doable with assignment expressions (regardless of spelling). Earlier, I wrote: "To showcase assignment expressions, we should be solving problems that don't have a good solution now." (I exclude "re-write your code as a for-loop statement" -- I consider that a last resort, not the best solution.) Now I realise that good solutions are in the eye of the beholder, but I think we (mostly) agree that: [process(x, 2*x, x**3) for obj in seq for x in [func(obj)]] is a hacky solution for assignments in an expression. It works, but it hardly speaks to the programmers intention. Whichever syntax we use, an explicit assignment expression is better: # verbose, Repeat Yourself syntax [process(x given x = func(obj), 2*x, x**3) for obj in seq] # concise, Don't Repeat Yourself syntax [process(x := func(obj), 2*x, x**3) for obj in seq] (I don't apologise for the editorial comments.) Do you have an equally compelling example for your given-comprehension syntax? I didn't think your example was obviously better than what we can already do: # calculate tx only once per x loop [process(tx, y) for x in xs given tx = transform(x) for y in ys] # existing solution [process(tx, y) for tx in (transform(x) for x in xs) for y in yz] Regardless of whether it is spelled := or given, I don't think that this example is a compelling use-case for assignment expressions. I think there are much better use-cases. (E.g. avoiding cascades of nested if statements.) -- Steve From steve at pearwood.info Thu May 31 08:53:15 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 31 May 2018 22:53:15 +1000 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530235334.GK12683@ando.pearwood.info> <20180531010104.GM12683@ando.pearwood.info> Message-ID: <20180531125315.GP12683@ando.pearwood.info> On Thu, May 31, 2018 at 02:22:21PM +0200, Peter O'Connor wrote: > There seems to be a lot of controversy about updating variables defined > outside a comprehension within a comprehension. Seems like it could lead > to a lot of bugs and unintended consequences, and that it's safer to not > allow side-effects of comprehensions. Gosh, you mean that a feature intended to have side-effects might have side-effects? Who would have predicted that! *wink* I have to keep pointing this out, because critics keep refusing to acknowledge this point: one of the major motivating use-cases of assignment expressions specifically relies on the ability to update a local variable from inside a comprehension. If not for that use-case, we probably wouldn't be having this discussion at all. I think it ought to take more than "controversy" to eliminate that use-case from consideration. It ought to take a good demonstration that this is a real, not just theoretical, problem, that it *encourages* bugs not just allows them. *Any* use of variables can "lead to a lot of bugs and unintended consequences" (just ask functional programmers). Yes, it can, but it usually doesn't. Bottom line is, if you think it is okay that the following assignment to x affects the local scope: results = [] for a in seq: # using "given" to avoid arguments about := y = (x given x = a)+1 results.append(y) assert "x" in locals() but then worry that changing the loop to a comprehension: results = [(x given x = a)+1 for a in seq] assert "x" in locals() will be a problem, then I think you are applying an unreasonably strict standard of functional purity towards comprehensions, one which is not justified by Python's consenting adults approach to side-effects or the fact that comprehensions can already have side-effects. (Sorry Nick!) [Aside: yes, I realise the assertions will fail if seq is empty. It's just an illustration, not production code.] -- Steve From rosuav at gmail.com Thu May 31 08:55:46 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 31 May 2018 22:55:46 +1000 Subject: [Python-ideas] A real life example of "given" In-Reply-To: <20180531122420.GO12683@ando.pearwood.info> References: <20180530235334.GK12683@ando.pearwood.info> <20180531010104.GM12683@ando.pearwood.info> <20180531122420.GO12683@ando.pearwood.info> Message-ID: On Thu, May 31, 2018 at 10:24 PM, Steven D'Aprano wrote: > Do you have an equally compelling example for your given-comprehension > syntax? I didn't think your example was obviously better than what we > can already do: > > # calculate tx only once per x loop > [process(tx, y) for x in xs given tx = transform(x) for y in ys] > > # existing solution > [process(tx, y) for tx in (transform(x) for x in xs) for y in yz] # alternate existing solution [process(tx, y) for x in xs for tx in [transform(x)] for y in yz] This syntax allows you to use both x and tx in the resultant expression. For instance: [process(value, row_total) for row in dataset for row_total in [sum(row)] for value in row] ret = [] for row in dataset: row_total = sum(row) for value in row: ret.append(process(value, row_total)) If done without optimization, each row would take O(n?) time, but this way it's O(n). I think Serhiy was trying to establish this form as a standard idiom, with optimization in the interpreter to avoid constructing a list and iterating over it (so it would be functionally identical to actual assignment). I'd rather see that happen than the creation of a messy 'given' syntax. ChrisA From peter.ed.oconnor at gmail.com Thu May 31 09:23:13 2018 From: peter.ed.oconnor at gmail.com (Peter O'Connor) Date: Thu, 31 May 2018 15:23:13 +0200 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530235334.GK12683@ando.pearwood.info> <20180531010104.GM12683@ando.pearwood.info> <20180531122420.GO12683@ando.pearwood.info> Message-ID: On Thu, May 31, 2018 at 2:55 PM, Chris Angelico wrote: > [process(tx, y) for x in xs for tx in [transform(x)] for y in yz] > ... I think Serhiy was trying to establish this form as a standard idiom, > with optimization in the interpreter to avoid constructing a list and > iterating over it (so it would be functionally identical to actual > assignment). I'd rather see that happen than the creation of a messy > 'given' syntax. > Perhaps it wouldn't be crazy to have "with name=initial" be that idiom instead of "for name in [initial]". As .. [process(tx, y) for x in xs with tx=transform(x) for y in yz] .. seems to convey the intention more clearly. More generally (outside of just comprehensions), "with name=expr:" could be used to temporarily bind "name" to "expr" inside the scope of the with-statement (and unbind it at the end). And then I could have my precious initialized generators (which I believe cannot be nicely implemented with ":=" unless we initialize the variable outside of the scope of the comprehension, which introduces the problem of unintended side-effects). smooth_signal = [average with average=0 for x in seq with average=(1-decay)*average + decay*x] -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu May 31 09:30:31 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 31 May 2018 23:30:31 +1000 Subject: [Python-ideas] A real life example of "given" In-Reply-To: References: <20180530235334.GK12683@ando.pearwood.info> <20180531010104.GM12683@ando.pearwood.info> <20180531122420.GO12683@ando.pearwood.info> Message-ID: On Thu, May 31, 2018 at 11:23 PM, Peter O'Connor wrote: > On Thu, May 31, 2018 at 2:55 PM, Chris Angelico wrote: >> >> [process(tx, y) for x in xs for tx in [transform(x)] for y in yz] >> >> ... >> >> I think Serhiy was trying to establish this form as a standard idiom, >> with optimization in the interpreter to avoid constructing a list and >> iterating over it (so it would be functionally identical to actual >> assignment). I'd rather see that happen than the creation of a messy >> 'given' syntax. > > > Perhaps it wouldn't be crazy to have "with name=initial" be that idiom > instead of "for name in [initial]". As .. > > [process(tx, y) for x in xs with tx=transform(x) for y in yz] > > .. seems to convey the intention more clearly. More generally (outside of > just comprehensions), "with name=expr:" could be used to temporarily bind > "name" to "expr" inside the scope of the with-statement (and unbind it at > the end). Except that 'with' means context managers, not just assignment. Also, it's not backward-compatible; if the "for var in [val]" syntax becomes an accepted idiom, it'll be valid in all versions of Python back to, what, 2.4? and just won't be optimized in older versions. Making a new syntax misses out on that benefit, so it needs to be a really good syntax, and 'with' isn't. > And then I could have my precious initialized generators (which I believe > cannot be nicely implemented with ":=" unless we initialize the variable > outside of the scope of the comprehension, which introduces the problem of > unintended side-effects). > > smooth_signal = [average with average=0 for x in seq with > average=(1-decay)*average + decay*x] I want to read this as "average with average=0" blah blah, which doesn't make a lot of sense. It'd be far FAR better to mess with things so that external assignment works. ChrisA From danilo.bellini at gmail.com Thu May 31 09:35:05 2018 From: danilo.bellini at gmail.com (Danilo J. S. Bellini) Date: Thu, 31 May 2018 10:35:05 -0300 Subject: [Python-ideas] Let try-except check the exception instance In-Reply-To: <20180531054148.GN12683@ando.pearwood.info> References: <20180531054148.GN12683@ando.pearwood.info> Message-ID: On 31 May 2018 at 02:41, Steven D'Aprano wrote: > Since error messages are rarely part of the exception API, testing for > them is fragile and prone to errors. For example, what if the message > changes to "no money" or "out of funds" or "insufficient funds" or > "keine Mittel"? Since the error message is not part of the API, that can > happen at any time, without warning. > The messages are from constraint triggers in the database. I'm the author of them. > The obvious way to do that is to create BeyondLimit and NoFunds > subclasses of DatabaseError. Why can't you do that? > Because I'm not the author of DatabaseError. It's an ORM exception or a database driver class exception, something I'd need to monkeypatch to change. That seems like a backwards way to do it to me. If you're the author of > the exception class, why not just subclass your exceptions the regular > way and get a real subclass? > I agree. Problem is: I'm not the author of the exception class, and I'm not raising/throwing their exception instances. On 31 May 2018 at 04:19, Terry Reedy wrote: > As Stephen said, messages are intentionally not part of the defined API. > [...] > We could check error numbers instead of meaningful strings. I'm thinking on technical error keys in either way. Previously to classes like FileNotFoundError, we would need to check an OSError errno. There were no alternative: Python users are not the author of "everywhere that raises OSError". Another example is urllib.error.HTTPError. We can have a specific handler code for a given HTTP error code. Yet we don't change urllib.request.urlopen to raise something else for that specific error code. On 31 May 2018 at 06:29, Stephan Houben wrote: > "[...] The exception matching machinery ignores the __instancecheck__ > mechanism." > The __subclasscheck__ gets bypassed, as well. -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap) -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenlhilton at gmail.com Thu May 31 10:26:42 2018 From: kenlhilton at gmail.com (Ken Hilton) Date: Thu, 31 May 2018 22:26:42 +0800 Subject: [Python-ideas] Let try-except check the exception instance Message-ID: In one of my own packages, I use the following to catch dynamic error messages: def handle_error(exc): # do stuff with exc with catch('error_code', handle_error): # do stuff that can raise an error The definition of "catch" is as follows, assuming the root exception actually raised is called "CodedError" whose "code" attribute is the error code: from contextlib import contextmanager @contextmanager def catch(code=None, caught=None, always=None): try: yield except CodedError as exc: if isinstance(code, str): is_caught = exc.code == code else: is_caught = exc.code in code #assume it's a container if (code is not None) and not is_caught: raise if caught is not None: caught(exc) finally: if always is not None: always() Perhaps you could make use of this? Suggesting, Ken Hilton; -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu May 31 10:36:43 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 1 Jun 2018 00:36:43 +1000 Subject: [Python-ideas] Let try-except check the exception instance In-Reply-To: References: Message-ID: On 31 May 2018 at 14:47, Danilo J. S. Bellini wrote: > The idea is to allow catching exceptions beyond checking their MRO, > using a class that checks the exception instance by implementing > a custom __instancecheck__. > The exception machinery deliberately attempts to avoid instantiating exception objects whenever it can, but that gets significantly more difficult if we always need to create the instance before we can decide whether or not the raised exception matches the given exception handler criteria. So quite aside from any philosophy-of-design questions, we're unlikely to ever implement this simply for laziness-of-evaluation reasons. (There are already lots of circumstances that force instantiation - that doesn't mean we're keen to add more) Cheers, Nick. P.S. There's a somewhat related issue aimed at getting ABCs to work correctly as exception handlers, although that still sticks to the principle that exception handler checks solely consider the exception type, not its value: https://bugs.python.org/issue12029 -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu May 31 10:49:58 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 31 May 2018 07:49:58 -0700 Subject: [Python-ideas] exception instantiation philosophy and practice [was: Let try-except check the exception instance] In-Reply-To: References: Message-ID: <5B100B96.6050900@stoneleaf.us> On 05/31/2018 07:36 AM, Nick Coghlan wrote: > The exception machinery deliberately attempts to avoid instantiating exception objects whenever it can, but that gets > significantly more difficult if we always need to create the instance before we can decide whether or not the raised > exception matches the given exception handler criteria. Why is this? Doesn't the exception have to be instantiated at some point, even if just to print to stderr? -- ~Ethan~ From solipsis at pitrou.net Thu May 31 10:52:00 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 31 May 2018 16:52:00 +0200 Subject: [Python-ideas] exception instantiation philosophy and practice [was: Let try-except check the exception instance] References: <5B100B96.6050900@stoneleaf.us> Message-ID: <20180531165200.7a385ed4@fsol> On Thu, 31 May 2018 07:49:58 -0700 Ethan Furman wrote: > On 05/31/2018 07:36 AM, Nick Coghlan wrote: > > > The exception machinery deliberately attempts to avoid instantiating exception objects whenever it can, but that gets > > significantly more difficult if we always need to create the instance before we can decide whether or not the raised > > exception matches the given exception handler criteria. > > Why is this? Doesn't the exception have to be instantiated at some point, even if just to print to stderr? Nick is talking about exceptions that are ultimately silenced, especially when caught from C code. You're right that, once caught from Python code, the exception *has* to be instantiated (since it is bound to a user-visible variable). Regards Antoine. From guido at python.org Thu May 31 10:53:43 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 31 May 2018 07:53:43 -0700 Subject: [Python-ideas] exception instantiation philosophy and practice [was: Let try-except check the exception instance] In-Reply-To: <5B100B96.6050900@stoneleaf.us> References: <5B100B96.6050900@stoneleaf.us> Message-ID: On Thu, May 31, 2018 at 7:49 AM, Ethan Furman wrote: > On 05/31/2018 07:36 AM, Nick Coghlan wrote: > > The exception machinery deliberately attempts to avoid instantiating >> exception objects whenever it can, but that gets >> significantly more difficult if we always need to create the instance >> before we can decide whether or not the raised >> exception matches the given exception handler criteria. >> > > Why is this? Doesn't the exception have to be instantiated at some point, > even if just to print to stderr? > Not if it gets caught and ignored. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Thu May 31 11:57:07 2018 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 31 May 2018 09:57:07 -0600 Subject: [Python-ideas] Let try-except check the exception instance In-Reply-To: References: Message-ID: On Wed, May 30, 2018 at 10:47 PM, Danilo J. S. Bellini wrote: >>>> try: > ... # [...] > ... session.commit() # Here it raises! > ... # [...] > ... except DatabaseError as exc: > ... msg = get_db_error_msg_from_exception(exc) > ... if msg == "beyond_limit": > ... # [...] > ... elif msg == "no_funds": > ... # [...] > ... else: > ... raise Matching on error messages is a recipe for pain. My experience with Go seared that lesson into my brain. (The boilerplate involved there to make custom error types leads to a lot of error message checking instead.) > That works, but I'd like to do something like: > >>>> try: > ... # [...] > ... except BeyondLimit: > ... # [...] > ... except NoFunds: > ... # [...] Using subclasses like this is the way to go. If you control the originating code then make it happen. :) If not then work to fix it upstream. If that fails then at least use a helper that converts the exception after the fact. It could look like following: # Subclassing DatabaseError means existing try-except block will still work the same. class CustomDatabaseError(DatabaseError): .... class BeyondLimitError(CustomDatabaseError): ... class NoFundsError(CustomDatabaseError): ... def convert_exception(exc): if isinstance(exc, CustomDatabaseError): raise msg = get_db_error_msg_from_exception(exc) if msg == "beyond_limit": raise BeyondLimitError(...) elif msg == "no_funds": raise NoFundsError(...) else: raise @contextmanager def converted_exceptions(): try: yield except CustomDatabaseError: raise except DatabaseError as exc: convert_exception(exc) def db_op(...): with converted_exceptions(): # Some code that does DB ops, e.g. session.commit(). ... try: db_op(...) except BeyondLimitError: ... except NoFundsError: ... except DatabaseError: # fallback ... The big wins there are with re-usability, maintainability, and separation of concerns. Notably, it consolidates your message-handling code to one spot and keeps it focused. -eric From alexander.belopolsky at gmail.com Thu May 31 14:00:24 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 31 May 2018 14:00:24 -0400 Subject: [Python-ideas] Let try-except check the exception instance In-Reply-To: References: Message-ID: On Thu, May 31, 2018 at 10:37 AM Nick Coghlan wrote: > > The exception machinery deliberately attempts to avoid instantiating exception objects whenever it can, but that gets significantly more difficult if we always need to create the instance before we can decide whether or not the raised exception matches the given exception handler criteria. Is this really true? Consider the following simple code class E(Exception): def __init__(self): print("instantiated") try: raise E except E: pass Is it truly necessary to instantiate E() in this case? Yet when I run it, I see "instantiated" printed on the console. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu May 31 14:03:57 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 1 Jun 2018 04:03:57 +1000 Subject: [Python-ideas] exception instantiation philosophy and practice [was: Let try-except check the exception instance] In-Reply-To: <20180531165200.7a385ed4@fsol> References: <5B100B96.6050900@stoneleaf.us> <20180531165200.7a385ed4@fsol> Message-ID: On Fri, Jun 1, 2018 at 12:52 AM, Antoine Pitrou wrote: > On Thu, 31 May 2018 07:49:58 -0700 > Ethan Furman wrote: >> On 05/31/2018 07:36 AM, Nick Coghlan wrote: >> >> > The exception machinery deliberately attempts to avoid instantiating exception objects whenever it can, but that gets >> > significantly more difficult if we always need to create the instance before we can decide whether or not the raised >> > exception matches the given exception handler criteria. >> >> Why is this? Doesn't the exception have to be instantiated at some point, even if just to print to stderr? > > Nick is talking about exceptions that are ultimately silenced, > especially when caught from C code. You're right that, once caught > from Python code, the exception *has* to be instantiated (since it is > bound to a user-visible variable). > Big and common example: Loop termination by raising StopIteration. ChrisA From brenbarn at brenbarn.net Thu May 31 14:03:56 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Thu, 31 May 2018 11:03:56 -0700 Subject: [Python-ideas] A real life example of "given" In-Reply-To: <20180531125315.GP12683@ando.pearwood.info> References: <20180530235334.GK12683@ando.pearwood.info> <20180531010104.GM12683@ando.pearwood.info> <20180531125315.GP12683@ando.pearwood.info> Message-ID: <5B10390C.7040802@brenbarn.net> On 2018-05-31 05:53, Steven D'Aprano wrote: > Bottom line is, if you think it is okay that the following assignment to > x affects the local scope: > > results = [] > for a in seq: > # using "given" to avoid arguments about := > y = (x given x = a)+1 > results.append(y) > assert "x" in locals() > > but then worry that changing the loop to a comprehension: > > results = [(x given x = a)+1 for a in seq] > assert "x" in locals() > > will be a problem, then I think you are applying an unreasonably strict > standard of functional purity towards comprehensions, one which is not > justified by Python's consenting adults approach to side-effects or the > fact that comprehensions can already have side-effects. What I don't understand is this: if we believe that, then why was comprehension-leaking EVER removed? Everything that I've seen advocating for this kind of leaking seems to me like it is much more logically consistent with allowing all comprehension variables to leak than it is with the current behavior, in which they don't leak. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From steve at pearwood.info Thu May 31 20:42:01 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 1 Jun 2018 10:42:01 +1000 Subject: [Python-ideas] A real life example of "given" In-Reply-To: <5B10390C.7040802@brenbarn.net> References: <20180531010104.GM12683@ando.pearwood.info> <20180531125315.GP12683@ando.pearwood.info> <5B10390C.7040802@brenbarn.net> Message-ID: <20180601004200.GQ12683@ando.pearwood.info> On Thu, May 31, 2018 at 11:03:56AM -0700, Brendan Barnwell wrote: > What I don't understand is this: if we believe that, then why was > comprehension-leaking EVER removed? Everything that I've seen > advocating for this kind of leaking seems to me like it is much more > logically consistent with allowing all comprehension variables to leak > than it is with the current behavior, in which they don't leak. Originally list comprehensions ran in the same scope as their surroundings. In Python 2: py> [1 for x in ("spam", "eggs")] [1, 1] py> x 'eggs' but when generator expressions were introduced a few years later, they ran in their own sub-local scope. This was, I believe, initially introduced to simplify the common situation that you return a generator from inside a function: def factory(): x = "something big" return (x for x in seq) would require holding onto a closure of the factory locals, including the "something big" object. Potentially forever, if the generator expression is never iterated over. (I hope someone will correct me if I have misunderstood.) To avoid that, the x in the generator was put into a distinct scope from the x in factory. Either way, the PEP introducing generator expressions makes it clear that it was a deliberate decision: The loop variable (if it is a simple variable or a tuple of simple variables) is not exposed to the surrounding function. This facilitates the implementation and makes typical use cases more reliable. https://www.python.org/dev/peps/pep-0289/ In the case of loop variables, there is an argument from "practicality beats purity" that they ought to run in their own scope. Loop variables tend to be short, generic names like "i", "x", "obj", all the more likely to clash with names in the surrounding scope, and hard to spot when they do. They're also likely to be used inside loops: for x in [1, 2, 3, 4]: alist = [expr for x in range(50)] # oops, x has been accidentally overridden (in Python 2) I don't *entirely* buy that argument, and I occasionally find it useful to inspect the loop variable of a list comprehension after it has run, but this is one windmill I'm not going to tilt against. Reversing the decision to put the loop variables in their own sublocal scope is *not* part of this PEP. But this is less likely to be a problem for explicit assignment. Outside of toy examples, we're more likely to assign to descriptive names and less likely to clash with any surrounding loop variable: for book in library: text = [content for chapter in books.chapters() if (content := chapter.get_text(all=True) and re.match(pattern, content)] Given an obvious and explicit assignment to "chapter", say, we are more likely to realise when we are reusing a name (compared to assigning to a loop variable i, say). At least, we should be no more likely to mess this up than we are for any other local-level assignment. -- Steve