From robert.kern at gmail.com Thu Sep 1 00:19:06 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 31 Aug 2011 17:19:06 -0500 Subject: [Python-ideas] Expanding statistical functions in Python's std. lib. In-Reply-To: References: <549901314286114@web119.yandex.ru> <4E56E859.3090504@canterbury.ac.nz> <4E5D05EE.2050502@gmail.com> Message-ID: On 8/31/11 2:05 PM, Guido van Rossum wrote: > You didn't get any responses AFAICT. That doesn't mean nobody is > interested -- perhaps your proposal is simply too general? Do you feel > up to making some more specific recommendations about the exact list > of functions to add? It's easier to criticize a concrete proposal. Do > you feel up to producing a patch that just adds the incomplete beta > function? It shows up deeply mis-threaded under "Create a StringBuilder class and use it everywhere" in my client. Perhaps Spectral One should try reposting it so that it shows up as a new thread. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From guido at python.org Thu Sep 1 00:23:12 2011 From: guido at python.org (Guido van Rossum) Date: Wed, 31 Aug 2011 15:23:12 -0700 Subject: [Python-ideas] Expanding statistical functions in Python's std. lib. In-Reply-To: References: <549901314286114@web119.yandex.ru> <4E56E859.3090504@canterbury.ac.nz> <4E5D05EE.2050502@gmail.com> Message-ID: On Wed, Aug 31, 2011 at 3:19 PM, Robert Kern wrote: > On 8/31/11 2:05 PM, Guido van Rossum wrote: >> >> You didn't get any responses AFAICT. That doesn't mean nobody is >> interested -- perhaps your proposal is simply too general? Do you feel >> up to making some more specific recommendations about the exact list >> of functions to add? It's easier to criticize a concrete proposal. Do >> you feel up to producing a patch that just adds the incomplete beta >> function? > > It shows up deeply mis-threaded under "Create a StringBuilder class and use > it everywhere" in my client. Perhaps Spectral One should try reposting it so > that it shows up as a new thread. What client is that? In my GMail (for once) it shows up as a new thread with subject "Re: [Python-ideas] Expanding statistical functions in Python's std. lib." I guess your client got confused by some of these headers: References: <549901314286114 at web119.yandex.ru> <4E56E859.3090504 at canterbury.ac.nz> In-Reply-To: <4E56E859.3090504 at canterbury.ac.nz> -- --Guido van Rossum (python.org/~guido) From greg.ewing at canterbury.ac.nz Thu Sep 1 00:52:11 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Sep 2011 10:52:11 +1200 Subject: [Python-ideas] aliasing In-Reply-To: <4E5E2C0F.7080603@pearwood.info> References: <4E5E2C0F.7080603@pearwood.info> Message-ID: <4E5EBB1B.1060802@canterbury.ac.nz> Steven D'Aprano wrote: > Peio Borthelle wrote: > >>>>> a = 2 >>>>> b = alias("a") >>>>> a = 'foo' >>>>> b >> >> 'foo' > > I have often thought that would be a nice to have feature, Seems to me it would be a confusing-to-have feature. By now it's deeply embedded in Python programmers' brains that assignment to a bare name can only change which object that name refers to, and can't have any other side effects. > You can always use one level of indirection: > > a = [2] > b = a > a[0] = 'foo' # Not a='foo' > b[0] > => print 'foo' Also, if you're willing to use an object attribute rather than a bare name, you can get the same effect using a property. class Switcheroo(object): def get_b(self): return self.a def set_b(self, value): self.a = value b = property(get_b, set_b) s = Switcheroo() s.a = 17 print s.a s.b = 42 print s.a -- Greg From greg.ewing at canterbury.ac.nz Thu Sep 1 01:04:30 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 01 Sep 2011 11:04:30 +1200 Subject: [Python-ideas] Add from __experimental__ import bla [was: Should we move to replace re with regex?] In-Reply-To: References: <1314569103.4640.19.camel@Gutsy> <58780826-87be-436f-b777-937e41691e90@m35g2000prl.googlegroups.com> <871uw2l0vs.fsf@uwakimon.sk.tsukuba.ac.jp> <20110831111946.22bac764@resist.wooz.org> <20110831185635.744c9624@pitrou.net> <1314811985.3505.0.camel@localhost.localdomain> Message-ID: <4E5EBDFE.3050504@canterbury.ac.nz> Guido van Rossum wrote: > 2 -> 3: galactic release Apocalyptic release? -- Greg From robert.kern at gmail.com Thu Sep 1 01:04:36 2011 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 31 Aug 2011 18:04:36 -0500 Subject: [Python-ideas] Expanding statistical functions in Python's std. lib. In-Reply-To: References: <549901314286114@web119.yandex.ru> <4E56E859.3090504@canterbury.ac.nz> <4E5D05EE.2050502@gmail.com> Message-ID: On 8/31/11 5:23 PM, Guido van Rossum wrote: > On Wed, Aug 31, 2011 at 3:19 PM, Robert Kern wrote: >> On 8/31/11 2:05 PM, Guido van Rossum wrote: >>> >>> You didn't get any responses AFAICT. That doesn't mean nobody is >>> interested -- perhaps your proposal is simply too general? Do you feel >>> up to making some more specific recommendations about the exact list >>> of functions to add? It's easier to criticize a concrete proposal. Do >>> you feel up to producing a patch that just adds the incomplete beta >>> function? >> >> It shows up deeply mis-threaded under "Create a StringBuilder class and use >> it everywhere" in my client. Perhaps Spectral One should try reposting it so >> that it shows up as a new thread. > > What client is that? Thunderbird, via GMane, which may or may not be adding more confusion to the mix. > In my GMail (for once) it shows up as a new > thread with subject "Re: [Python-ideas] Expanding statistical > functions in Python's std. lib." I guess your client got confused by > some of these headers: > > References: <549901314286114 at web119.yandex.ru> > <4E56E859.3090504 at canterbury.ac.nz> > In-Reply-To: <4E56E859.3090504 at canterbury.ac.nz> And indeed, it shows up threaded under Greg Ewing's Aug 25 post to the StringBuilder thread. Email threading is something of an art, but I'm not sure it's right to say that my client is getting "confused" by taking the In-Reply-To header at its word. ;-) Anyways, that's why I suspect he's not getting many responses. As to the substance of the proposal, I'm -0 on having the full complement of statistical distribution functions and +0 on adding just the incomplete beta function. Personally, I will never use any of them since I can get them from scipy. I am at least going to be using numpy to generate any of the test statistics that I would pass through these functions. I don't see anything particularly compelling about having them in the math module as opposed to a third party module (be it scipy or something lighter-weight). That said, having a good complement of common special functions that can be used to build up a variety of less-common functions is a good thing to have in a standard library. I think you could defend adding the incomplete beta function on that principle, if nothing else. You could make a similar argument for the Bessel functions j0(), j1() and jn(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ncoghlan at gmail.com Thu Sep 1 01:14:49 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 1 Sep 2011 09:14:49 +1000 Subject: [Python-ideas] Add from __experimental__ import bla [was: Should we move to replace re with regex?] In-Reply-To: <20110831110536.111822cf@resist.wooz.org> References: <1314569103.4640.19.camel@Gutsy> <58780826-87be-436f-b777-937e41691e90@m35g2000prl.googlegroups.com> <20110831110536.111822cf@resist.wooz.org> Message-ID: On Thu, Sep 1, 2011 at 1:05 AM, Barry Warsaw wrote: > In the face of PEP 402, how could you enforce that? ?Even if you can't or > don't want to enforce it, how would a user be able to verify that it was the > case for something in experimental? PEP 402 doesn't really change anything regarding corporate checks of code provenance - those relate to how the code is acquired and installed in the first place, not how you use it at runtime. If engineers are doing end runs around corporate policies, that's not a problem that technology can realistically address :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Thu Sep 1 01:31:41 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 1 Sep 2011 09:31:41 +1000 Subject: [Python-ideas] Add from __experimental__ import bla [was: Should we move to replace re with regex?] In-Reply-To: References: <1314569103.4640.19.camel@Gutsy> <58780826-87be-436f-b777-937e41691e90@m35g2000prl.googlegroups.com> <871uw2l0vs.fsf@uwakimon.sk.tsukuba.ac.jp> <20110831111946.22bac764@resist.wooz.org> <20110831185635.744c9624@pitrou.net> Message-ID: On Thu, Sep 1, 2011 at 3:36 AM, Mike Meyer wrote: > On Wed, Aug 31, 2011 at 10:23 AM, Guido van Rossum wrote: >> >> (*) Can we pick a terminology so we all agree that "3.3.3" is a "minor >> release", "3.3" is a "major release", and "3" an "earthshattering >> release"? Or other terms -- but something that is both agreed upon and >> clear enough without explanation. I'm tired of having to clarify minor >> and major every time I use them out of fear they'll be mistaken for >> "3" and "3.3". > > Given that there's no general industry agreement on what those things mean, > "clear enough without explanation" is probably unrealistic. On the other > hand, a PEP or some similar document that lays out the terminology used for > python releases (and encouraged for python libraries) would let you assume > that python people had read that explanation, and make a reference to it > easy (i.e."see PEP XXXX"). That PEP is PEP 101 (at least as far as Python releases go), and it is currently a little inconsistent (using major.minor.micro in some sections, as the devguide and sys.version_info apparently also do, but also referring to feature releases as major releases on at least one occasion). PEP 386 (which aims to standardise version numbering within the wider Python ecosystem) doesn't mandate a naming scheme (only a numbering one), but does refer to the major.minor.micro names in the explanatory text. Personally, I agree with others that "feature release" and "bug fix release" are as close as we're likely to get to unambiguous names for the way Python itself uses x.y and x.y.z release numbering. Creating a new version of the language itself is rare enough that I don't think it really needs a generic name. Although with the accumulated baggage that 3.0 swept away, "spring cleaning release" might be appropriate :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From aquavitae69 at gmail.com Thu Sep 1 06:39:03 2011 From: aquavitae69 at gmail.com (David Townshend) Date: Thu, 1 Sep 2011 06:39:03 +0200 Subject: [Python-ideas] Expanding statistical functions in Python's std. lib. In-Reply-To: References: <549901314286114@web119.yandex.ru> <4E56E859.3090504@canterbury.ac.nz> <4E5D05EE.2050502@gmail.com> Message-ID: This had probably been thought about before, but why not include numpy in stdlib? On Sep 1, 2011 1:05 AM, "Robert Kern" wrote: > On 8/31/11 5:23 PM, Guido van Rossum wrote: >> On Wed, Aug 31, 2011 at 3:19 PM, Robert Kern wrote: >>> On 8/31/11 2:05 PM, Guido van Rossum wrote: >>>> >>>> You didn't get any responses AFAICT. That doesn't mean nobody is >>>> interested -- perhaps your proposal is simply too general? Do you feel >>>> up to making some more specific recommendations about the exact list >>>> of functions to add? It's easier to criticize a concrete proposal. Do >>>> you feel up to producing a patch that just adds the incomplete beta >>>> function? >>> >>> It shows up deeply mis-threaded under "Create a StringBuilder class and use >>> it everywhere" in my client. Perhaps Spectral One should try reposting it so >>> that it shows up as a new thread. >> >> What client is that? > > Thunderbird, via GMane, which may or may not be adding more confusion to the mix. > >> In my GMail (for once) it shows up as a new >> thread with subject "Re: [Python-ideas] Expanding statistical >> functions in Python's std. lib." I guess your client got confused by >> some of these headers: >> >> References: <549901314286114 at web119.yandex.ru> >> <4E56E859.3090504 at canterbury.ac.nz> >> In-Reply-To: <4E56E859.3090504 at canterbury.ac.nz> > > And indeed, it shows up threaded under Greg Ewing's Aug 25 post to the > StringBuilder thread. Email threading is something of an art, but I'm not sure > it's right to say that my client is getting "confused" by taking the In-Reply-To > header at its word. ;-) > > Anyways, that's why I suspect he's not getting many responses. As to the > substance of the proposal, I'm -0 on having the full complement of statistical > distribution functions and +0 on adding just the incomplete beta function. > Personally, I will never use any of them since I can get them from scipy. I am > at least going to be using numpy to generate any of the test statistics that I > would pass through these functions. I don't see anything particularly compelling > about having them in the math module as opposed to a third party module (be it > scipy or something lighter-weight). > > That said, having a good complement of common special functions that can be used > to build up a variety of less-common functions is a good thing to have in a > standard library. I think you could defend adding the incomplete beta function > on that principle, if nothing else. You could make a similar argument for the > Bessel functions j0(), j1() and jn(). > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it had > an underlying truth." > -- Umberto Eco > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Sep 1 07:32:57 2011 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 01 Sep 2011 00:32:57 -0500 Subject: [Python-ideas] Expanding statistical functions in Python's std. lib. In-Reply-To: References: <549901314286114@web119.yandex.ru> <4E56E859.3090504@canterbury.ac.nz> <4E5D05EE.2050502@gmail.com> Message-ID: On 8/31/11 11:39 PM, David Townshend wrote: > This had probably been thought about before, but why not include numpy in stdlib? Well, it's not particularly germane to this thread since most of the requested functions exist in scipy but not numpy. In any case, numpy is too large, too C, and too actively developed to be a part of the standard library at this time. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From stephen at xemacs.org Thu Sep 1 10:43:47 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 01 Sep 2011 17:43:47 +0900 Subject: [Python-ideas] Terminology for "earthshattering" [was: Add from __experimental__ import bla] In-Reply-To: References: <1314569103.4640.19.camel@Gutsy> <58780826-87be-436f-b777-937e41691e90@m35g2000prl.googlegroups.com> <871uw2l0vs.fsf@uwakimon.sk.tsukuba.ac.jp> <20110831111946.22bac764@resist.wooz.org> <20110831185635.744c9624@pitrou.net> Message-ID: <87mxeok5z0.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > (*) Can we pick a terminology so we all agree that "3.3.3" is a "minor > release", "3.3" is a "major release", and "3" an "earthshattering > release"? I'm not sure "earthshattering" is Pythonic enough. If you're willing to go with something pedestrian, at least the Debian types, and I think Gentoo, use "epoch" to mean that upstream decided to redo its version numbering. In a sense, I think that applies to Python 3 (which has to be judged a success, since it induced Linus to go to Linux 3!) and Linux 3 for that matter. How about the leading digit being the "Pythonic epoch", and "Python 3 was an epochal release"? Or "epoch-making release"? From peio.borthelle at gmail.com Thu Sep 1 15:19:16 2011 From: peio.borthelle at gmail.com (Peio Borthelle) Date: Thu, 1 Sep 2011 15:19:16 +0200 Subject: [Python-ideas] aliasing In-Reply-To: References: Message-ID: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> Le 31 ao?t 2011 ? 17:20, python-ideas-request at python.org a ?crit : > I have often thought that would be a nice to have feature, but to be > honest, I have never found a use for it that was absolutely necessary. I > have always found another way to solve the problem. > > To make it work using just ordinary assignment syntax, as you suggest, > requires more than just an "alias" function. It would need changes to > the Python internals. Possibly very large changes. Without a convincing > use-case, I don't think that will happen. > > So even though I think this would be a neat feature to have, and > possibly even useful, I don't think it is useful enough to justify the > work needed to make it happen. > > -- > Steven I don't agree with you when you say there is no necessary use case, e.g. I have two images in a gui and they must stay at the same position (relatively one to the other) but follow the movements of the arrow keys (truthful!). You can't do it by using an intermediate list because the list object is mutable but his data not: ------------------------ >>> list = [3,] >>> a = list[0] >>> list[0] = 6 >>> a 3 ------------------------- I really don't see any other solution, even I'm sure it would be a twisted operation and I don't find it very pythonic... Amicalement, Peio -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruce at leapyear.org Thu Sep 1 17:38:57 2011 From: bruce at leapyear.org (Bruce Leban) Date: Thu, 1 Sep 2011 08:38:57 -0700 Subject: [Python-ideas] aliasing In-Reply-To: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> References: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> Message-ID: On Thu, Sep 1, 2011 at 6:19 AM, Peio Borthelle wrote: > > I don't agree with you when you say there is no necessary use case, e.g. I > have two images in a gui and they must stay at the same position (relatively > one to the other) but follow the movements of the arrow keys > (truthful!). You can't do it by using an intermediate list because the list > object is mutable but his data not: > ------------------------ > >>> list = [3,] > >>> a = list[0] > >>> list[0] = 6 > >>> a > 3 > This little snippet is unsurprising and would be extremely surprising if it printed 6 instead. > ------------------------- > I really don't see any other solution, even I'm sure it would be a twisted > operation and I don't find it very pythonic... > > If you don't see any other solution than you're not looking hard enough. What you're saying is you have objects in a gui which must be manipulated by simple variables rather than be encapsulated in a class. I don't buy it. How does this work with N objects which must lock together? I note that you say "relative to one another" so what you really mean is x2 locks to x1 + A and y2 locks to x1 + B. That's not a simple alias. Can I assign to x2? If you think assigning to x2 will change x1, you haven't thought it through. If you think it won't but instead will break the alias, then what is the advantage of not just writing x2()?? You save two characters (and a lambda somewhere else) at the expense of making the code much harder to understand. It's like a programmer building a spreadsheet app in Python saying this is the only way he can implement cell recalc. Fortran has a statement called COMMON which aliases variables. It's useful for reshaping arrays. Other than that, it's an easy way to confuse programmers and the compiler. --- Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Sep 1 18:17:53 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 01 Sep 2011 12:17:53 -0400 Subject: [Python-ideas] aliasing In-Reply-To: References: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> Message-ID: On 9/1/2011 11:38 AM, Bruce Leban wrote: > On Thu, Sep 1, 2011 at 6:19 AM, Peio Borthelle > > wrot > e.g. I have two images in a gui and they must stay at the same > position (relatively one to the other) but follow the movements of > the arrow keys (truthful!). You put them in a frame so that their positions are relative to a frame rather than absolute pixel positions. Then you move the frame. Relative positioning is fundamental to computer graphics. > What you're saying is you have objects in a gui which must be > manipulated by simple variables rather than be encapsulated in a class. The position should be an attribute of a class and relative to something. > Fortran has a statement called COMMON which aliases variables. This works because in Fortran a 'variable' is a block of memory. As I remember, COMMON says to reuse memory for different variables. > It's > useful for reshaping arrays. Other than that, it's an easy way to > confuse programmers and the compiler. -- Terry Jan Reedy From sklass at pointcircle.com Thu Sep 1 17:40:19 2011 From: sklass at pointcircle.com (Steven Klass) Date: Thu, 1 Sep 2011 08:40:19 -0700 Subject: [Python-ideas] Improving this? - foo() takes at least 6 arguments (6 given) Message-ID: <3587ACF7-8B90-40B4-89E1-BC9AF2D4E5CC@pointcircle.com> Hi all, Most of us who have been coding for awhile have often seen this error and understand the "self" concept. I was helping a junior colleague out and he came to me with this problem. Colleague: "So my code is apparently asking for 6 arguments and it appears that I gave it 6 arguments" Me: "Oh that - Yeah... your short one. It really wants 7.." That got me thinking and here is the post. Now while this seems simple to fix I know this isn't that simple and a healthy dose of introspection will be required. --- Steven Klass (480) 225-1112 sklass at pointcircle.com From ethan at stoneleaf.us Thu Sep 1 19:19:56 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 01 Sep 2011 10:19:56 -0700 Subject: [Python-ideas] Improving this? - foo() takes at least 6 arguments (6 given) In-Reply-To: <3587ACF7-8B90-40B4-89E1-BC9AF2D4E5CC@pointcircle.com> References: <3587ACF7-8B90-40B4-89E1-BC9AF2D4E5CC@pointcircle.com> Message-ID: <4E5FBEBC.1030604@stoneleaf.us> Steven Klass wrote: > Hi all, > > Most of us who have been coding for awhile have often seen this error and understand the "self" concept. I was helping a junior colleague out and he came to me with this problem. > > Colleague: "So my code is apparently asking for 6 arguments and it appears that I gave it 6 arguments" > Me: "Oh that - Yeah... your short one. It really wants 7.." > > That got me thinking and here is the post. Now while this seems simple to fix I know this isn't that simple and a healthy dose of introspection will be required. How did you get this? --> class test(object): ... def __init__(self): ... pass ... --> t = test(1) 2.5 --- Traceback (most recent call last): File "", line 1, in TypeError: __init__() takes exactly 1 argument (2 given) 2.7 --- Traceback (most recent call last): File "", line 1, in TypeError: __init__() takes exactly 1 argument (2 given) 3.2 --- Traceback (most recent call last): File "", line 1, in TypeError: __init__() takes exactly 1 positional argument (2 given) ~Ethan~ From barry at python.org Thu Sep 1 20:26:35 2011 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Sep 2011 14:26:35 -0400 Subject: [Python-ideas] Add from __experimental__ import bla [was: Should we move to replace re with regex?] References: <1314569103.4640.19.camel@Gutsy> <58780826-87be-436f-b777-937e41691e90@m35g2000prl.googlegroups.com> <20110831110536.111822cf@resist.wooz.org> Message-ID: <20110901142635.3f766a9b@resist.wooz.org> On Aug 31, 2011, at 09:29 AM, Guido van Rossum wrote: >On Wed, Aug 31, 2011 at 8:05 AM, Barry Warsaw wrote: >> In the face of PEP 402, how could you enforce that? ?Even if you can't or >> don't want to enforce it, how would a user be able to verify that it was the >> case for something in experimental? > >I'm sorry, I don't follow. The experimental package would only contain >code distributed as part of the stdlib, and the code put in the >stdlib's experimental package would get the same care from the core >developers as the rest of the stdlib. The only difference would be >that we'd drop the guarantee that the APIs offered would still be >present in the next release (i.e. from 3.3 -> 3.4; the guarantees >would hold from 3.3.1 -> 3.3.2). As long as the "guarantees" only cover code distributed in the experimental package with Python, that's okay. Maybe I was reading too much into Nick's comment, but when third party code can situate itself under 'experimental', we obviously can't make those same guarantees of the code we don't distribute. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Thu Sep 1 20:28:37 2011 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Sep 2011 14:28:37 -0400 Subject: [Python-ideas] Add from __experimental__ import bla [was: Should we move to replace re with regex?] References: <1314569103.4640.19.camel@Gutsy> <58780826-87be-436f-b777-937e41691e90@m35g2000prl.googlegroups.com> <20110831110536.111822cf@resist.wooz.org> Message-ID: <20110901142837.5137d173@resist.wooz.org> On Sep 01, 2011, at 09:14 AM, Nick Coghlan wrote: >On Thu, Sep 1, 2011 at 1:05 AM, Barry Warsaw wrote: >> In the face of PEP 402, how could you enforce that? ?Even if you can't or >> don't want to enforce it, how would a user be able to verify that it was the >> case for something in experimental? > >PEP 402 doesn't really change anything regarding corporate checks of >code provenance - those relate to how the code is acquired and >installed in the first place, not how you use it at runtime. If >engineers are doing end runs around corporate policies, that's not a >problem that technology can realistically address :) Agreed! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Thu Sep 1 20:34:14 2011 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Sep 2011 14:34:14 -0400 Subject: [Python-ideas] multi-version libraries (was Re: Add from __experimental__ import bla) References: <1314569103.4640.19.camel@Gutsy> <58780826-87be-436f-b777-937e41691e90@m35g2000prl.googlegroups.com> <871uw2l0vs.fsf@uwakimon.sk.tsukuba.ac.jp> <20110831111946.22bac764@resist.wooz.org> Message-ID: <20110901143414.6a25255b@resist.wooz.org> On Aug 31, 2011, at 09:31 AM, Guido van Rossum wrote: >On Wed, Aug 31, 2011 at 8:19 AM, Barry Warsaw wrote: >> On Aug 30, 2011, at 08:53 PM, Guido van Rossum wrote: >> >>>TBH the best use case I can think of would actually be the ipaddr >>>package, which is somewhat controversial but not overly so, and seems >>>to lack a push to ever get it accepted. Putting it in experimental for >>>3.3 would let us distribute it with Python without committing 100% to >>>the solution it offers over its nearest competitor. However the >>>downside is that that's a very long wait, still provides a pretty >>>strong bias (unless we were to include both competing packages), and >>>still doesn't look like it might get enough beta testing, unless the >>>uptake of 3.3 is huge. So maybe we should just count PyPI downloads >>>and decide that way. "50,000,000 Elvis fans can't be wrong." (An >>>interesting meme by itself. :-) >> >> I think an experimental namespace is actually attacking the wrong problem, >> or maybe the right problem in the wrong way. ;) >> >> I'd much prefer to see some brainstorming on multi-version support, which is >> something that Robert Collins is always bugging me about. ?Something like >> pkg_resource's require() function seems like a good place to start. > >It's a great idea to brainstorm about, just not in this thread. I see >the two issues completely orthogonal. It seems related though. It seems like the primary use case for experimental, especially in reference to the ipaddr module, is "we're including this battery, but it's shape might change, so don't count on it fitting into your socket next release". With multi-version libraries, we don't have to be so wishy-washy. We could say "here's version 0.5 of ipaddr" and the next release could safely include both that version and a stable, completely redesigned API of ipaddr 1.0. The two could co-exist (for a while), supporting both the legacy API, and the new whizzy one we actually like. ;) >> If we had this built-in, then including ipaddr in the stdlib as it currently >> stands would be a no-brainer, even with a suboptimal API. ?It would get lots >> of testing, and a completely different API could be designed for Python 3.4 >> without break packages that relied on the old API. ?Python's deprecation >> policy could be adjusted to include diminishing support for older versions of >> a module in the stdlib, and we'd avoid ugliness like unittest2 and such. > >You sound like you have a solution for using multiple versions of an >API in the same program. Do you? Then out with it! Otherwise, no, it >wouldn't be a no-brainer. I wish I did! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Thu Sep 1 20:37:29 2011 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Sep 2011 14:37:29 -0400 Subject: [Python-ideas] Add from __experimental__ import bla [was: Should we move to replace re with regex?] References: <1314569103.4640.19.camel@Gutsy> <58780826-87be-436f-b777-937e41691e90@m35g2000prl.googlegroups.com> <871uw2l0vs.fsf@uwakimon.sk.tsukuba.ac.jp> <20110831111946.22bac764@resist.wooz.org> <20110831185635.744c9624@pitrou.net> <1314811985.3505.0.camel@localhost.localdomain> Message-ID: <20110901143729.516af5fe@resist.wooz.org> On Aug 31, 2011, at 10:51 AM, Guido van Rossum wrote: >2 -> 3: galactic release OMGWTFBBQ release >3.2 -> 3.3: feature release (also 3 -> 3.1) Don't you mean '3.0 -> 3.1' ? >3.2.1 -> 3.2.2: bugfix release (also 3.2 -> 3.2.1) +1! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Thu Sep 1 20:42:08 2011 From: barry at python.org (Barry Warsaw) Date: Thu, 1 Sep 2011 14:42:08 -0400 Subject: [Python-ideas] Terminology for "earthshattering" [was: Add from __experimental__ import bla] References: <1314569103.4640.19.camel@Gutsy> <58780826-87be-436f-b777-937e41691e90@m35g2000prl.googlegroups.com> <871uw2l0vs.fsf@uwakimon.sk.tsukuba.ac.jp> <20110831111946.22bac764@resist.wooz.org> <20110831185635.744c9624@pitrou.net> <87mxeok5z0.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20110901144208.66115e96@resist.wooz.org> On Sep 01, 2011, at 05:43 PM, Stephen J. Turnbull wrote: >How about the leading digit being the "Pythonic epoch", and "Python 3 >was an epochal release"? Or "epoch-making release"? Or maybe: snake reference: skin shedding release funny brit reference: spanish inquisition release -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From guido at python.org Thu Sep 1 21:57:47 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 1 Sep 2011 12:57:47 -0700 Subject: [Python-ideas] multi-version libraries (was Re: Add from __experimental__ import bla) In-Reply-To: <20110901143414.6a25255b@resist.wooz.org> References: <1314569103.4640.19.camel@Gutsy> <58780826-87be-436f-b777-937e41691e90@m35g2000prl.googlegroups.com> <871uw2l0vs.fsf@uwakimon.sk.tsukuba.ac.jp> <20110831111946.22bac764@resist.wooz.org> <20110901143414.6a25255b@resist.wooz.org> Message-ID: On Thu, Sep 1, 2011 at 11:34 AM, Barry Warsaw wrote: > On Aug 31, 2011, at 09:31 AM, Guido van Rossum wrote: > >>On Wed, Aug 31, 2011 at 8:19 AM, Barry Warsaw wrote: >>> On Aug 30, 2011, at 08:53 PM, Guido van Rossum wrote: >>> >>>>TBH the best use case I can think of would actually be the ipaddr >>>>package, which is somewhat controversial but not overly so, and seems >>>>to lack a push to ever get it accepted. Putting it in experimental for >>>>3.3 would let us distribute it with Python without committing 100% to >>>>the solution it offers over its nearest competitor. However the >>>>downside is that that's a very long wait, still provides a pretty >>>>strong bias (unless we were to include both competing packages), and >>>>still doesn't look like it might get enough beta testing, unless the >>>>uptake of 3.3 is huge. So maybe we should just count PyPI downloads >>>>and decide that way. "50,000,000 Elvis fans can't be wrong." (An >>>>interesting meme by itself. :-) >>> >>> I think an experimental namespace is actually attacking the wrong problem, >>> or maybe the right problem in the wrong way. ;) >>> >>> I'd much prefer to see some brainstorming on multi-version support, which is >>> something that Robert Collins is always bugging me about. ?Something like >>> pkg_resource's require() function seems like a good place to start. >> >>It's a great idea to brainstorm about, just not in this thread. I see >>the two issues completely orthogonal. > > It seems related though. ?It seems like the primary use case for experimental, > especially in reference to the ipaddr module, is "we're including this > battery, but it's shape might change, so don't count on it fitting into your > socket next release". ?With multi-version libraries, we don't have to be so > wishy-washy. ?We could say "here's version 0.5 of ipaddr" and the next release > could safely include both that version and a stable, completely redesigned API > of ipaddr 1.0. ?The two could co-exist (for a while), supporting both the > legacy API, and the new whizzy one we actually like. ;) It still sounds like they are (nearly) orthogonal though. Even if we had such a feature we'd still want to put ipaddr 0.5 into experimental, but most other new stdlib modules would not have to go there. And even if we didn't have such a feature, I'd still consider putting things in experimental, and just telling people to deal with it. Especially since we *don't* have the feature. :-) >>> If we had this built-in, then including ipaddr in the stdlib as it currently >>> stands would be a no-brainer, even with a suboptimal API. ?It would get lots >>> of testing, and a completely different API could be designed for Python 3.4 >>> without break packages that relied on the old API. ?Python's deprecation >>> policy could be adjusted to include diminishing support for older versions of >>> a module in the stdlib, and we'd avoid ugliness like unittest2 and such. >> >>You sound like you have a solution for using multiple versions of an >>API in the same program. Do you? Then out with it! Otherwise, no, it >>wouldn't be a no-brainer. > > I wish I did! That's what I'm talking about! :-) -- --Guido van Rossum (python.org/~guido) From guido at python.org Thu Sep 1 22:05:44 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 1 Sep 2011 13:05:44 -0700 Subject: [Python-ideas] Add from __experimental__ import bla [was: Should we move to replace re with regex?] In-Reply-To: <20110901142635.3f766a9b@resist.wooz.org> References: <1314569103.4640.19.camel@Gutsy> <58780826-87be-436f-b777-937e41691e90@m35g2000prl.googlegroups.com> <20110831110536.111822cf@resist.wooz.org> <20110901142635.3f766a9b@resist.wooz.org> Message-ID: On Thu, Sep 1, 2011 at 11:26 AM, Barry Warsaw wrote: > On Aug 31, 2011, at 09:29 AM, Guido van Rossum wrote: > >>On Wed, Aug 31, 2011 at 8:05 AM, Barry Warsaw wrote: >>> In the face of PEP 402, how could you enforce that? ?Even if you can't or >>> don't want to enforce it, how would a user be able to verify that it was the >>> case for something in experimental? >> >>I'm sorry, I don't follow. The experimental package would only contain >>code distributed as part of the stdlib, and the code put in the >>stdlib's experimental package would get the same care from the core >>developers as the rest of the stdlib. The only difference would be >>that we'd drop the guarantee that the APIs offered would still be >>present in the next release (i.e. from 3.3 -> 3.4; the guarantees >>would hold from 3.3.1 -> 3.3.2). > > As long as the "guarantees" only cover code distributed in the experimental > package with Python, that's okay. ?Maybe I was reading too much into Nick's > comment, but when third party code can situate itself under 'experimental', we > obviously can't make those same guarantees of the code we don't distribute. That certainly wasn't part of *my* plan any more than letting 3rd parties install themselves anywhere else inside stdlib packages. On the other hand it probably wouldn't matter because the review process doesn't look at what you import -- it only looks at dependencies (i.e. stuff that needs to be downloaded). Whether it goes under experimental would be orthogonal -- the litmus test for "is this covered by the PSF" would be "do I have to install it separately" (when using a Python install built from source, so there are no packages pre-installed by the distro). -- --Guido van Rossum (python.org/~guido) From g.nius.ck at gmail.com Thu Sep 1 22:11:54 2011 From: g.nius.ck at gmail.com (Christopher King) Date: Thu, 1 Sep 2011 16:11:54 -0400 Subject: [Python-ideas] aliasing In-Reply-To: References: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> Message-ID: Wrong list again On Thu, Sep 1, 2011 at 4:10 PM, Christopher King wrote: > ------------------------ >> >>> list = [3,] >> >>> a = list >> >>> list[0] = 6 >> >>> a[0] >> 3 >> ------------------------- >> > Slight error in my code. It should be. > ------------------------ > >>> list = [3,] > >>> a = list > >>> list[0] = 6 > >>> a[0] > 6 > ------------------------- > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.nius.ck at gmail.com Thu Sep 1 22:16:02 2011 From: g.nius.ck at gmail.com (Christopher King) Date: Thu, 1 Sep 2011 16:16:02 -0400 Subject: [Python-ideas] Terminology for "earthshattering" [was: Add from __experimental__ import bla] In-Reply-To: <87mxeok5z0.fsf@uwakimon.sk.tsukuba.ac.jp> References: <1314569103.4640.19.camel@Gutsy> <58780826-87be-436f-b777-937e41691e90@m35g2000prl.googlegroups.com> <871uw2l0vs.fsf@uwakimon.sk.tsukuba.ac.jp> <20110831111946.22bac764@resist.wooz.org> <20110831185635.744c9624@pitrou.net> <87mxeok5z0.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: How about irreversible-release. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.nius.ck at gmail.com Thu Sep 1 22:20:52 2011 From: g.nius.ck at gmail.com (Christopher King) Date: Thu, 1 Sep 2011 16:20:52 -0400 Subject: [Python-ideas] Expanding statistical functions in Python's std. lib. In-Reply-To: <4E5D05EE.2050502@gmail.com> References: <549901314286114@web119.yandex.ru> <4E56E859.3090504@canterbury.ac.nz> <4E5D05EE.2050502@gmail.com> Message-ID: I think this is very Pythonic. On blogs describing python, one of the features they list is batteries included. Statistical functions would be great. I give it +1. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Sep 1 22:24:01 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 1 Sep 2011 13:24:01 -0700 Subject: [Python-ideas] Terminology for "earthshattering" [was: Add from __experimental__ import bla] In-Reply-To: References: <1314569103.4640.19.camel@Gutsy> <58780826-87be-436f-b777-937e41691e90@m35g2000prl.googlegroups.com> <871uw2l0vs.fsf@uwakimon.sk.tsukuba.ac.jp> <20110831111946.22bac764@resist.wooz.org> <20110831185635.744c9624@pitrou.net> <87mxeok5z0.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Thu, Sep 1, 2011 at 1:16 PM, Christopher King wrote: > How about?irreversible-release. "Killer release". (An oblique reference to Monty Python's killer joke.) -- --Guido van Rossum (python.org/~guido) From aquavitae69 at gmail.com Thu Sep 1 22:25:29 2011 From: aquavitae69 at gmail.com (David Townshend) Date: Thu, 1 Sep 2011 22:25:29 +0200 Subject: [Python-ideas] Add from __experimental__ import bla [was: Should we move to replace re with regex?] In-Reply-To: <20110901143729.516af5fe@resist.wooz.org> References: <1314569103.4640.19.camel@Gutsy> <58780826-87be-436f-b777-937e41691e90@m35g2000prl.googlegroups.com> <871uw2l0vs.fsf@uwakimon.sk.tsukuba.ac.jp> <20110831111946.22bac764@resist.wooz.org> <20110831185635.744c9624@pitrou.net> <1314811985.3505.0.camel@localhost.localdomain> <20110901143729.516af5fe@resist.wooz.org> Message-ID: Another approach to __experimental__ could be a semi-official "python-experimental" package on pypi which contains all the stuff that's already been talked about. It's just one extra package to depend on and could follow its own release cycle. I don't know if this is better than including it in the stdlib (I like the idea of __experimental__ myself), but maybe its another idea to consider. David On Sep 1, 2011 8:40 PM, "Barry Warsaw" wrote: > On Aug 31, 2011, at 10:51 AM, Guido van Rossum wrote: > >>2 -> 3: galactic release > > OMGWTFBBQ release > >>3.2 -> 3.3: feature release (also 3 -> 3.1) > > Don't you mean '3.0 -> 3.1' ? > >>3.2.1 -> 3.2.2: bugfix release (also 3.2 -> 3.2.1) > > +1! > -Barry > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Sep 1 22:28:36 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 1 Sep 2011 13:28:36 -0700 Subject: [Python-ideas] Add from __experimental__ import bla [was: Should we move to replace re with regex?] In-Reply-To: References: <1314569103.4640.19.camel@Gutsy> <58780826-87be-436f-b777-937e41691e90@m35g2000prl.googlegroups.com> <871uw2l0vs.fsf@uwakimon.sk.tsukuba.ac.jp> <20110831111946.22bac764@resist.wooz.org> <20110831185635.744c9624@pitrou.net> <1314811985.3505.0.camel@localhost.localdomain> <20110901143729.516af5fe@resist.wooz.org> Message-ID: On Thu, Sep 1, 2011 at 1:25 PM, David Townshend wrote: > Another approach to __experimental__ could be a semi-official > "python-experimental" package on pypi which contains all the stuff that's > already been talked about. It's just one extra package to depend on and > could follow its own release cycle. I don't know if this is better than > including it in the stdlib (I like the idea of __experimental__ myself), but > maybe its another idea to consider. No. It wouldn't have any of the advantages of a bundled experimental package. -- --Guido van Rossum (python.org/~guido) From mikegraham at gmail.com Thu Sep 1 22:56:20 2011 From: mikegraham at gmail.com (Mike Graham) Date: Thu, 1 Sep 2011 16:56:20 -0400 Subject: [Python-ideas] Expanding statistical functions in Python's std. lib. In-Reply-To: <4E5D05EE.2050502@gmail.com> References: <549901314286114@web119.yandex.ru> <4E56E859.3090504@canterbury.ac.nz> <4E5D05EE.2050502@gmail.com> Message-ID: On Tue, Aug 30, 2011 at 11:46 AM, Spectral One wrote: > > Wandering about, looking up statistics info for a program I was writing, I > found a recommendation to add various useful 'special functions' to C's math > library: > http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1069.pdf > > The arguments in that paper make a lot of sense to me, and apply well to > Python. They came up with a good list, IMnsHO. I'd recommend implementing > this list in some form as library functions in Python. > > Blindly copying wouldn't end up particularly 'Pythonic;' tweaking the API is > required. ? ?Some of the selection choices, such as returning real only, > ought to be reevaluated, for example. Obviously, any of the decisions to > keep things C-like rather than object-oriented ought to shift, as well. > > Function names are only important as far as they are clear. I suggest naming > per e.g. distribution_t(), or dist_F(), > and include modification for algebraic order, as well, so gamma() and > log_gamma(). That said, anything clear is fine. > > Thoughts on the matter? I noticed that the math library in 2.7+ added the > gamma and log(gamma) functions, already, which was nice. Obviously, most, if > not all, are already present in extensions modules such as NumPy, but there > is value in having these things built into the language. "Batteries > included, "and all that. > > > > By the by, if that is far too much for one suggestion, then please just > treat this as a suggestion to add just the incomplete beta function. > (P-values for binomial, F, and t are all nice, too, though with inc. beta, > they aren't terrible to generate. I really think they should be included in > the standard library.) I'm not sure that many people who could make tons of use from statistical functions don't already have cause to be using numpy/scipy. I would certainly be unfortunate if having a little more statistics functionality in the stdlib discouraged people who should be using numpy from doing so. "Batteries included" has always been a bit of an oversell, and as a Python user I don't have any expectation of being able to do fairly-specialized work without third-party modules, nor do I think it's necessarily a net gain for Python if I could. -0 Mike From greg.ewing at canterbury.ac.nz Fri Sep 2 02:25:07 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 02 Sep 2011 12:25:07 +1200 Subject: [Python-ideas] aliasing In-Reply-To: References: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> Message-ID: <4E602263.2000109@canterbury.ac.nz> Terry Reedy wrote: > This works because in Fortran a 'variable' is a block of memory. As I > remember, COMMON says to reuse memory for different variables. Also I think in early versions of Fortran it was the only way of sharing variables between subroutines, because there was no concept of a global namespace -- each subroutine was compiled independently, even if they were in the same source file. -- Greg From ethan at stoneleaf.us Fri Sep 2 02:50:11 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 01 Sep 2011 17:50:11 -0700 Subject: [Python-ideas] Expanding statistical functions in Python's std. lib. In-Reply-To: References: <549901314286114@web119.yandex.ru> <4E56E859.3090504@canterbury.ac.nz> <4E5D05EE.2050502@gmail.com> Message-ID: <4E602843.8010803@stoneleaf.us> Mike Graham wrote: > "Batteries included" has always been a bit of an oversell, and as a > Python user I don't have any expectation of being able to do > fairly-specialized work without third-party modules, nor do I think > it's necessarily a net gain for Python if I could. Well, it is "batteries" not "nuclear reactors" that are included. One must keep these things in perspective. ~Ethan~ From python at mrabarnett.plus.com Fri Sep 2 03:17:08 2011 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 02 Sep 2011 02:17:08 +0100 Subject: [Python-ideas] aliasing In-Reply-To: <4E602263.2000109@canterbury.ac.nz> References: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> <4E602263.2000109@canterbury.ac.nz> Message-ID: <4E602E94.20505@mrabarnett.plus.com> On 02/09/2011 01:25, Greg Ewing wrote: > Terry Reedy wrote: > >> This works because in Fortran a 'variable' is a block of memory. As I >> remember, COMMON says to reuse memory for different variables. > > Also I think in early versions of Fortran it was the only > way of sharing variables between subroutines, because there > was no concept of a global namespace -- each subroutine > was compiled independently, even if they were in the same > source file. > Fortran also pre-dated stack allocation (I think), so local storage was static (and no recursion!). COMMON enabled re-use of memory where two or more subroutines or functions weren't 'active' at the same time. From steve at pearwood.info Fri Sep 2 04:27:52 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 02 Sep 2011 12:27:52 +1000 Subject: [Python-ideas] aliasing In-Reply-To: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> References: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> Message-ID: <4E603F28.2010802@pearwood.info> Peio Borthelle wrote: > Le 31 ao?t 2011 ? 17:20, python-ideas-request at python.org a ?crit : > >> I have often thought that would be a nice to have feature, but to >> be honest, I have never found a use for it that was absolutely >> necessary. I have always found another way to solve the problem. >> >> To make it work using just ordinary assignment syntax, as you >> suggest, requires more than just an "alias" function. It would need >> changes to the Python internals. Possibly very large changes. >> Without a convincing use-case, I don't think that will happen. >> >> So even though I think this would be a neat feature to have, and >> possibly even useful, I don't think it is useful enough to justify >> the work needed to make it happen. >> >> -- Steven > > I don't agree with you when you say there is no necessary use case, I didn't say that there *is* no use-case, only that I have never found one. Perhaps you have misunderstood the English idiom "Without a convincing use-case...". This isn't meant to imply that there is no use-case, only that nobody has yet found one. The implication is that, if you wish to promote this change to Python's behaviour, showing a convincing use-case is a necessary first step. (But not the only step.) -- Steven From wuwei23 at gmail.com Fri Sep 2 05:51:32 2011 From: wuwei23 at gmail.com (alex23) Date: Thu, 1 Sep 2011 20:51:32 -0700 (PDT) Subject: [Python-ideas] aliasing In-Reply-To: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> References: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> Message-ID: <6c887fe3-29c8-4c36-a9f2-dfd66b77cd1d@p37g2000prp.googlegroups.com> On Sep 1, 11:19?pm, Peio Borthelle wrote: > I have two images in a gui and they must stay at the same position > (relatively one to the other) but follow the movements of the arrow > keys (truthful!). While I think that Terry's suggestion of a frame object that takes care of the movement is the better approach, you can also use the package Trellis to keep object attributes in lockstep (amongst other features). http://pypi.python.org/pypi/Trellis From stephen at xemacs.org Fri Sep 2 06:04:33 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 02 Sep 2011 13:04:33 +0900 Subject: [Python-ideas] aliasing In-Reply-To: <4E602E94.20505@mrabarnett.plus.com> References: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> <4E602263.2000109@canterbury.ac.nz> <4E602E94.20505@mrabarnett.plus.com> Message-ID: <87aaank2su.fsf@uwakimon.sk.tsukuba.ac.jp> MRAB writes: > Fortran also pre-dated stack allocation (I think), so local storage was > static (and no recursion!). If FORTRAN [sic] predated stack allocation, what did Lisp use to handle recursion? From steve at pearwood.info Fri Sep 2 07:19:58 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 02 Sep 2011 15:19:58 +1000 Subject: [Python-ideas] aliasing In-Reply-To: <87aaank2su.fsf@uwakimon.sk.tsukuba.ac.jp> References: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> <4E602263.2000109@canterbury.ac.nz> <4E602E94.20505@mrabarnett.plus.com> <87aaank2su.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4E60677E.8060602@pearwood.info> Stephen J. Turnbull wrote: > MRAB writes: > > > Fortran also pre-dated stack allocation (I think), so local storage was > > static (and no recursion!). > > If FORTRAN [sic] predated stack allocation, what did Lisp use to > handle recursion? I don't see the connection. The first proposal for Fortran was 1953, with the first public release in 1957, while John McCarthy didn't start work on Lisp until 1958. I'm not sure when a Lisp compiler was first available, but there's no doubt that Fortran pre-dates Lisp. But even if it didn't, there's no logical inconsistency between Lisp using stack allocation and Fortran not. (There's no evidence that I can see that either language influenced the other directly.) -- Steven From ncoghlan at gmail.com Fri Sep 2 07:22:44 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 2 Sep 2011 15:22:44 +1000 Subject: [Python-ideas] IP addressing library, aka reviving PEP 3144 (Re: Add from __experimental__ import bla) In-Reply-To: References: Message-ID: On Fri, Sep 2, 2011 at 8:03 AM, Peter Moody wrote: > I'll make an implementation with these changes this weekend. Great to hear! > It sounds like it's going to be clunky, but that's definitely with a grain or two of bias. Hopefully it won't be too inconvenient - in theory, it just means that when you derive the network details from a host address and masking information you can't throw the original host address information away any more (since IPNetwork own't be keeping it around). > Documentation isn't my strong suit, but assuming the implementation is > palatable to all involved, that can be forthcoming as well. Splitting the API up a little should actually make the documentation task itself easier. For folks that are already familiar with networking concepts, it isn't that hard to mentally map the "IPNetwork" name as meaning "either an actual IP network or else an arbitrary host/netmask pair". We might grumble about it (*cough*), but we wouldn't be confused once we made the connection. However, once ipaddr goes into the standard library, its module documentation is going to be the first introduction many future Python programmers will have to the details of how IP addressing actually works. We'll refer them off to other resources, of course, but having clean semantics in the API itself is important in helping people that are *not* already networking experts use the module correctly. That means having the class model map cleanly to the real world entities it represents, and while IP networks and specific interfaces on those networks are both represented accurately by the combination of an IP address with masking information, they're *not* the same kind of thing. It's the difference between using Pair(a, b) for both 2-D coordinates and rational numbers vs using separate Point(x, y) and Rational(n, d) classes. Yes, the information stored in both cases is the same, but the semantics assigned to that information changes. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From anacrolix at gmail.com Fri Sep 2 07:33:01 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Fri, 2 Sep 2011 15:33:01 +1000 Subject: [Python-ideas] allow line break at operators In-Reply-To: <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4e424208$0$29965$c3e8da3$5496439d@news.astraweb.com> <1312981104.89312.YahooMailNeo@web121520.mail.ne1.yahoo.com> <1312982377.95657.YahooMailNeo@web121508.mail.ne1.yahoo.com> <1313031175.38817.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> Message-ID: I guess the issue here is that you can't tell if an expression is complete without checking the indent of the following line. This is likely not desirable. On Thu, Sep 1, 2011 at 11:43 PM, Yingjie Lan wrote: > Hi Matt, > ======================================================= > From: Matt Joiner > > The "trailing \" workaround is nonobvious. Wrapping in () is noisy and > already heavily used by other syntactical structures. > ======================================================= > How about only require indentation > to freely break lines? Here is an example: > x = firstpart * secondpart #line breaks here > + anotherpart #continue by indentation > + stillanother #continue on. > #until here, another line starts by dedentation > y = some_expression?- another_one > All this would be completely compatible with former code, while > having almost free line breaking! Plus, indentation makes it pretty. > Really hope Python can have freedom in breaking lines. > Yingjie From greg.ewing at canterbury.ac.nz Fri Sep 2 08:03:13 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 02 Sep 2011 18:03:13 +1200 Subject: [Python-ideas] aliasing In-Reply-To: <87aaank2su.fsf@uwakimon.sk.tsukuba.ac.jp> References: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> <4E602263.2000109@canterbury.ac.nz> <4E602E94.20505@mrabarnett.plus.com> <87aaank2su.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4E6071A1.2090004@canterbury.ac.nz> Stephen J. Turnbull wrote: > If FORTRAN [sic] predated stack allocation, what did Lisp use to > handle recursion? According to Wikipedia, the first Lisp implementation was an interpreter written in machine code, not Fortran. However, even if an interpreter were written in Fortran, it wouldn't be constrained by Fortran's limitations, any more than CPython is contstrained by C's limitations. -- Greg From gahtune at gmail.com Fri Sep 2 08:14:12 2011 From: gahtune at gmail.com (Gabriel AHTUNE) Date: Fri, 2 Sep 2011 14:14:12 +0800 Subject: [Python-ideas] allow line break at operators In-Reply-To: References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4e424208$0$29965$c3e8da3$5496439d@news.astraweb.com> <1312981104.89312.YahooMailNeo@web121520.mail.ne1.yahoo.com> <1312982377.95657.YahooMailNeo@web121508.mail.ne1.yahoo.com> <1313031175.38817.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> Message-ID: So can be done with this syntax: > x = firstpart * secondpart + #line breaks here > anotherpart + #continue > stillanother #continue on. after a "+" operator the line is clearly not finished yet. Gabriel AHTUNE 2011/9/2 Matt Joiner > I guess the issue here is that you can't tell if an expression is > complete without checking the indent of the following line. This is > likely not desirable. > > On Thu, Sep 1, 2011 at 11:43 PM, Yingjie Lan wrote: > > Hi Matt, > > ======================================================= > > From: Matt Joiner > > > > The "trailing \" workaround is nonobvious. Wrapping in () is noisy and > > already heavily used by other syntactical structures. > > ======================================================= > > How about only require indentation > > to freely break lines? Here is an example: > > x = firstpart * secondpart #line breaks here > > + anotherpart #continue by indentation > > + stillanother #continue on. > > #until here, another line starts by dedentation > > y = some_expression - another_one > > All this would be completely compatible with former code, while > > having almost free line breaking! Plus, indentation makes it pretty. > > Really hope Python can have freedom in breaking lines. > > Yingjie > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Fri Sep 2 08:22:44 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 02 Sep 2011 15:22:44 +0900 Subject: [Python-ideas] aliasing In-Reply-To: <4E60677E.8060602@pearwood.info> References: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> <4E602263.2000109@canterbury.ac.nz> <4E602E94.20505@mrabarnett.plus.com> <87aaank2su.fsf@uwakimon.sk.tsukuba.ac.jp> <4E60677E.8060602@pearwood.info> Message-ID: <878vq7jwej.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > Stephen J. Turnbull wrote: > > MRAB writes: > > > > > Fortran also pre-dated stack allocation (I think), so local storage was > > > static (and no recursion!). > > > > If FORTRAN [sic] predated stack allocation, what did Lisp use to > > handle recursion? > > I don't see the connection. The first proposal for Fortran was 1953, > with the first public release in 1957, while John McCarthy didn't start > work on Lisp until 1958. He didn't start work on Lisp the language, or Lisp the interpreter? The language existed for some time before it occurred to people to implement an interpreter, but implementing the interpreter was fairly quick once somebody thought of it as I understand it. After all, once you have the read-eval-print loop, lambda, cond, cons, car, cdr, a data structure for symbols, and a data structure for the environment that supports recursion (thanks to Bruce Leban for pointing out off-list that this was an association list, not local variables on the stack), you can define everything else from that. AFAIK the public release of FORTRAN and the academic introduction of a Lisp interpreter were pretty much simultaneous. > But even if it didn't, there's no logical inconsistency between > Lisp using stack allocation and Fortran not. (There's no evidence > that I can see that either language influenced the other directly.) As quoted above, "stack allocation" refers to the concept, not its use in FORTRAN. There are lots of concepts that existed for decades before being implemented in Fortran.[1] I was just curious whether this was one of them. I'll go see if the Dragon Book has anything to say about it. Footnotes: [1] Just as some concepts have waited for decades to be implemented in Python. I suspect "alias" is one that will have to wait for a lot more decades! Back on topic! Whee! From stephen at xemacs.org Fri Sep 2 09:28:16 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 02 Sep 2011 16:28:16 +0900 Subject: [Python-ideas] allow line break at operators In-Reply-To: References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4e424208$0$29965$c3e8da3$5496439d@news.astraweb.com> <1312981104.89312.YahooMailNeo@web121520.mail.ne1.yahoo.com> <1312982377.95657.YahooMailNeo@web121508.mail.ne1.yahoo.com> <1313031175.38817.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> Message-ID: <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> Gabriel AHTUNE writes: > So can be done with this syntax: > > > x = firstpart * secondpart + #line breaks here > > anotherpart + #continue > > stillanother #continue on. > > after a "+" operator the line is clearly not finished yet. Sure, but IIRC one design principle of Python is that the keyword that denotes the syntax should be the first thing on the line, making it easy to scan down the left side of the code to see the syntactic structure. The required indentation of the controlled suite also helps emphasize that keyword. Analogously, if operators are going to denote continuation, they should come first on the line. I just don't think this idea is going anywhere. Explicit continuation with backslash or implicit continuation of parenthesized expressions is just not that heavy a price to pay. Perhaps historically some of these ideas could have been implemented, but now they're just going to confuse a host of editors and code analysis tools. From steve at pearwood.info Fri Sep 2 10:30:41 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 02 Sep 2011 18:30:41 +1000 Subject: [Python-ideas] aliasing In-Reply-To: <878vq7jwej.fsf@uwakimon.sk.tsukuba.ac.jp> References: <6BA1535A-8DB5-425A-94B5-8AD3AFA96AEE@gmail.com> <4E602263.2000109@canterbury.ac.nz> <4E602E94.20505@mrabarnett.plus.com> <87aaank2su.fsf@uwakimon.sk.tsukuba.ac.jp> <4E60677E.8060602@pearwood.info> <878vq7jwej.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4E609431.8090005@pearwood.info> Stephen J. Turnbull wrote: > Steven D'Aprano writes: > > Stephen J. Turnbull wrote: > > > MRAB writes: > > > > > > > Fortran also pre-dated stack allocation (I think), so local storage was > > > > static (and no recursion!). > > > > > > If FORTRAN [sic] predated stack allocation, what did Lisp use to > > > handle recursion? > > > > I don't see the connection. The first proposal for Fortran was 1953, > > with the first public release in 1957, while John McCarthy didn't start > > work on Lisp until 1958. > > He didn't start work on Lisp the language, or Lisp the interpreter? Define "start work on" :) At what point do idle musings about a possibility become actual work? According to McCarthy, early key ideas and experiments in Lisp occurred between 1956 and 1958, some of which using a Fortran-based back end (so I was wrong to say there was no connection -- McCarthy was certainly aware of Fortran and at least mildly influenced by its "formula translation" aspect). Between 1958 and 1962 Lisp was implemented and used for experiments in AI. http://www-formal.stanford.edu/jmc/history/lisp.ps In Fortran's case, I haven't been able to find when John Backus first started thinking about the idea, but he submitted a proposal to IBM in late 1953. A draft specification followed in 1954, followed by a manual in 1956 and finally a compiler in 1957. I guess the pace of computer software releases was slower back then, and people more forgiving of vapourware. -- Steven From guido at python.org Fri Sep 2 21:30:34 2011 From: guido at python.org (Guido van Rossum) Date: Fri, 2 Sep 2011 12:30:34 -0700 Subject: [Python-ideas] allow line break at operators In-Reply-To: <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4e424208$0$29965$c3e8da3$5496439d@news.astraweb.com> <1312981104.89312.YahooMailNeo@web121520.mail.ne1.yahoo.com> <1312982377.95657.YahooMailNeo@web121508.mail.ne1.yahoo.com> <1313031175.38817.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Fri, Sep 2, 2011 at 12:28 AM, Stephen J. Turnbull wrote: > Gabriel AHTUNE writes: > ?> So can be done with this syntax: > ?> > ?> > x = firstpart * secondpart ?+ ?#line breaks here > ?> > anotherpart + #continue > ?> > stillanother #continue on. > ?> > ?> after a "+" operator the line is clearly not finished yet. > > Sure, but IIRC one design principle of Python is that the keyword that > denotes the syntax should be the first thing on the line, making it > easy to scan down the left side of the code to see the syntactic > structure. ?The required indentation of the controlled suite also > helps emphasize that keyword. That's true for *statements* (except assignments and calls). > Analogously, if operators are going to denote continuation, they > should come first on the line. That doesn't follow. My preferred style is actually to put the binary operator at the end of the line. This also matches the prevailing style for breaking lines after commas (a comma can be seen as a kind of binary operator). > I just don't think this idea is going anywhere. ?Explicit continuation > with backslash or implicit continuation of parenthesized expressions > is just not that heavy a price to pay. ?Perhaps historically some of > these ideas could have been implemented, but now they're just going to > confuse a host of editors and code analysis tools. Totally agreed that this isn't going to happen. -- --Guido van Rossum (python.org/~guido) From stephen at xemacs.org Sat Sep 3 06:38:15 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 03 Sep 2011 13:38:15 +0900 Subject: [Python-ideas] allow line break at operators In-Reply-To: References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4e424208$0$29965$c3e8da3$5496439d@news.astraweb.com> <1312981104.89312.YahooMailNeo@web121520.mail.ne1.yahoo.com> <1312982377.95657.YahooMailNeo@web121508.mail.ne1.yahoo.com> <1313031175.38817.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87bov2jl54.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > On Fri, Sep 2, 2011 at 12:28 AM, Stephen J. Turnbull wrote: > > Sure, but IIRC one design principle of Python is that the keyword that > > denotes the syntax should be the first thing on the line, [...] > That's true for *statements* (except assignments and calls). > > > Analogously, if operators are going to denote continuation, they > > should come first on the line. > That doesn't follow. Agreed, it's not a logical implication. The analogy is only an analogy, but my eyes do work that way. My conclusion is that we shouldn't try to encourage either style, because people "see" continuation differently. Legislating a style isn't going to change that, I think. From stephen at xemacs.org Sat Sep 3 08:10:26 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 03 Sep 2011 15:10:26 +0900 Subject: [Python-ideas] allow line break at operators In-Reply-To: <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4e424208$0$29965$c3e8da3$5496439d@news.astraweb.com> <1312981104.89312.YahooMailNeo@web121520.mail.ne1.yahoo.com> <1312982377.95657.YahooMailNeo@web121508.mail.ne1.yahoo.com> <1313031175.38817.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> Message-ID: <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> Yingjie Lan writes: > Have you considered line continuation by indentation? It seems to > meet the design principle. I think it is the most natural way to > allow free line breaking in Python. Briefly, yes, and I think it would need a lot of tuning and probably complex rules. Unlike statements, where everybody (except the judges of the Obfuscated C Contest) agrees on a simple rule: "In a control structure, the controlled suite should be uniformly indented one level", line breaking and indentation of long expressions is an art, and people have different opinions on "readability" and "beauty." Achieving a compromise that is workable even for a few major styles is likely to be annoying and bug-prone. Pretty much every program I write seems to have a continued list of data or a multi-line dictionary display as data. It's not unusual for me to comment the formal arguments in a function definition, or the parent classes of a class definition. The exception for parenthesized objects is something I depend on for what I consider good style. Of course I could use explicit continuation, but in a long table that's ugly and error-prone. Long expressions that need to be broken across lines, on the other hand, often indication that I haven't thought carefully enough about that component of the program, and an extra pair of parentheses or a terminal backslash just isn't that "heavy" or ugly in the context of such long expressions. For me, they're also pretty rare; many programs I write have no explicit continuations in them at all. YMMV, of course, but I find the compromise that Python arrived at to be very useful, and I must suppose that it was substantially easier to implement than "fully free" line breaking (whatever that means to you). From wickedgrey at gmail.com Sat Sep 3 10:59:57 2011 From: wickedgrey at gmail.com (Eli Stevens (Gmail)) Date: Sat, 3 Sep 2011 01:59:57 -0700 Subject: [Python-ideas] Improving this? - foo() takes at least 6 arguments (6 given) In-Reply-To: <4E5FBEBC.1030604@stoneleaf.us> References: <3587ACF7-8B90-40B4-89E1-BC9AF2D4E5CC@pointcircle.com> <4E5FBEBC.1030604@stoneleaf.us> Message-ID: On Thu, Sep 1, 2011 at 10:19 AM, Ethan Furman wrote: >> Colleague: ?"So my code is apparently asking for 6 arguments and it >> appears that I gave it 6 arguments" >> Me: ?"Oh that - Yeah... your short one. ?It really wants 7.." > > How did you get this? Having seen this bite someone recently, I have an example readily at hand: >>> def f(a,b,c,d=None): pass ... >>> f(1,2,d=4) Traceback (most recent call last): File "", line 1, in TypeError: f() takes at least 3 arguments (3 given) The mention of self might be a bit of a red herring, since unless there's another way to generate the same error message, the error really is "f() takes at least 3 arguments (2 given, plus 1 keyword argument not counted here)". Cheers, Eli From stephen at xemacs.org Sat Sep 3 11:29:03 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 03 Sep 2011 18:29:03 +0900 Subject: [Python-ideas] allow line break at operators In-Reply-To: <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4e424208$0$29965$c3e8da3$5496439d@news.astraweb.com> <1312981104.89312.YahooMailNeo@web121520.mail.ne1.yahoo.com> <1312982377.95657.YahooMailNeo@web121508.mail.ne1.yahoo.com> <1313031175.38817.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> Message-ID: <878vq6j7og.fsf@uwakimon.sk.tsukuba.ac.jp> Yingjie Lan writes: > Python uses indentation for blocks, and by the same mechanism, line > breaking can be?accommodated without requiring parenthesis or > ending backslashes. Possibly, but now you have a problem that a dedent has ambiguous meaning. It might mean that you're ending a suite, or it might mean you're ending a continued expression. This probably can be disambiguated, but I don't know how easy that will be to do perfectly, including in reporting ill-formed programs. > Most people seems to like an indentation on the continuing lines, Most of the time, yes, but sometimes not. For example, in generating text, it's often useful to dedent substantially so you can have a nearly normal length line in the literal strings being concatenated. Or you might have a pattern like this: x = long_named_variable_a - long_named_variable_a_base + long_named_variable_b - long_named_variable_b_base which your parser would raise an error on, I presume. That's not freedom! From dickinsm at gmail.com Sat Sep 3 13:02:34 2011 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 3 Sep 2011 12:02:34 +0100 Subject: [Python-ideas] Improving this? - foo() takes at least 6 arguments (6 given) In-Reply-To: References: <3587ACF7-8B90-40B4-89E1-BC9AF2D4E5CC@pointcircle.com> <4E5FBEBC.1030604@stoneleaf.us> Message-ID: On Sat, Sep 3, 2011 at 9:59 AM, Eli Stevens (Gmail) wrote: > On Thu, Sep 1, 2011 at 10:19 AM, Ethan Furman wrote: >>> Colleague: ?"So my code is apparently asking for 6 arguments and it >>> appears that I gave it 6 arguments" >>> Me: ?"Oh that - Yeah... your short one. ?It really wants 7.." >> >> How did you get this? > > Having seen this bite someone recently, I have an example readily at hand: > >>>> def f(a,b,c,d=None): pass > ... >>>> f(1,2,d=4) > Traceback (most recent call last): > ?File "", line 1, in > TypeError: f() takes at least 3 arguments (3 given) Isn't this already fixed for Python 3.3? Python 3.3.0a0 (default:a8748022504f, Sep ?2 2011, 12:32:36) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> def f(a, b, c, d=None): pass ... [63593 refs] >>> f(1, 2, d=4) Traceback (most recent call last): ?File "", line 1, in TypeError: f() missing 1 required positional argument: 'c' [63624 refs] From benjamin at python.org Sat Sep 3 16:16:16 2011 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 3 Sep 2011 14:16:16 +0000 (UTC) Subject: [Python-ideas] Improving this? - foo() takes at least 6 arguments (6 given) References: <3587ACF7-8B90-40B4-89E1-BC9AF2D4E5CC@pointcircle.com> <4E5FBEBC.1030604@stoneleaf.us> Message-ID: Eli Stevens (Gmail writes: > > On Thu, Sep 1, 2011 at 10:19 AM, Ethan Furman wrote: > >> Colleague: ?"So my code is apparently asking for 6 arguments and it > >> appears that I gave it 6 arguments" > >> Me: ?"Oh that - Yeah... your short one. ?It really wants 7.." > > > > How did you get this? > > Having seen this bite someone recently, I have an example readily at hand: > > >>> def f(a,b,c,d=None): pass > ... > >>> f(1,2,d=4) > Traceback (most recent call last): > File "", line 1, in > TypeError: f() takes at least 3 arguments (3 given) > > The mention of self might be a bit of a red herring, since unless > there's another way to generate the same error message, the error > really is "f() takes at least 3 arguments (2 given, plus 1 keyword > argument not counted here)". Coming your way soon: Python 3.3.0a0 (default:fe0497bd7354, Sep 3 2011, 10:14:20) [GCC 4.4.5] on linux Type "help", "copyright", "credits" or "license" for more information. >>> def f(a,b,c,d=None): pass ... >>> f(1,2,d=4) Traceback (most recent call last): File "", line 1, in TypeError: f() missing 1 required positional argument: 'c' From ron3200 at gmail.com Sun Sep 4 17:22:32 2011 From: ron3200 at gmail.com (ron3200) Date: Sun, 04 Sep 2011 10:22:32 -0500 Subject: [Python-ideas] allow line break at operators In-Reply-To: <87bov2jl54.fsf@uwakimon.sk.tsukuba.ac.jp> References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4e424208$0$29965$c3e8da3$5496439d@news.astraweb.com> <1312981104.89312.YahooMailNeo@web121520.mail.ne1.yahoo.com> <1312982377.95657.YahooMailNeo@web121508.mail.ne1.yahoo.com> <1313031175.38817.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <87bov2jl54.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1315149752.24283.18.camel@Gutsy> On Sat, 2011-09-03 at 13:38 +0900, Stephen J. Turnbull wrote: > Guido van Rossum writes: > > On Fri, Sep 2, 2011 at 12:28 AM, Stephen J. Turnbull wrote: > > > > Sure, but IIRC one design principle of Python is that the keyword that > > > denotes the syntax should be the first thing on the line, > [...] > > That's true for *statements* (except assignments and calls). > > > > > Analogously, if operators are going to denote continuation, they > > > should come first on the line. > > > That doesn't follow. > > Agreed, it's not a logical implication. The analogy is only an > analogy, but my eyes do work that way. > > My conclusion is that we shouldn't try to encourage either style, > because people "see" continuation differently. Legislating a style > isn't going to change that, I think. I like to start continued lines with an operator as well. I also think it helps me keep it in my head a bit easier when I do that. I think this is one of those areas where computers and people differ, but it may also depend on the persons native language as to what works better for them. Ron From python at mrabarnett.plus.com Sun Sep 4 19:59:31 2011 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 04 Sep 2011 18:59:31 +0100 Subject: [Python-ideas] allow line break at operators In-Reply-To: <1315130288.86666.YahooMailNeo@web121515.mail.ne1.yahoo.com> References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <1313031175.38817.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> <1315092177.54288.YahooMailNeo@web121504.mail.ne1.yahoo.com> <4E62DCC6.8000906@mrabarnett.plus.com> <1315130288.86666.YahooMailNeo@web121515.mail.ne1.yahoo.com> Message-ID: <4E63BC83.9000401@mrabarnett.plus.com> On 04/09/2011 00:22, Yingjie Lan wrote: > On 04/09/2011 03:04, MRAB wrote: >> I think that the rules would be: >> >> If a line ends with a colon and the next line is indented, then >> it's the start of a block, and the following lines which belong to >> that block have the same indent. >> >> If a line doesn't end with a colon but the next line is indented, >> then it's the start of a continuation, and the following lines >> which belong to that continuation have the same indent. >> >> In both cases there could be blocks nested in blocks and possibly >> continuations nested in continuations, as well as blocks nested in >> continuations and continuations nested in blocks. >> >> I'm not sure what the effect would be if you had mis-indented >> lines. For example, if a line was accidentally indented after a >> comment, then it would be treated as part of the comment. It's in >> cases like those that syntax colouring would be helpful. It would >> be a good idea to use an editor which could indicate in some way >> when a line is a continuation. >> > Thanks, I think that's the rule described in its full glory. > Currently I am not quite sure of the use case for continuation > nested in continuation -- it seems to be still a single continuation, > but it allows for some additional freedom in formatting the continued > line. Do you have other use cases for that? > I don't have a use-case, I was just wondering whether in this: first second third fourth "third" is a continuation, giving this: first second third fourth which has 2 continuations, leading to this: first second third fourth > For the case of mis-indentation, as demonstrated in your scenario, I > think it is better that the rule is not applied to a comment > continued onto the next line. The effect of a '#' only carries to > the end of a line, if one would like the next line to be a comment, > just use another '#'. It remains to consider a mis-indentation that > only involves code lines. However, that is not a new problem, so we > should not worry (it is like a sunken cost). > As well as still limiting a comment to a line, I'd also still limit a string literal (except a triple-quoted string literal) to a line. From ben+python at benfinney.id.au Mon Sep 5 00:26:49 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 05 Sep 2011 08:26:49 +1000 Subject: [Python-ideas] allow line break at operators References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <1313031175.38817.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> <1315092177.54288.YahooMailNeo@web121504.mail.ne1.yahoo.com> <4E62DCC6.8000906@mrabarnett.plus.com> <1315130288.86666.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E63BC83.9000401@mrabarnett.plus.com> Message-ID: <87liu4oseu.fsf@benfinney.id.au> MRAB writes: > I don't have a use-case, I was just wondering whether in this: > > first > second > third > fourth > > "third" is a continuation, giving this: > > first > second third > fourth > > which has 2 continuations, leading to this: > > first second third fourth I do this routinely in my code, using bracketing syntax. It's useful for visually showing the structure of moderately complex generator expressions or function calls, for instance. > As well as still limiting a comment to a line, I'd also still limit a > string literal (except a triple-quoted string literal) to a line. How many string literals do you count in the following statement? I count one: raise HoustonWeHaveAProblemError( "Lorem ipsum dolor sit amet," " consectetur adipiscing elit.") -- \ ?Creativity can be a social contribution, but only in so far as | `\ society is free to use the results.? ?Richard M. Stallman | _o__) | Ben Finney From ben+python at benfinney.id.au Mon Sep 5 00:39:39 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 05 Sep 2011 08:39:39 +1000 Subject: [Python-ideas] allow line break at operators References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <1313031175.38817.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> <1315092177.54288.YahooMailNeo@web121504.mail.ne1.yahoo.com> <4E62DCC6.8000906@mrabarnett.plus.com> <1315130288.86666.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E63BC83.9000401@mrabarnett.plus.com> <87liu4oseu.fsf@benfinney.id.au> Message-ID: <87fwkcortg.fsf@benfinney.id.au> Ben Finney writes: > MRAB writes: > > > As well as still limiting a comment to a line, I'd also still limit > > a string literal (except a triple-quoted string literal) to a line. > > How many string literals do you count in the following statement? I > count one: > > raise HoustonWeHaveAProblemError( > "Lorem ipsum dolor sit amet," > " consectetur adipiscing elit.") The Python compiler agrees with me: >>> import dis >>> def foo(): ... raise ValueError( ... "Lorem ipsum dolor sit amet," ... " consectetur adipiscing elit.") ... >>> dis.dis(foo) 2 0 LOAD_GLOBAL 0 (ValueError) 3 3 LOAD_CONST 1 ('Lorem ipsum dolor sit amet, consectetur adipiscing elit.') 6 CALL_FUNCTION 1 9 RAISE_VARARGS 1 12 LOAD_CONST 0 (None) 15 RETURN_VALUE So if you do mean ?limit a string literal to a line? to cover the above case, I disagree. I frequently use the fact that a single-quoted string literal can be broken over multiple lines like the above, to make code more readable. -- \ ?The Vatican is not a state.? a state must have territory. This | `\ is a palace with gardens, about as big as an average golf | _o__) course.? ?Geoffrey Robertson, 2010-09-18 | Ben Finney From python at mrabarnett.plus.com Mon Sep 5 01:01:08 2011 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 05 Sep 2011 00:01:08 +0100 Subject: [Python-ideas] allow line break at operators In-Reply-To: <87liu4oseu.fsf@benfinney.id.au> References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <1313031175.38817.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> <1315092177.54288.YahooMailNeo@web121504.mail.ne1.yahoo.com> <4E62DCC6.8000906@mrabarnett.plus.com> <1315130288.86666.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E63BC83.9000401@mrabarnett.plus.com> <87liu4oseu.fsf@benfinney.id.au> Message-ID: <4E640334.10309@mrabarnett.plus.com> On 04/09/2011 23:26, Ben Finney wrote: > MRAB writes: > >> I don't have a use-case, I was just wondering whether in this: >> >> first >> second >> third >> fourth >> >> "third" is a continuation, giving this: >> >> first >> second third >> fourth >> >> which has 2 continuations, leading to this: >> >> first second third fourth > > I do this routinely in my code, using bracketing syntax. It's useful for > visually showing the structure of moderately complex generator > expressions or function calls, for instance. > >> As well as still limiting a comment to a line, I'd also still limit a >> string literal (except a triple-quoted string literal) to a line. > > How many string literals do you count in the following statement? I > count one: > > raise HoustonWeHaveAProblemError( > "Lorem ipsum dolor sit amet," > " consectetur adipiscing elit.") > It depends on how you count them. :-) What I mean is that I'd still forbid a newline between the quotes. Is this acceptable? "Lorem ipsum dolor sit amet, consectetur adipiscing elit." I'd say not. From ben+python at benfinney.id.au Mon Sep 5 01:21:23 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 05 Sep 2011 09:21:23 +1000 Subject: [Python-ideas] allow line break at operators References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <1313031175.38817.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> <1315092177.54288.YahooMailNeo@web121504.mail.ne1.yahoo.com> <4E62DCC6.8000906@mrabarnett.plus.com> <1315130288.86666.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E63BC83.9000401@mrabarnett.plus.com> <87liu4oseu.fsf@benfinney.id.au> <4E640334.10309@mrabarnett.plus.com> Message-ID: <87aaajq4gc.fsf@benfinney.id.au> MRAB writes: > What I mean is that I'd still forbid a newline between the quotes. Is > this acceptable? > > "Lorem ipsum dolor sit amet, > consectetur adipiscing elit." > > I'd say not. I agree with you on that. But I don't know how to terminologically distinguish that from my example; they're both one string literal. Perhaps we who hold this position should say that ?bracketing syntax?, for the purpose of statement continuation, should not include string quotes ?'? or ?"?; we already have triple-quoted strings for that purpose. -- \ ?It is wrong to think that the task of physics is to find out | `\ how nature *is*. Physics concerns what we can *say* about | _o__) nature?? ?Niels Bohr | Ben Finney From hetchkay at gmail.com Mon Sep 5 06:55:10 2011 From: hetchkay at gmail.com (H Krishnan) Date: Mon, 5 Sep 2011 10:25:10 +0530 Subject: [Python-ideas] relaxing keyword usage restrictions Message-ID: Hi, Apologies if this has been asked already: why can't the usage of at least some keywords as identifiers be allowed with the context of the usage being used to identify if a keyword or a variable is being referred to. For example, if = 3 # if is identifier if if = 3: # first if is keyword, second if is identifer A more complex usage may require some rules: a = not or and and ==> parse left to right to interpret as a = ( (not or) and (and) ). 'or' and the last 'and' are identifiers While some of the usage may be confusing to the user, that is not for the language to dictate but for the user to choose. Almost all keywords have specific semantics which (in my limited understanding) do not conflict with their usage as identifiers. Regards, Krishnan From mwm at mired.org Mon Sep 5 07:18:40 2011 From: mwm at mired.org (Mike Meyer) Date: Sun, 04 Sep 2011 22:18:40 -0700 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: Message-ID: H Krishnan wrote: >Hi, > >Apologies if this has been asked already: why can't the usage of at >least some keywords as identifiers be allowed with the context of the >usage being used to identify if a keyword or a variable is being >referred to. > >For example, > >if = 3 # if is identifier >if if = 3: # first if is keyword, second if is identifer By itself, enough to show this is a bad idea. >While some of the usage may be confusing to the user, that is not for >the language to dictate but for the user to choose. Wrong. It's for the community to decide. The python community regularly decides to reject changes for no more reason than they might confuse a reader and provide no real benefit. Your examples show no benefit, and are obviously confusing. However, there may be some merit to the basic idea. In particular, allowing class to be used as a method parameter has an obvious use case (class methods), and would not produce those confusing cases you used as examples. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. From ncoghlan at gmail.com Mon Sep 5 07:50:20 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 5 Sep 2011 15:50:20 +1000 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: Message-ID: On Mon, Sep 5, 2011 at 3:18 PM, Mike Meyer wrote: > However, there may be some merit to the basic idea. In particular, allowing class to be used as a method parameter has an obvious use case (class methods), and would not produce those confusing cases you used as examples. The main advantage of making things out-and-out keywords is that it greatly simplifies the task of *parsing* the language. It's a trade-off between occasional inconvenience in users having to choose slightly clunky variable and attribute names (such as 'cls', 'class_' or 'klass') and the simplicity Python parser developers gain by defining certain words as keywords that can never appear as variable names. One benefit users gain directly from that simplicity is that it makes syntax highlighters more likely to work correctly - the highlighters can just always mark keywords as keywords, without needing to worry about the usage context. It's easy to say "hey, let's allow keywords to be identifiers in places where they aren't being used as syntax". However, it doesn't look like such a great idea when you start considering the amount of change that would be needed to update the likes CPython, PyPy, IronPython, Jython, Cython/Pyrex, assorted syntax highlighting rules for various editors, Pygments, etc, etc. So, not impossible, but not worth the hassle either, especially since it would actually make the language *harder* to learn. Learning not to use keywords as variable names is fairly easy. Learning the ins and outs of when pseudo-keywords could be used as identifiers and when they were disallowed would be annoying. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ben+python at benfinney.id.au Mon Sep 5 08:19:46 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 05 Sep 2011 16:19:46 +1000 Subject: [Python-ideas] relaxing keyword usage restrictions References: Message-ID: <87wrdno6il.fsf@benfinney.id.au> H Krishnan writes: > While some of the usage may be confusing to the user, that is not for > the language to dictate but for the user to choose. This statement makes the entirely unjustified assumption that the code will be read by exactly one ?user?. Python is a language designed for writing code that isn't limited to just one reader. So no, every individual author of code doesn't get to choose what is readable for others; the Python community as a whole has a large say in that. -- \ ?There are no chaplains in foxholes.? ?Sergeant Justin | `\ Griffith, 2011-07-27 | _o__) | Ben Finney From hetchkay at gmail.com Mon Sep 5 09:23:32 2011 From: hetchkay at gmail.com (H Krishnan) Date: Mon, 5 Sep 2011 12:53:32 +0530 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: Message-ID: > > Wrong. It's for the community to decide. The python community regularly decides to reject changes for no more reason than they might confuse a reader and provide no real benefit. Your examples show no benefit, and are obviously confusing. > Not sure I understand this point very well. Could I start a separate thread to say that Python should grab 'was' and 'were' as keywords so that a user is not able to write: if was is were: was = not were Sorry, that was tongue in cheek... Regards, Krishnan From stephen at xemacs.org Mon Sep 5 10:41:11 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 05 Sep 2011 17:41:11 +0900 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: Message-ID: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> H Krishnan writes: > > Wrong. It's for the community to decide. The python community > > regularly decides to reject changes for no more reason than they > > might confuse a reader and provide no real benefit. Your examples > > show no benefit, and are obviously confusing. > Not sure I understand this point very well. Could I start a separate > thread to say that Python should grab 'was' and 'were' as keywords so > that a user is not able to write: No, if you're all that serious about messing with the sematics of keywords, what you should do instead is skim PEPs 221, 318, 343, and 359,[1] and note how much attention is paid to the cost of adding keywords in the rationales. There may be other good examples. Executive summary of the PEP list: the cost of keywords like "class" is well understood; there was a deliberate decision that this cost would be paid. But there is also strong resistence to adding new ones that are not absolutely required to implement a new feature. These same considerations apply to any new syntax in the language; see PEP 3003. Also read PEP 20, aka "python -m this" aka "The Zen of Python." The Python community considers language development to be an art. Implicit in that philosophical position is the possibility that you may not like the result. It's also possible that you may be able to convince the community of your position. But that is going to be much more likely (and you are much less likely to be defeated by windmills) if you understand the philosophy and tastes behind Python language features. Some of them may not be obvious unless you are Dutch (I guess you aren't :-), so PEP 3009 is required reading. Footnotes: [1] http://www.python.org/dev/peps/ From g.rodola at gmail.com Mon Sep 5 11:36:28 2011 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Mon, 5 Sep 2011 11:36:28 +0200 Subject: [Python-ideas] Improving this? - foo() takes at least 6 arguments (6 given) In-Reply-To: References: <3587ACF7-8B90-40B4-89E1-BC9AF2D4E5CC@pointcircle.com> <4E5FBEBC.1030604@stoneleaf.us> Message-ID: 2011/9/3 Mark Dickinson : > On Sat, Sep 3, 2011 at 9:59 AM, Eli Stevens (Gmail) > wrote: >> On Thu, Sep 1, 2011 at 10:19 AM, Ethan Furman wrote: >>>> Colleague: ?"So my code is apparently asking for 6 arguments and it >>>> appears that I gave it 6 arguments" >>>> Me: ?"Oh that - Yeah... your short one. ?It really wants 7.." >>> >>> How did you get this? >> >> Having seen this bite someone recently, I have an example readily at hand: >> >>>>> def f(a,b,c,d=None): pass >> ... >>>>> f(1,2,d=4) >> Traceback (most recent call last): >> ?File "", line 1, in >> TypeError: f() takes at least 3 arguments (3 given) > > Isn't this already fixed for Python 3.3? > > Python 3.3.0a0 (default:a8748022504f, Sep ?2 2011, 12:32:36) > [GCC 4.2.1 (Apple Inc. build 5664)] on darwin > Type "help", "copyright", "credits" or "license" for more information. >>>> def f(a, b, c, d=None): pass > ... > [63593 refs] >>>> f(1, 2, d=4) > Traceback (most recent call last): > ?File "", line 1, in > TypeError: f() missing 1 required positional argument: 'c' > [63624 refs] > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > This is a very good improvement I wasn't aware of! Glad to see that. Regards, --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ From ncoghlan at gmail.com Mon Sep 5 12:26:51 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 5 Sep 2011 20:26:51 +1000 Subject: [Python-ideas] Improving this? - foo() takes at least 6 arguments (6 given) In-Reply-To: References: <3587ACF7-8B90-40B4-89E1-BC9AF2D4E5CC@pointcircle.com> <4E5FBEBC.1030604@stoneleaf.us> Message-ID: On Mon, Sep 5, 2011 at 7:36 PM, Giampaolo Rodol? wrote: > This is a very good improvement I wasn't aware of! > Glad to see that. You can see some more examples of the results of Benjamin's work on this problem in the updated test suite: http://hg.python.org/cpython/file/31b3218794e2/Lib/test/test_extcall.py#l298 Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From jeanpierreda at gmail.com Mon Sep 5 14:10:53 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 5 Sep 2011 08:10:53 -0400 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: Message-ID: The "as" keyword used to be a pseudokeyword that functioned this way. Apparently the experiment didn't go well, as it was changed to a full keyword in 2.6. Devin On Mon, Sep 5, 2011 at 12:55 AM, H Krishnan wrote: > Hi, > > Apologies if this has been asked already: why can't the usage of at > least some keywords as identifiers be allowed with the context of the > usage being used to identify if a keyword or a variable is being > referred to. > > For example, > > if = 3 ?# if is identifier > if if = 3: # first if is keyword, second if is identifer > > A more complex usage may require some rules: > > a = not or and and > ==> parse left to right to interpret as a = ( (not or) and (and) ). > 'or' and the last 'and' are identifiers > > While some of the usage may be confusing to the user, that is not for > the language to dictate but for the user to choose. > > Almost all keywords have specific semantics which (in my limited > understanding) do not conflict with their usage as identifiers. > > Regards, > Krishnan > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From jkbbwr at gmail.com Mon Sep 5 17:16:35 2011 From: jkbbwr at gmail.com (Jakob Bowyer) Date: Mon, 5 Sep 2011 16:16:35 +0100 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: Message-ID: -1 to the concept. It seems to create readability issues? I mean this sounds bad really if if == 3: On Mon, Sep 5, 2011 at 1:10 PM, Devin Jeanpierre wrote: > The "as" keyword used to be a pseudokeyword that functioned this way. > Apparently the experiment didn't go well, as it was changed to a full > keyword in 2.6. > > Devin > > On Mon, Sep 5, 2011 at 12:55 AM, H Krishnan wrote: > > Hi, > > > > Apologies if this has been asked already: why can't the usage of at > > least some keywords as identifiers be allowed with the context of the > > usage being used to identify if a keyword or a variable is being > > referred to. > > > > For example, > > > > if = 3 # if is identifier > > if if = 3: # first if is keyword, second if is identifer > > > > A more complex usage may require some rules: > > > > a = not or and and > > ==> parse left to right to interpret as a = ( (not or) and (and) ). > > 'or' and the last 'and' are identifiers > > > > While some of the usage may be confusing to the user, that is not for > > the language to dictate but for the user to choose. > > > > Almost all keywords have specific semantics which (in my limited > > understanding) do not conflict with their usage as identifiers. > > > > Regards, > > Krishnan > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > http://mail.python.org/mailman/listinfo/python-ideas > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Sep 6 00:05:30 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 06 Sep 2011 10:05:30 +1200 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: Message-ID: <4E6547AA.4010200@canterbury.ac.nz> Devin Jeanpierre wrote: > The "as" keyword used to be a pseudokeyword that functioned this way. > Apparently the experiment didn't go well, I think that was only done because the __future__ mechanism hadn't been invented then. As far as I know, it was always intended to become a full keyword at some point. -- Greg From stephen at xemacs.org Tue Sep 6 04:26:32 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 06 Sep 2011 11:26:32 +0900 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: <4E6547AA.4010200@canterbury.ac.nz> References: <4E6547AA.4010200@canterbury.ac.nz> Message-ID: <87r53uiexz.fsf@uwakimon.sk.tsukuba.ac.jp> Greg Ewing writes: > Devin Jeanpierre wrote: > > The "as" keyword used to be a pseudokeyword that functioned this way. > > Apparently the experiment didn't go well, > > I think that was only done because the __future__ mechanism > hadn't been invented then. As far as I know, it was always > intended to become a full keyword at some point. That may be true, but the PEP says "this word 'as' is not intended to become a keyword." For my point it doesn't matter exactly what the intention was; what matters is that people pay attention to the cost of reserving keywords globally, but also accept the need for a few keywords. From masklinn at masklinn.net Tue Sep 6 17:00:56 2011 From: masklinn at masklinn.net (Masklinn) Date: Tue, 6 Sep 2011 17:00:56 +0200 Subject: [Python-ideas] xmlrpc.client to deprecate Binary for bytes? Message-ID: Not sure if this is an actual idea/feature request or a bug. Might be half/half In Python 2, many actual strings were "packaged" through the str type, so xmlrpclib had to provide an other wrapper (xmlrpclib.Binary) to differentiate between actual strings (to insert as-is) and binary blobs, to b64encode. Among Python 3 fixes are the stricter separation between bytestrings (bytes) and text strings (str), but xmlrpc.client *still* requires the usage of a Binary wrapper, and trying to encode a bytes instance just blows up: >>> xmlrpc.client.dumps((b'foo',)) Traceback (most recent call last): File "", line 1, in File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/xmlrpc/client.py", line 956, in dumps data = m.dumps(params) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/xmlrpc/client.py", line 506, in dumps dump(v, write) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/xmlrpc/client.py", line 530, in __dump f(self, value, write) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/xmlrpc/client.py", line 569, in dump_string write(escape(value)) File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/xmlrpc/client.py", line 156, in escape s = s.replace("&", "&") TypeError: expected an object with the buffer interface >>> xmlrpc.client.dumps((xmlrpc.client.Binary(b'foo'),)) '\n\n\nZm9v\n\n\n\n\n' I think it would be worthwhile to do the following: * Have xmlrpc.client treat dumping `bytes` (and related objects, such as bytearray or memoryview) as if they were `Binary`-wrapped (by b64encoding them and shipping them as binary data, which they are) * Add a flag to xmlrpc.client's `ServerProxy` and `loads` to have binary data ( nodes of xmlrpc) decode to `bytes` rather than `Binary` instances. This option would likely be `False` by default, at least to start with, so as not to break compatibility with Python 3.1 and 3.2 code. PS: as seen above, xmlrpc claims to require objects "with the buffer interface", Python's documentation (http://docs.python.org/py3k/library/stdtypes.html?highlight=memoryview#memoryview) claims bytes and bytearray implement "the buffer protocol". Who is lying? From ethan at stoneleaf.us Tue Sep 6 18:51:36 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 06 Sep 2011 09:51:36 -0700 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: Message-ID: <4E664F98.2040509@stoneleaf.us> H Krishnan wrote: > Hi, > > Apologies if this has been asked already: why can't the usage of at > least some keywords as identifiers be allowed with the context of the > usage being used to identify if a keyword or a variable is being > referred to. > > For example, > > if = 3 # if is identifier > if if = 3: # first if is keyword, second if is identifer Awkward, confusing, horrible. -1 From solipsis at pitrou.net Tue Sep 6 18:35:51 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 6 Sep 2011 18:35:51 +0200 Subject: [Python-ideas] relaxing keyword usage restrictions References: <4E664F98.2040509@stoneleaf.us> Message-ID: <20110906183551.40f00cde@pitrou.net> On Tue, 06 Sep 2011 09:51:36 -0700 Ethan Furman wrote: > H Krishnan wrote: > > Hi, > > > > Apologies if this has been asked already: why can't the usage of at > > least some keywords as identifiers be allowed with the context of the > > usage being used to identify if a keyword or a variable is being > > referred to. > > > > For example, > > > > if = 3 # if is identifier > > if if = 3: # first if is keyword, second if is identifer > > > Awkward, confusing, horrible. We really need an enum class. Antoine. From barry at python.org Tue Sep 6 22:41:03 2011 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Sep 2011 16:41:03 -0400 Subject: [Python-ideas] relaxing keyword usage restrictions References: Message-ID: <20110906164103.772ed18b@resist.wooz.org> On Sep 05, 2011, at 03:50 PM, Nick Coghlan wrote: >The main advantage of making things out-and-out keywords is that it >greatly simplifies the task of *parsing* the language. While I agree with this, and the other justifications Nick mentions, I *could* see a potential middle ground, by allowing keywords as attributes. I think I'd still be -0 (at least) on it, but IIRC Jython did allow these at one point (e.g. `foo.class`), mostly through an accident of its implementation. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Tue Sep 6 23:54:05 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Sep 2011 07:54:05 +1000 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: <20110906164103.772ed18b@resist.wooz.org> References: <20110906164103.772ed18b@resist.wooz.org> Message-ID: On Wed, Sep 7, 2011 at 6:41 AM, Barry Warsaw wrote: > On Sep 05, 2011, at 03:50 PM, Nick Coghlan wrote: > >>The main advantage of making things out-and-out keywords is that it >>greatly simplifies the task of *parsing* the language. > > While I agree with this, and the other justifications Nick mentions, I *could* > see a potential middle ground, by allowing keywords as attributes. ?I think > I'd still be -0 (at least) on it, but IIRC Jython did allow these at one point > (e.g. `foo.class`), mostly through an accident of its implementation. The main objection to that approach is that it breaks down when you go to explicitly define class (or module) attributes that make use of the feature: class Foo: for = "Demonstrating a flaw in the 'keywords as attributes' concept" While there's a definite appeal to the idea of allowing keywords-as-attributes, it comes at the cost of breaking the symmetry between the identifiers that are permitted in module and class definition code and the attributes that are accessible via those namespaces. You could answer "but the feature is for programmatically generated attributes, such as CSV column names, not ones you type in a class definition" and make a reasonable case for the feature on that basis, but the conceptual symmetry is still broken. Ultimately, it's the "keep the rules simple" argument that holds the most weight with me on this topic, so the twin rules of "keywords can only be used as keywords" and "attribute names must be legal identifiers" end up being my preferred approach. The occasional inconvenience of a deliberate misspelling or the addition of a trailing underscore to convert a keyword into a legal identifier seems like a worthwhile trade-off for the corresponding simplification of the language syntax and semantics. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From barry at python.org Wed Sep 7 00:33:46 2011 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Sep 2011 18:33:46 -0400 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <20110906164103.772ed18b@resist.wooz.org> Message-ID: <20110906183346.2bcdfbf8@resist.wooz.org> On Sep 07, 2011, at 07:54 AM, Nick Coghlan wrote: >Ultimately, it's the "keep the rules simple" argument that holds the >most weight with me on this topic, so the twin rules of "keywords can >only be used as keywords" and "attribute names must be legal >identifiers" end up being my preferred approach. The occasional >inconvenience of a deliberate misspelling or the addition of a >trailing underscore to convert a keyword into a legal identifier seems >like a worthwhile trade-off for the corresponding simplification of >the language syntax and semantics. Still not disagreeing with you, but in some sense, you *can* use keywords as attributes today: Python 3.2.2 (default, Sep 5 2011, 21:17:14) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> class Foo: pass ... >>> f = Foo() >>> setattr(f, 'class', 'evil') >>> f.class File "", line 1 f.class ^ SyntaxError: invalid syntax >>> getattr(f, 'class') 'evil' >>> >>> dir(f) ['__class__', '__delattr__', '__dict__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'class'] You just have to use "weird" syntax to set and get the values of those attributes. ;) So much for consistency. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From anacrolix at gmail.com Wed Sep 7 01:44:09 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Wed, 7 Sep 2011 09:44:09 +1000 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: <20110906183346.2bcdfbf8@resist.wooz.org> References: <20110906164103.772ed18b@resist.wooz.org> <20110906183346.2bcdfbf8@resist.wooz.org> Message-ID: I was -1 until you brought up consistency. I'm going to wave the flag of "least surprising behaviour" and change my vote to +0. Yes it makes it harder to parse, but it's easier for humans :P On Wed, Sep 7, 2011 at 8:33 AM, Barry Warsaw wrote: > On Sep 07, 2011, at 07:54 AM, Nick Coghlan wrote: > >>Ultimately, it's the "keep the rules simple" argument that holds the >>most weight with me on this topic, so the twin rules of "keywords can >>only be used as keywords" and "attribute names must be legal >>identifiers" end up being my preferred approach. The occasional >>inconvenience of a deliberate misspelling or the addition of a >>trailing underscore to convert a keyword into a legal identifier seems >>like a worthwhile trade-off for the corresponding simplification of >>the language syntax and semantics. > > Still not disagreeing with you, but in some sense, you *can* use keywords as > attributes today: > > Python 3.2.2 (default, Sep ?5 2011, 21:17:14) > [GCC 4.6.1] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> class Foo: pass > ... >>>> f = Foo() >>>> setattr(f, 'class', 'evil') >>>> f.class > ?File "", line 1 > ? ?f.class > ? ? ? ? ?^ > SyntaxError: invalid syntax >>>> getattr(f, 'class') > 'evil' >>>> >>>> dir(f) > ['__class__', '__delattr__', '__dict__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'class'] > > You just have to use "weird" syntax to set and get the values of those > attributes. ;) ?So much for consistency. > > -Barry > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > From ncoghlan at gmail.com Wed Sep 7 02:21:59 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Sep 2011 10:21:59 +1000 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: <20110906183346.2bcdfbf8@resist.wooz.org> References: <20110906164103.772ed18b@resist.wooz.org> <20110906183346.2bcdfbf8@resist.wooz.org> Message-ID: On Wed, Sep 7, 2011 at 8:33 AM, Barry Warsaw wrote: > Still not disagreeing with you, but in some sense, you *can* use keywords as > attributes today: You can actually use arbitrary strings, such as "1234" and "not a legal identifier" that way, so I see that behaviour as still being consistent. Some implementations, including CPython, even let you use arbitrary non-string objects via the attribute dict interface, but whether or not that works has been explicitly deemed an implementation detail by Guido. If you treat a namespace like a string-keyed dictionary, you can use any string, even those that aren't valid identifiers. Treat it like an actual namespace, though, and the identifier restrictions come into play (including those disallowing the use of keywords). While it is indeed another rule, it's still a fairly simple and consistent one, since the identifier rules are there to allow consistent parsing without dedicated delimiters. That does give me an idea, though. Rather than allowing keywords-as-attributes, it may make more sense to allow string literals to stand in for identifiers without obeying the normal rules, permitting things like: class Foo: normal = 1 'class' = 'This is probably a terrible idea' '1234' = 'as is this' 'or does it' = 'have some merit?' >>> Foo.normal 1 >>> Foo.'normal' 1 >>> Foo.'class' 'This is probably a terrible idea' >>> Foo.'1234' 'as is this' >>> Foo.'or does it' 'have some merit?' I'm still -1 myself, but such a delimited approach at least has the virtue of addressing the *entire* scope of the identifier syntax limitation rather than singling out keywords for special treatment. Legitimate use cases for such a feature would include at least those currently handled by the 'rename' flag on the namedtuple constructor (the latter would still be needed to handle duplicate field names, but illegal identifiers could just be quoted). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From steve at pearwood.info Wed Sep 7 02:31:34 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 07 Sep 2011 10:31:34 +1000 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <20110906164103.772ed18b@resist.wooz.org> <20110906183346.2bcdfbf8@resist.wooz.org> Message-ID: <4E66BB66.3030409@pearwood.info> Matt Joiner wrote: > I was -1 until you brought up consistency. I'm going to wave the flag > of "least surprising behaviour" and change my vote to +0. Yes it makes > it harder to parse, but it's easier for humans :P I see nothing "least surprising" about allowing keywords to be used as identifiers. I see that as causing more, not less, confusion in a language that allows me to write things like: from as import import as from for for in in: if = elif if if else else if break: continue elif continue: break Easier for humans to parse? I don't think so. I think this is an anti-feature that is good for obfuscating Python code, and nothing else. If "consistency" is an argument in favour for allowing code like the above, then it's a foolish consistency. In my opinion, we can't save this idea by only allowing keywords as attributes. If you want to talk about surprising behaviour, try explaining to a beginner why of = instance.of is allowed, but if = instance.if is prohibited. -- Steven From ghostwriter402 at gmail.com Wed Sep 7 02:51:29 2011 From: ghostwriter402 at gmail.com (Spectral One) Date: Tue, 06 Sep 2011 19:51:29 -0500 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <20110906164103.772ed18b@resist.wooz.org> Message-ID: <4E66C011.6070905@gmail.com> Seems that unadorned keywords and adorned keywords could be a touch independent. By that I mean the the following could be illegal: if if>5: pass #...and... class thingy: class="example of bad syntax" while possible to allow the following: class thingy: thingy.class="example of possible syntax" However, even if you could, I don't think you should. I just don't see sufficient value in this idea to be worth the change. -1 ( I guess that's how that works) -Nate From greg.ewing at canterbury.ac.nz Wed Sep 7 02:56:04 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 07 Sep 2011 12:56:04 +1200 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <20110906164103.772ed18b@resist.wooz.org> Message-ID: <4E66C124.3050806@canterbury.ac.nz> Nick Coghlan wrote: > You could answer "but the feature is for programmatically generated > attributes, such as CSV column names, I'd argue that it's for interfacing with libraries written in a different language. Your point about consistency still stands, but consistency doesn't necessarily trump all other considerations. Having to translate the API of a wrapped library by prepending underscores, or whatever convention is chosen, carries a mental burden as well. Not only do you have to remember to do it, you have to remember what convention the author of the wrapper used. -- Greg From greg.ewing at canterbury.ac.nz Wed Sep 7 03:01:47 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 07 Sep 2011 13:01:47 +1200 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <20110906164103.772ed18b@resist.wooz.org> <20110906183346.2bcdfbf8@resist.wooz.org> Message-ID: <4E66C27B.2080906@canterbury.ac.nz> Nick Coghlan wrote: > class Foo: > normal = 1 > 'class' = 'This is probably a terrible idea' > '1234' = 'as is this' > 'or does it' = 'have some merit?' Could be rather confusing, since the interpretation of a string literal would depend somewhat subtly on its position in an expression. And it's not just a LHS/RHS distinction, because presumably the following would still parse with all the string literals being literals... or would it? 'foo'['blarg'] = 'asdf' -- Greg From stephen at xemacs.org Wed Sep 7 03:15:32 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 07 Sep 2011 10:15:32 +0900 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: <20110906183346.2bcdfbf8@resist.wooz.org> References: <20110906164103.772ed18b@resist.wooz.org> <20110906183346.2bcdfbf8@resist.wooz.org> Message-ID: <87aaahi24r.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > >>> setattr(f, 'class', 'evil') > You just have to use "weird" syntax to set and get the values of those > attributes. ;) So much for consistency. I don't see this as anything but an optimization of dict. You could easily encapsulate the syntax above in a class, no? From greg.ewing at canterbury.ac.nz Wed Sep 7 03:13:49 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 07 Sep 2011 13:13:49 +1200 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: <4E66C011.6070905@gmail.com> References: <20110906164103.772ed18b@resist.wooz.org> <4E66C011.6070905@gmail.com> Message-ID: <4E66C54D.90900@canterbury.ac.nz> Spectral One wrote: > class thingy: > thingy.class="example of possible syntax" That wouldn't work, BTW, because the class name is not bound until after the class body code has been executed. -- Greg From barry at python.org Wed Sep 7 03:57:17 2011 From: barry at python.org (Barry Warsaw) Date: Tue, 6 Sep 2011 21:57:17 -0400 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <20110906164103.772ed18b@resist.wooz.org> <20110906183346.2bcdfbf8@resist.wooz.org> Message-ID: <20110906215717.62adf161@resist.wooz.org> On Sep 07, 2011, at 10:21 AM, Nick Coghlan wrote: >>> class Foo: ... """I'm still -1 myself, but such a delimited approach at least has the ... virtue of addressing the *entire* scope of the identifier syntax ... limitation rather than singling out keywords for special treatment. ... ... Legitimate use cases for such a feature would include at least those ... currently handled by the 'rename' flag on the namedtuple constructor ... (the latter would still be needed to handle duplicate field names, ... but illegal identifiers could just be quoted). ... """ = False frightening-ly y'rs, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Wed Sep 7 04:47:32 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Sep 2011 12:47:32 +1000 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: <20110906215717.62adf161@resist.wooz.org> References: <20110906164103.772ed18b@resist.wooz.org> <20110906183346.2bcdfbf8@resist.wooz.org> <20110906215717.62adf161@resist.wooz.org> Message-ID: On Wed, Sep 7, 2011 at 11:57 AM, Barry Warsaw wrote: > On Sep 07, 2011, at 10:21 AM, Nick Coghlan wrote: > >>>> class Foo: > ... ? ?"""I'm still -1 myself, but such a delimited approach at least has the > ... ? ?virtue of addressing the *entire* scope of the identifier syntax > ... ? ?limitation rather than singling out keywords for special treatment. > ... > ... ? ?Legitimate use cases for such a feature would include at least those > ... ? ?currently handled by the 'rename' flag on the namedtuple constructor > ... ? ?(the latter would still be needed to handle duplicate field names, > ... ? ?but illegal identifiers could just be quoted). > ... ? ?""" = False > > frightening-ly y'rs, Even worse: class Foo: '' = True Utterly untenable :) It would also be horribly ambiguous in several places where strings and identifiers are already both allowed, so the full concept fails on multiple counts. That means the scope would have to be limited to potentially supporting string literals on the RHS of the dot operator. Essentially it would just become further syntactic sugar for the getattr(), setattr() and delattr() builtins, with the string delimiters remaining optional for valid identifiers. I still don't really like the idea, but I have to admit that it would at least arguably be more elegant than attribute renaming or builtin function based workarounds for attribute names from external sources that aren't legal Python identifiers. As with other recurring discussions, I suggest that someone particularly interested in the topic put together a PEP so we can avoid rehashing the same details every couple of years - even rejected PEPs can serve as useful repositories for the reasons why certain things *aren't* done. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 7 07:26:03 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 7 Sep 2011 15:26:03 +1000 Subject: [Python-ideas] Deprecating bytes.swapcase and friends [was: Maintenance burden of str.swapcase] In-Reply-To: <874o0phstx.fsf@uwakimon.sk.tsukuba.ac.jp> References: <785989C3-66EA-4ED9-B6D2-E55FE2A30DE8@voidspace.org.uk> <87ehztip2r.fsf@uwakimon.sk.tsukuba.ac.jp> <4E66763B.7080707@pearwood.info> <4E668029.6080106@v.loewis.de> <7EA8F302-1DC4-43D9-B124-09A51172E9BF@voidspace.org.uk> <4E66845F.3060708@v.loewis.de> <4271E434-0E35-4D7E-98D0-097402B8C3FD@gmail.com> <877h5li0dk.fsf@uwakimon.sk.tsukuba.ac.jp> <874o0phstx.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Wed, Sep 7, 2011 at 2:36 PM, Stephen J. Turnbull wrote: > I don't know if it's worth the effort to deprecate them, though. I could live with a purely documentation based deprecation, although I'd prefer to *actually* deprecate at least those four methods on bytes and bytearray objects (since we switched mailing lists, reproducing the list for reference: 'capitalize', 'istitle', 'swapcase', 'title'). > There is a school of thought (represented on python-dev by Philip Eby > and Antoine Pitrou, among others, I would say) that says that text > with an implicit encoding is still text if you can figure out what the > encoding is, and the syntactically important tokens are invariably > ASCII, which often is enough information to do the work. ?So if you > can do some operation without first converting to str, let's save the > cycles and the bytes (especially in bit-shoveling applications like > WSGI)! ?I disagree, but "consenting adults" and all that. FWIW, I actually used to be in that school myself, *until* I took on the task of making more of the urllib.parse APIs take a polymorphic bytes-in-bytes-out, str-in-str-out approach for 3.2. The difference in complexity between the "right" way (i.e. decoding with the ascii codec, manipulating as Unicode, encoding back to bytes with the ascii codec) and a hackier approach that tried to manipulate the bytes directly was such that I didn't even end up benchmarking the two approaches to decide between them - I ended up having zero interest in attempting to maintain the latter version, so the implicit decode/encode is the version that went into the release. That experience pushed me solidly in the direction of arbitrary fast ASCII text manipulation without encoding/decoding overhead in Python 3 being a task for a third party type - neither bytes nor str fit the bill. To be really effective, such a type either needs algorithms dedicated to using it so that all the associated 'literals' are predefined as objects of the relevant type and don't need to worry about handling actual strings being passed in or else they need to transparently interoperate with builtin str objects. The potential viability and utility of such a tagged string type, however, isn't a particularly strong argument for anything relating to the bytes API - it's pretty clear that Guido's plan to break the 8-bit-data-as-text paradigm in Python 3 has succeeded to that extent. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From stephen at xemacs.org Wed Sep 7 06:36:26 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 07 Sep 2011 13:36:26 +0900 Subject: [Python-ideas] Deprecating bytes.swapcase and friends [was: Maintenance burden of str.swapcase] In-Reply-To: References: <785989C3-66EA-4ED9-B6D2-E55FE2A30DE8@voidspace.org.uk> <87ehztip2r.fsf@uwakimon.sk.tsukuba.ac.jp> <4E66763B.7080707@pearwood.info> <4E668029.6080106@v.loewis.de> <7EA8F302-1DC4-43D9-B124-09A51172E9BF@voidspace.org.uk> <4E66845F.3060708@v.loewis.de> <4271E434-0E35-4D7E-98D0-097402B8C3FD@gmail.com> <877h5li0dk.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <874o0phstx.fsf@uwakimon.sk.tsukuba.ac.jp> This is all speculation and no hint of implementation at this point ... redirecting this subthread to Python-Ideas. Reply-To set accordingly. Nick Coghlan writes: > Heh, I knew as soon as I sent that message that someone would be able > to point out a counter example. I agree that RFC 822 (and > case-insensitive ASCII comparison in general) is enough to save > lower() and upper() and co, but what about this even further reduced > list of text-specific methods: > > 'capitalize' > 'istitle' > 'swapcase' > 'title' > > While case-insensitive comparison makes sense for wire level data, > where do these methods fit in, even when embedded ASCII text fragments > are involved? Well, 'capitalize' could theoretically be used to "beautify" RFC 822 field names, but realistically, to me they're a litmus test for packages I probably don't want on my system.<0.5 wink> I don't know if it's worth the effort to deprecate them, though. There is a school of thought (represented on python-dev by Philip Eby and Antoine Pitrou, among others, I would say) that says that text with an implicit encoding is still text if you can figure out what the encoding is, and the syntactically important tokens are invariably ASCII, which often is enough information to do the work. So if you can do some operation without first converting to str, let's save the cycles and the bytes (especially in bit-shoveling applications like WSGI)! I disagree, but "consenting adults" and all that. It occurs to me that the bit-shoveling applications would generally be sufficiently well-served with a special "codec" that just stuffs the data pointer in a bytes object into the latin1 member of the data pointer union in a PEP 393 Unicode object, and marks the Unicode object as "ascii-compatible", ie, anything ASCII can be manipulated as text, but anything non-ASCII is like a private character that Python doesn't know anything about, and can't do anything useful with, except delete or pass through verbatim (perhaps as a slice). This may be nonsense; I don't know enough about Python internals to be sure. And it would be a change to PEP 393, since the encoding of the 8-bit representation would no longer be Unicode. I wouldn't blame Martin one bit if he hated the idea in principle! On the other hand, the "Latin-1 can be used to decode any binary content" end-around makes that point moot IMO. This would give a somewhat safer way of doing that. But if feasible and a Pythonic implementation could be devised, that would take much of the wind out of the sails of the "implicitly it's ASCII text" crowd. The whole "it's inefficient in time and space to work with 'str'" argument goes away, leaving them with "it's verbose" as the only reason for not doing the conversion. I don't know if there would be any use case left for bytes at that point ... but that's clearly a py4k discussion. From jeanpierreda at gmail.com Wed Sep 7 15:40:39 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 7 Sep 2011 09:40:39 -0400 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: <4E66C27B.2080906@canterbury.ac.nz> References: <20110906164103.772ed18b@resist.wooz.org> <20110906183346.2bcdfbf8@resist.wooz.org> <4E66C27B.2080906@canterbury.ac.nz> Message-ID: On Tue, Sep 6, 2011 at 9:01 PM, Greg Ewing wrote: > Nick Coghlan wrote: > >> class Foo: >> ?normal = 1 >> ?'class' = 'This is probably a terrible idea' >> ?'1234' = 'as is this' >> ?'or does it' = 'have some merit?' > > Could be rather confusing, since the interpretation of a string > literal would depend somewhat subtly on its position in an > expression. And it's not just a LHS/RHS distinction, because > presumably the following would still parse with all the string > literals being literals... or would it? > > ?'foo'['blarg'] = 'asdf' I seem to remember them being banned for all use in Python forever, but backticks are used for a similar purpose in SQL without ambiguity. `blarg baz` = 3 `foo`[`blarg baz`] = `asdf` I'd definitely prefer them to overloaded quotes. Devin From guido at python.org Wed Sep 7 16:39:00 2011 From: guido at python.org (Guido van Rossum) Date: Wed, 7 Sep 2011 07:39:00 -0700 Subject: [Python-ideas] Deprecating bytes.swapcase and friends [was: Maintenance burden of str.swapcase] In-Reply-To: References: <785989C3-66EA-4ED9-B6D2-E55FE2A30DE8@voidspace.org.uk> <87ehztip2r.fsf@uwakimon.sk.tsukuba.ac.jp> <4E66763B.7080707@pearwood.info> <4E668029.6080106@v.loewis.de> <7EA8F302-1DC4-43D9-B124-09A51172E9BF@voidspace.org.uk> <4E66845F.3060708@v.loewis.de> <4271E434-0E35-4D7E-98D0-097402B8C3FD@gmail.com> <877h5li0dk.fsf@uwakimon.sk.tsukuba.ac.jp> <874o0phstx.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: TBH, your experience showed that trying to write "polymorphic" code manipulating either-str-or-bytes-meaning-text is too ugly to care. I don't know if the same is true if one were to just set out to manipulate bytes-meaning-text. FWIW, I haven't changed my mind on swapcase -- I regret it, but (despite acknowledging your experience) value the consistency more than the cost of implementing it. I could live with deprecating it across the board, if only to ease life for PyPy and others. --Guido On Tue, Sep 6, 2011 at 10:26 PM, Nick Coghlan wrote: > On Wed, Sep 7, 2011 at 2:36 PM, Stephen J. Turnbull wrote: >> I don't know if it's worth the effort to deprecate them, though. > > I could live with a purely documentation based deprecation, although > I'd prefer to *actually* deprecate at least those four methods on > bytes and bytearray objects (since we switched mailing lists, > reproducing the list for reference: 'capitalize', 'istitle', > 'swapcase', 'title'). > >> There is a school of thought (represented on python-dev by Philip Eby >> and Antoine Pitrou, among others, I would say) that says that text >> with an implicit encoding is still text if you can figure out what the >> encoding is, and the syntactically important tokens are invariably >> ASCII, which often is enough information to do the work. ?So if you >> can do some operation without first converting to str, let's save the >> cycles and the bytes (especially in bit-shoveling applications like >> WSGI)! ?I disagree, but "consenting adults" and all that. > > FWIW, I actually used to be in that school myself, *until* I took on > the task of making more of the urllib.parse APIs take a polymorphic > bytes-in-bytes-out, str-in-str-out approach for 3.2. The difference in > complexity between the "right" way (i.e. decoding with the ascii > codec, manipulating as Unicode, encoding back to bytes with the ascii > codec) and a hackier approach that tried to manipulate the bytes > directly was such that I didn't even end up benchmarking the two > approaches to decide between them - I ended up having zero interest in > attempting to maintain the latter version, so the implicit > decode/encode is the version that went into the release. > > That experience pushed me solidly in the direction of arbitrary fast > ASCII text manipulation without encoding/decoding overhead in Python 3 > being a task for a third party type - neither bytes nor str fit the > bill. To be really effective, such a type either needs algorithms > dedicated to using it so that all the associated 'literals' are > predefined as objects of the relevant type and don't need to worry > about handling actual strings being passed in or else they need to > transparently interoperate with builtin str objects. > > The potential viability and utility of such a tagged string type, > however, isn't a particularly strong argument for anything relating to > the bytes API - it's pretty clear that Guido's plan to break the > 8-bit-data-as-text paradigm in Python 3 has succeeded to that extent. > > Regards, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (python.org/~guido) From pyideas at rebertia.com Wed Sep 7 20:39:52 2011 From: pyideas at rebertia.com (Chris Rebert) Date: Wed, 7 Sep 2011 11:39:52 -0700 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <20110906164103.772ed18b@resist.wooz.org> <20110906183346.2bcdfbf8@resist.wooz.org> <4E66C27B.2080906@canterbury.ac.nz> Message-ID: On Wed, Sep 7, 2011 at 6:40 AM, Devin Jeanpierre wrote: > On Tue, Sep 6, 2011 at 9:01 PM, Greg Ewing wrote: >> Nick Coghlan wrote: >> >>> class Foo: >>> ?normal = 1 >>> ?'class' = 'This is probably a terrible idea' >>> ?'1234' = 'as is this' >>> ?'or does it' = 'have some merit?' >> >> Could be rather confusing, since the interpretation of a string >> literal would depend somewhat subtly on its position in an >> expression. And it's not just a LHS/RHS distinction, because >> presumably the following would still parse with all the string >> literals being literals... or would it? >> >> ?'foo'['blarg'] = 'asdf' > > I seem to remember them being banned for all use in Python > forever, but backticks Indeed; see PEP 3099. "No more backticks." Cheers, Chris From tjreedy at udel.edu Wed Sep 7 23:26:49 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 07 Sep 2011 17:26:49 -0400 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <20110906164103.772ed18b@resist.wooz.org> <20110906183346.2bcdfbf8@resist.wooz.org> <20110906215717.62adf161@resist.wooz.org> Message-ID: On 9/6/2011 10:47 PM, Nick Coghlan wrote: > On Wed, Sep 7, 2011 at 11:57 AM, Barry Warsaw wrote: >> On Sep 07, 2011, at 10:21 AM, Nick Coghlan wrote: >> >>>>> class Foo: >> ... """I'm still -1 myself, but such a delimited approach at least has the >> ... virtue of addressing the *entire* scope of the identifier syntax >> ... limitation rather than singling out keywords for special treatment. >> ... >> ... Legitimate use cases for such a feature would include at least those >> ... currently handled by the 'rename' flag on the namedtuple constructor >> ... (the latter would still be needed to handle duplicate field names, >> ... but illegal identifiers could just be quoted). >> ... """ = False >> >> frightening-ly y'rs, > > Even worse: > > class Foo: > '' = True > > Utterly untenable :) And completely possible with set/getattr ;=) I personally think the current de facto rule is fine: any identifier can be and must be acceptable as an attribute name; the use of anything else by indirect non-attribute-reference means (set/getattr, .__dict__, or otherwise) is implementation dependent. -- Terry Jan Reedy From hetchkay at gmail.com Fri Sep 9 08:04:28 2011 From: hetchkay at gmail.com (H Krishnan) Date: Fri, 9 Sep 2011 11:34:28 +0530 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Hi, I have read through the comments and the (largely negative) votes for this idea. The main argument against it seems to be readability. I feel the examples I chose were distracting. The intent of the examples was to show the semantics and not to suggest that anyone should write such code. One can write totally unreadable code even with the current syntax: if not was is can: return take One cannot however write readable code such as the following: if yield > principal: return = yield - principal def send(from, to, message): .. Syntax is something people will get used to. The following is perfectly valid 'scheme' code: (define define 3) (* define 5) In my 10+ years in scheme, I have never used 'define' as a symbol, nor have I seen anyone else do it, even though there is no language restriction. 'self' is not a keyword in python, but I have not so far seen anyone use 'self' other than as the first argument of a class method. The second argument was about the domino effect of such a change on parsing and syntax highlighting tools. That is true for any change to the core language, though the proposed change might cause more impact than most others. If syntax highlighting tools can use python's ast and other modules to identify keywords (instead of using their own parsers), the domino effect will be lesser. The use-case I had in mind was related to a customization tool we are developing. We want to allow users to define expressions and assign them to names. We 'eval' the expressions subsequently. We have to currently prevent users from using any python keyword as a name, which I felt was an unnecessary restriction. I don't think a partial solution like allowing keywords as attributes will work. It will be quite confusing and actually complicate the syntax highlighters more than a full solution would. Anyway, thanks for all the comments. Regards, Krishnan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruce at leapyear.org Fri Sep 9 08:46:12 2011 From: bruce at leapyear.org (Bruce Leban) Date: Thu, 8 Sep 2011 23:46:12 -0700 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Thu, Sep 8, 2011 at 11:04 PM, H Krishnan wrote: > The use-case I had in mind was related to a customization tool we are > developing. We want to allow users to define expressions and assign them to > names. We 'eval' the expressions subsequently. We have to currently prevent > users from using any python keyword as a name, which I felt was an > unnecessary restriction. > Now that you've defined the actual use case you had in mind, it's a good reason not to use this. Using 'eval' on user-provided expressions is risky in many ways. If you restrict the expressions to simple operators like +-*/ then that's easy (and safer) to process without using eval (or if you are using eval, it's easy to replace every variable X with something like data["X"] completely eliminating the possibility of information leakage). If you're allowing calls to arbitrary functions then you're probably going to have some security holes in your app. For example, you better make sure to disallow expressions like: __import__('sys').exit() or [i for i in range(2**100) if i < 1] and lots of others. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com P.S.. Yes, that's valid scheme? So what? Redefining 'define' pretty much guarantees that the rest of your program will fail. (define define 3)(display (* define 5)) => 15(define x 4) => FAIL The fact that another language allows you to shoot yourself in the foot isn't a good argument that Python should allow that too. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hetchkay at gmail.com Fri Sep 9 09:21:18 2011 From: hetchkay at gmail.com (H Krishnan) Date: Fri, 9 Sep 2011 12:51:18 +0530 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Hi, > __import__('sys').exit() > > or > > [i for i in range(2**100) if i < 1] > > > The execution is not in a server but in the user's machine/environment. If the user wants to do something like that, it would be the user's problem. Nothing prevents the user from doing the same things in a python process invoked by her, right? > P.S.. Yes, that's valid scheme? So what? Redefining 'define' pretty much > guarantees that the rest of your program will fail. > > (define define 3)(display (* define 5)) => 15(define x 4) => FAIL > > The fact that another language allows you to shoot yourself in the foot > isn't a good argument that Python should allow that too. > > Yes, scheme does not have special syntax for 'keywords' but still allows keywords to be overridden. But nothing (but syntax highlighters) will break in Python by allowing keywords as identifiers, and so it is a less drastic change for Python. Regards, Krishnan -------------- next part -------------- An HTML attachment was scrubbed... URL: From pyideas at rebertia.com Fri Sep 9 09:36:51 2011 From: pyideas at rebertia.com (Chris Rebert) Date: Fri, 9 Sep 2011 00:36:51 -0700 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Thu, Sep 8, 2011 at 11:04 PM, H Krishnan wrote: > Hi, > I have read through the comments and the (largely negative) votes for this > idea. > The main argument against it seems to be readability. I feel the examples I > chose were distracting. The intent of the examples was to show the semantics > and not to suggest that anyone should write such code. One can write totally > unreadable code even with the current syntax: > if not was is can: > ? ?return take "You can write FORTRAN [(or rather, bad code)] in any language" > One cannot however write readable code such as the following: > if yield > principal: > ? ?return = yield - principal As an example of how problematic the proposal is, that last line (modulo some var names, obviously) contains a valid yield-expression (see PEP 342 - http://www.python.org/dev/peps/pep-0342/ ). Subtle bugs and parser complexity ahoy! > Syntax is something people will get used to. The following is perfectly > valid 'scheme' code: > (define define 3) > (* define 5) > In my 10+ years in scheme, I have never used 'define' as a symbol, nor have > I seen anyone else do it, even though there is no language restriction. > 'self' is not a keyword in python, but I have not so far seen anyone use > 'self' other than as the first argument of a class method. This would seem to argue against the proposed language feature. If nobody does it even when they can do it, then why should we bother to change our rules to allow it? > The use-case I had in mind was related to a customization tool we are > developing. We want to allow users to define expressions and assign them to > names. We 'eval' the expressions subsequently. We have to currently prevent > users from using any python keyword as a name, which I felt was an > unnecessary restriction. Define a name-mangling convention then, and escape the problematic (or just all) names. Or use a dictionary instead of a normal namespace to store and refer to the expressions (as Bruce suggested excellently). Or accept the fact that since the users are really writing in Python (and not some custom scripting language), they should observe Python's rules. Cheers, Chris -- http://rebertia.com From tjreedy at udel.edu Fri Sep 9 09:48:41 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 09 Sep 2011 03:48:41 -0400 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 9/9/2011 2:04 AM, H Krishnan wrote: > Hi, > I have read through the comments and the (largely negative) votes for > this idea. > > The main argument against it seems to be readability. I feel the > examples I chose were distracting. The intent of the examples was to > show the semantics and not to suggest that anyone should write such > code. The parser has to be ready for and work correctly with the worst horror anyone *could* write. if in in in or and and and: One can write totally unreadable code even with the current syntax: > if not was is can: > return take Except that *is* readable *because* keywords are keywords and identifiers are identifiers. 'if not x is y' implies that x and y should be boolean values and we want them to have opposite values to proceed. > One cannot however write readable code such as the following: > if yield > principal: > return = yield - principal Funny you should choose that example. With a slight change myreturn = yield - principal it is legal syntax today with 'yield' interpreted as a keyword. So it cannot be interpreted as an identifier without making Python grammar ambiguous and unparseable with its current parser. Because C grammar allows typedef names, which are just identifiers, to be used as pseudo-keywords, without being distinguished by a real keyword or other mechanism, it is ambiguous. The ambiguity is resolved by making C globally context sensitive. Therefore, C cannot be parsed and compiled with a LR or LL compiler, such as produced by YACC. The parser has to be escaped to resolve the ambiguity and everyone who writes C parsers has to patch the output of any automatic parser generator. Having a sequence of characters be ambiguously either a keyword or an identifier is a bad idea. They are really different things. They have different grammatical functions. If one wants everything that looks like an identifier by be usable as an identifier, then there should be no identifier-like keywords. Require that all keystrings not be legal identifiers. Make them consist of or include some non-identifier character. Guido could had decided that all keystrings should start or end with a symbol such as '$' or '`', : $if, $yield, etc. Your problem would have been solved. And most everyone would have a problem with harder code entry. > The second argument was about the domino effect of such a change on > parsing and syntax highlighting tools. That is true for any change to > the core language, though the proposed change might cause more impact > than most others. Hardly in the same ballpark, I think. 3.2 had no core language changes. 3.3 might add 'yield from'. Syntax highlighters should not be affected. Adding a new keyword should also have no impact as any such tool should get the current keywords from keyword.kwlist. > If syntax highlighting tools can use python's ast and > other modules to identify keywords (instead of using their own parsers), > the domino effect will be lesser. Right now, it is a simple lookup. > The use-case I had in mind was related to a customization tool we are > developing. We want to allow users to define expressions and assign them > to names. We 'eval' the expressions subsequently. We have to currently > prevent users from using any python keyword as a name, which I felt was > an unnecessary restriction. For the present, you can remove the restriction on keywords that cannot appear in expressions by mechanically transforming them. -- Terry Jan Reedy From p.f.moore at gmail.com Fri Sep 9 10:11:23 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 Sep 2011 09:11:23 +0100 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 9 September 2011 08:48, Terry Reedy wrote: >> One cannot however write readable code such as the following: >> if yield > principal: >> ? ?return = yield - principal > > Funny you should choose that example. With a slight change > ?myreturn = yield - principal > it is legal syntax today with 'yield' interpreted as a keyword. So it cannot > be interpreted as an identifier without making Python grammar ambiguous and > unparseable with its current parser. Here is some currently legal syntax: globals()['return'] = lambda x:x def f(): return(12) print(f()) Currently, that prints 12. If keywords are allowed as variable names, presumably it should produce 'None'? Or should we break the semantics of globals()? Or what? And given that the globals() line could be arbitrarily far away from the definition of f(), the parser would need arbitrary lookbehind. And lookahead, as the globals() line could be in a function defined after f but called before it. If you don't like me using globals(), the line could be replaced by def g(): global return return = lambda x:x g() but that's not currently legal syntax, so it obscures my point slightly... Python's parser is simple by design. This isn't going to be accepted (even if it were a good idea :-)) Paul. From hetchkay at gmail.com Fri Sep 9 10:18:03 2011 From: hetchkay at gmail.com (H Krishnan) Date: Fri, 9 Sep 2011 13:48:03 +0530 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Hi, As someone pointed out, I think PEP 342 (coroutines via enhanced generators) anyway kills this proposal since it is currently possible to do something like: val = yield where yield is a keyword. The fact that another language allows you to shoot yourself in the foot > isn't a good argument that Python should allow that too. > If I do: __builtins__.exit = os._exit = os.kill = sys.exit = lambda *args: False is there a way to programmatically exit? Regards, Krishnan -------------- next part -------------- An HTML attachment was scrubbed... URL: From pyideas at rebertia.com Fri Sep 9 10:21:32 2011 From: pyideas at rebertia.com (Chris Rebert) Date: Fri, 9 Sep 2011 01:21:32 -0700 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Fri, Sep 9, 2011 at 1:18 AM, H Krishnan wrote: > If I do: > __builtins__.exit = os._exit = os.kill = sys.exit = lambda *args: False > is there a way to programmatically exit? Yes: raise SystemExit Cheers, Chris From tjreedy at udel.edu Fri Sep 9 18:05:52 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 09 Sep 2011 12:05:52 -0400 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 9/9/2011 3:21 AM, H Krishnan wrote: > P.S.. Yes, that's valid scheme? So what? Redefining 'define' pretty > much guarantees that the rest of your program will fail. > > (define define 3) > (display (* define 5))=> 15 > (define x4)=> FAIL > > The fact that another language allows you to shoot yourself in the > foot isn't a good argument that Python should allow that too. > > Yes, scheme does not have special syntax for 'keywords' but still allows > keywords to be overridden. In the above, 'define' is not a keyword, it is a built-in function or macro name. Python allows those to be over-ridden too, with the same consequence. list = 3 list(1,2,3) # TypeError: 'int' object is not callable Because of its simple one-syntax-fits-all-needs design, I do not think Scheme has or needs functional keywords in the way Python does. For instance, the class statement, with it 'class' keyword, can be replaced by a call to the builtin 'type' class. The name 'type' can be over-ridden just like 'define' above. Perhaps Scheme has value keywords like Common Lisp (NIL?), I don't know. From https://secure.wikimedia.org/wikipedia/en/wiki/Keyword_%28computer_programming%29 "In Common Lisp, the term "keyword" (or "keyword symbol") is used for a special sort of symbol, or identifier. Unlike other symbols, which usually stand for variables or functions, keywords are self-quoting and evaluate to themselves. Keywords are usually used to label named arguments to functions, and to represent symbolic values." Python has keyword values None, False, and True. These used to just be builtin identifiers that could be over-ridden, but were made keywords to prevent that. > But nothing (but syntax highlighters) and the parser -- see my other post and Paul Moore's followup. > will break in Python by allowing keywords as identifiers, If you disagree, try changing the grammar to do what you want while keeping Python an LL(1) language. > and so it is a less drastic change for Python. We seem to have different ideas of 'drastic' ;-). -- Terry Jan Reedy From tjreedy at udel.edu Fri Sep 9 18:38:39 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 09 Sep 2011 12:38:39 -0400 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 9/9/2011 4:11 AM, Paul Moore wrote: > On 9 September 2011 08:48, Terry Reedy wrote: >>> One cannot however write readable code such as the following: >>> if yield> principal: >>> return = yield - principal >> >> Funny you should choose that example. With a slight change >> myreturn = yield - principal >> it is legal syntax today with 'yield' interpreted as a keyword. So it cannot >> be interpreted as an identifier without making Python grammar ambiguous and >> unparseable with its current parser. > > Here is some currently legal syntax: > > globals()['return'] = lambda x:x > def f(): > return(12) > > print(f()) > > Currently, that prints 12. If keywords are allowed as variable names, > presumably it should produce 'None'? Or should we break the semantics > of globals()? Or what? C now has a very similar problem. (According to Eli Bendersky) The C statememnt f(a); can either be a function call (presumably with side-effects) or a declaration that a is a variable of type f. > And given that the globals() line could be arbitrarily far away from > the definition of f(), the parser would need arbitrary lookbehind. And > lookahead, as the globals() line could be in a function defined after > f but called before it. The C typedef ambiguity has to be resolved according to whether there is or is not a typedef for f in effect at that particular spot. This can even change within a statement. That determination requires arbitrary lookbehind, including the effect of #include directives. For Python, the runtime statements from module import * # or somewhat equivalently for k,v in parse_config_file(config_file): globals()[k] = v would makes global lookbehinds unresolvable at compile time. So Python solves the potential keyword/namespace-name ambiguity with a builltin keyword definition list. Similarly, avoiding global/local ambiguity is the reason why 'import *' is now prohibited within function defs and why "locals[k] = v" has no effect on the actual function local namespace. Python requires that the status of names within functions be determined and determinable at compile time. -- Terry Jan Reedy From g.brandl at gmx.net Fri Sep 9 19:28:55 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 09 Sep 2011 19:28:55 +0200 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Am 09.09.2011 10:21, schrieb Chris Rebert: > On Fri, Sep 9, 2011 at 1:18 AM, H Krishnan wrote: > >> If I do: >> __builtins__.exit = os._exit = os.kill = sys.exit = lambda *args: False >> is there a way to programmatically exit? > > Yes: raise SystemExit Or reload(sys); sys.exit() Georg From g.brandl at gmx.net Fri Sep 9 19:31:59 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 09 Sep 2011 19:31:59 +0200 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Am 09.09.2011 09:48, schrieb Terry Reedy: > On 9/9/2011 2:04 AM, H Krishnan wrote: >> Hi, >> I have read through the comments and the (largely negative) votes for >> this idea. >> >> The main argument against it seems to be readability. I feel the >> examples I chose were distracting. The intent of the examples was to >> show the semantics and not to suggest that anyone should write such >> code. > > The parser has to be ready for and work correctly with the worst horror > anyone *could* write. > if in in in or and and and: > > One can write totally unreadable code even with the current syntax: >> if not was is can: >> return take > > Except that *is* readable *because* keywords are keywords and > identifiers are identifiers. 'if not x is y' implies that x and y should > be boolean values and we want them to have opposite values to proceed. > >> One cannot however write readable code such as the following: >> if yield > principal: >> return = yield - principal > > Funny you should choose that example. With a slight change > myreturn = yield - principal > it is legal syntax today with 'yield' interpreted as a keyword. So it > cannot be interpreted as an identifier without making Python grammar > ambiguous and unparseable with its current parser. Actually, it isn't: "yield" expressions, like generator expressions, need to be inside parentheses. (A rule that's an ambiguousness restriction and a nice readability helper.) Georg From pyideas at rebertia.com Fri Sep 9 20:20:47 2011 From: pyideas at rebertia.com (Chris Rebert) Date: Fri, 9 Sep 2011 11:20:47 -0700 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Fri, Sep 9, 2011 at 10:31 AM, Georg Brandl wrote: > Am 09.09.2011 09:48, schrieb Terry Reedy: >> On 9/9/2011 2:04 AM, H Krishnan wrote: >>> One cannot however write readable code such as the following: >>> if yield > principal: >>> ? ? return = yield - principal >> >> Funny you should choose that example. With a slight change >> ? ?myreturn = yield - principal >> it is legal syntax today with 'yield' interpreted as a keyword. So it >> cannot be interpreted as an identifier without making Python grammar >> ambiguous and unparseable with its current parser. > > Actually, it isn't: "yield" expressions, like generator expressions, need > to be inside parentheses. ?(A rule that's an ambiguousness restriction and > a nice readability helper.) Not always. That code is indeed valid (unless something changed in Python 3.2). Python 3.1.2: def foo(): bar = yield 42 # look Ma, no parens! print(bar) a = foo() print(next(a)) a.send(7) Output: 42 7 Traceback (most recent call last): File "prog.py", line 7, in a.send(7) StopIteration Cheers, Chris From jeanpierreda at gmail.com Fri Sep 9 20:24:21 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Fri, 9 Sep 2011 14:24:21 -0400 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Even so, there's the difference between (yield) - principal and yield (- principal) Devin On Fri, Sep 9, 2011 at 2:20 PM, Chris Rebert wrote: > On Fri, Sep 9, 2011 at 10:31 AM, Georg Brandl wrote: >> Am 09.09.2011 09:48, schrieb Terry Reedy: >>> On 9/9/2011 2:04 AM, H Krishnan wrote: > >>>> One cannot however write readable code such as the following: >>>> if yield > principal: >>>> ? ? return = yield - principal >>> >>> Funny you should choose that example. With a slight change >>> ? ?myreturn = yield - principal >>> it is legal syntax today with 'yield' interpreted as a keyword. So it >>> cannot be interpreted as an identifier without making Python grammar >>> ambiguous and unparseable with its current parser. >> >> Actually, it isn't: "yield" expressions, like generator expressions, need >> to be inside parentheses. ?(A rule that's an ambiguousness restriction and >> a nice readability helper.) > > Not always. That code is indeed valid (unless something changed in Python 3.2). > > Python 3.1.2: > > def foo(): > ? ?bar = yield 42 # look Ma, no parens! > ? ?print(bar) > > a = foo() > print(next(a)) > a.send(7) > > Output: > 42 > 7 > Traceback (most recent call last): > ?File "prog.py", line 7, in > ? ?a.send(7) > StopIteration > > Cheers, > Chris > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From jh at improva.dk Fri Sep 9 20:21:23 2011 From: jh at improva.dk (Jacob Holm) Date: Fri, 09 Sep 2011 20:21:23 +0200 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4E6A5923.3030603@improva.dk> On 2011-09-09 19:31, Georg Brandl wrote: > Am 09.09.2011 09:48, schrieb Terry Reedy: >> On 9/9/2011 2:04 AM, H Krishnan wrote: >>> [snip] >> >> Funny you should choose that example. With a slight change >> myreturn = yield - principal >> it is legal syntax today with 'yield' interpreted as a keyword. So it >> cannot be interpreted as an identifier without making Python grammar >> ambiguous and unparseable with its current parser. > > Actually, it isn't: "yield" expressions, like generator expressions, need > to be inside parentheses. (A rule that's an ambiguousness restriction and > a nice readability helper.) Actually it is, at least in 3.1 (I don't have a more recent version to test on ATM). It is the same as myreturn = yield (-principal) It computes and yields the value (-principal). When resumed by a "send" on the generator the argument is assigned to myreturn. If resumed by "next" instead, this is treated as send(None). I agree that this is confusing. In spite of all my work hashing out the details of PEP 380, I still initially thought it would work like myreturn = (yield) - principal probably because of the context where "yield" was supposed to be a regular variable name. (An idea I'm -1 on, btw). Based on this, I would support a change to outlaw myreturn = yield - principal forcing you to add the parentheses to clearly show which interpretation is intended. However, I would still want to keep the statement form yield -principal legal, so I am not sure if that proposal has a chance to fly. - Jacob From mwm at mired.org Fri Sep 9 20:50:14 2011 From: mwm at mired.org (Mike Meyer) Date: Fri, 9 Sep 2011 11:50:14 -0700 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Thu, Sep 8, 2011 at 11:04 PM, H Krishnan wrote: > Syntax is something people will get used to. The following is perfectly > valid 'scheme' code: > (define define 3) > (* define 5) > In my 10+ years in scheme, I have never used 'define' as a symbol, nor have > I seen anyone else do it, even though there is no language restriction. > 'self' is not a keyword in python, but I have not so far seen anyone use > 'self' other than as the first argument of a class method. > Others have pointed out the parsing problems this would create. I'll point out that Python is not Scheme. I'm a fan of both languages, but they are not the same. Scheme has a history of valuing power, and adding it by removing restrictions. The example above - of allowing people to redefine what are primitives in other languages - fits that model perfectly. Python, on the other hand, has a history of valuing readability. With very few exceptions, changes to Python make some use case more readable, without adding a way to write really unreadable code. The change to have non-reserved keywords clashes badly with that model. From g.brandl at gmx.net Fri Sep 9 21:23:45 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 09 Sep 2011 21:23:45 +0200 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: <4E6A5923.3030603@improva.dk> References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> <4E6A5923.3030603@improva.dk> Message-ID: Am 09.09.2011 20:21, schrieb Jacob Holm: > On 2011-09-09 19:31, Georg Brandl wrote: >> Am 09.09.2011 09:48, schrieb Terry Reedy: >>> On 9/9/2011 2:04 AM, H Krishnan wrote: >>>> > [snip] >>> >>> Funny you should choose that example. With a slight change >>> myreturn = yield - principal >>> it is legal syntax today with 'yield' interpreted as a keyword. So it >>> cannot be interpreted as an identifier without making Python grammar >>> ambiguous and unparseable with its current parser. >> >> Actually, it isn't: "yield" expressions, like generator expressions, need >> to be inside parentheses. (A rule that's an ambiguousness restriction and >> a nice readability helper.) > > Actually it is, at least in 3.1 (I don't have a more recent version to > test on ATM). It is the same as > > myreturn = yield (-principal) > > It computes and yields the value (-principal). When resumed by a "send" > on the generator the argument is assigned to myreturn. If resumed by > "next" instead, this is treated as send(None). Of course. I'm not new to PEP 342 :) But I have to apologize: what I did test was the confusingly similar return yield - principal which isn't allowed (yes, I know that even return (yield -principal) isn't allowed, but that's not for syntactical reasons.) Now I checked properly: In fact, "yield" expressions after assignment operators are special-cased by the grammar, so that they don't need to be parenthesized [1]. In all other places, yield expressions must occur in parentheses. For example: myreturn = principal - yield Georg [1] I guess that's because it was thought to be a common case. I agree that it's not really helping readability. From g.nius.ck at gmail.com Fri Sep 9 23:12:21 2011 From: g.nius.ck at gmail.com (Christopher King) Date: Fri, 9 Sep 2011 17:12:21 -0400 Subject: [Python-ideas] List Revolution Message-ID: The first element in a list is element zero, the second is one, the third it two, and so on. This some times confuses newbies to the language or programming in general. This system was invited when single bits where precious. It's time to update. Keep in mind this is something for version 4, since its not reverse compatible. I say we make the first element 1, second 2, third 3, and so on. Other languages would follow. We are python, made for easiness and readability, and we are in the age where you can't even get something as small as a kilobyte USB. We must make first one, second 2, and third 3, like it is supposed to be. I give this: *+1* -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Sep 9 23:22:56 2011 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Sep 2011 14:22:56 -0700 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: On Fri, Sep 9, 2011 at 2:12 PM, Christopher King wrote: > The first element in a list is element zero, the second is one, the third it > two, and so on. This some times confuses newbies to the language or > programming in?general. This system was?invited?when single bits where > precious. It's time to update. Keep in mind this is something for version 4, > since its not reverse?compatible. I say we make the first element 1, second > 2, third 3, and so on. Other languages would follow. We are python, made for > easiness and?readability, and we are in the age where you can't even get > something as small as a kilobyte USB. We must make first one, second 2, and > third 3, like it is supposed to be. I give this: > +1 Consider it done. -- --Guido van Rossum (python.org/~guido) From dwblas at gmail.com Fri Sep 9 23:42:09 2011 From: dwblas at gmail.com (David Blaschke) Date: Fri, 9 Sep 2011 14:42:09 -0700 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: On Fri, Sep 9, 2011 at 2:12 PM, Christopher King wrote: > The first element in a list is element zero, the second is one, the third it > two, and so on. Those are not element numbers, but offsets. What is confusing is calling them element numbers. Back in the days of dinosaurs, you would write the access to containers yourself, so you if you wanted to access the first element, the memory offset is zero from the beginning. If you want the second element the memory offset is 1 X length of element from the beginning, etc. This has carried forward as languages have moved forward. A dictionary would be the choice for indexing by 1, 2, 3, etc. -- Unless your heart is open and serene, with nothing touching your feelings, how can you respond completely without error,,,,,Yuanwu From anacrolix at gmail.com Sat Sep 10 03:31:46 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Sat, 10 Sep 2011 11:31:46 +1000 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: I can't believe nobody else thought of this first. +1!! On Sat, Sep 10, 2011 at 7:12 AM, Christopher King wrote: > The first element in a list is element zero, the second is one, the third it > two, and so on. This some times confuses newbies to the language or > programming in?general. This system was?invited?when single bits where > precious. It's time to update. Keep in mind this is something for version 4, > since its not reverse?compatible. I say we make the first element 1, second > 2, third 3, and so on. Other languages would follow. We are python, made for > easiness and?readability, and we are in the age where you can't even get > something as small as a kilobyte USB. We must make first one, second 2, and > third 3, like it is supposed to be. I give this: > +1 > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > From anacrolix at gmail.com Sat Sep 10 03:32:59 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Sat, 10 Sep 2011 11:32:59 +1000 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: Recommended reading: http://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html On Sat, Sep 10, 2011 at 7:12 AM, Christopher King wrote: > The first element in a list is element zero, the second is one, the third it > two, and so on. This some times confuses newbies to the language or > programming in?general. This system was?invited?when single bits where > precious. It's time to update. Keep in mind this is something for version 4, > since its not reverse?compatible. I say we make the first element 1, second > 2, third 3, and so on. Other languages would follow. We are python, made for > easiness and?readability, and we are in the age where you can't even get > something as small as a kilobyte USB. We must make first one, second 2, and > third 3, like it is supposed to be. I give this: > +1 > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > From guido at python.org Sat Sep 10 04:08:13 2011 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Sep 2011 19:08:13 -0700 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: On Fri, Sep 9, 2011 at 6:32 PM, Matt Joiner wrote: > Recommended reading: > http://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html Nice one! Reminds me of these two lines from the zen of Python: There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch. -- --Guido van Rossum (python.org/~guido) From stephen at xemacs.org Sat Sep 10 04:38:55 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 10 Sep 2011 11:38:55 +0900 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: <87ty8lce9s.fsf@uwakimon.sk.tsukuba.ac.jp> Christopher King writes: > We must make first one, second 2, and third 3, like it is supposed > to be. There's a better way: def first (sequence): return sequence['one'] # def second (sequence): return sequence[1] def third (sequence): return sequence[2] def fourth (sequence): return sequence[3] def fifth (sequence): return sequence[4] def sixth (sequence): return sequence[5] def seventh (sequence): return sequence[6] def eighth (sequence): return sequence[7] There's probably an Ordinal module that would allow you to generate such functions to your heart's content. I'll have to wait for Guido's pronouncement about whether it's "more Pythonic" as well as "just plain better" though. From bruce at leapyear.org Sat Sep 10 04:41:08 2011 From: bruce at leapyear.org (Bruce Leban) Date: Fri, 9 Sep 2011 19:41:08 -0700 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: On Fri, Sep 9, 2011 at 2:12 PM, Christopher King wrote: > I say we make the first element 1, second 2, third 3, and so on. > Other languages would follow. We are python, made for easiness > and readability, and we are in the age where you can't even get something as > small as a kilobyte USB. We must make first one, second 2, and third 3, like > it is supposed to be. I give this: > *+1* > Well, given Guido's endorsement, I think we need to change how these +1/-1 votes work. Clearly, +1 means what +0 used to mean. So you have to say +2 to vote in favor of something and -0 to vote against. I know it will be confusing during the transition period but it will be so much easier to use when we are done. On Fri, Sep 9, 2011 at 7:08 PM, Guido van Rossum wrote: > Nice one! Reminds me of these two lines from the zen of Python: > > There should be one-- and preferably only one --obvious way to do it. > Although that way may not be obvious at first unless you're Dutch. > Given this change, I think you mean: There should be two -- and preferably only two -- obvious ways to do it. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sat Sep 10 05:09:16 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 10 Sep 2011 12:09:16 +0900 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87sjo5ccv7.fsf@uwakimon.sk.tsukuba.ac.jp> H Krishnan writes: > In my 10+ years in scheme, I have never used 'define' as a symbol, Of course you have used 'define' as a symbol. The rules of expression evaluation in Scheme say "evaluate the first element of a list, and apply the result to the rest of the list as arguments." When 'define' is the first element of a list, it is evaluated as a symbol, ie, the content of the value cell of the symbol 'define' is retrieved. Conventionally, that cell contains a function that sets the value of the second argument to the value of the third argument. You can't really get away with saying, "I haven't used it as a variable", either, since taking the value of a symbol is using it as a variable. So the design of Scheme (and Lisps in general) is very elegant, because everything is determined by the arrangement of parentheses. There is no need for reserving keywords. However, not all languages are Lisps. Vive la difference![1] Except that as somebody pointed out, there is a need. It turns out that users want keywords in the form of "constants that evaluate to themselves". So in the end you're at the same place, just a different balance. (I know that Scheme does not have them, although they can be created using define -- that's Schemingly correct, of course. But other languages have made a deliberate choice to introduce them, cf. http://stackoverflow.com/questions/1527548/why-does-clojure-have-keywords-in-addition-to-symbols.) > If syntax highlighting tools can use python's ast and other modules > to identify keywords (instead of using their own parsers), the > domino effect will be lesser. They can't. Think editors written in Lisp. Footnotes: [1] With apologies to the Francophones present for my spelling. From hetchkay at gmail.com Sat Sep 10 05:20:12 2011 From: hetchkay at gmail.com (H Krishnan) Date: Sat, 10 Sep 2011 08:50:12 +0530 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: <87sjo5ccv7.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> <87sjo5ccv7.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Sat, Sep 10, 2011 at 8:39 AM, Stephen J. Turnbull wrote: > H Krishnan writes: > > > In my 10+ years in scheme, I have never used 'define' as a symbol, > > Of course you have used 'define' as a symbol. The rules of expression > evaluation in Scheme say "evaluate the first element of a list, and > apply the result to the rest of the list as arguments." When 'define' > is the first element of a list, it is evaluated as a symbol, ie, the > content of the value cell of the symbol 'define' is retrieved. > Conventionally, that cell contains a function that sets the value > of the second argument to the value of the third argument. > > You can't really get away with saying, "I haven't used it as a > variable", either, since taking the value of a symbol is using it as a > variable. > > So the design of Scheme (and Lisps in general) is very elegant, > because everything is determined by the arrangement of parentheses. > There is no need for reserving keywords. However, not all languages > are Lisps. Vive la difference![1] > > Except that as somebody pointed out, there is a need. It turns out > that users want keywords in the form of "constants that evaluate to > themselves". So in the end you're at the same place, just a different > balance. (I know that Scheme does not have them, although they can be > created using define -- that's Schemingly correct, of course. But > other languages have made a deliberate choice to introduce them, cf. > > http://stackoverflow.com/questions/1527548/why-does-clojure-have-keywords-in-addition-to-symbols > .) > > > If syntax highlighting tools can use python's ast and other modules > > to identify keywords (instead of using their own parsers), the > > domino effect will be lesser. > > They can't. Think editors written in Lisp. > > > Footnotes: > [1] With apologies to the Francophones present for my spelling. > > Thanks to the python community, I have learnt a lot in these mail exchanges. I didn't realize that my proposal was not backward compatible. As some of you pointed out, currently, "return (val)" and "yield (val)" are legal python with 'return' and 'yield' used as keywords. So, I'll withdraw my proposal. I did want to reply to a few specific comments (e.g. 'python parser is simple') but I don't subscribe to the mailing list and didn't want to break the threading in the list. Thanks, Krishnan -------------- next part -------------- An HTML attachment was scrubbed... URL: From cmjohnson.mailinglist at gmail.com Sat Sep 10 06:10:41 2011 From: cmjohnson.mailinglist at gmail.com (Carl Matthew Johnson) Date: Fri, 9 Sep 2011 18:10:41 -1000 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: <743DA5B0-E3A0-4FEE-A32E-195EBF9319AE@gmail.com> On Sep 9, 2011, at 3:32 PM, Matt Joiner wrote: > Recommended reading: > http://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html The reasoning in this letter is so terrible. Why do people quote this whenever this comes up? "We should start lists at 0 because when writing for-loops the <= look goofy otherwise." Why on earth should Pythonistas care what the for-loops are like when we only have for-each loops! It's crazy. Look, EWD was a good computer scientist (=MATHEMATICIAN), but he was terrible when it comes to the ART of programming language design. There's no reason to think that being a good mathematician would make one also good at designing a user interface, and that's what a programming language is--a user interface for highly advanced users. Other than "Go To Considered Harmful" everything I've read of his about programming has been wrong. (Correctness proofs, really?) And much of that wrongness stems from EWD's false belief that programming should be the same as math. Now, as it happens, I don't think we should change to using 1 (although I do note that Lua gets along fine with it). But the reason not to change is because 0 is a well entrenched standard so most programmers are more used to it than to 1. In other words, it's the standard because in UI design you stick with what your users will expect. It has nothing to do with some arcane mathematician's gobbledygook writings about greater than and less than signs. Programming language design is the art of making something that helps humans work more efficiently, not some pure mathematics descended from Platonic heaven. Sorry to rant, but that article always comes up and so I have grudge against EWD. From adam.jorgensen.za at gmail.com Sat Sep 10 08:13:08 2011 From: adam.jorgensen.za at gmail.com (Adam Jorgensen) Date: Sat, 10 Sep 2011 08:13:08 +0200 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: This is a joke, right? -1 On 9 September 2011 23:12, Christopher King wrote: > The first element in a list is element zero, the second is one, the third > it two, and so on. This some times confuses newbies to the language or > programming in general. This system was invited when single bits where > precious. It's time to update. Keep in mind this is something for version 4, > since its not reverse compatible. I say we make the first element 1, second > 2, third 3, and so on. Other languages would follow. We are python, made for > easiness and readability, and we are in the age where you can't even get > something as small as a kilobyte USB. We must make first one, second 2, and > third 3, like it is supposed to be. I give this: > *+1* > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anacrolix at gmail.com Sat Sep 10 12:08:21 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Sat, 10 Sep 2011 20:08:21 +1000 Subject: [Python-ideas] List Revolution In-Reply-To: <743DA5B0-E3A0-4FEE-A32E-195EBF9319AE@gmail.com> References: <743DA5B0-E3A0-4FEE-A32E-195EBF9319AE@gmail.com> Message-ID: Well it makes complete sense to me. Furthermore, Python even properly supports the style of conditional he uses. a <= b < c IS a great way to do things. 0 based indexing IS right for computing. Not sure if you're trolling or I'm not counter-trolling... -.- EWD is the man. Dutch programmers in decades to come have a lot to live up to. On Sat, Sep 10, 2011 at 2:10 PM, Carl Matthew Johnson wrote: > > On Sep 9, 2011, at 3:32 PM, Matt Joiner wrote: > >> Recommended reading: >> http://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html > > The reasoning in this letter is so terrible. Why do people quote this whenever this comes up? "We should start lists at 0 because when writing for-loops the <= look goofy otherwise." Why on earth should Pythonistas care what the for-loops are like when we only have for-each loops! It's crazy. > > Look, EWD was a good computer scientist (=MATHEMATICIAN), but he was terrible when it comes to the ART of programming language design. There's no reason to think that being a good mathematician would make one also good at designing a user interface, and that's what a programming language is--a user interface for highly advanced users. Other than "Go To Considered Harmful" everything I've read of his about programming has been wrong. (Correctness proofs, really?) And much of that wrongness stems from EWD's false belief that programming should be the same as math. > > Now, as it happens, I don't think we should change to using 1 (although I do note that Lua gets along fine with it). But the reason not to change is because 0 is a well entrenched standard so most programmers are more used to it than to 1. In other words, it's the standard because in UI design you stick with what your users will expect. It has nothing to do with some arcane mathematician's gobbledygook writings about greater than and less than signs. Programming language design is the art of making something that helps humans work more efficiently, not some pure mathematics descended from Platonic heaven. > > Sorry to rant, but that article always comes up and so I have grudge against EWD. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From jkbbwr at gmail.com Sat Sep 10 12:24:14 2011 From: jkbbwr at gmail.com (Jakob Bowyer) Date: Sat, 10 Sep 2011 11:24:14 +0100 Subject: [Python-ideas] List Revolution In-Reply-To: References: <743DA5B0-E3A0-4FEE-A32E-195EBF9319AE@gmail.com> Message-ID: Im dead confused. Is this a joke? Or are we actually switching to index-1 based lists like Lua? Because I like index-0. On Sat, Sep 10, 2011 at 11:08 AM, Matt Joiner wrote: > Well it makes complete sense to me. Furthermore, Python even properly > supports the style of conditional he uses. a <= b < c IS a great way > to do things. 0 based indexing IS right for computing. Not sure if > you're trolling or I'm not counter-trolling... -.- > > EWD is the man. Dutch programmers in decades to come have a lot to live up > to. > > On Sat, Sep 10, 2011 at 2:10 PM, Carl Matthew Johnson > wrote: > > > > On Sep 9, 2011, at 3:32 PM, Matt Joiner wrote: > > > >> Recommended reading: > >> http://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html > > > > The reasoning in this letter is so terrible. Why do people quote this > whenever this comes up? "We should start lists at 0 because when writing > for-loops the <= look goofy otherwise." Why on earth should Pythonistas care > what the for-loops are like when we only have for-each loops! It's crazy. > > > > Look, EWD was a good computer scientist (=MATHEMATICIAN), but he was > terrible when it comes to the ART of programming language design. There's no > reason to think that being a good mathematician would make one also good at > designing a user interface, and that's what a programming language is--a > user interface for highly advanced users. Other than "Go To Considered > Harmful" everything I've read of his about programming has been wrong. > (Correctness proofs, really?) And much of that wrongness stems from EWD's > false belief that programming should be the same as math. > > > > Now, as it happens, I don't think we should change to using 1 (although I > do note that Lua gets along fine with it). But the reason not to change is > because 0 is a well entrenched standard so most programmers are more used to > it than to 1. In other words, it's the standard because in UI design you > stick with what your users will expect. It has nothing to do with some > arcane mathematician's gobbledygook writings about greater than and less > than signs. Programming language design is the art of making something that > helps humans work more efficiently, not some pure mathematics descended from > Platonic heaven. > > > > Sorry to rant, but that article always comes up and so I have grudge > against EWD. > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > http://mail.python.org/mailman/listinfo/python-ideas > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Fri Sep 9 23:34:45 2011 From: ron3200 at gmail.com (ron3200) Date: Fri, 09 Sep 2011 16:34:45 -0500 Subject: [Python-ideas] yield statements (was relaxing keyword usage restrictions) In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> <4E6A5923.3030603@improva.dk> Message-ID: <1315604085.16635.70.camel@Gutsy> On Fri, 2011-09-09 at 21:23 +0200, Georg Brandl wrote: > Now I checked properly: In fact, "yield" expressions after assignment > operators are special-cased by the grammar, so that they don't need to > be parenthesized [1]. In all other places, yield expressions must occur > in parentheses. For example: > > myreturn = principal - yield > > Georg > > [1] I guess that's because it was thought to be a common case. I agree > that it's not really helping readability. Concerning yield without the parentheses. x = yield I really think it should have been "x = (yield)" or "x yield None". Here is my reasoning. What if yield was an assignment operator with similar usage rules as '='? x = y Assign x the object(s) y is assigned to. x yield y assign x from yield, yield out y In other words, the "yield" would never appear to represent a value directly. That's one of the things that makes them confusing to read and write. Heres how the examples in pep 342 would change if yield worked more like '='. x = yield 42 -> x yield 42 x = yield -> x yield None x = 12 + (yield 42) y yield 42 x = 12 + y x = 12 + (yield) y yield None # explicit is better than special casing None here. x = 12 + y foo(yield 42) y yield 42 foo(y) foo(yield) y yield None foo(y) So it only adds one line to some things, but it makes for easier to read code. Once we have yield statements like these, we can then add the ability to do yield expressions by wrapping a yield statement inside parentheses. And special casing (yield None) as (yield) Oh, wait.. we already have that part. It looks to me, that having yield statements, could help with explaining yield expressions. It may be worth doing just for that. Cheers, Ron From guido at python.org Sat Sep 10 17:13:36 2011 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Sep 2011 08:13:36 -0700 Subject: [Python-ideas] List Revolution In-Reply-To: References: <743DA5B0-E3A0-4FEE-A32E-195EBF9319AE@gmail.com> Message-ID: On Sat, Sep 10, 2011 at 3:08 AM, Matt Joiner wrote: > Well it makes complete sense to me. Furthermore, Python even properly > supports the style of conditional he uses. a <= b < c IS a great way > to do things. 0 based indexing IS right for computing. Not sure if > you're trolling or I'm not counter-trolling... -.- > > EWD is the man. Dutch programmers in decades to come have a lot to live up to. I think I'm doing okay. :-) (And to those taking the thread seriously: this is all in jest. We won't change the indexing base. The idea is so preposterous that the only kind of response possible is to laugh with it.) --Guido > On Sat, Sep 10, 2011 at 2:10 PM, Carl Matthew Johnson > wrote: >> >> On Sep 9, 2011, at 3:32 PM, Matt Joiner wrote: >> >>> Recommended reading: >>> http://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html >> >> The reasoning in this letter is so terrible. Why do people quote this whenever this comes up? "We should start lists at 0 because when writing for-loops the <= look goofy otherwise." Why on earth should Pythonistas care what the for-loops are like when we only have for-each loops! It's crazy. >> >> Look, EWD was a good computer scientist (=MATHEMATICIAN), but he was terrible when it comes to the ART of programming language design. There's no reason to think that being a good mathematician would make one also good at designing a user interface, and that's what a programming language is--a user interface for highly advanced users. Other than "Go To Considered Harmful" everything I've read of his about programming has been wrong. (Correctness proofs, really?) And much of that wrongness stems from EWD's false belief that programming should be the same as math. >> >> Now, as it happens, I don't think we should change to using 1 (although I do note that Lua gets along fine with it). But the reason not to change is because 0 is a well entrenched standard so most programmers are more used to it than to 1. In other words, it's the standard because in UI design you stick with what your users will expect. It has nothing to do with some arcane mathematician's gobbledygook writings about greater than and less than signs. Programming language design is the art of making something that helps humans work more efficiently, not some pure mathematics descended from Platonic heaven. >> >> Sorry to rant, but that article always comes up and so I have grudge against EWD. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (python.org/~guido) From stefan_ml at behnel.de Sat Sep 10 18:09:42 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 10 Sep 2011 18:09:42 +0200 Subject: [Python-ideas] yield statements (was relaxing keyword usage restrictions) In-Reply-To: <1315604085.16635.70.camel@Gutsy> References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> <4E6A5923.3030603@improva.dk> <1315604085.16635.70.camel@Gutsy> Message-ID: ron3200, 09.09.2011 23:34: > x = yield > > I really think it should have been "x = (yield)" or "x yield None". >[...] > x = y Assign x the object(s) y is assigned to. > x yield y assign x from yield, yield out y Whoa, that's ugly. It totally hides the fact that an assignment is taking place at all. Here, yield isn't operating on x, it's operating on y. Your syntax adds symmetry where there is none. (And, BTW, I agree that the yield should always be in parentheses, but I guess it's too late to change that before Py4.) Stefan From massimo.dipierro at gmail.com Sat Sep 10 18:35:21 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Sat, 10 Sep 2011 11:35:21 -0500 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> Here is my proposal which hopefully would settle the issue to rest class List(list): """ >>> a=List() >>> for i in range(100000): a.append(i) >>> print a.first 0 >>> print a.second 1 >>> print a.third 2 >>> print a.twentieth 19 >>> print a.twentysecond 21 >>> print a.onehundredthirtyfifth 134 >>> print a.onethousandfivehundredandthirtyeighth 1537 """ def __getattr__(self,name): import re if name.endswith('first'): name = name[:-5]+'one' elif name.endswith('second'): name = name[:-6]+'two' elif name.endswith('third'): name = name[:-5]+'three' elif name.endswith('fth'): name = name[:-3]+'ve' elif name.endswith('hth'): name = name[:-3]+'th' elif name.endswith('ieth'): name = name[:-4]+'y' elif name.endswith('th'): name = name[:-2] subs = { 'one':'+1', 'two':'+2', 'three':'+3', 'four':'+4', 'five':'+5', 'six':'+6', 'seven':'+7', 'eigth':'+8', 'nine':'+9', 'ten':'+10', 'eleven':'+11', 'twelve':'+12', 'thirteen':'+13', 'fourteen':'+14', 'fiftheen':'+15', 'sixteen':'+16', 'seventeen':'+17', 'eighteen':'+18', 'nineteen':'+19', 'ten':'+10', 'twenty':'+20', 'thirty':'+30', 'fourty':'+40', 'fifthy':'+50', 'sixty':'+60', 'seventy':'+70', 'eighty':'+80', 'ninety':'+90', 'hundred':')*100+(', 'thousand':')*1000+(', 'million':')*1000000+(', 'billion':')*1000000000+(', 'trillion':')*100000000000+(', 'and',''} for key,value in subs.items(): name = name.replace(key,value) if '(' in name: name='('+name+')' name.replace('()','1') if not re.compile('[\d\+\*\(\)]+').match(name): return AttributeError try: return self[eval(name)-1] except: raise AttributeError I would like to stress that the use of eval in this case is safe as the expression is first validated against a regex. Massimo From guido at python.org Sat Sep 10 18:39:27 2011 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Sep 2011 09:39:27 -0700 Subject: [Python-ideas] yield statements (was relaxing keyword usage restrictions) In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> <4E6A5923.3030603@improva.dk> <1315604085.16635.70.camel@Gutsy> Message-ID: On Sat, Sep 10, 2011 at 9:09 AM, Stefan Behnel wrote: > (And, BTW, I agree that the yield should always be in parentheses, but I > guess it's too late to change that before Py4.) You can make that a personal style today. And you could lobby for inclusion in PEP 8 (though personally I don't care either way). -- --Guido van Rossum (python.org/~guido) From stefan_ml at behnel.de Sat Sep 10 18:45:58 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 10 Sep 2011 18:45:58 +0200 Subject: [Python-ideas] List Revolution In-Reply-To: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> References: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> Message-ID: Massimo Di Pierro, 10.09.2011 18:35: > [...] > if not re.compile('[\d\+\*\(\)]+').match(name): return AttributeError >[...] > I would like to stress that the use of eval in this case is safe as the expression is first validated against a regex. Nice try. And it would even be ok if you had assured that the *entire* expression was validated against the regex. Stefan From massimo.dipierro at gmail.com Sat Sep 10 19:05:44 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Sat, 10 Sep 2011 12:05:44 -0500 Subject: [Python-ideas] List Revolution In-Reply-To: References: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> Message-ID: <3E0F0EE4-6117-4263-805C-BA2D74F425C8@gmail.com> LOL. Damn dollar signs. if not re.compile('^[\d\+\*\(\)]+$').match(name): return AttributeError On Sep 10, 2011, at 11:45 AM, Stefan Behnel wrote: > Massimo Di Pierro, 10.09.2011 18:35: >> [...] >> if not re.compile('[\d\+\*\(\)]+').match(name): return AttributeError >> [...] >> I would like to stress that the use of eval in this case is safe as the expression is first validated against a regex. > > Nice try. And it would even be ok if you had assured that the *entire* expression was validated against the regex. > > Stefan > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From g.nius.ck at gmail.com Sat Sep 10 19:41:06 2011 From: g.nius.ck at gmail.com (Christopher King) Date: Sat, 10 Sep 2011 13:41:06 -0400 Subject: [Python-ideas] List Revolution In-Reply-To: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> References: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> Message-ID: > > >>> print a.first > 0 > >>> print a.second > 1 > >>> print a.third > 2 > >>> print a.twentieth > 19 > >>> print a.twentysecond > 21 > >>> print a.onehundredthirtyfifth > 134 > >>> print a.onethousandfivehundredandthirtyeighth > 1537 This isn't good, because it requires to much typing. That's like saying that we should use the names of integers instead of the number. Also, the range would probably be changed to anyway. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Sat Sep 10 19:43:53 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sat, 10 Sep 2011 13:43:53 -0400 Subject: [Python-ideas] List Revolution In-Reply-To: <3E0F0EE4-6117-4263-805C-BA2D74F425C8@gmail.com> References: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> <3E0F0EE4-6117-4263-805C-BA2D74F425C8@gmail.com> Message-ID: > Damn dollar signs. > if not re.compile('^[\d\+\*\(\)]+$').match(name): return AttributeError '(2**1024)**(2**1024)' etc. Devin On Sat, Sep 10, 2011 at 1:05 PM, Massimo Di Pierro wrote: > LOL. > > Damn dollar signs. > if not re.compile('^[\d\+\*\(\)]+$').match(name): return AttributeError > > On Sep 10, 2011, at 11:45 AM, Stefan Behnel wrote: > >> Massimo Di Pierro, 10.09.2011 18:35: >>> [...] >>> ? ? ? ? if not re.compile('[\d\+\*\(\)]+').match(name): return AttributeError >>> [...] >>> I would like to stress that the use of eval in this case is safe as the expression is first validated against a regex. >> >> Nice try. And it would even be ok if you had assured that the *entire* expression was validated against the regex. >> >> Stefan >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From massimo.dipierro at gmail.com Sat Sep 10 19:48:46 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Sat, 10 Sep 2011 12:48:46 -0500 Subject: [Python-ideas] List Revolution In-Reply-To: References: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> <3E0F0EE4-6117-4263-805C-BA2D74F425C8@gmail.com> Message-ID: remember name is an attribute. It cannot contain special symbols (*) by definition. There is no way you can build "**". On Sep 10, 2011, at 12:43 PM, Devin Jeanpierre wrote: >> Damn dollar signs. >> if not re.compile('^[\d\+\*\(\)]+$').match(name): return AttributeError > > '(2**1024)**(2**1024)' etc. > > Devin > > On Sat, Sep 10, 2011 at 1:05 PM, Massimo Di Pierro > wrote: >> LOL. >> >> Damn dollar signs. >> if not re.compile('^[\d\+\*\(\)]+$').match(name): return AttributeError >> >> On Sep 10, 2011, at 11:45 AM, Stefan Behnel wrote: >> >>> Massimo Di Pierro, 10.09.2011 18:35: >>>> [...] >>>> if not re.compile('[\d\+\*\(\)]+').match(name): return AttributeError >>>> [...] >>>> I would like to stress that the use of eval in this case is safe as the expression is first validated against a regex. >>> >>> Nice try. And it would even be ok if you had assured that the *entire* expression was validated against the regex. >>> >>> Stefan >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> http://mail.python.org/mailman/listinfo/python-ideas >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas >> From massimo.dipierro at gmail.com Sat Sep 10 19:50:06 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Sat, 10 Sep 2011 12:50:06 -0500 Subject: [Python-ideas] List Revolution In-Reply-To: References: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> <3E0F0EE4-6117-4263-805C-BA2D74F425C8@gmail.com> Message-ID: <618FDEB2-AC5B-4853-B72B-7ABB9A405606@gmail.com> Anyway ... mine was not a serious proposal. I like python lists as they are. On Sep 10, 2011, at 12:43 PM, Devin Jeanpierre wrote: >> Damn dollar signs. >> if not re.compile('^[\d\+\*\(\)]+$').match(name): return AttributeError > > '(2**1024)**(2**1024)' etc. > > Devin > > On Sat, Sep 10, 2011 at 1:05 PM, Massimo Di Pierro > wrote: >> LOL. >> >> Damn dollar signs. >> if not re.compile('^[\d\+\*\(\)]+$').match(name): return AttributeError >> >> On Sep 10, 2011, at 11:45 AM, Stefan Behnel wrote: >> >>> Massimo Di Pierro, 10.09.2011 18:35: >>>> [...] >>>> if not re.compile('[\d\+\*\(\)]+').match(name): return AttributeError >>>> [...] >>>> I would like to stress that the use of eval in this case is safe as the expression is first validated against a regex. >>> >>> Nice try. And it would even be ok if you had assured that the *entire* expression was validated against the regex. >>> >>> Stefan >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> http://mail.python.org/mailman/listinfo/python-ideas >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas >> From guido at python.org Sat Sep 10 20:00:24 2011 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Sep 2011 11:00:24 -0700 Subject: [Python-ideas] List Revolution In-Reply-To: <618FDEB2-AC5B-4853-B72B-7ABB9A405606@gmail.com> References: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> <3E0F0EE4-6117-4263-805C-BA2D74F425C8@gmail.com> <618FDEB2-AC5B-4853-B72B-7ABB9A405606@gmail.com> Message-ID: I thought we were all just having a little bit of fun with what's obviously the least-likely-to-be-accepted proposal to have hit python-ideas in a long time; and now I see serious questions about it on Twitter and Quora... Time to start adding smileys to all posts! :-) (Though seriously, Massimo, you should be able to implement the same idea without using eval() at all.) --Guido On Sat, Sep 10, 2011 at 10:50 AM, Massimo Di Pierro wrote: > Anyway ... mine was not a serious proposal. I like python lists as they are. > > On Sep 10, 2011, at 12:43 PM, Devin Jeanpierre wrote: > >>> Damn dollar signs. >>> if not re.compile('^[\d\+\*\(\)]+$').match(name): return AttributeError >> >> '(2**1024)**(2**1024)' etc. >> >> Devin >> >> On Sat, Sep 10, 2011 at 1:05 PM, Massimo Di Pierro >> wrote: >>> LOL. >>> >>> Damn dollar signs. >>> if not re.compile('^[\d\+\*\(\)]+$').match(name): return AttributeError >>> >>> On Sep 10, 2011, at 11:45 AM, Stefan Behnel wrote: >>> >>>> Massimo Di Pierro, 10.09.2011 18:35: >>>>> [...] >>>>> ? ? ? ? if not re.compile('[\d\+\*\(\)]+').match(name): return AttributeError >>>>> [...] >>>>> I would like to stress that the use of eval in this case is safe as the expression is first validated against a regex. >>>> >>>> Nice try. And it would even be ok if you had assured that the *entire* expression was validated against the regex. >>>> >>>> Stefan >>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> http://mail.python.org/mailman/listinfo/python-ideas >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> http://mail.python.org/mailman/listinfo/python-ideas >>> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (python.org/~guido) From massimo.dipierro at gmail.com Sat Sep 10 20:50:46 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Sat, 10 Sep 2011 13:50:46 -0500 Subject: [Python-ideas] List Revolution In-Reply-To: References: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> <3E0F0EE4-6117-4263-805C-BA2D74F425C8@gmail.com> <618FDEB2-AC5B-4853-B72B-7ABB9A405606@gmail.com> Message-ID: <46293340-AFAA-4A78-B36B-C0B859C23B60@gmail.com> Of course. It was a proof of concept. Here is with an inline parser: class List(list): """ >>> a=List() >>> for i in range(100000): a.append(i) >>> print a.first 0 >>> print a.second 1 >>> print a.third 2 >>> print a.twentieth 19 >>> print a.twentysecond 21 >>> print a.onehundredthirtyfifth 134 >>> print a.onethousandfivehundredthirtyeighth 1537 """ def __getattr__(self,name): if name.endswith('first'): name = name[:-5]+'one' elif name.endswith('second'): name = name[:-6]+'two' elif name.endswith('third'): name = name[:-5]+'three' elif name.endswith('fth'): name = name[:-3]+'ve' elif name.endswith('hth'): name = name[:-3]+'th' elif name.endswith('ieth'): name = name[:-4]+'y' elif name.endswith('th'): name = name[:-2] subs = [ ("eleven","+11"), ("twelve","+12"), ("thirteen","+13"), ("fourteen","+14"), ("fiftheen","+15"), ("sixteen","+16"), ("seventeen","+17"), ("eighteen","+18"), ("nineteen","+19"), ("ten","+10"), ("twenty","+20"), ("thirty","+30"), ("fourty","+40"), ("fifty","+50"), ("sixty","+60"), ("seventy","+70"), ("eighty","+80"), ("ninety","+90"), ("one","+1"), ("two","+2"), ("three","+3"), ("four","+4"), ("five","+5"), ("six","+6"), ("seven","+7"), ("eigth","+8"), ("nine","+9"), ("ten","+10"), ("hundred",")*100+("), ("thousand",")*1000+("), ("million",")*1000000+("), ("billion",")*1000000000+("), ("trillion",")*100000000000+("), ("and","")] for key,value in subs: name = name.replace(key,value) if '(' in name: name='('+name+')' name.replace('()','1') ## parser import re tokenizer = re.compile('\+|\(\+|\(|\)|\*|[\d]+') stack = [0] while name: token = tokenizer.match(name) if not token: raise AttributeError if token.group() in ('(','(+'): stack.append(0) if token.group().isdigit(): stack.append(int(token.group())) if token.group() in ')+' and len(stack)>1: stack[-2:]=[stack[-1]+stack[-2]] if token.group()=='*': name = name[1:] token = tokenizer.match(name) if not token or not token.group().isdigit(): raise AttrbuteError stack[-1:]=[stack[-1]*int(token.group())] name = name[token.end():] number = sum(stack) ## end parser try: return self[number-1] except: raise IndexError On Sep 10, 2011, at 1:00 PM, Guido van Rossum wrote: > I thought we were all just having a little bit of fun with what's > obviously the least-likely-to-be-accepted proposal to have hit > python-ideas in a long time; and now I see serious questions about it > on Twitter and Quora... Time to start adding smileys to all posts! :-) > > (Though seriously, Massimo, you should be able to implement the same > idea without using eval() at all.) > > --Guido > > On Sat, Sep 10, 2011 at 10:50 AM, Massimo Di Pierro > wrote: >> Anyway ... mine was not a serious proposal. I like python lists as they are. >> >> On Sep 10, 2011, at 12:43 PM, Devin Jeanpierre wrote: >> >>>> Damn dollar signs. >>>> if not re.compile('^[\d\+\*\(\)]+$').match(name): return AttributeError >>> >>> '(2**1024)**(2**1024)' etc. >>> >>> Devin >>> >>> On Sat, Sep 10, 2011 at 1:05 PM, Massimo Di Pierro >>> wrote: >>>> LOL. >>>> >>>> Damn dollar signs. >>>> if not re.compile('^[\d\+\*\(\)]+$').match(name): return AttributeError >>>> >>>> On Sep 10, 2011, at 11:45 AM, Stefan Behnel wrote: >>>> >>>>> Massimo Di Pierro, 10.09.2011 18:35: >>>>>> [...] >>>>>> if not re.compile('[\d\+\*\(\)]+').match(name): return AttributeError >>>>>> [...] >>>>>> I would like to stress that the use of eval in this case is safe as the expression is first validated against a regex. >>>>> >>>>> Nice try. And it would even be ok if you had assured that the *entire* expression was validated against the regex. >>>>> >>>>> Stefan >>>>> >>>>> _______________________________________________ >>>>> Python-ideas mailing list >>>>> Python-ideas at python.org >>>>> http://mail.python.org/mailman/listinfo/python-ideas >>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> http://mail.python.org/mailman/listinfo/python-ideas >>>> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas >> > > > > -- > --Guido van Rossum (python.org/~guido) From massimo.dipierro at gmail.com Sat Sep 10 21:09:25 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Sat, 10 Sep 2011 14:09:25 -0500 Subject: [Python-ideas] List Revolution In-Reply-To: <4E20DA04-E7BD-4141-AC5B-6AD42DF6191F@cs.depaul.edu> References: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> <3E0F0EE4-6117-4263-805C-BA2D74F425C8@gmail.com> <618FDEB2-AC5B-4853-B72B-7ABB9A405606@gmail.com> <46293340-AFAA-4A78-B36B-C0B859C23B60@gmail.com> <4E20DA04-E7BD-4141-AC5B-6AD42DF6191F@cs.depaul.edu> Message-ID: On Sep 10, 2011, at 2:07 PM, Massimo Di Pierro wrote: > Not to open a can of worms but why not use eval in this example? > It is faster. It is 5x more compact. It allows me to take advantage of the Python parser instead of reinventing the wheel. It is less error prone and easier to understand. It is a python keyword. > > # with eval > if not re.compile('^[\*\+\(\)\d]+$').match(name): raise AttributeError > try: number = eval(name) > except: raise AttributeError > > # without eval > tokenizer = re.compile('\+|\(\+|\(|\)|\*|[\d]+') > stack = [0] > while name: > token = tokenizer.match(name) > if not token: raise AttributeError > elif token.group() in ('(','(+'): stack.append(0) > elif token.group().isdigit(): stack.append(int(token.group())) > elif token.group() in ')+' and len(stack)>1: stack[-2:]=[stack[-1]+stack[-2]] > elif token.group()=='*': > name = name[1:] > token = tokenizer.match(name) > if not token or not token.group().isdigit(): raise AttrbuteError > stack[-1:]=[stack[-1]*int(token.group())] > name = name[token.end():] > number = sum(stack) > > > > Massimo > > > On Sep 10, 2011, at 1:50 PM, Massimo Di Pierro wrote: > >> Of course. It was a proof of concept. Here is with an inline parser: >> >> class List(list): >> """ >>>>> a=List() >>>>> for i in range(100000): a.append(i) >>>>> print a.first >> 0 >>>>> print a.second >> 1 >>>>> print a.third >> 2 >>>>> print a.twentieth >> 19 >>>>> print a.twentysecond >> 21 >>>>> print a.onehundredthirtyfifth >> 134 >>>>> print a.onethousandfivehundredthirtyeighth >> 1537 >> """ >> def __getattr__(self,name): >> if name.endswith('first'): name = name[:-5]+'one' >> elif name.endswith('second'): name = name[:-6]+'two' >> elif name.endswith('third'): name = name[:-5]+'three' >> elif name.endswith('fth'): name = name[:-3]+'ve' >> elif name.endswith('hth'): name = name[:-3]+'th' >> elif name.endswith('ieth'): name = name[:-4]+'y' >> elif name.endswith('th'): name = name[:-2] >> subs = [ >> ("eleven","+11"), >> ("twelve","+12"), >> ("thirteen","+13"), >> ("fourteen","+14"), >> ("fiftheen","+15"), >> ("sixteen","+16"), >> ("seventeen","+17"), >> ("eighteen","+18"), >> ("nineteen","+19"), >> ("ten","+10"), >> ("twenty","+20"), >> ("thirty","+30"), >> ("fourty","+40"), >> ("fifty","+50"), >> ("sixty","+60"), >> ("seventy","+70"), >> ("eighty","+80"), >> ("ninety","+90"), >> ("one","+1"), >> ("two","+2"), >> ("three","+3"), >> ("four","+4"), >> ("five","+5"), >> ("six","+6"), >> ("seven","+7"), >> ("eigth","+8"), >> ("nine","+9"), >> ("ten","+10"), >> ("hundred",")*100+("), >> ("thousand",")*1000+("), >> ("million",")*1000000+("), >> ("billion",")*1000000000+("), >> ("trillion",")*100000000000+("), >> ("and","")] >> for key,value in subs: name = name.replace(key,value) >> if '(' in name: name='('+name+')' >> name.replace('()','1') >> ## parser >> import re >> tokenizer = re.compile('\+|\(\+|\(|\)|\*|[\d]+') >> stack = [0] >> while name: >> token = tokenizer.match(name) >> if not token: raise AttributeError >> if token.group() in ('(','(+'): stack.append(0) >> if token.group().isdigit(): stack.append(int(token.group())) >> if token.group() in ')+' and len(stack)>1: stack[-2:]=[stack[-1]+stack[-2]] >> if token.group()=='*': >> name = name[1:] >> token = tokenizer.match(name) >> if not token or not token.group().isdigit(): raise AttrbuteError >> stack[-1:]=[stack[-1]*int(token.group())] >> name = name[token.end():] >> number = sum(stack) >> ## end parser >> try: return self[number-1] >> except: raise IndexError >> >> >> >> >> On Sep 10, 2011, at 1:00 PM, Guido van Rossum wrote: >> >>> I thought we were all just having a little bit of fun with what's >>> obviously the least-likely-to-be-accepted proposal to have hit >>> python-ideas in a long time; and now I see serious questions about it >>> on Twitter and Quora... Time to start adding smileys to all posts! :-) >>> >>> (Though seriously, Massimo, you should be able to implement the same >>> idea without using eval() at all.) >>> >>> --Guido >>> >>> On Sat, Sep 10, 2011 at 10:50 AM, Massimo Di Pierro >>> wrote: >>>> Anyway ... mine was not a serious proposal. I like python lists as they are. >>>> >>>> On Sep 10, 2011, at 12:43 PM, Devin Jeanpierre wrote: >>>> >>>>>> Damn dollar signs. >>>>>> if not re.compile('^[\d\+\*\(\)]+$').match(name): return AttributeError >>>>> >>>>> '(2**1024)**(2**1024)' etc. >>>>> >>>>> Devin >>>>> >>>>> On Sat, Sep 10, 2011 at 1:05 PM, Massimo Di Pierro >>>>> wrote: >>>>>> LOL. >>>>>> >>>>>> Damn dollar signs. >>>>>> if not re.compile('^[\d\+\*\(\)]+$').match(name): return AttributeError >>>>>> >>>>>> On Sep 10, 2011, at 11:45 AM, Stefan Behnel wrote: >>>>>> >>>>>>> Massimo Di Pierro, 10.09.2011 18:35: >>>>>>>> [...] >>>>>>>> if not re.compile('[\d\+\*\(\)]+').match(name): return AttributeError >>>>>>>> [...] >>>>>>>> I would like to stress that the use of eval in this case is safe as the expression is first validated against a regex. >>>>>>> >>>>>>> Nice try. And it would even be ok if you had assured that the *entire* expression was validated against the regex. >>>>>>> >>>>>>> Stefan >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Python-ideas mailing list >>>>>>> Python-ideas at python.org >>>>>>> http://mail.python.org/mailman/listinfo/python-ideas >>>>>> >>>>>> _______________________________________________ >>>>>> Python-ideas mailing list >>>>>> Python-ideas at python.org >>>>>> http://mail.python.org/mailman/listinfo/python-ideas >>>>>> >>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> http://mail.python.org/mailman/listinfo/python-ideas >>>> >>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >> > From massimo.dipierro at gmail.com Sat Sep 10 21:14:13 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Sat, 10 Sep 2011 14:14:13 -0500 Subject: [Python-ideas] List Revolution In-Reply-To: References: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> <3E0F0EE4-6117-4263-805C-BA2D74F425C8@gmail.com> <618FDEB2-AC5B-4853-B72B-7ABB9A405606@gmail.com> Message-ID: <0DA77FB5-97E4-45F3-85E1-A8A5C5BA15AB@gmail.com> Not to open a can of worms but why not use eval in this example? It is faster. It is 5x more compact. It allows me to take advantage of the Python parser instead of reinventing the wheel. It is less error prone and easier to understand. It is a python keyword. # with eval if not re.compile('^[\*\+\(\)\d]+$').match(name): raise AttributeError try: number = eval(name) except: raise AttributeError # without eval tokenizer = re.compile('\+|\(\+|\(|\)|\*|[\d]+') stack = [0] while name: token = tokenizer.match(name) if not token: raise AttributeError elif token.group() in ('(','(+'): stack.append(0) elif token.group().isdigit(): stack.append(int(token.group())) elif token.group() in ')+' and len(stack)>1: stack[-2:]=[stack[-1]+stack[-2]] elif token.group()=='*': name = name[1:] token = tokenizer.match(name) if not token or not token.group().isdigit(): raise AttrbuteError stack[-1:]=[stack[-1]*int(token.group())] name = name[token.end():] number = sum(stack) Massimo On Sep 10, 2011, at 1:00 PM, Guido van Rossum wrote: > I thought we were all just having a little bit of fun with what's > obviously the least-likely-to-be-accepted proposal to have hit > python-ideas in a long time; and now I see serious questions about it > on Twitter and Quora... Time to start adding smileys to all posts! :-) > > (Though seriously, Massimo, you should be able to implement the same > idea without using eval() at all.) > > --Guido From matt at vazor.com Sat Sep 10 21:22:29 2011 From: matt at vazor.com (Matt Billenstein) Date: Sat, 10 Sep 2011 20:22:29 +0100 Subject: [Python-ideas] List Revolution Message-ID: <4tjo0h8x3n28.1l1k6x6f@elasticemail.com> On Sat, Sep 10, 2011 at 02:14:13PM -0500, Massimo Di Pierro wrote: > Not to open a can of worms but why not use eval in this example? > It is faster. It is 5x more compact. It allows me to take advantage of the > Python parser instead of reinventing the wheel. It is less error prone and > easier to understand. It is a python keyword. I wouldn't worry too much about the implementation details of a clearly insane idea... And eval generally has high "code-smell" -- it can lead to code injection exploits. m -- Matt Billenstein matt at vazor.com http://www.vazor.com/ From massimo.dipierro at gmail.com Sat Sep 10 21:27:01 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Sat, 10 Sep 2011 14:27:01 -0500 Subject: [Python-ideas] List Revolution In-Reply-To: <4tjo0h8x3n28.1l1k6w6e@elasticemail.com> References: <4tjo0h8x3n28.1l1k6w6e@elasticemail.com> Message-ID: On Sep 10, 2011, at 2:22 PM, Matt Billenstein wrote: > On Sat, Sep 10, 2011 at 02:14:13PM -0500, Massimo Di Pierro wrote: >> Not to open a can of worms but why not use eval in this example? >> It is faster. It is 5x more compact. It allows me to take advantage of the >> Python parser instead of reinventing the wheel. It is less error prone and >> easier to understand. It is a python keyword. > > I wouldn't worry too much about the implementation details of a clearly insane > idea... agreed > And eval generally has high "code-smell" -- it can lead to code injection exploits. Not in this example. So why not? Any code can introduce vulnerabilities if incorrect. From jeanpierreda at gmail.com Sat Sep 10 21:37:26 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sat, 10 Sep 2011 15:37:26 -0400 Subject: [Python-ideas] List Revolution In-Reply-To: References: <4tjo0h8x3n28.1l1k6w6e@elasticemail.com> Message-ID: > Not in this example. So why not? Yes it did in this example. You introduced a vulnerability in your first attempt, and it was found by somebody else. eval makes accidental vulnerabilities way easier than they should be, even for people that are trying to be careful. Even your modified version had something that in some circumstances would be DoS-attackable because your verification still allowed for unintended operations. But yeah, if you can guarantee that there won't be any ill-effects from using eval, _of course_ it's OK. The problem is that these guarantees are too easy to get wrong, or to forget, or to misunderstand. Devin On Sat, Sep 10, 2011 at 3:27 PM, Massimo Di Pierro wrote: > On Sep 10, 2011, at 2:22 PM, Matt Billenstein wrote: > >> On Sat, Sep 10, 2011 at 02:14:13PM -0500, Massimo Di Pierro wrote: >>> Not to open a can of worms but why not use eval in this example? >>> It is faster. It is 5x more compact. It allows me to take advantage of the >>> Python parser instead of reinventing the wheel. It is less error prone and >>> easier to understand. It is a python keyword. >> >> I wouldn't worry too much about the implementation details of a clearly insane >> idea... > > agreed > >> And eval generally has high "code-smell" -- it can lead to code injection exploits. > > Not in this example. So why not? > Any code can introduce vulnerabilities if incorrect. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From tjreedy at udel.edu Sat Sep 10 21:43:42 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 10 Sep 2011 15:43:42 -0400 Subject: [Python-ideas] List Revolution In-Reply-To: References: <743DA5B0-E3A0-4FEE-A32E-195EBF9319AE@gmail.com> Message-ID: On 9/10/2011 6:24 AM, Jakob Bowyer wrote: > Im dead confused. Is this a joke? Yes, a September Fools' Day joke. -- Terry Jan Reedy From massimo.dipierro at gmail.com Sat Sep 10 21:51:08 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Sat, 10 Sep 2011 14:51:08 -0500 Subject: [Python-ideas] List Revolution In-Reply-To: References: <4tjo0h8x3n28.1l1k6w6e@elasticemail.com> Message-ID: First, the "vulnerability" was fixed. (*) Second, the parser could also have vulnerabilities. In fact I the vulnerability in the eval code was easy to find. How do you know there is no bug in the parser loop that I implemented that will cause some strings to go into an infinite loop?The explicit parser may still have a vulnerability, it is just more difficult to debug. Perhaps I put both there on purpose to see which one you could find. Massimo (*) I put it in quote because the possible values in name are already limited by the fact that name was an attribute (can only contain alphanumeric characters, no ".", no spaces, etc. The fact that the regex had a typo does not imply a vulnerability, although I will not exclude it either. It just was not proved although I acknowledge the regex needed fixing. On Sep 10, 2011, at 2:37 PM, Devin Jeanpierre wrote: >> Not in this example. So why not? > > Yes it did in this example. You introduced a vulnerability in your > first attempt, and it was found by somebody else. eval makes > accidental vulnerabilities way easier than they should be, even for > people that are trying to be careful. Even your modified version had > something that in some circumstances would be DoS-attackable because > your verification still allowed for unintended operations. > > But yeah, if you can guarantee that there won't be any ill-effects > from using eval, _of course_ it's OK. The problem is that these > guarantees are too easy to get wrong, or to forget, or to > misunderstand. > > Devin > > On Sat, Sep 10, 2011 at 3:27 PM, Massimo Di Pierro > wrote: >> On Sep 10, 2011, at 2:22 PM, Matt Billenstein wrote: >> >>> On Sat, Sep 10, 2011 at 02:14:13PM -0500, Massimo Di Pierro wrote: >>>> Not to open a can of worms but why not use eval in this example? >>>> It is faster. It is 5x more compact. It allows me to take advantage of the >>>> Python parser instead of reinventing the wheel. It is less error prone and >>>> easier to understand. It is a python keyword. >>> >>> I wouldn't worry too much about the implementation details of a clearly insane >>> idea... >> >> agreed >> >>> And eval generally has high "code-smell" -- it can lead to code injection exploits. >> >> Not in this example. So why not? >> Any code can introduce vulnerabilities if incorrect. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas >> From guido at python.org Sat Sep 10 23:35:57 2011 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Sep 2011 14:35:57 -0700 Subject: [Python-ideas] List Revolution In-Reply-To: <4E20DA04-E7BD-4141-AC5B-6AD42DF6191F@cs.depaul.edu> References: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> <3E0F0EE4-6117-4263-805C-BA2D74F425C8@gmail.com> <618FDEB2-AC5B-4853-B72B-7ABB9A405606@gmail.com> <46293340-AFAA-4A78-B36B-C0B859C23B60@gmail.com> <4E20DA04-E7BD-4141-AC5B-6AD42DF6191F@cs.depaul.edu> Message-ID: On Sat, Sep 10, 2011 at 12:07 PM, Massimo Di Pierro wrote: > Not to open a can of worms but why not use eval in this example? > It is faster. It is 5x more compact. It allows me to take advantage of the Python parser instead of reinventing the wheel. It is less error prone and easier to understand. It is a python keyword. It is unsafe (as the bug in your first version inadvertently illustrated :-). I don't actually quite follow what your example does (sorry), but, even though it is less powerful than exec(), eval() invokes the full Python parser+compiler, which can take quite a bit of time to start up and uses a fair bit of memory. Also in other Python versions this is often much slower than in CPython, or has a big startup overhead. But, of course, it is great for quick demos! -- --Guido van Rossum (python.org/~guido) From g.nius.ck at gmail.com Sun Sep 11 00:11:48 2011 From: g.nius.ck at gmail.com (Christopher King) Date: Sat, 10 Sep 2011 18:11:48 -0400 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: I think there is some confusion about my idea (I didn't intend it as a joke, but I was unsure, I was really surprised when Guido endorsed. I think people have been thinking that my idea was as follows: >>> items=['a', 'b', 'c'] >>> items.first 'a' >>> items.second 'b' >>> items.third 'c' It is *not* that. I would not want it to be in words. That would be dumb what my idea was was this: >>> items=['a', 'b', 'c'] >>> items[1] 'a' >>> items[2] 'b' >>> items[3] 'c' I'm not sure if this one is good, but if not, I'm sure you we come up with another line of jokes (I like jokes, so its alright.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Sep 11 00:34:45 2011 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Sep 2011 15:34:45 -0700 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: On Sat, Sep 10, 2011 at 3:11 PM, Christopher King wrote: > I think there is some confusion about my idea (I didn't intend it as a joke, > but I was unsure, I was really?surprised?when Guido endorsed. I think people > have been thinking that my idea was as follows: >>>> items=['a', 'b', 'c'] >>>> items.first > 'a' >>>> items.second > 'b' >>>> items.third > 'c' > It is not?that. I would not want it to be in words. That would be dumb what > my idea was was this: >>>> items=['a', 'b', 'c'] >>>> items[1] > 'a' >>>> items[2] > 'b' >>>> items[3] > 'c' > I'm not sure if this one is good, but if not, I'm sure you we come up with > another line of jokes (I like jokes, so its alright.) I understood that. -- I think others did too but found the proposal so preposterous that they started posting nonsensical "solutions" based on an intentionally literalistic misreading of your original post. (If you think this is odd, realize that Python was named after Monty Python's Flying Circus -- we occasionally like to show off our warped sense of humor. :-) Anyway, the reason your proposal is not going to fly, quite apart of whether it would be a good idea for a brand new language design(*), is that there is over 20 years of existing Python code that would have to be changed, not to mention the brains of millions of users, and hundreds of books about Python. Plus pretty much every other language in widespread use today (C, C++, Java, C#, JavaScript, Ruby, to name a few; presumably also Objective C given its C inheritance) agrees that indexes start at zero. It is a cultural battle that has been fought and won long ago (all the old languages used 1-based indexing: Fortran, Algol, Pascal) and it's really not that important in the grand scheme of things, so the status quo wins. (*) I personally think 0-based indexing is better, and the referenced EWD expresses why better than I could. But I'm sure that if we lived in a world where 1-based indexing was the norm I'd get by just fine. -- --Guido van Rossum (python.org/~guido) From brian.curtin at gmail.com Sun Sep 11 00:39:31 2011 From: brian.curtin at gmail.com (Brian Curtin) Date: Sat, 10 Sep 2011 17:39:31 -0500 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: On Sat, Sep 10, 2011 at 17:11, Christopher King wrote: > I think there is some confusion about my idea (I didn't intend it as a joke, > but I was unsure, I was really?surprised?when Guido endorsed. I think people > have been thinking that my idea was as follows: >>>> items=['a', 'b', 'c'] >>>> items.first > 'a' >>>> items.second > 'b' >>>> items.third > 'c' > It is not?that. I would not want it to be in words. That would be dumb what > my idea was was this: >>>> items=['a', 'b', 'c'] >>>> items[1] > 'a' >>>> items[2] > 'b' >>>> items[3] > 'c' > I'm not sure if this one is good, but if not, I'm sure you we come up with > another line of jokes (I like jokes, so its alright.) I think everyone knows what your suggestion was - list indexes starting at 1 instead of 0. Simply put, it won't happen. Think of this: by the time Python 4 even rolls around, it might be 10+ years down the road, putting Python at 30+ years old. We can't change something so fundamental to not only Python, but almost all other languages, after 30+ years of existence in Python and even more elsewhere. From python at mrabarnett.plus.com Sun Sep 11 02:26:57 2011 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 11 Sep 2011 01:26:57 +0100 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: <4E6C0051.2080501@mrabarnett.plus.com> On 10/09/2011 23:34, Guido van Rossum wrote: [snip] > Anyway, the reason your proposal is not going to fly, quite apart of > whether it would be a good idea for a brand new language design(*), is > that there is over 20 years of existing Python code that would have to > be changed, not to mention the brains of millions of users, and > hundreds of books about Python. Plus pretty much every other language > in widespread use today (C, C++, Java, C#, JavaScript, Ruby, to name a > few; presumably also Objective C given its C inheritance) agrees that > indexes start at zero. It is a cultural battle that has been fought > and won long ago (all the old languages used 1-based indexing: > Fortran, Algol, Pascal) and it's really not that important in the > grand scheme of things, so the status quo wins. > In Pascal you specify the index range, which can be any enumerated type, not just integer. > (*) I personally think 0-based indexing is better, and the referenced > EWD expresses why better than I could. But I'm sure that if we lived > in a world where 1-based indexing was the norm I'd get by just fine. > From steve at pearwood.info Sun Sep 11 02:37:21 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 11 Sep 2011 10:37:21 +1000 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: <4E6C02C1.9050704@pearwood.info> Guido van Rossum wrote: > Plus pretty much every other language > in widespread use today (C, C++, Java, C#, JavaScript, Ruby, to name a > few; presumably also Objective C given its C inheritance) agrees that > indexes start at zero. It is a cultural battle that has been fought > and won long ago (all the old languages used 1-based indexing: > Fortran, Algol, Pascal) Lua appears to be a conspicuous counter-example of a relatively recent popular language with 1-based indexing. See the (extremely incomplete!) list here: http://en.wikipedia.org/wiki/Comparison_of_programming_languages_(string_functions) 0-based indexes are useful for some tasks, and less useful for other tasks. In my experience, I find that 0-based indexing is more useful most of the time: it leads to fewer off-by-one errors. 1-based indexes are particularly well-suited for programming languages using a natural language metaphor, usually aimed at non-programmers. Examples include Xion, Applescript, and Inform-7. The Ubiquity desktop appears to be aiming for a natural language scripting language, like Applescript only more so: http://mitcho.com/blog/projects/how-natural-should-a-natural-interface-be/ -- Steven From ncoghlan at gmail.com Sun Sep 11 03:06:51 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Sep 2011 11:06:51 +1000 Subject: [Python-ideas] yield statements (was relaxing keyword usage restrictions) In-Reply-To: References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> <4E6A5923.3030603@improva.dk> <1315604085.16635.70.camel@Gutsy> Message-ID: On Sun, Sep 11, 2011 at 2:39 AM, Guido van Rossum wrote: > On Sat, Sep 10, 2011 at 9:09 AM, Stefan Behnel wrote: >> (And, BTW, I agree that the yield should always be in parentheses, but I >> guess it's too late to change that before Py4.) > > You can make that a personal style today. And you could lobby for > inclusion in PEP 8 (though personally I don't care either way). As the examples in the PEP suggest, the "no parens" usage was for simple evolutions from the pre-existing yield statement syntax: yield None => x = yield yield 42 => x = yield 42 yield y => x = yield y For simple cases like that, the parens would be redundant visual clutter (similar to the ill-advised practice of including redundant parens in if statements: "if (expr):"). Beyond those simple cases, I don't think there needs to be a yield specific recommendation - the general guidelines to use parentheses and named subexpressions to improve readability should cover it. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From greg.ewing at canterbury.ac.nz Sun Sep 11 03:10:36 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 11 Sep 2011 13:10:36 +1200 Subject: [Python-ideas] relaxing keyword usage restrictions In-Reply-To: <87sjo5ccv7.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87vct7idp4.fsf@uwakimon.sk.tsukuba.ac.jp> <87sjo5ccv7.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4E6C0A8C.5030209@canterbury.ac.nz> Stephen J. Turnbull wrote: > Except that as somebody pointed out, there is a need. It turns out > that users want keywords in the form of "constants that evaluate to > themselves". You don't strictly *need* them, though, because a variable that evaluates to the desired constant has much the same effect. (Except for the possibility of shadowing, to which the Pythonic answer is "don't do that, then".) -- Greg From greg.ewing at canterbury.ac.nz Sun Sep 11 03:26:33 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 11 Sep 2011 13:26:33 +1200 Subject: [Python-ideas] List Revolution In-Reply-To: <743DA5B0-E3A0-4FEE-A32E-195EBF9319AE@gmail.com> References: <743DA5B0-E3A0-4FEE-A32E-195EBF9319AE@gmail.com> Message-ID: <4E6C0E49.7080205@canterbury.ac.nz> Carl Matthew Johnson wrote: > The reasoning in this letter is so terrible... "We should start lists at 0 > because when writing for-loops the <= look goofy otherwise." Why on earth > should Pythonistas care what the for-loops are like when we only have > for-each loops! It's crazy. Um... have you actually read that article? He doesn't mention "for loops" at all, or any other kind of loop, for that matter. Neither does he say anything about the appearance of the comparison operators. He does talk about the values of the range endpoints, and mentions notations for ranges and array lower bounds in various languages. Python has equivalents of those -- sequence indexes, range() parameters, slice indexes, etc. Most of what he says applies perfectly well to Python. -- Greg From guido at python.org Sun Sep 11 03:40:30 2011 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Sep 2011 18:40:30 -0700 Subject: [Python-ideas] List Revolution In-Reply-To: <4E6C0E49.7080205@canterbury.ac.nz> References: <743DA5B0-E3A0-4FEE-A32E-195EBF9319AE@gmail.com> <4E6C0E49.7080205@canterbury.ac.nz> Message-ID: On Sat, Sep 10, 2011 at 6:26 PM, Greg Ewing wrote: > Carl Matthew Johnson wrote: > >> The reasoning in this letter is so terrible... "We should start lists at 0 >> because when writing for-loops the <= look goofy otherwise." Why on earth > >> should Pythonistas care what the for-loops are like when we only have >> for-each loops! It's crazy. > > Um... have you actually read that article? He doesn't mention "for loops" at > all, or any other kind of loop, for that matter. Neither does he say > anything about the appearance of the comparison operators. > > He does talk about the values of the range endpoints, and mentions notations > for ranges and array lower bounds in various languages. Python has equivalents > of those -- sequence indexes, range() parameters, slice indexes, etc. Most > of what he says applies perfectly well to Python. EWD points out the power of half-open intervals, which I independently discovered when I considered the issue of indexing and slicing in Python. If you consider that arcane mathematics I'm not sure I really want to use any code you wrote... -- --Guido van Rossum (python.org/~guido) From greg.ewing at canterbury.ac.nz Sun Sep 11 03:44:48 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 11 Sep 2011 13:44:48 +1200 Subject: [Python-ideas] List Revolution In-Reply-To: References: <743DA5B0-E3A0-4FEE-A32E-195EBF9319AE@gmail.com> Message-ID: <4E6C1290.1000507@canterbury.ac.nz> Jakob Bowyer wrote: > Im dead confused. Is this a joke? Or are we actually switching to > index-1 based lists like Lua? Because I like index-0. Clearly a compromise is called for here. We should switch to fermionic sequences with half-integer indices: 1/2, 3/2, 5/2, ... -- Greg From ncoghlan at gmail.com Sun Sep 11 03:51:14 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 11 Sep 2011 11:51:14 +1000 Subject: [Python-ideas] List Revolution In-Reply-To: <4E6C02C1.9050704@pearwood.info> References: <4E6C02C1.9050704@pearwood.info> Message-ID: On Sun, Sep 11, 2011 at 10:37 AM, Steven D'Aprano wrote: > 0-based indexes are useful for some tasks, and less useful for other tasks. > In my experience, I find that 0-based indexing is more useful most of the > time: it leads to fewer off-by-one errors. > > 1-based indexes are particularly well-suited for programming languages using > a natural language metaphor, usually aimed at non-programmers. Examples > include Xion, Applescript, and Inform-7. Indeed, the concepts of half-open ranges and 0-based indexing go hand in hand (as described in the EWD article), and it ties directly in to the notion of *index arithmetic*. A case that illustrates this nicely is that of partitioning a sequence. Suppose we want the first 5 items in one subsequence and the rest in another. This is easy to write, and requires no adjustments to the numbers: head = seq[:5] assert len(head) == 5 tail = seq[5:] assert len(tail) == len(seq) - 5 Zero based indexing (in conjunction with half-open ranges) makes the arithmetic work out nicely, and, in practice, that turns out to be important when it comes to writing correct programs. However, it comes at the cost of breaking the intuitive mapping to the counting numbers: the first item is at offset 0, the second is at offset 1, etc. This is a definite downside, but the collective judgment of many language designers is that the reduction in off-by-one errors when manipulating indices is worth the additional difficulty in learning the language for programming novices. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From massimo.dipierro at gmail.com Sun Sep 11 04:45:03 2011 From: massimo.dipierro at gmail.com (Massimo DiPierro) Date: Sat, 10 Sep 2011 21:45:03 -0500 Subject: [Python-ideas] List Revolution In-Reply-To: References: <4E6C02C1.9050704@pearwood.info> Message-ID: <8E66CA38-7D08-445A-A556-D2E907416612@gmail.com> I was about to propose this: class List(list): def __getitem__(self,i): """ a = List() a.append(4) a.appned(5) print a[0.5] 4.5 """ if isinstance(i,(int,long)): return list.__getitem__(self,i) else: j = int(i) x = i - j return self[j]*(1.0-x)+self[j+1]*x On Sep 10, 2011, at 8:51 PM, Nick Coghlan wrote: > On Sun, Sep 11, 2011 at 10:37 AM, Steven D'Aprano wrote: >> 0-based indexes are useful for some tasks, and less useful for other tasks. >> In my experience, I find that 0-based indexing is more useful most of the >> time: it leads to fewer off-by-one errors. >> >> 1-based indexes are particularly well-suited for programming languages using >> a natural language metaphor, usually aimed at non-programmers. Examples >> include Xion, Applescript, and Inform-7. > > Indeed, the concepts of half-open ranges and 0-based indexing go hand > in hand (as described in the EWD article), and it ties directly in to > the notion of *index arithmetic*. > > A case that illustrates this nicely is that of partitioning a > sequence. Suppose we want the first 5 items in one subsequence and the > rest in another. This is easy to write, and requires no adjustments to > the numbers: > > head = seq[:5] > assert len(head) == 5 > tail = seq[5:] > assert len(tail) == len(seq) - 5 > > Zero based indexing (in conjunction with half-open ranges) makes the > arithmetic work out nicely, and, in practice, that turns out to be > important when it comes to writing correct programs. > > However, it comes at the cost of breaking the intuitive mapping to the > counting numbers: the first item is at offset 0, the second is at > offset 1, etc. This is a definite downside, but the collective > judgment of many language designers is that the reduction in > off-by-one errors when manipulating indices is worth the additional > difficulty in learning the language for programming novices. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From ericsnowcurrently at gmail.com Sun Sep 11 05:45:25 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 10 Sep 2011 21:45:25 -0600 Subject: [Python-ideas] List Revolution In-Reply-To: References: <4E6C02C1.9050704@pearwood.info> Message-ID: On Sat, Sep 10, 2011 at 7:51 PM, Nick Coghlan wrote: > On Sun, Sep 11, 2011 at 10:37 AM, Steven D'Aprano wrote: >> 0-based indexes are useful for some tasks, and less useful for other tasks. >> In my experience, I find that 0-based indexing is more useful most of the >> time: it leads to fewer off-by-one errors. >> >> 1-based indexes are particularly well-suited for programming languages using >> a natural language metaphor, usually aimed at non-programmers. Examples >> include Xion, Applescript, and Inform-7. > > Indeed, the concepts of half-open ranges and 0-based indexing go hand > in hand (as described in the EWD article), and it ties directly in to > the notion of *index arithmetic*. > > A case that illustrates this nicely is that of partitioning a > sequence. Suppose we want the first 5 items in one subsequence and the > rest in another. This is easy to write, and requires no adjustments to > the numbers: > > ?head = seq[:5] > ?assert len(head) == 5 > ?tail = seq[5:] > ?assert len(tail) == len(seq) - 5 > > Zero based indexing (in conjunction with half-open ranges) makes the > arithmetic work out nicely, and, in practice, that turns out to be > important when it comes to writing correct programs. > > However, it comes at the cost of breaking the intuitive mapping to the > counting numbers: the first item is at offset 0, the second is at > offset 1, etc. This is a definite downside, but the collective > judgment of many language designers is that the reduction in > off-by-one errors when manipulating indices is worth the additional > difficulty in learning the language for programming novices. This discussion has reminded me of a post from Tim Peters I stumbled on a few weeks ago: http://mail.python.org/pipermail/python-list/2000-October/637704.html "The trick is that indices in Python point *between* array elements"... Terry Reedy also explained this pretty well, I thought: http://mail.python.org/pipermail/python-list/2005-September/927859.html -eric * Tim also had an earlier, but similar post that I found when looking for the remembered one: http://mail.python.org/pipermail/python-list/1999-August/622024.html > > Cheers, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From dag.odenhall at gmail.com Sun Sep 11 12:46:29 2011 From: dag.odenhall at gmail.com (Dag Odenhall) Date: Sun, 11 Sep 2011 12:46:29 +0200 Subject: [Python-ideas] List Revolution In-Reply-To: References: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> <3E0F0EE4-6117-4263-805C-BA2D74F425C8@gmail.com> Message-ID: <1315737989.3871.0.camel@ganesh> On Sat, 2011-09-10 at 12:48 -0500, Massimo Di Pierro wrote: > remember name is an attribute. It cannot contain special symbols (*) by definition. There is no way you can build "**". Perfectly possible with getattr(). From ron3200 at gmail.com Sun Sep 11 06:52:50 2011 From: ron3200 at gmail.com (ron3200) Date: Sat, 10 Sep 2011 23:52:50 -0500 Subject: [Python-ideas] List Revolution In-Reply-To: References: <4E6C02C1.9050704@pearwood.info> Message-ID: <1315716770.22156.150.camel@Gutsy> On Sun, 2011-09-11 at 11:51 +1000, Nick Coghlan wrote: > On Sun, Sep 11, 2011 at 10:37 AM, Steven D'Aprano wrote: > > 0-based indexes are useful for some tasks, and less useful for other tasks. > > In my experience, I find that 0-based indexing is more useful most of the > > time: it leads to fewer off-by-one errors. > > > > 1-based indexes are particularly well-suited for programming languages using > > a natural language metaphor, usually aimed at non-programmers. Examples > > include Xion, Applescript, and Inform-7. > > Indeed, the concepts of half-open ranges and 0-based indexing go hand > in hand (as described in the EWD article), and it ties directly in to > the notion of *index arithmetic*. > > A case that illustrates this nicely is that of partitioning a > sequence. Suppose we want the first 5 items in one subsequence and the > rest in another. This is easy to write, and requires no adjustments to > the numbers: > > head = seq[:5] > assert len(head) == 5 > tail = seq[5:] > assert len(tail) == len(seq) - 5 > > Zero based indexing (in conjunction with half-open ranges) makes the > arithmetic work out nicely, and, in practice, that turns out to be > important when it comes to writing correct programs. > > However, it comes at the cost of breaking the intuitive mapping to the > counting numbers: the first item is at offset 0, the second is at > offset 1, etc. This is a definite downside, but the collective > judgment of many language designers is that the reduction in > off-by-one errors when manipulating indices is worth the additional > difficulty in learning the language for programming novices. Python slicing was designed with indexing in mind. If we had used counts instead, we could have just added a bit of syntax to make things work in a nice way. The biggest difference isn't the math, but having a way to specify weather a count, is inclusive or exclusive. Given that, the math will work just as nice. (note: using '...' instead of ':' to make it clear it's different.) head = seq[...n] # the first n items tail = seq[n>...] # items after first n items assert len(head) = n assert len(head) + len(tail) = len(seq) And the reverse works too. tail = seq[-n...] # the last n items head = seq[...<-n] # items before the last n items assert len(tail) == n assert len(head) + len(tail) == len(seq) And then you also have. before = seq[......] And while we're at it. inner = seq[n1...n2] # items from n1 to n2, including n2 inner = seq[n1>......] # everything but item n thewholething = seq[...] Adding items to the front and back. seq[...<1] = seq # prepend items seq[-1>...] = seq # append items etc... So it can be done either way. It's just not good to try and mix index's with counts. (Not suggesting we do this, but if we were starting from scratch, I don't see any problems with it.) Cheers, Ron From guido at python.org Sun Sep 11 18:32:57 2011 From: guido at python.org (Guido van Rossum) Date: Sun, 11 Sep 2011 09:32:57 -0700 Subject: [Python-ideas] List Revolution In-Reply-To: <1315716770.22156.150.camel@Gutsy> References: <4E6C02C1.9050704@pearwood.info> <1315716770.22156.150.camel@Gutsy> Message-ID: On Sat, Sep 10, 2011 at 9:52 PM, ron3200 wrote: > So it can be done either way. ?It's just not good to try and mix index's > with counts. ?(Not suggesting we do this, but if we were starting from > scratch, I don't see any problems with it.) I do. It violates TOOWTDI. -- --Guido van Rossum (python.org/~guido) From fdrake at acm.org Sun Sep 11 20:52:23 2011 From: fdrake at acm.org (Fred Drake) Date: Sun, 11 Sep 2011 14:52:23 -0400 Subject: [Python-ideas] Columns for pprint.PrettyPrinter In-Reply-To: <987401CE-FF96-4E83-A5B3-19190AFCFE3B@GrayVines.com> References: <987401CE-FF96-4E83-A5B3-19190AFCFE3B@GrayVines.com> Message-ID: On Fri, Aug 19, 2011 at 7:23 PM, Julian Berman wrote: > it could be coerced into something like > > ? ?>>> pprint.pprint(range(30), columns=5) > ? ?[0, 1, 2, 3, 4 > ? ? 3, 4, 5, 6, 7, ... ] I'd love to see something like this added. Another possibility would be something that figures out how many columns would be appropriate based on the available (remaining) width. > or for something nested, which I'm less thrilled with, and haven't thought > out how to implement unless you have a somewhat balanced structure, but for > posterity: > > ? ?{"foo" : > ? ? ? ? ? ? ? {"bar" : 1, ? ? ? ? ? {"hello" : 2, ? ? ? ? ? ? ? {"other" : 1, > ? ? ? ? ? ? ? ?"baz" : 2, ? ? ? ? ? ?"world" : 1}, ? ? ? ? ? ? ?"thing" : 2, > ? ? ? ? ? ? ? ? "foo" : 3}, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? "here" : 3}, > > ? ? ?... > ? ? ?} This is just a little frightening. -Fred -- Fred L. Drake, Jr.? ? "A person who won't read has no advantage over one who can't read." ?? --Samuel Langhorne Clemens From massimo.dipierro at gmail.com Mon Sep 12 01:37:32 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Sun, 11 Sep 2011 18:37:32 -0500 Subject: [Python-ideas] List Revolution In-Reply-To: <1315737989.3871.0.camel@ganesh> References: <54A38A59-66B5-481B-8DA4-8380D3C424DC@gmail.com> <3E0F0EE4-6117-4263-805C-BA2D74F425C8@gmail.com> <1315737989.3871.0.camel@ganesh> Message-ID: So you are saying that with my code (which was meant as joke not as serious proposal) you can do a = List() getattr(a,'(2**1024)**(2**1024)') and that is a vulnerability? But you can already do a = [] a[(2**1024)**(2**1024)] or simply (2**1024)**(2**1024) The ability write insane code is not a vulnerability. It is a feature. Exposing that to users may constitute a vulnerability. Massimo On Sep 11, 2011, at 5:46 AM, Dag Odenhall wrote: > On Sat, 2011-09-10 at 12:48 -0500, Massimo Di Pierro wrote: >> remember name is an attribute. It cannot contain special symbols (*) by definition. There is no way you can build "**". > > Perfectly possible with getattr(). > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From fuzzyman at gmail.com Mon Sep 12 01:41:45 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Mon, 12 Sep 2011 00:41:45 +0100 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: On 9 September 2011 22:12, Christopher King wrote: > The first element in a list is element zero, the second is one, the third > it two, and so on. This some times confuses newbies to the language or > programming in general. This system was invited when single bits where > precious. It's time to update. Keep in mind this is something for version 4, > since its not reverse compatible. I say we make the first element 1, second > 2, third 3, and so on. Other languages would follow. We are python, made for > easiness and readability, and we are in the age where you can't even get > something as small as a kilobyte USB. We must make first one, second 2, and > third 3, like it is supposed to be. How about this (works with current and future pythons): >>> class Index(int): ... def __new__(cls, val): ... return int.__new__(cls, val -1) ... def __repr__(self): ... return 'Index(%s)' % (int(self) + 1) ... >>> thing = list(range(1, 11)) >>> >>> thing[Index(1)] 1 >>> thing [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] >>> thing[Index(2)] 2 >>> thing[Index(3)] 3 >>> Index(1) Index(1) Michael Foord > I give this: > *+1* > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Sun Sep 11 23:04:46 2011 From: ron3200 at gmail.com (Ron Adam) Date: Sun, 11 Sep 2011 16:04:46 -0500 Subject: [Python-ideas] List Revolution In-Reply-To: References: <4E6C02C1.9050704@pearwood.info> <1315716770.22156.150.camel@Gutsy> Message-ID: <1315775086.26528.29.camel@Gutsy> On Sun, 2011-09-11 at 09:32 -0700, Guido van Rossum wrote: > On Sat, Sep 10, 2011 at 9:52 PM, ron3200 wrote: > > So it can be done either way. It's just not good to try and mix index's > > with counts. (Not suggesting we do this, but if we were starting from > > scratch, I don't see any problems with it.) > > I do. It violates TOOWTDI. I meant that I didn't see a problem with using counts instead of indexes, if we were starting from scratch, and if that was the only way to do it. Here's a good guideline that isn't in The Zen of Python... Just because "it can be done", isn't a reason to do it! ;-) Cheers, Ron (Fixed my email name, so maybe this will get posted faster instead of taking a whole day to show up.) From nevesagar at gmail.com Mon Sep 12 04:54:28 2011 From: nevesagar at gmail.com (Sagar Neve) Date: Sun, 11 Sep 2011 19:54:28 -0700 (PDT) Subject: [Python-ideas] =?utf-8?q?threading_=3A_AttributeE=E2=80=8Brror=3A?= =?utf-8?q?_=27NoneType=27_object_has_no_attribute_=27start=27?= Message-ID: I am new to the python and I need a small script to be written using threads which reads the specific files present in a directory and does some selections of lines and then write those lines into different files. However the error I am getting is: AttributeError: 'NoneType' object has no attribute 'start' Here is the code. Can somebody please help me. This is very urgent. def getFiles(fname): fname=preprocessLogFileDir + "/" + fname print fname fin = open(fname, "r") line=fin.readline() print line .... # some extraction logic. fout = open("out-single-file/1xx_2xx_total", "w") fout.write(line) fout.close() files=os.listdir(preprocessLogFileDir) cnt=1; for fname in files: print fname if fnmatch.fnmatch(fname,'*.log'): #and os.path.isfile(fname): thr = "thr" + str(cnt) print thr thr=getFiles(fname) thr.start() thr.join cnt = cnt + 1 else: print "\npreprocessLogFileDir/fname is either a directory or does not end with 'log'." print "Probabally not a pre-process file. Ignoring the file.\n\n" From ctb at msu.edu Mon Sep 12 05:00:12 2011 From: ctb at msu.edu (C. Titus Brown) Date: Sun, 11 Sep 2011 20:00:12 -0700 Subject: [Python-ideas] threading : AttributeE???rror: 'NoneType' object has no attribute 'start' In-Reply-To: References: Message-ID: <20110912030012.GG20463@idyll.org> On Sun, Sep 11, 2011 at 07:54:28PM -0700, Sagar Neve wrote: > I am new to the python and I need a small script to be written using > threads which reads the specific files present in a directory and does > some selections of lines and then write those lines into different > files. > > However the error I am getting is: > AttributeError: 'NoneType' object has no attribute 'start' [ ... ] Hi Sagar, this isn't the right place to post questions about how to use Python -- this is python-ideas, a list for discussing future changes to the language. Please go to http://mail.python.org/mailman/listinfo/python-list with future requests for help. > Here is the code. Can somebody please help me. This is very urgent. ...and note that your urgency is not ours, since you're not paying us! That having been said, at least one source of your problems is that 'getFiles' doesn't return a value. It's kind of hard to figure out what you want it to do; 'thr.start()' followed by 'thr.join()' immediately isn't going to result in much threading, unless there's more going on here than meets the eye. You might want to use something like thr = threading.Thread(target=getFiles) instead. See http://docs.python.org/library/threading.html#thread-objects And please ask future questions elsewhere. thanks! cheers, --titus > def getFiles(fname): > fname=preprocessLogFileDir + "/" + fname > print fname > fin = open(fname, "r") > line=fin.readline() > print line > .... # some extraction logic. > fout = open("out-single-file/1xx_2xx_total", "w") > fout.write(line) > fout.close() > > files=os.listdir(preprocessLogFileDir) > cnt=1; > for fname in files: > print fname > if fnmatch.fnmatch(fname,'*.log'): #and os.path.isfile(fname): > thr = "thr" + str(cnt) > print thr > thr=getFiles(fname) > thr.start() > thr.join > cnt = cnt + 1 > else: > print "\npreprocessLogFileDir/fname is either a > directory or does > not end with 'log'." > print "Probabally not a pre-process file. Ignoring the > file.\n\n" > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- C. Titus Brown, ctb at msu.edu From adam.jorgensen.za at gmail.com Mon Sep 12 07:48:43 2011 From: adam.jorgensen.za at gmail.com (Adam Jorgensen) Date: Mon, 12 Sep 2011 07:48:43 +0200 Subject: [Python-ideas] List Revolution In-Reply-To: <1315775086.26528.29.camel@Gutsy> References: <4E6C02C1.9050704@pearwood.info> <1315716770.22156.150.camel@Gutsy> <1315775086.26528.29.camel@Gutsy> Message-ID: Hahaha, I figured it was a joke but you can never be too careful ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwm at mired.org Mon Sep 12 08:05:28 2011 From: mwm at mired.org (mike.w.meyer@gmail.com) Date: Sun, 11 Sep 2011 23:05:28 -0700 Subject: [Python-ideas] List Revolution In-Reply-To: References: <4E6C02C1.9050704@pearwood.info> <1315716770.22156.150.camel@Gutsy> <1315775086.26528.29.camel@Gutsy> Message-ID: <2f456184-f51c-4f85-bf9e-17aebb43058c@email.android.com> Ok, how about enhancing the list type to allow specifying an offset, defaulting to 0? The offset should be positive, and that many elements simple aren't there. Asking for an element that isn't there, or a slice including such an element, raises value error exception. The iterators & etc for this list would all start at offset instead of zero. -- Sent from my Android tablet with K-9 Mail. Please excuse my brevity. From matt at vazor.com Mon Sep 12 08:18:37 2011 From: matt at vazor.com (Matt Billenstein) Date: Mon, 12 Sep 2011 07:18:37 +0100 Subject: [Python-ideas] List Revolution Message-ID: <4tjogj0natts.1lcr005u@elasticemail.com> Ugh, I hate the idea of library code having to think about dealing with the possibility that you chose 42 as the starting index for some list you passed into a function in said library. Then I suppose we'll have to extend this to strings and tuples and so forth as well? Seriously, any further "solutions" for this non-problem are just mental masturbation -- please stop. m On Sun, Sep 11, 2011 at 11:05:28PM -0700, mike.w.meyer at gmail.com wrote: > Ok, how about enhancing the list type to allow specifying an offset, > defaulting to 0? The offset should be positive, and that many elements simple > aren't there. Asking for an element that isn't there, or a slice including > such an element, raises value error exception. The iterators & etc for this > list would all start at offset instead of zero. > -- > Sent from my Android tablet with K-9 Mail. Please excuse my brevity. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- Matt Billenstein matt at vazor.com http://www.vazor.com/ From tlesher at gmail.com Mon Sep 12 09:53:51 2011 From: tlesher at gmail.com (Tim Lesher) Date: Mon, 12 Sep 2011 03:53:51 -0400 Subject: [Python-ideas] =?utf-8?q?threading_=3A_AttributeE=E2=80=8Brror=3A?= =?utf-8?q?_=27NoneType=27_object_has_no_attribute_=27start=27?= In-Reply-To: References: Message-ID: On Sun, Sep 11, 2011 at 22:54, Sagar Neve wrote: > Here is the code. Can somebody please help me. This is very urgent. First off, python-ideas is not the right place to ask for programming help--it's for suggesting changes to the Python language itself. You will have better results posting to a help site like stackoverflow.com. In your case, it looks like you want to create a thread, but "thr = getFiles(fname)" is not doing that--it's just calling a function. You probably want to change this to "thr = threading.Thread(target=getFiles, args=(fname,))" -- Tim Lesher From steve at pearwood.info Mon Sep 12 12:36:20 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 12 Sep 2011 20:36:20 +1000 Subject: [Python-ideas] List Revolution In-Reply-To: <2f456184-f51c-4f85-bf9e-17aebb43058c@email.android.com> References: <4E6C02C1.9050704@pearwood.info> <1315716770.22156.150.camel@Gutsy> <1315775086.26528.29.camel@Gutsy> <2f456184-f51c-4f85-bf9e-17aebb43058c@email.android.com> Message-ID: <4E6DE0A4.7060303@pearwood.info> mike.w.meyer at gmail.com wrote: > Ok, how about enhancing the list type to allow specifying an offset, > defaulting to 0? The offset should be positive, and that many > elements simple aren't there. Asking for an element that isn't there, > or a slice including such an element, raises value error exception. > The iterators & etc for this list would all start at offset instead > of zero. I believe that Ada allows that. Any Ada users want to comment on how useful or annoying this is? Another alternative would be to specify the indexing scheme at call time, rather than when you create the list. Something like: item = mylist[7 starting from 5] will set item to mylist[2] (using zero-based indexing). Best of all, with a slight change of syntax, that is available right now: item = mylist[7-5] and so we can simulate variable-offset indexing with no new syntax and only a very small run-time cost. Guido's time machine strikes again! -- Steven From steve at pearwood.info Mon Sep 12 12:41:17 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 12 Sep 2011 20:41:17 +1000 Subject: [Python-ideas] List Revolution In-Reply-To: <4E6DE0A4.7060303@pearwood.info> References: <4E6C02C1.9050704@pearwood.info> <1315716770.22156.150.camel@Gutsy> <1315775086.26528.29.camel@Gutsy> <2f456184-f51c-4f85-bf9e-17aebb43058c@email.android.com> <4E6DE0A4.7060303@pearwood.info> Message-ID: <4E6DE1CD.8080407@pearwood.info> Steven D'Aprano wrote: > item = mylist[7-5] > > and so we can simulate variable-offset indexing with no new syntax and > only a very small run-time cost. Oh, I forgot to mention... if both values are literals, as in the example above, there may not even be a run-time cost. CPython's peek-hole optimizer will compile it to mylist[2]. -- Steven From stefan_ml at behnel.de Mon Sep 12 17:09:45 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 12 Sep 2011 17:09:45 +0200 Subject: [Python-ideas] =?utf-8?q?threading_=3A_AttributeE=E2=80=8Brror=3A?= =?utf-8?q?_=27NoneType=27_object_has_no_attribute_=27start=27?= In-Reply-To: References: Message-ID: Sagar Neve, 12.09.2011 04:54: > I am new to the python and I need a small script to be written using > threads which reads the specific files present in a directory and does > some selections of lines and then write those lines into different > files. As others have told you, there are better lists to ask this kind of question. Specifically the python-tutor mailing list is a dedicated and friendly place for new users to ask questions. http://mail.python.org/mailman/listinfo/tutor Stefan From lists at andros.org.uk Mon Sep 12 21:09:07 2011 From: lists at andros.org.uk (Andrew McLean) Date: Mon, 12 Sep 2011 20:09:07 +0100 Subject: [Python-ideas] List Revolution In-Reply-To: References: Message-ID: <4E6E58D3.2010802@andros.org.uk> On 10/09/2011 23:34, Guido van Rossum wrote: > It is a cultural battle that has been fought and won long ago (all the > old languages used 1-based indexing: Fortran, Algol, Pascal) and it's > really not that important in the grand scheme of things, so the status > quo wins. (*) I personally think 0-based indexing is better, and the > referenced EWD expresses why better than I could. But I'm sure that if > we lived in a world where 1-based indexing was the norm I'd get by > just fine. To be pedantic, Fortran *defaults* to 1 based indexing, but it lets you choose any initial index (on a per variable basis). This has been the case at least as far back as the Fortran 77 standard, possibly further back than that. I'm really not seriously suggesting this... but it might be possible to do something similar for Python. You could add an optional initial_index argument (with a default of zero) to the constructor for sequence types and then modify any operations that involve indexing. Andrew From lukas.lueg at googlemail.com Mon Sep 12 22:20:11 2011 From: lukas.lueg at googlemail.com (Lukas Lueg) Date: Mon, 12 Sep 2011 22:20:11 +0200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context Message-ID: Hi, the proposal is to reduce the number of occurences where Python automatically and inevitably uses pure boolean objects (bool()), instead of relying on rich object behaviour. My goal is to give objects finer control over how they want to be interpreted in a boolean context. This reaches down as far as implementing the boolean operations as an object-protocol. In Python, no kind of object is in any way special. Instead of relying on what an object actually IS, we rely on what an object can DO. This approach of anti-discrimination is the source of great confidence about the language and it's aversion to black magic. The bool()-type however is a primus inter pares: In numerous locations, Python strongly favors this type of object either explicitly by casting to it or implicitly through the compiled code. Here are a three examples (protocol-, object- and code-based) where Python grinds a boolean context down to pure boolean values: In protocol behaviour: The documentation about how to implement __contains__ vaguely states that the function "should return true, false otherwise". In fact we may return any object that can be interpreted as a boolean, this is intented behaviour. However the implementation of COMPARE_OP in Python/ceval.c inevitably casts the result of "x in y" to Py_True or Py_False, throwing away our object and leaving us with this bool only. This kills any chance of producing useful objects in __contains__, even objects that may have a meaning outside the pure boolean context. For example: x = mycontainer(['foo', 'bar', 'bar', 'foo', 'foo']) y = 'foo' in mycontainer print y == True >> True print y >> mycontainer('foo', 'foo', 'foo') This has been discussed on python-ideas in mid-2010 (http://mail.python.org/pipermail/python-ideas/2010-July/007733.html) without distinct outcome. In object behaviour: Rich Comparision allows numeric- or set-like comparision in any way we like, returing any kind of object. The documentation clearly states that the returned object is only(!) interpreted as a boolean when used in a boolean context (__contains__ falls short here). For example: x = set((1,2,3,4,5)) y = set((2,3,4)) z = x > y print z == True >> True print z >> set([1,5]) # equivalent to x - y z = x < y print z == True >> False print z >> set([]) # equivalent to y - x print 3 > 2 >> 1 print (3 > 2) == True >> True In code behaviour: Why is it, that we can override the behaviour of arithmetic operations like "+, -, *, /", but not the boolean operations like "and, or, xor, not" ? Why do we force any value being generated in a boolean context like "if x == 1 or y == 1" to actually be of boolean type? The result of "(x/y) == 1" may be any kind of object coming from __eq__. However the "or"-operator here grinds them both down to being just True or False. This is due to the fact that the generated bytecode evaluates one expression at a time and does not keep track of the objects it got while doing so. The pure boolean behaviour arises from the use of JUMP_IF_FALSE/TRUE after each comparision. Instead of having boolean operations being a part of the language, we could implement them as a an extension to the Rich Comparision Protocol, giving rise to functions like object.__bor__, object.__bnot__ or object.__band__: The expression "if x == 1 or y == 1" would then become equivalent to tmp1 = x.__eq__(1) if tmp1: return tmp1 tmp2 = y.__eq__(1) if tmp2: return tmp2 return = tmp1.__bor__(tmp2) Likewise, the expression "if x == 1 and y == 1" would become tmp1 = x.__eq__(1) if not tmp1: return tmp1 tmp2 = y.__eq__(1) if not tmp2: return tmp2 return tmp1.__band__(tmp2) The object-example from above now tells us how boolean behaviour and arithmetic behaviour go hand in hand: "(setA > setB) or (setA < setB)" is True because "set([1,5]).__bor__(set([]))" is the same as "set([1,5]) + set([])" and equivalent to True in a boolean context. Likewise, "(setA > setB) and (setA < setB)" is False because "set([1,5]).__band__set([])" is just "set([])". It follows that "(setA > setB) == (setA - setB) == (setA & setB)". We can't do this with boolean operations being part of the language only. Summing all up, I really think that we should break the dominance of bool() and take a look at how we can implement boolean contexts without relying on boolean values all the time. None of this can be implemented without breaking at least the CPython API. For example, the behaviour of __contains__ can't be changed in the proposed way without changing the signature of "int PySequence_Contains()" to "PyObject* PySequence_Contains()". From alexander.belopolsky at gmail.com Mon Sep 12 22:29:43 2011 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 12 Sep 2011 16:29:43 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: On Mon, Sep 12, 2011 at 4:20 PM, Lukas Lueg wrote: .. > In code behaviour: Why is it, that we can override the behaviour of > arithmetic operations like "+, -, *, /", but not the boolean > operations like "and, or, xor, not" ? You may want to take a look at PEP 335: http://www.python.org/dev/peps/pep-0335/ From mikegraham at gmail.com Mon Sep 12 22:44:22 2011 From: mikegraham at gmail.com (Mike Graham) Date: Mon, 12 Sep 2011 16:44:22 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: On Mon, Sep 12, 2011 at 4:20 PM, Lukas Lueg wrote: > Hi, > > the proposal is to reduce the number of occurences where Python > automatically and inevitably uses pure boolean objects (bool()), > instead of relying on rich object behaviour. My goal is to give > objects finer control over how they want to be interpreted in a > boolean context. This reaches down as far as implementing the boolean > operations as an object-protocol. > > In Python, no kind of object is in any way special. Instead of relying > on what an object actually IS, we rely on what an object can DO. This > approach of anti-discrimination is the source of great confidence > about the language and it's aversion to black magic. The bool()-type > however is a primus inter pares: In numerous locations, Python > strongly favors this type of object either explicitly by casting to it > or implicitly through the compiled code. > > > Here are a three examples (protocol-, object- and code-based) where > Python grinds a boolean context down to pure boolean values: > > In protocol behaviour: The documentation about how to implement > __contains__ vaguely states that the function "should return true, > false otherwise". In fact we may return any object that can be > interpreted as a boolean, this is intented behaviour. However the > implementation of COMPARE_OP in Python/ceval.c inevitably casts > the result of "x in y" to Py_True or Py_False, throwing away our object > and leaving us with this bool only. > This kills any chance of producing useful objects in __contains__, > even objects that may have a meaning outside the pure boolean context. > For example: > > x = mycontainer(['foo', 'bar', 'bar', 'foo', 'foo']) > y = 'foo' in mycontainer > print y == True >>> True > print y >>> mycontainer('foo', 'foo', 'foo') > > This has been discussed on python-ideas in mid-2010 > (http://mail.python.org/pipermail/python-ideas/2010-July/007733.html) > without distinct outcome. > > > In object behaviour: Rich Comparision allows numeric- or set-like > comparision in any way we like, returing any kind of object. The > documentation clearly states that the returned object is only(!) > interpreted as a boolean when used in a boolean context (__contains__ > falls short here). > For example: > > x = set((1,2,3,4,5)) > y = set((2,3,4)) > z = x > y > print z == True >>> True > print z >>> set([1,5]) # equivalent to x - y > > z = x < y > print z == True >>> False > print z >>> set([]) # equivalent to y - x > > print 3 > 2 >>> 1 > print (3 > 2) == True >>> True > > > In code behaviour: Why is it, that we can override the behaviour of > arithmetic operations like "+, -, *, /", but not the boolean > operations like "and, or, xor, not" ? Why do we force any value being > generated in a boolean context like "if x == 1 or y == 1" to actually > be of boolean type? The result of "(x/y) == 1" may be any kind of > object coming from __eq__. However the "or"-operator here grinds them > both down to being just True or False. This is due to the fact that > the generated bytecode evaluates one expression at a time and does not > keep track of the objects it got while doing so. The pure boolean > behaviour arises from the use of JUMP_IF_FALSE/TRUE after each > comparision. > Instead of having boolean operations being a part of the language, we > could implement them as a an extension to the Rich Comparision > Protocol, giving rise to functions like object.__bor__, > object.__bnot__ or object.__band__: > > The expression "if x == 1 or y == 1" would then become equivalent to > > tmp1 = x.__eq__(1) > if tmp1: > ?return tmp1 > tmp2 = y.__eq__(1) > if tmp2: > ?return tmp2 > return = tmp1.__bor__(tmp2) > > Likewise, the expression "if x == 1 and y == 1" would become > > tmp1 = x.__eq__(1) > if not tmp1: > ?return tmp1 > tmp2 = y.__eq__(1) > if not tmp2: > ?return tmp2 > return tmp1.__band__(tmp2) > > The object-example from above now tells us how boolean behaviour and > arithmetic behaviour go hand in hand: "(setA > setB) or (setA < setB)" > is True because "set([1,5]).__bor__(set([]))" is the same as > "set([1,5]) + set([])" and equivalent to True in a boolean context. > Likewise, "(setA > setB) and (setA < setB)" is False because > "set([1,5]).__band__set([])" is just "set([])". It follows that "(setA >> setB) == (setA - setB) == (setA & setB)". We can't do this with > boolean operations being part of the language only. > > > Summing all up, I really think that we should break the dominance of > bool() and take a look at how we can implement boolean contexts > without relying on boolean values all the time. > > None of this can be implemented without breaking at least the CPython > API. For example, the behaviour of __contains__ can't be changed in > the proposed way without changing the signature of "int > PySequence_Contains()" to "PyObject* PySequence_Contains()". Your basic idea seems like a good one to me at an abstract level -- methods like __contains__ have no good design reason to typecheck. However, I don't think any of the concrete changes here serve to make Python a nicer or saner language. There is no reason why set should be changed either of the ways you discuss, and there is no excuse I can think of to implement something that works like mycontainer. Mike From tjreedy at udel.edu Mon Sep 12 23:26:26 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 12 Sep 2011 17:26:26 -0400 Subject: [Python-ideas] List Revolution In-Reply-To: <2f456184-f51c-4f85-bf9e-17aebb43058c@email.android.com> References: <4E6C02C1.9050704@pearwood.info> <1315716770.22156.150.camel@Gutsy> <1315775086.26528.29.camel@Gutsy> <2f456184-f51c-4f85-bf9e-17aebb43058c@email.android.com> Message-ID: On 9/12/2011 2:05 AM, mike.w.meyer at gmail.com wrote: > Ok, how about enhancing the list type to allow specifying an offset, Because then every list index/slice access would require more time to subtract the offset. > defaulting to 0? The offset should be positive, and that many > elements simple aren't there. Asking for an element that isn't there, > or a slice including such an element, raises value error exception. Slicing never raises a ValueError. In particular, seq[0:1] always returns a sequence with 1 or 0 items. This is an important feature. > The iterators& etc for this list would all start at offset instead > of zero. Write a list subclass. Decide how you would handle indexing from the right (currently all negative indexes) and implement that. -- Terry Jan Reedy From sklass at pointcircle.com Mon Sep 12 23:36:41 2011 From: sklass at pointcircle.com (Steven Klass) Date: Mon, 12 Sep 2011 14:36:41 -0700 Subject: [Python-ideas] List Revolution In-Reply-To: References: <4E6C02C1.9050704@pearwood.info> <1315716770.22156.150.camel@Gutsy> <1315775086.26528.29.camel@Gutsy> <2f456184-f51c-4f85-bf9e-17aebb43058c@email.android.com> Message-ID: <912FC93A-71B9-4817-8635-625BB7A78983@pointcircle.com> Stop - please for all of mankind - stop this thread? --- Steven Klass (480) 225-1112 sklass at pointcircle.com On Sep 12, 2011, at 2:26 PM, Terry Reedy wrote: > On 9/12/2011 2:05 AM, > mike.w.meyer at gmail.com wrote: >> Ok, how about enhancing the list type to allow specifying an offset, > > Because then every list index/slice access would require more time to subtract the offset. > >> defaulting to 0? The offset should be positive, and that many >> elements simple aren't there. Asking for an element that isn't there, >> or a slice including such an element, raises value error exception. > > Slicing never raises a ValueError. In particular, seq[0:1] always returns a sequence with 1 or 0 items. This is an important feature. > >> The iterators& etc for this list would all start at offset instead >> of zero. > > Write a list subclass. Decide how you would handle indexing from the right (currently all negative indexes) and implement that. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From jxo6948 at rit.edu Tue Sep 13 00:00:46 2011 From: jxo6948 at rit.edu (John O'Connor) Date: Mon, 12 Sep 2011 18:00:46 -0400 Subject: [Python-ideas] List Revolution In-Reply-To: <912FC93A-71B9-4817-8635-625BB7A78983@pointcircle.com> References: <4E6C02C1.9050704@pearwood.info> <1315716770.22156.150.camel@Gutsy> <1315775086.26528.29.camel@Gutsy> <2f456184-f51c-4f85-bf9e-17aebb43058c@email.android.com> <912FC93A-71B9-4817-8635-625BB7A78983@pointcircle.com> Message-ID: > Stop - please for all of mankind - stop this thread? +1 From jxo6948 at rit.edu Tue Sep 13 00:02:52 2011 From: jxo6948 at rit.edu (John O'Connor) Date: Mon, 12 Sep 2011 18:02:52 -0400 Subject: [Python-ideas] List Revolution In-Reply-To: References: <4E6C02C1.9050704@pearwood.info> <1315716770.22156.150.camel@Gutsy> <1315775086.26528.29.camel@Gutsy> <2f456184-f51c-4f85-bf9e-17aebb43058c@email.android.com> <912FC93A-71B9-4817-8635-625BB7A78983@pointcircle.com> Message-ID: I mean +2 :-P - John On Mon, Sep 12, 2011 at 6:00 PM, John O'Connor wrote: >> Stop - please for all of mankind - stop this thread? > > +1 > From tjreedy at udel.edu Tue Sep 13 00:27:15 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 12 Sep 2011 18:27:15 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: On 9/12/2011 4:20 PM, Lukas Lueg wrote: > However the "or"-operator here grinds them > both down to being just True or False. Both 'and' and 'or' return one of their inputs unchanged and unground. >>> a = 'a' >>> b = 'b' >>> a or b 'a' >>> a and b 'b' >>> c = '' >>> a and c '' >>> c and a '' >>> a or c 'a' >>> c or a 'a' If both inputs are True or False, then or course you always get one of them back. But again, no 'grind'ing. > Why is it, that we can override the behaviour of > arithmetic operations like "+, -, *, /", but not the boolean > operations like "and, or, xor, not" ? In Python, 'and' and 'or' are *not* (functional) boolean operations. They are flow-control keywords that abbreviate in expression form an if-else statement pattern. They are compiled to similar bytecode. def f1(a,b): return a and b def f2(a,b): if not a: return a else: return b from dis import dis dis(f1) dis(f2) ### 0 LOAD_FAST 0 (a) 3 JUMP_IF_FALSE_OR_POP 9 6 LOAD_FAST 1 (b) 9 RETURN_VALUE 0 LOAD_FAST 0 (a) 3 POP_JUMP_IF_TRUE 10 6 LOAD_FAST 0 (a) 9 RETURN_VALUE 10 LOAD_FAST 1 (b) 13 RETURN_VALUE The difference is that the compiler does not notice that the 'a' to be returned is the same 'a' that it just tested and which is still sitting on the stack, so it reloads it or, if is were an expression, would recalculate it. There is no internal function that *could* be overloaded. In this sense, they are similar to 'lambda expressions, which abbreviate a simple def statement pattern and which compile to essentially the same bytecode as the def statement. The xor bitwise int operator '^' can be overloaded: __xor__ and __rxor__. Logical xor can be implemented as bool(a)^bool(b). Before there was a bool class, 'not not a' was the direct way to coerce a to a boolean value. Anything that could be done with a hidden .__not__ method could also be done, and I would say, better done, with an overt .invert method. For instance, a mutable binary array class should likely have .invert to flip each member in-place. As it is now, anyone reading 'not a' can depend on the result being a bool. And I think that is good. -- Terry Jan Reedy From ericsnowcurrently at gmail.com Tue Sep 13 00:29:13 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 12 Sep 2011 16:29:13 -0600 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: On Mon, Sep 12, 2011 at 2:20 PM, Lukas Lueg wrote: > Hi, > > the proposal is to reduce the number of occurences where Python > automatically and inevitably uses pure boolean objects (bool()), > instead of relying on rich object behaviour. My goal is to give > objects finer control over how they want to be interpreted in a > boolean context. This reaches down as far as implementing the boolean > operations as an object-protocol. > > In Python, no kind of object is in any way special. Instead of relying > on what an object actually IS, we rely on what an object can DO. This > approach of anti-discrimination is the source of great confidence > about the language and it's aversion to black magic. The bool()-type > however is a primus inter pares: In numerous locations, Python > strongly favors this type of object either explicitly by casting to it > or implicitly through the compiled code. > > > Here are a three examples (protocol-, object- and code-based) where > Python grinds a boolean context down to pure boolean values: > > In protocol behaviour: The documentation about how to implement > __contains__ vaguely states that the function "should return true, > false otherwise". In fact we may return any object that can be > interpreted as a boolean, this is intented behaviour. However the > implementation of COMPARE_OP in Python/ceval.c inevitably casts > the result of "x in y" to Py_True or Py_False, throwing away our object > and leaving us with this bool only. > This kills any chance of producing useful objects in __contains__, > even objects that may have a meaning outside the pure boolean context. > For example: > > x = mycontainer(['foo', 'bar', 'bar', 'foo', 'foo']) > y = 'foo' in mycontainer > print y == True >>> True > print y >>> mycontainer('foo', 'foo', 'foo') > So you want "give me the item 'foo' in mycontainer" rather than "is item 'foo' in mycontainer"? Isn't that what __getitem__ does already? > This has been discussed on python-ideas in mid-2010 > (http://mail.python.org/pipermail/python-ideas/2010-July/007733.html) > without distinct outcome. > > > In object behaviour: Rich Comparision allows numeric- or set-like > comparision in any way we like, returing any kind of object. The > documentation clearly states that the returned object is only(!) > interpreted as a boolean when used in a boolean context (__contains__ > falls short here). > For example: > > x = set((1,2,3,4,5)) > y = set((2,3,4)) > z = x > y > print z == True >>> True > print z >>> set([1,5]) # equivalent to x - y Why not just use x - y in the first place? -eric > > z = x < y > print z == True >>> False > print z >>> set([]) # equivalent to y - x > > print 3 > 2 >>> 1 > print (3 > 2) == True >>> True > > > In code behaviour: Why is it, that we can override the behaviour of > arithmetic operations like "+, -, *, /", but not the boolean > operations like "and, or, xor, not" ? Why do we force any value being > generated in a boolean context like "if x == 1 or y == 1" to actually > be of boolean type? The result of "(x/y) == 1" may be any kind of > object coming from __eq__. However the "or"-operator here grinds them > both down to being just True or False. This is due to the fact that > the generated bytecode evaluates one expression at a time and does not > keep track of the objects it got while doing so. The pure boolean > behaviour arises from the use of JUMP_IF_FALSE/TRUE after each > comparision. > Instead of having boolean operations being a part of the language, we > could implement them as a an extension to the Rich Comparision > Protocol, giving rise to functions like object.__bor__, > object.__bnot__ or object.__band__: > > The expression "if x == 1 or y == 1" would then become equivalent to > > tmp1 = x.__eq__(1) > if tmp1: > ?return tmp1 > tmp2 = y.__eq__(1) > if tmp2: > ?return tmp2 > return = tmp1.__bor__(tmp2) > > Likewise, the expression "if x == 1 and y == 1" would become > > tmp1 = x.__eq__(1) > if not tmp1: > ?return tmp1 > tmp2 = y.__eq__(1) > if not tmp2: > ?return tmp2 > return tmp1.__band__(tmp2) > > The object-example from above now tells us how boolean behaviour and > arithmetic behaviour go hand in hand: "(setA > setB) or (setA < setB)" > is True because "set([1,5]).__bor__(set([]))" is the same as > "set([1,5]) + set([])" and equivalent to True in a boolean context. > Likewise, "(setA > setB) and (setA < setB)" is False because > "set([1,5]).__band__set([])" is just "set([])". It follows that "(setA >> setB) == (setA - setB) == (setA & setB)". We can't do this with > boolean operations being part of the language only. > > > Summing all up, I really think that we should break the dominance of > bool() and take a look at how we can implement boolean contexts > without relying on boolean values all the time. > > None of this can be implemented without breaking at least the CPython > API. For example, the behaviour of __contains__ can't be changed in > the proposed way without changing the signature of "int > PySequence_Contains()" to "PyObject* PySequence_Contains()". > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From g.nius.ck at gmail.com Tue Sep 13 00:32:15 2011 From: g.nius.ck at gmail.com (Christopher King) Date: Mon, 12 Sep 2011 18:32:15 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: I do like the idea, but here is the problem >>> s= StrangeObjectofDeath(True) >>> if s: print "True" True >>> s and True False >>> True and s True >>> s or False False >>> False or s True >>> not s "Bananas" See how this could mess up a third party module if it is given this mess. It also makes stuff less readable. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Sep 13 01:55:30 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 13 Sep 2011 11:55:30 +1200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: <4E6E9BF2.1010707@canterbury.ac.nz> Eric Snow wrote: > So you want "give me the item 'foo' in mycontainer" rather than "is > item 'foo' in mycontainer"? Isn't that what __getitem__ does already? The canonical use case for this kind of thing is numpy arrays. To be consistent with the numpy philosophy, 'a in b' where b is an array should give you an array of values indicating whether a is in each element of b. -- Greg From greg.ewing at canterbury.ac.nz Tue Sep 13 01:59:39 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 13 Sep 2011 11:59:39 +1200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: <4E6E9CEB.5090107@canterbury.ac.nz> Lukas Lueg wrote: > The expression "if x == 1 or y == 1" would then become equivalent to > > tmp1 = x.__eq__(1) > if tmp1: > return tmp1 > tmp2 = y.__eq__(1) > if tmp2: > return tmp2 > return = tmp1.__bor__(tmp2) This scheme would be of limited usefulness. It still assumes that the operands *can* be treated as boolean values, which is not always the case -- numpy arrays can't, for example. Also, the short-circuiting behaviour is still defined by their boolean values. I suggested a more complete way of addressing this in PEP 335. -- Greg From steve at pearwood.info Tue Sep 13 03:00:35 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 13 Sep 2011 11:00:35 +1000 Subject: [Python-ideas] Proposal: add a calculator statistics module Message-ID: <4E6EAB33.50006@pearwood.info> I propose adding a basic calculator statistics module to the standard library, similar to the sorts of functions you would get on a scientific calculator: mean (average) variance (population and sample) standard deviation (population and sample) correlation coefficient and similar. I am volunteering to provide, and support, this module, written in pure Python so other implementations will be able to use it. Simple calculator-style statistics seem to me to be a fairly obvious "battery" to be included, more useful in practice than some functions already available such as factorial and the hyperbolic functions. The lack of a standard solution leads people who need basic stats to roll their own. This seems seductively simple, as the basic stats formulae are quite simple. Unfortunately doing it *correctly* is much harder than it seems. Variance, in particular, is prone to serious inaccuracies. Here is the most obvious algorithm, using the so-called "computational formula for the variance": def variance(data): # ?2 = 1/n**2 * (n*?(x**2) - (?x)**2) n = len(data) s1 = sum(x**2 for x in data) s2 = sum(data) return (n*s1 - s2**2)/(n*n) Many stats text books recommend this as the best way to calculate variance, advice which makes sense when you're talking about hand calculations of small numbers of moderate sized data, but not for floating point. It appears to work: >>> data = [1, 2, 4, 5, 8] >>> variance(data) # exact value = 6 6.0 but unfortunately it is numerically unstable. Shifting all the data points by a constant amount shouldn't change the variance, but it does: >>> data = [x+1e12 for x in data] >>> variance(data) 171798691.84 Even worse, variance should never be negative: >>> variance(data*100) -1266637395.197952 Note that using math.fsum instead of the built-in sum does not fix the numeric instability problem, and it adds the additional problem that it coerces the data points to float. (If you use Decimal, this may not be what you want.) Here is an example of published code which suffers from exactly this problem: https://bitbucket.org/larsyencken/simplestats/src/c42e048a6625/src/basic.py and here is an example on StackOverflow. Note the most popular answer given is to use the Computational Formula, which is the wrong answer. http://stackoverflow.com/questions/2341340/calculate-mean-and-variance-with-one-iteration I would like to add a module to the standard library to solve these sorts of simple stats problems the right way, once and for all. Thoughts, comments, objections or words of encouragement are welcome. -- Steven From ethan at stoneleaf.us Tue Sep 13 03:24:30 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 12 Sep 2011 18:24:30 -0700 Subject: [Python-ideas] Proposal: add a calculator statistics module In-Reply-To: <4E6EAB33.50006@pearwood.info> References: <4E6EAB33.50006@pearwood.info> Message-ID: <4E6EB0CE.2060002@stoneleaf.us> Steven D'Aprano wrote: > I propose adding a basic calculator statistics module to the standard > library, similar to the sorts of functions you would get on a scientific > calculator: Looks like a good candidate for __experimental__! :) +1 ~Ethan~ From massimo.dipierro at gmail.com Tue Sep 13 03:14:47 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Mon, 12 Sep 2011 20:14:47 -0500 Subject: [Python-ideas] Proposal: add a calculator statistics module In-Reply-To: <4E6EAB33.50006@pearwood.info> References: <4E6EAB33.50006@pearwood.info> Message-ID: <09157BD0-3FB0-4301-8D91-E198DE989AB1@gmail.com> You only need: def E(f,data): return sum(f(x) for x in data)/len(data) Than you can compute ANY expectation value data = range(0,10) average = E(lambda x:x, data) variance = E(lambda x:(x-mu)**2,data) skewness = E(lambda x:(x-mu)**3,data)/variance**(2.0/3.0) X = [random() for i in range(N)] Y = [random() for i in range(N)] XY = [(X[i],Y[i]) for i in range(N)] covariance = E[lambda x,y: x*y, XY] - E(lambda x,y:x, XY)*E(lambda x,y:y, XY) Hope it makes sense. etc.etc. Massimo On Sep 12, 2011, at 8:00 PM, Steven D'Aprano wrote: > I propose adding a basic calculator statistics module to the standard library, similar to the sorts of functions you would get on a scientific calculator: > > mean (average) > variance (population and sample) > standard deviation (population and sample) > correlation coefficient > > and similar. I am volunteering to provide, and support, this module, written in pure Python so other implementations will be able to use it. > > Simple calculator-style statistics seem to me to be a fairly obvious "battery" to be included, more useful in practice than some functions already available such as factorial and the hyperbolic functions. > > The lack of a standard solution leads people who need basic stats to roll their own. This seems seductively simple, as the basic stats formulae are quite simple. Unfortunately doing it *correctly* is much harder than it seems. Variance, in particular, is prone to serious inaccuracies. Here is the most obvious algorithm, using the so-called "computational formula for the variance": > > > def variance(data): > # ?2 = 1/n**2 * (n*?(x**2) - (?x)**2) > n = len(data) > s1 = sum(x**2 for x in data) > s2 = sum(data) > return (n*s1 - s2**2)/(n*n) > > Many stats text books recommend this as the best way to calculate variance, advice which makes sense when you're talking about hand calculations of small numbers of moderate sized data, but not for floating point. It appears to work: > > >>> data = [1, 2, 4, 5, 8] > >>> variance(data) # exact value = 6 > 6.0 > > but unfortunately it is numerically unstable. Shifting all the data points by a constant amount shouldn't change the variance, but it does: > > >>> data = [x+1e12 for x in data] > >>> variance(data) > 171798691.84 > > Even worse, variance should never be negative: > > >>> variance(data*100) > -1266637395.197952 > > Note that using math.fsum instead of the built-in sum does not fix the numeric instability problem, and it adds the additional problem that it coerces the data points to float. (If you use Decimal, this may not be what you want.) > > Here is an example of published code which suffers from exactly this problem: > > https://bitbucket.org/larsyencken/simplestats/src/c42e048a6625/src/basic.py > > and here is an example on StackOverflow. Note the most popular answer given is to use the Computational Formula, which is the wrong answer. > > http://stackoverflow.com/questions/2341340/calculate-mean-and-variance-with-one-iteration > > I would like to add a module to the standard library to solve these sorts of simple stats problems the right way, once and for all. > > Thoughts, comments, objections or words of encouragement are welcome. > > > > -- > Steven > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From bruce at leapyear.org Tue Sep 13 03:08:30 2011 From: bruce at leapyear.org (Bruce Leban) Date: Mon, 12 Sep 2011 18:08:30 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: Philosophically, the idea that boolean operations like x in y should be able to return any value for True makes sense. The details leave much to be desired. It wouldn't make sense for it to return x since 0 in [0,1] and '' in 'string' should both return a true value and 0 and '' are false. On Mon, Sep 12, 2011 at 1:20 PM, Lukas Lueg wrote: > > x = set((1,2,3,4,5)) > y = set((2,3,4)) > z = x > y > print z == True > >> True > print z > >> set([1,5]) # equivalent to x - y > You've lost me where you suggest changing the behavior of == True. And where you propose that x > y should return x - y. You also don't mention what x < y should return. The real problem is the lack of a compelling scenario why you'd want to make this change. Aside from that, checking to see if one set is contained in another does not require computation of the full difference. Why should very_big_set > (1,2) go to the overhead of constructing a still very big set that's going to be immediately discarded? > > z = x < y > print z == True > >> False > print z > >> set([]) # equivalent to y - x > > print 3 > 2 > >> 1 > print (3 > 2) == True > >> True > I don't know the scenario where 3 > 2 returning 1 is useful. Would 2 < 3 also return 1? The Icon programming language (and others) has 2 < 3 return 3. This makes expressions like 2 < 3 < 4 work. And in fact python and/or operators work in exactly the same way returning the value that determines the answer (2 and 3) equals 3. Icon can do this because it has a special result 'fail' which blocks further evaluation, roughly equivalent to 1 < 0 throwing an exception. It wouldn't work in Python for lots of reasons. > The object-example from above now tells us how boolean behaviour and > arithmetic behaviour go hand in hand: "(setA > setB) or (setA < setB)" > is True because "set([1,5]).__bor__(set([]))" is the same as > "set([1,5]) + set([])" and equivalent to True in a boolean context. > Likewise, "(setA > setB) and (setA < setB)" is False because > "set([1,5]).__band__set([])" is just "set([])". It follows that "(setA > > setB) == (setA - setB) == (setA & setB)". So what? This isn't a general principle or anything. All you've done is say that if you define set setB) or (setA < setB)" is always true or anything (because it isn't of course). > We can't do this with > boolean operations being part of the language only. > Nor do we need to. Figuring out the value of "(setA > setB) or (setA < setB)" is much more less convoluted than that. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Tue Sep 13 03:40:34 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 12 Sep 2011 21:40:34 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: Can we not allow things like `a < b` to return non-boolean values, without altering the behaviour of existing Python types? There _are_ use cases (numpy, ORMs, DSLs, and numpy). These don't really fit the bill. Devin On Mon, Sep 12, 2011 at 4:20 PM, Lukas Lueg wrote: > Hi, > > the proposal is to reduce the number of occurences where Python > automatically and inevitably uses pure boolean objects (bool()), > instead of relying on rich object behaviour. My goal is to give > objects finer control over how they want to be interpreted in a > boolean context. This reaches down as far as implementing the boolean > operations as an object-protocol. > > In Python, no kind of object is in any way special. Instead of relying > on what an object actually IS, we rely on what an object can DO. This > approach of anti-discrimination is the source of great confidence > about the language and it's aversion to black magic. The bool()-type > however is a primus inter pares: In numerous locations, Python > strongly favors this type of object either explicitly by casting to it > or implicitly through the compiled code. > > > Here are a three examples (protocol-, object- and code-based) where > Python grinds a boolean context down to pure boolean values: > > In protocol behaviour: The documentation about how to implement > __contains__ vaguely states that the function "should return true, > false otherwise". In fact we may return any object that can be > interpreted as a boolean, this is intented behaviour. However the > implementation of COMPARE_OP in Python/ceval.c inevitably casts > the result of "x in y" to Py_True or Py_False, throwing away our object > and leaving us with this bool only. > This kills any chance of producing useful objects in __contains__, > even objects that may have a meaning outside the pure boolean context. > For example: > > x = mycontainer(['foo', 'bar', 'bar', 'foo', 'foo']) > y = 'foo' in mycontainer > print y == True >>> True > print y >>> mycontainer('foo', 'foo', 'foo') > > This has been discussed on python-ideas in mid-2010 > (http://mail.python.org/pipermail/python-ideas/2010-July/007733.html) > without distinct outcome. > > > In object behaviour: Rich Comparision allows numeric- or set-like > comparision in any way we like, returing any kind of object. The > documentation clearly states that the returned object is only(!) > interpreted as a boolean when used in a boolean context (__contains__ > falls short here). > For example: > > x = set((1,2,3,4,5)) > y = set((2,3,4)) > z = x > y > print z == True >>> True > print z >>> set([1,5]) # equivalent to x - y > > z = x < y > print z == True >>> False > print z >>> set([]) # equivalent to y - x > > print 3 > 2 >>> 1 > print (3 > 2) == True >>> True > > > In code behaviour: Why is it, that we can override the behaviour of > arithmetic operations like "+, -, *, /", but not the boolean > operations like "and, or, xor, not" ? Why do we force any value being > generated in a boolean context like "if x == 1 or y == 1" to actually > be of boolean type? The result of "(x/y) == 1" may be any kind of > object coming from __eq__. However the "or"-operator here grinds them > both down to being just True or False. This is due to the fact that > the generated bytecode evaluates one expression at a time and does not > keep track of the objects it got while doing so. The pure boolean > behaviour arises from the use of JUMP_IF_FALSE/TRUE after each > comparision. > Instead of having boolean operations being a part of the language, we > could implement them as a an extension to the Rich Comparision > Protocol, giving rise to functions like object.__bor__, > object.__bnot__ or object.__band__: > > The expression "if x == 1 or y == 1" would then become equivalent to > > tmp1 = x.__eq__(1) > if tmp1: > ?return tmp1 > tmp2 = y.__eq__(1) > if tmp2: > ?return tmp2 > return = tmp1.__bor__(tmp2) > > Likewise, the expression "if x == 1 and y == 1" would become > > tmp1 = x.__eq__(1) > if not tmp1: > ?return tmp1 > tmp2 = y.__eq__(1) > if not tmp2: > ?return tmp2 > return tmp1.__band__(tmp2) > > The object-example from above now tells us how boolean behaviour and > arithmetic behaviour go hand in hand: "(setA > setB) or (setA < setB)" > is True because "set([1,5]).__bor__(set([]))" is the same as > "set([1,5]) + set([])" and equivalent to True in a boolean context. > Likewise, "(setA > setB) and (setA < setB)" is False because > "set([1,5]).__band__set([])" is just "set([])". It follows that "(setA >> setB) == (setA - setB) == (setA & setB)". We can't do this with > boolean operations being part of the language only. > > > Summing all up, I really think that we should break the dominance of > bool() and take a look at how we can implement boolean contexts > without relying on boolean values all the time. > > None of this can be implemented without breaking at least the CPython > API. For example, the behaviour of __contains__ can't be changed in > the proposed way without changing the signature of "int > PySequence_Contains()" to "PyObject* PySequence_Contains()". > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From ubershmekel at gmail.com Tue Sep 13 03:49:08 2011 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Tue, 13 Sep 2011 04:49:08 +0300 Subject: [Python-ideas] Proposal: add a calculator statistics module In-Reply-To: <09157BD0-3FB0-4301-8D91-E198DE989AB1@gmail.com> References: <4E6EAB33.50006@pearwood.info> <09157BD0-3FB0-4301-8D91-E198DE989AB1@gmail.com> Message-ID: I like the idea of including batteries for statistics. Not to leave without shedding a bike - I know the "math" module says it "provides access to the mathematical functions defined by the C standard", but I think it's the perfect home for such functionality. http://docs.python.org/library/math.html begs for another header named "statistical functions". --Yuval Greenfield -------------- next part -------------- An HTML attachment was scrubbed... URL: From massimo.dipierro at gmail.com Tue Sep 13 03:56:10 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Mon, 12 Sep 2011 20:56:10 -0500 Subject: [Python-ideas] win32 extensions Message-ID: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> Just a question. What is the reason the win32 extensions are shipped separately? Is it a licensing issue or is there another reason? Massimo From eliben at gmail.com Tue Sep 13 04:32:22 2011 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 13 Sep 2011 05:32:22 +0300 Subject: [Python-ideas] Proposal: add a calculator statistics module In-Reply-To: <4E6EAB33.50006@pearwood.info> References: <4E6EAB33.50006@pearwood.info> Message-ID: On Tue, Sep 13, 2011 at 04:00, Steven D'Aprano wrote: > I propose adding a basic calculator statistics module to the standard > library, similar to the sorts of functions you would get on a scientific > calculator: > > mean (average) > variance (population and sample) > standard deviation (population and sample) > correlation coefficient > +1 Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Tue Sep 13 04:35:56 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 13 Sep 2011 11:35:56 +0900 Subject: [Python-ideas] Proposal: add a calculator statistics module In-Reply-To: <4E6EAB33.50006@pearwood.info> References: <4E6EAB33.50006@pearwood.info> Message-ID: <87fwk12mpf.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > I propose adding a basic calculator statistics module to the standard > library, similar to the sorts of functions you would get on a scientific > calculator: > > mean (average) > variance (population and sample) > standard deviation (population and sample) > correlation coefficient +1 for these. > and similar. I'm not sure about that. I immediately thought "how about bivariate linear regression?", but that's actually misleading in many cases. Doing anything more general is beyond the scope of "like a typical scientific/statistical hand calculator". On the other hand, well-commented code showing how to do the bivariate case would help people to implement more general cases. Ditto for other common tasks like chi-square tests. From massimo.dipierro at gmail.com Tue Sep 13 04:50:26 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Mon, 12 Sep 2011 21:50:26 -0500 Subject: [Python-ideas] Proposal: add a calculator statistics module In-Reply-To: <87fwk12mpf.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4E6EAB33.50006@pearwood.info> <87fwk12mpf.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <24CA2D88-0629-4613-B4DF-936E17309E70@gmail.com> Technically linear regression can be implemented in one line of code assuming a good linear algebra package: http://code.google.com/p/fin2py/source/browse/numeric.py#791 (here used for fitting polynomials or any other bivariate or multivariate function linear in coefficients) Unless python is going to have a built-in linear algebra packages, I think linear regressions fits better in scipy. Massimo On Sep 12, 2011, at 9:35 PM, Stephen J. Turnbull wrote: > Steven D'Aprano writes: > >> I propose adding a basic calculator statistics module to the standard >> library, similar to the sorts of functions you would get on a scientific >> calculator: >> >> mean (average) >> variance (population and sample) >> standard deviation (population and sample) >> correlation coefficient > > +1 for these. > >> and similar. > > I'm not sure about that. I immediately thought "how about bivariate > linear regression?", but that's actually misleading in many cases. > Doing anything more general is beyond the scope of "like a typical > scientific/statistical hand calculator". > > On the other hand, well-commented code showing how to do the bivariate > case would help people to implement more general cases. Ditto for > other common tasks like chi-square tests. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Tue Sep 13 05:22:12 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 13 Sep 2011 13:22:12 +1000 Subject: [Python-ideas] win32 extensions References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> Message-ID: <878vptkty3.fsf@benfinney.id.au> Massimo Di Pierro writes: > Just a question. What is the reason the win32 extensions are shipped > separately? This might better be asked on the developers's forum; this forum is for discussing changes to Python. > Is it a licensing issue or is there another reason? Here is a message to consider for information about this issue . -- \ ?I think Western civilization is more enlightened precisely | `\ because we have learned how to ignore our religious leaders.? | _o__) ?Bill Maher, 2003 | Ben Finney From hetchkay at gmail.com Tue Sep 13 05:24:47 2011 From: hetchkay at gmail.com (H. Krishnan) Date: Mon, 12 Sep 2011 20:24:47 -0700 (PDT) Subject: [Python-ideas] return value of yield expressions Message-ID: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> Hi, I should start off by saying that my knowledge of PEP342 is less than a week old, and I cannot claim any use-case for what I am about to propose. The PEP itself says "In effect, a yield-expression is like an inverted function call; the argument to yield is in fact returned (yielded) from the currently executing function, and the "return value" of yield is the argument passed in via send()." One could, I guess, wrap a function to be 'called' via this mechanism: def genwrap(func): def gen(): ret = None while True: args, kwds = (yield ret) ret = func(*args, **kwds) del args, kwds g = gen() g.next() return g f = genwrap(func) f.send( (args, kwds) ) However, 'send' takes only one argument (and hence the poor syntax in the last statement). Has the option of the return value of the yield expression treated very much like function arguments been discussed? (a1, a2, a3 = default, *args, **kwds) = (yield ret) 'send' could support positional and keyword arguments. I guess this particular form would break backward compatibility but there might be other alternatives. Regards, Krishnan From greg.ewing at canterbury.ac.nz Tue Sep 13 01:25:12 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 13 Sep 2011 11:25:12 +1200 Subject: [Python-ideas] List Revolution In-Reply-To: <4E6DE0A4.7060303@pearwood.info> References: <4E6C02C1.9050704@pearwood.info> <1315716770.22156.150.camel@Gutsy> <1315775086.26528.29.camel@Gutsy> <2f456184-f51c-4f85-bf9e-17aebb43058c@email.android.com> <4E6DE0A4.7060303@pearwood.info> Message-ID: <4E6E94D8.2040006@canterbury.ac.nz> Steven D'Aprano wrote: > I believe that Ada allows that. Any Ada users want to comment on how > useful or annoying this is? I think that sort of thing works better in a statically-typed language, because anything you pass to a library needs to be of a data type defined by that library, or at least known to the library, so both the library and the caller are aware of the indexing scheme being used. With duck-typing, on the other hand, it just seems like a recipe for confusion. -- Greg From ncoghlan at gmail.com Tue Sep 13 05:39:38 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Sep 2011 13:39:38 +1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: On Tue, Sep 13, 2011 at 11:40 AM, Devin Jeanpierre wrote: > Can we not allow things like `a < b` to return non-boolean values, > without altering the behaviour of existing Python types? We already do. As far as I am aware, the only holdouts are: x in a (coerces to bool) not a (coerces to bool) a and b (doesn't coerce as such, but roughly equivalent to "a if not a else b") a or b (doesn't coerce as such, but roughly equivalent to "a if a else b") The first case can already be overridden (via __contains__) and I don't believe there's any specific reason for retaining the coercion to bool (aside from nobody writing and championing a patch that eliminates the restriction). PEP 207 (which added rich comparisons in the first place) is silent on the matter - it only talks about the comparison and ordering operations. The latter 3 cases are discussed in depth in PEP 335 (Overloadable Boolean Operators). Again, I don't believe there's anything fundamentally wrong with the idea, but it requires both an up to date reference implementation and a champion willing to gather use cases, assess the performance impact and generally argue in its favour. We don't have the numpy folks knocking down our door asking for distributable versions of these operations, after all. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Tue Sep 13 06:06:45 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Sep 2011 14:06:45 +1000 Subject: [Python-ideas] Proposal: add a calculator statistics module In-Reply-To: <4E6EAB33.50006@pearwood.info> References: <4E6EAB33.50006@pearwood.info> Message-ID: On Tue, Sep 13, 2011 at 11:00 AM, Steven D'Aprano wrote: > I propose adding a basic calculator statistics module to the standard > library, similar to the sorts of functions you would get on a scientific > calculator: > > mean (average) > variance (population and sample) > standard deviation (population and sample) > correlation coefficient > > and similar. I am volunteering to provide, and support, this module, written > in pure Python so other implementations will be able to use it. > > Simple calculator-style statistics seem to me to be a fairly obvious > "battery" to be included, more useful in practice than some functions > already available such as factorial and the hyperbolic functions. And since some folks may not have seen it, Steven's proposal here is following up on a suggestion Raymond Hettinger posted to this last year: http://mail.python.org/pipermail/python-ideas/2010-October/008267.html >From my point of view, I'd make the following suggestions: 1. We should start very small (similar to the way itertools grew over time) To me that means: mean, median, mode variance standard deviation Anything beyond that (including coroutine-style running calculations) is probably better left until 3.4. In the specific case of running calculations, this is to give us a chance to see how coroutine APIs are best written in a world where generators can return values as well as yielding them. Any APIs that would benefit from having access to running variants (such as being able to collect multiple statistics in a single pass) should also be postponed. Some more advanced algorithms could be included as recipes in the initial docs. The docs should also include pointers to more full-featured stats modules for reference when users needs outgrow the included batteries. 2. The 'math' module is not the place for this, a new, dedicated module is more appropriate. This is mainly due to the fact that the math module is focused primarily on binary floating point, while these algorithms should be neutral with regard to the specific numeric type involved. However, the practical issues with math being a builtin module are also a factor. There are many colours the naming bikeshed could be painted, but I'd be inclined to just call it 'statistics' ('statstools' is unwieldy, and other variants like 'stats', 'simplestats', 'statlib' and 'stats-tools' all exist on PyPI). Since the opportunity to just use the full word is there, we may as well take it. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Tue Sep 13 06:09:09 2011 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Sep 2011 21:09:09 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: On Mon, Sep 12, 2011 at 8:39 PM, Nick Coghlan wrote: > On Tue, Sep 13, 2011 at 11:40 AM, Devin Jeanpierre > wrote: >> Can we not allow things like `a < b` to return non-boolean values, >> without altering the behaviour of existing Python types? > > We already do. As far as I am aware, the only holdouts are: > > x in a (coerces to bool) > not a (coerces to bool) > a and b (doesn't coerce as such, but roughly equivalent to "a if not a else b") > a or b (doesn't coerce as such, but roughly equivalent to "a if a else b") > > The first case can already be overridden (via __contains__) But does the C code wrapping it preserve the non-bool value? (IIRC originally many of these were constrained by the C implementation more than by any philosophical desire to keep bool pure.) > and I > don't believe there's any specific reason for retaining the coercion > to bool (aside from nobody writing and championing a patch that > eliminates the restriction). Oh I see, you are saying that there is no need *in principle*. Agreed for __contains__. (BTW there is also a need for the reverse operator. Maybe it could be called __in__?) > PEP 207 (which added rich comparisons in > the first place) is silent on the matter - it only talks about the > comparison and ordering operations. It wasn't a priority at the time. > The latter 3 cases are discussed in depth in PEP 335 (Overloadable > Boolean Operators). Again, I don't believe there's anything > fundamentally wrong with the idea, but it requires both an up to date > reference implementation and a champion willing to gather use cases, > assess the performance impact and generally argue in its favour. We > don't have the numpy folks knocking down our door asking for > distributable versions of these operations, after all. I'm skeptical. It would mean that the compiler no longer has the option to translate "if not" into a reversal of the control flow; it must generate code to execute the "not" operator. That seems like a pessimization. Here's some disassembler output to prove it: >>> dis.dis(lambda x: 1 if not x else 2) 1 0 LOAD_FAST 0 (x) 3 POP_JUMP_IF_TRUE 10 6 LOAD_CONST 1 (1) 9 RETURN_VALUE >> 10 LOAD_CONST 2 (2) 13 RETURN_VALUE >>> dis.dis(lambda x: 1 if x else 2) 1 0 LOAD_FAST 0 (x) 3 POP_JUMP_IF_FALSE 10 6 LOAD_CONST 1 (1) 9 RETURN_VALUE >> 10 LOAD_CONST 2 (2) 13 RETURN_VALUE >>> It looks like the same argument cannot be made for 'and' / 'or', so perhaps those are fine -- though I also expect that there aren't many use cases. (For all these, I suspect the most common use case is actually symbolic algebra packages which want to overload operators so they can return a parse tree instead of an outcome. This works for regular operators and for comparisons, but not for in, not, and, or. But I'm not sure I want to pay the price for this flexibility everywhere where those operators are used in their traditional meanings. Admittedly I haven't read PEP 335 thoroughly. -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Tue Sep 13 06:26:28 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Sep 2011 14:26:28 +1000 Subject: [Python-ideas] win32 extensions In-Reply-To: <878vptkty3.fsf@benfinney.id.au> References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> Message-ID: On Tue, Sep 13, 2011 at 1:22 PM, Ben Finney wrote: > Massimo Di Pierro > writes: > >> Just a question. What is the reason the win32 extensions are shipped >> separately? > > This might better be asked on the developers's forum; this forum is for > discussing changes to Python. 'Why is it so?' questions are perfectly reasonable fodder for python-ideas (since they're usually a precursor to "let's do it differently"). Part of the reason this list exists is so misunderstandings of why Python is the way it is can be clarified here without generating traffic on the main development list. >> Is it a licensing issue or is there another reason? > > Here is a message to consider for information about this issue > . I'm not sure why you linked that - it's about the need to use Visual Studio to safely build C extensions on Windows, not about why the CPython Windows installer doesn't bundle pywin32 the way ActiveState (and, IIRC, Enthought) do. The Python 3 series defines a stable ABI (PEP 384) that eliminates that restriction. As far as the original question goes, Mark Hammond would be the person in the best position to answer that. I don't know much about how pywin32 development works, and I'm not sure any of the other core devs do, either. The 'pywin32' web presence is fragmented enough that it isn't easy to find out more. (The SF project pages are at http://sourceforge.net/projects/pywin32/ and there seems to be some discussion at http://mail.python.org/mailman/listinfo/python-win32, but there's all sorts of stale information floating around which means that Google doesn't help much, either) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Tue Sep 13 06:51:25 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Sep 2011 14:51:25 +1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: On Tue, Sep 13, 2011 at 2:09 PM, Guido van Rossum wrote: > On Mon, Sep 12, 2011 at 8:39 PM, Nick Coghlan wrote: >> On Tue, Sep 13, 2011 at 11:40 AM, Devin Jeanpierre >> wrote: >>> Can we not allow things like `a < b` to return non-boolean values, >>> without altering the behaviour of existing Python types? >> >> We already do. As far as I am aware, the only holdouts are: >> >> x in a (coerces to bool) >> not a (coerces to bool) >> a and b (doesn't coerce as such, but roughly equivalent to "a if not a else b") >> a or b (doesn't coerce as such, but roughly equivalent to "a if a else b") >> >> The first case can already be overridden (via __contains__) > > But does the C code wrapping it preserve the non-bool value? (IIRC > originally many of these were constrained by the C implementation more > than by any philosophical desire to keep bool pure.) > >> and I >> don't believe there's any specific reason for retaining the coercion >> to bool (aside from nobody writing and championing a patch that >> eliminates the restriction). > > Oh I see, you are saying that there is no need *in principle*. Agreed > for __contains__. (BTW there is also a need for the reverse operator. > Maybe it could be called __in__?) The two issues seem somewhat orthogonal to me, but yes, the general idea would be to make 'in' behave more like a rich comparison operator rather than an explicitly boolean operation as it does now. It occurs to me that adding __in__ could also address a slight performance oddity with 3.2 range objects: the __contains__ check in 3.2 has an O(1) fast path for containment checks on actual Python integers, but falls back to the O(n) sequential search for objects that only implement __index__(). If __in__() was available, such objects could conceivably map containment tests to checks against the corresponding real Python integer (although doing that carelessly would do weird things to other containers, such as identity-keyed dictionaries. That would be a quality of implementation issue on __in__ methods, though). > But I'm not sure I want to pay the price for this flexibility > everywhere where those operators are used in their traditional > meanings. Admittedly I haven't read PEP 335 thoroughly. I think that's a pretty good explanation for why PEP 335 hasn't been pushed too hard - there's a lot of value in having not/and/or behave more like control flow structures rather than ordinary operations. PEP 335 does show that it's theoretically possible to make them behave more like operators, but that doesn't make it a good idea. To be honest, I don't think anyone would cry too much if you decided to explicitly reject it on the basis of continuing to allow control flow optimisations for code involving not/and/or. While CPython doesn't do it, I believe there *are* control flow transformations that the current semantics permit that PEP 335 would disallow, such as automatically applying De Morgan's Law (I don't actually have a use case for doing that, I'm just mentioning it as a consequence of the semantics change proposed by the PEP). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From p.f.moore at gmail.com Tue Sep 13 11:23:35 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 13 Sep 2011 10:23:35 +0100 Subject: [Python-ideas] Proposal: add a calculator statistics module In-Reply-To: References: <4E6EAB33.50006@pearwood.info> Message-ID: On 13 September 2011 05:06, Nick Coghlan wrote: > On Tue, Sep 13, 2011 at 11:00 AM, Steven D'Aprano wrote: >> I propose adding a basic calculator statistics module to the standard >> library, similar to the sorts of functions you would get on a scientific >> calculator: >> >> mean (average) >> variance (population and sample) >> standard deviation (population and sample) >> correlation coefficient >> >> and similar. I am volunteering to provide, and support, this module, written >> in pure Python so other implementations will be able to use it. >> >> Simple calculator-style statistics seem to me to be a fairly obvious >> "battery" to be included, more useful in practice than some functions >> already available such as factorial and the hyperbolic functions. > > And since some folks may not have seen it, Steven's proposal here is > following up on a suggestion Raymond Hettinger posted to this last > year: > > http://mail.python.org/pipermail/python-ideas/2010-October/008267.html > > >From my point of view, I'd make the following suggestions: > > 1. We should start very small (similar to the way itertools grew over time) > > To me that means: > ?mean, median, mode > ?variance > ?standard deviation > > Anything beyond that (including coroutine-style running calculations) > is probably better left until 3.4. In the specific case of running > calculations, this is to give us a chance to see how coroutine APIs > are best written in a world where generators can return values as well > as yielding them. Any APIs that would benefit from having access to > running variants (such as being able to collect multiple statistics in > a single pass) should also be postponed. > > Some more advanced algorithms could be included as recipes in the > initial docs. The docs should also include pointers to more > full-featured stats modules for reference when users needs outgrow the > included batteries. > > 2. The 'math' module is not the place for this, a new, dedicated > module is more appropriate. This is mainly due to the fact that the > math module is focused primarily on binary floating point, while these > algorithms should be neutral with regard to the specific numeric type > involved. However, the practical issues with math being a builtin > module are also a factor. > > There are many colours the naming bikeshed could be painted, but I'd > be inclined to just call it 'statistics' ('statstools' is unwieldy, > and other variants like 'stats', 'simplestats', 'statlib' and > 'stats-tools' all exist on PyPI). Since the opportunity to just use > the full word is there, we may as well take it. +1 (both on the Steven's original suggestion, and Nick's follow-up comment). I like the suggestion of having a running calculation version, but agree that it's probably a bit soon to decide on the best API for such things. Recipes in the documentation would be a good start, though. One place I'd disagree with Nick, though, I'd like to see correlation coefficient and linear regression in there. They are common on calculators, and I do tend to use them reasonably often. Please save me from starting Excel to calculate them! :-) Paul. From ncoghlan at gmail.com Tue Sep 13 12:03:53 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Sep 2011 20:03:53 +1000 Subject: [Python-ideas] Proposal: add a calculator statistics module In-Reply-To: References: <4E6EAB33.50006@pearwood.info> Message-ID: On Tue, Sep 13, 2011 at 7:23 PM, Paul Moore wrote: > One place I'd disagree with Nick, though, I'd like to see correlation > coefficient and linear regression in there. They are common on > calculators, and I do tend to use them reasonably often. Please save > me from starting Excel to calculate them! :-) I'm not strongly against those two, I mainly wanted to emphasise the "start small and grow" aspect to try to keep the initial version focused and maintainable. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From p.f.moore at gmail.com Tue Sep 13 12:08:09 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 13 Sep 2011 11:08:09 +0100 Subject: [Python-ideas] Proposal: add a calculator statistics module In-Reply-To: References: <4E6EAB33.50006@pearwood.info> Message-ID: On 13 September 2011 11:03, Nick Coghlan wrote: > On Tue, Sep 13, 2011 at 7:23 PM, Paul Moore wrote: >> One place I'd disagree with Nick, though, I'd like to see correlation >> coefficient and linear regression in there. They are common on >> calculators, and I do tend to use them reasonably often. Please save >> me from starting Excel to calculate them! :-) > > I'm not strongly against those two, I mainly wanted to emphasise the > "start small and grow" aspect to try to keep the initial version > focused and maintainable. That's how I read your comments, and I agree. Paul. From jh at improva.dk Tue Sep 13 13:02:16 2011 From: jh at improva.dk (Jacob Holm) Date: Tue, 13 Sep 2011 13:02:16 +0200 Subject: [Python-ideas] return value of yield expressions In-Reply-To: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> Message-ID: <4E6F3838.5030701@improva.dk> Hi On 2011-09-13 05:24, H. Krishnan wrote: > Hi, > I should start off by saying that my knowledge of PEP342 is less than > a week old, and I cannot claim any use-case for what I am about to > propose. That's OK. Welcome to the wonderful world of PEP342 generators :) > > The PEP itself says "In effect, a yield-expression is like an inverted > function call; the argument to yield is in fact returned (yielded) > from the currently executing function, and the "return value" of yield > is the argument passed in via send()." > > One could, I guess, wrap a function to be 'called' via this mechanism: > def genwrap(func): > def gen(): > ret = None > while True: > args, kwds = (yield ret) > ret = func(*args, **kwds) > del args, kwds > g = gen() > g.next() > return g > > f = genwrap(func) > f.send( (args, kwds) ) > > However, 'send' takes only one argument (and hence the poor syntax in > the last statement). > Has the option of the return value of the yield expression treated > very much like function arguments been discussed? > > (a1, a2, a3 = default, *args, **kwds) = (yield ret) > > 'send' could support positional and keyword arguments. > > I guess this particular form would break backward compatibility but > there might be other alternatives. > There are really two independent proposals here. I'll call them "function argument unpacking" and "allow arbitrary arguments to send". To start with the latter, I don't think that is going to fly. All arguments to send must be returned as a single value by yield or cause most of the examples in the PEP380 discussion to fail. Allowing arbitrary arguments to send would have to change that value in a backward-incompatible way when sending a single argument. However, you don't really need to change send itself to improve the situation. A small helper function like: def send(*args, **kwds): if not args: raise TypeError( 'send() takes at least 1 positional argument (0 given)' ) return args[0].send((args[1:], kwds)) would turn your "f.send( (args, kwds) )" into "send(f, *args, **kwds)", which is already much nicer syntax. I would be +0 on adding such a helper as builtin, similar to the "next" builtin. Your other proposal is really independent of generators I think. I too would like to see a way to do "function argument unpacking". args, kwds = (yield ret) # any expression really (a1, a2, a3, *args), kwds = (lambda a1,a2,a3=default,*args, **kwds: (a1,a2,a3)+args, kwds )(*args, **kwds) is about the shortest way I can come up with that works today, and that is way too much repetition for my taste. If locals() were writable (a change that is unlikely to happen btw), this could be reduced to: args, kwds = (yield ret) # any expression really locals().update( (lambda a1,a2,a3=default,*args,**kwds:locals())(*args, **kwds) ) which would avoid some of the duplication but is really not that more readable. Your version seems like a nice extension of the regular tuple unpacking for the special case of (args, kwds) tuples. The main problem is how to distinguish the normal tuple unpacking from the new form. E.g. does a, b = (), {'a':1, 'b':2} result in a==() and b=={'a':1, 'b':2} or in a==1 and b==2? (it would have to be the first, for obvious backward-compatibility reasons) A possible solution would be: *(a1, a2, a3 = default, *args, **kwds) = (yield ret) Ie, let '*(argument list) = args, kwds' be the syntax for "function argument unpacking". I am not sure if this is parseable within the constraints that we have for the python parser, but I think it would be. It is currently invalid syntax, so should be backwards compatible. To summarize -1 to "allow arbitrary arguments to generator.send". +0 to adding a builtin "send" helper function as described above. +1 for "function argument unpacking". Best regards - Jacob From stefan_ml at behnel.de Tue Sep 13 14:21:31 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 13 Sep 2011 14:21:31 +0200 Subject: [Python-ideas] return value of yield expressions In-Reply-To: <4E6F3838.5030701@improva.dk> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> Message-ID: Jacob Holm, 13.09.2011 13:02: > args, kwds = (yield ret) # any expression really > (a1, a2, a3, *args), kwds = (lambda a1,a2,a3=default,*args, **kwds: > (a1,a2,a3)+args, kwds > )(*args, **kwds) Note that recent Python versions support extended argument unpacking, so this works: a1, *other, a2 = return_some_sequence() If you use the last value of the returned sequence (such as a tuple) to pass a dict, or if you return a tuple with two arguments (posargs, kwargdict), you basically get what you wanted above. Time machine keys are back were they were. Nothing to see here, keep passing. Stefan From jh at improva.dk Tue Sep 13 14:32:01 2011 From: jh at improva.dk (Jacob Holm) Date: Tue, 13 Sep 2011 14:32:01 +0200 Subject: [Python-ideas] return value of yield expressions In-Reply-To: References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> Message-ID: <4E6F4D41.1080904@improva.dk> Hi Stefan On 2011-09-13 14:21, Stefan Behnel wrote: > Jacob Holm, 13.09.2011 13:02: >> args, kwds = (yield ret) # any expression really >> (a1, a2, a3, *args), kwds = (lambda a1,a2,a3=default,*args, **kwds: >> (a1,a2,a3)+args, kwds >> )(*args, **kwds) > > Note that recent Python versions support extended argument unpacking, so > this works: > > a1, *other, a2 = return_some_sequence() > If you look closer you'll see I am actually using that feature already in the code snippet you quoted. > If you use the last value of the returned sequence (such as a tuple) to > pass a dict, or if you return a tuple with two arguments (posargs, > kwargdict), you basically get what you wanted above. basically no. The suggested "function argument unpacking" includes support for default values, and for passing positional arguments by name. Everything that happens when you call a function using (*arg, **kwds) really. > Time machine keys are back were they were. Nothing to see here, keep > passing. I disagree. You overlooked most of the requested feature... - Jacob From anacrolix at gmail.com Tue Sep 13 14:49:05 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Tue, 13 Sep 2011 22:49:05 +1000 Subject: [Python-ideas] return value of yield expressions In-Reply-To: <4E6F4D41.1080904@improva.dk> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> Message-ID: I don't see any value in this. Please give an example where keyword unpacking is clearly superior. On Sep 13, 2011 10:42 PM, "Jacob Holm" wrote: > Hi Stefan > > On 2011-09-13 14:21, Stefan Behnel wrote: >> Jacob Holm, 13.09.2011 13:02: >>> args, kwds = (yield ret) # any expression really >>> (a1, a2, a3, *args), kwds = (lambda a1,a2,a3=default,*args, **kwds: >>> (a1,a2,a3)+args, kwds >>> )(*args, **kwds) >> >> Note that recent Python versions support extended argument unpacking, so >> this works: >> >> a1, *other, a2 = return_some_sequence() >> > > If you look closer you'll see I am actually using that feature already > in the code snippet you quoted. > > >> If you use the last value of the returned sequence (such as a tuple) to >> pass a dict, or if you return a tuple with two arguments (posargs, >> kwargdict), you basically get what you wanted above. > > basically no. The suggested "function argument unpacking" includes > support for default values, and for passing positional arguments by > name. Everything that happens when you call a function using (*arg, > **kwds) really. > > >> Time machine keys are back were they were. Nothing to see here, keep >> passing. > > I disagree. You overlooked most of the requested feature... > > - Jacob > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From hetchkay at gmail.com Tue Sep 13 15:00:59 2011 From: hetchkay at gmail.com (H. Krishnan) Date: Tue, 13 Sep 2011 06:00:59 -0700 (PDT) Subject: [Python-ideas] return value of yield expressions In-Reply-To: References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> Message-ID: <1694141.1337.1315918859671.JavaMail.geo-discussion-forums@vbac9> In the particular situation of send() --> yield, one could do something like (forgeting the backward compatibility with respect to the return value): gen.send(2, a2=3, e=5) and in the gen: (a1, a2, a3 = 3, *args, **kwds) = (yield ) without having to do whatever Jacob has written: args, kwds = (yield ) (a1, a2, a3, *args), kwds = (lambda a1,a2,a3=3,*args, **kwds: (a1,a2,a3)+args, kwds )(*args, **kwds) -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Tue Sep 13 15:00:59 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 13 Sep 2011 15:00:59 +0200 Subject: [Python-ideas] return value of yield expressions In-Reply-To: <4E6F4D41.1080904@improva.dk> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> Message-ID: Jacob Holm, 13.09.2011 14:32: > On 2011-09-13 14:21, Stefan Behnel wrote: >> Jacob Holm, 13.09.2011 13:02: >>> args, kwds = (yield ret) # any expression really >>> (a1, a2, a3, *args), kwds = (lambda a1,a2,a3=default,*args, **kwds: >>> (a1,a2,a3)+args, kwds >>> )(*args, **kwds) >> >> Note that recent Python versions support extended argument unpacking, so >> this works: >> >> a1, *other, a2 = return_some_sequence() >> > > If you look closer you'll see I am actually using that feature already > in the code snippet you quoted. Ah, ok, I thought you were *proposing* to add this. That happens on python-ideas a lot more often than you seem to expect. >> If you use the last value of the returned sequence (such as a tuple) to >> pass a dict, or if you return a tuple with two arguments (posargs, >> kwargdict), you basically get what you wanted above. > > basically no. The suggested "function argument unpacking" includes > support for default values, and for passing positional arguments by > name. Everything that happens when you call a function using (*arg, > **kwds) really. Well, I really don't see how this is a wide-spread use case (I certainly never stumbled over it), but if you feel like needing it, write a utility function that does the unpacking for you in a couple of lines and wrap the call with that. I have my doubts that it would make your code much clearer. Especially default values for keyword dict return values do not appear to be any useful to me, given that you'd most likely unpack them one by one anyway. So you could just use d.get() with a default argument there, thus making it explicit and obvious in your code what is going on. Stefan From hetchkay at gmail.com Tue Sep 13 15:03:50 2011 From: hetchkay at gmail.com (H. Krishnan) Date: Tue, 13 Sep 2011 06:03:50 -0700 (PDT) Subject: [Python-ideas] return value of yield expressions In-Reply-To: <4E6F3838.5030701@improva.dk> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> Message-ID: <24129449.2208.1315919030106.JavaMail.geo-discussion-forums@yqap15> > To start with the latter, I don't think that is going to fly. All > arguments to send must be returned as a single value by yield or cause > most of the examples in the PEP380 discussion to fail. Allowing > arbitrary arguments to send would have to change that value in a > backward-incompatible way when sending a single argument. > Actually, the issue is with how the return value is gathered rather than in send() itself, right? If the return value is specially gathered for unpacking (as you suggested), would there be any other issue with backward compatibility? -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas.lueg at googlemail.com Tue Sep 13 16:01:29 2011 From: lukas.lueg at googlemail.com (Lukas Lueg) Date: Tue, 13 Sep 2011 16:01:29 +0200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: > I think that's a pretty good explanation for why PEP 335 hasn't been > pushed too hard - there's a lot of value in having not/and/or behave > more like control flow structures rather than ordinary operations. PEP > 335 does show that it's theoretically possible to make them behave > more like operators, but that doesn't make it a good idea. The question is if we want to treat bools more special than any other type. Right now bools are the only place where this is done. Do we really want to specialize the language in order to favour control flow in the implementation? Forever? That's a bold statement. > To be honest, I don't think anyone would cry too much if you decided > to explicitly reject it on the basis of continuing to allow control > flow optimisations for code involving not/and/or. While CPython > doesn't do it, I believe there *are* control flow transformations that > the current semantics permit that PEP 335 would disallow, such as > automatically applying De Morgan's Law (I don't actually have a use > case for doing that, I'm just mentioning it as a consequence of the > semantics change proposed by the PEP). Performance is always an argument. Python as a language, and most especially CPython itself, has a history of favouring design over application; the same should be true for bools. Concerning De Morgan's Law, I think in Python this should be treated as a convention in the implementation, not a rule of the language. After all, we also have [1,2,3] + [4,5,6] = [1,2,3,4,5,6] not by rule but by simple reasoning. Also: If an object does not override a certain boolean operation, the compiler can still take advantage of optimisation. Progress on the AST may pave the road for that. From brian.curtin at gmail.com Tue Sep 13 16:14:14 2011 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 13 Sep 2011 09:14:14 -0500 Subject: [Python-ideas] win32 extensions In-Reply-To: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> Message-ID: On Mon, Sep 12, 2011 at 20:56, Massimo Di Pierro wrote: > Just a question. What is the reason the win32 extensions are shipped separately? Is it a licensing issue or is there another reason? It's certainly a mature project having been around for quite some time, and has also provided Python 3 support since the beginning. It's also a project for which there is no replacement, outside of hand-writing extension modules to expose the APIs you need, and it has given many developers access to the Windows APIs from the ease of a few simple imports. For those reasons, I think it's a great candidate for inclusion. There are negatives I see to inclusion, mainly the sheer size at 213K SLOC [0] (CPython is 713K). We're currently mulling over the inclusion of a new regular expression library which clocks in well below that at 28K, but it's potentially above the level at which anyone could provide a thorough review and gain an understanding, while simultaneously reducing the bus factor. While Mark is a respected member of the community and has already contributed directly to CPython, we should be careful to include something so large. I may be missing something due to the fragmented web presence, but documentation could be a challenge, although I remember some form of documentation being available on the ActiveState. Perhaps that could be converted to reStructuredText, or that might just have to start over. Maybe I'll seem crazy, but I think if we are to include this, the "py" prefix of the name would have to go. In fact, moving forward with this, even "win32" is probably a less than appropriate name, as Microsoft has moved to "Windows API" some time ago, which "more accurately reflects its roots in 16-bit Windows and its support on 64-bit Windows" [1]. I don't know the track record of stdlib inclusions and name changes, but I feel like this should be called "windows". Summary: although my negative talk outweighs the dinky positive paragraph, I think it's something worth looking into if Mark is on board with it. [0] generated using David A. Wheeler's SLOCCount. [1] http://en.wikipedia.org/wiki/Windows_API From jh at improva.dk Tue Sep 13 16:17:08 2011 From: jh at improva.dk (Jacob Holm) Date: Tue, 13 Sep 2011 16:17:08 +0200 Subject: [Python-ideas] return value of yield expressions In-Reply-To: References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> Message-ID: <4E6F65E4.20507@improva.dk> On 2011-09-13 15:00, Stefan Behnel wrote: > Jacob Holm, 13.09.2011 14:32: >>> If you use the last value of the returned sequence (such as a tuple) to >>> pass a dict, or if you return a tuple with two arguments (posargs, >>> kwargdict), you basically get what you wanted above. >> >> basically no. The suggested "function argument unpacking" includes >> support for default values, and for passing positional arguments by >> name. Everything that happens when you call a function using (*arg, >> **kwds) really. > > Well, I really don't see how this is a wide-spread use case (I certainly > never stumbled over it), but if you feel like needing it, write a > utility function that does the unpacking for you in a couple of lines > and wrap the call with that. I have my doubts that it would make your > code much clearer. I remember running into this exact problem at one point when working with decorators, but don't remember the details. IIRC you really can't do much in terms of writing a helper function. You can get far, but not all the way. a, b, c, args, kwds = helper( (lambda a, b, c=3, *args, **kwds:locals()), (args, kwds) ) is IIRC the best you can do. You need the tuple-unpacking assignment to get the names into locals, and you need an actual function (the lambda) to specify the arguments in a natural way. You still end up repeating the names of each argument, which is a DRY violation. No helper function can get you even close to the readability and non-DRYness of *(a, b, c=42, *args, **kwds) = (args, kwds) > > Especially default values for keyword dict return values do not appear > to be any useful to me, given that you'd most likely unpack them one by > one anyway. So you could just use d.get() with a default argument there, > thus making it explicit and obvious in your code what is going on. The point is to be able to get from a (args, kwds) tuple to some actual locally assigned names, based on the rules we already know and love from function arguments. I know this is a weak argument, based on purity rather than practicality, but until I run into the issue again I really can't give you an example. - Jacob From brian.curtin at gmail.com Tue Sep 13 16:18:22 2011 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 13 Sep 2011 09:18:22 -0500 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> Message-ID: On Tue, Sep 13, 2011 at 09:14, Brian Curtin wrote: > Maybe I'll seem crazy, but I think if we are to include this, the "py" > prefix of the name would have to go. In fact, moving forward with > this, even "win32" is probably a less than appropriate name, as > Microsoft has moved to "Windows API" some time ago, which "more > accurately reflects its roots in 16-bit Windows and its support on > 64-bit Windows" [1]. I don't know the track record of stdlib > inclusions and name changes, but I feel like this should be called > "windows". I should finish this thought: I think "windows" should be a top-level package, and the sub-packages like "win32com" or "win32api" should drop the "win32" prefix. From jh at improva.dk Tue Sep 13 16:33:07 2011 From: jh at improva.dk (Jacob Holm) Date: Tue, 13 Sep 2011 16:33:07 +0200 Subject: [Python-ideas] return value of yield expressions In-Reply-To: <24129449.2208.1315919030106.JavaMail.geo-discussion-forums@yqap15> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <24129449.2208.1315919030106.JavaMail.geo-discussion-forums@yqap15> Message-ID: <4E6F69A3.5030602@improva.dk> Hi On 2011-09-13 15:03, H. Krishnan wrote: >> To start with the latter, I don't think that is going to fly. All >> arguments to send must be returned as a single value by yield or cause >> most of the examples in the PEP380 discussion to fail. Allowing >> arbitrary arguments to send would have to change that value in a >> backward-incompatible way when sending a single argument. >> > > Actually, the issue is with how the return value is gathered rather than in > send() itself, right? If the return value is specially gathered for > unpacking (as you suggested), would there be any other issue with backward > compatibility? Not quite sure I understand the question, but I'll try to answer anyway. You need to be able to use "var1 = yield var2" in the generator independent on what send is called with, or you can't wrap a generator using another generator (different ways of doing that was a large part of the PEP380 discussion). So what should "var1" be in each of the following cases? g.send() # currently illegal g.send(None) # None g.send((), {}) # currently illegal g.send(((), {})) # ((), {}) I suppose you could change it so passing exactly one argument did something different from passing zero or multiple arguments, but then you have a problem distinguishing when you actually want to use the value in the other end. Hope this helps. - Jacob From alexander.belopolsky at gmail.com Tue Sep 13 16:35:05 2011 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 13 Sep 2011 10:35:05 -0400 Subject: [Python-ideas] Proposal: add a calculator statistics module In-Reply-To: References: <4E6EAB33.50006@pearwood.info> Message-ID: On Tue, Sep 13, 2011 at 12:06 AM, Nick Coghlan wrote: .. > 2. The 'math' module is not the place for this, a new, dedicated > module is more appropriate. This is mainly due to the fact that the > math module is focused primarily on binary floating point, while these > algorithms should be neutral with regard to the specific numeric type > involved. While I agree that a new dedicated module is an appropriate place for this functionality, I am concerned that generic type-neutral algorithms may be sub-optimal for float-only calculations. I think at least at the start, the new module should follow the same design as the math module - namely provide numerically stable efficient algorithms for double precision floating point data. Other types should be supported by coercing to float. This means that support for long integers, decimal type, fractions, complex numbers will necessarily be limited. For example, you will not be able to compute average of 100-digit integers accurately even when the exact answer is an integer. Once the semantics of operations on general iterables yielding numbers are in place, the first optimization should probably be to special case operations on arrays of doubles. Proper support for decimal type, fractions, etc would probably need a separate project. From ron3200 at gmail.com Tue Sep 13 04:13:55 2011 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 12 Sep 2011 21:13:55 -0500 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: <1315880035.30361.3.camel@Gutsy> On Mon, 2011-09-12 at 21:40 -0400, Devin Jeanpierre wrote: > Can we not allow things like `a < b` to return non-boolean values, > without altering the behaviour of existing Python types? Would that return 'a' or 'b', or something else? From aquavitae69 at gmail.com Tue Sep 13 17:03:10 2011 From: aquavitae69 at gmail.com (David Townshend) Date: Tue, 13 Sep 2011 17:03:10 +0200 Subject: [Python-ideas] Object interface to path names Message-ID: I've recently been developing a tool to track changes to a fairly large structured file system, and in the process got to thinking about how working with path names could be improved. The problems I've had with using just os and os.path have lead to three objectives of any new implementation: 1. Cleaner handling of paths names, specifically constructing path names without the need for a lot of nested os.path.join() and os.split() functions. 2. Allow a validation of paths names based on predefined rules. (Although this requirement might be very specific to my use case) 3. Allow caching of file attribute data so that queries do not have to wait the disk or network to respond (although at the cost of accuracy). The first can be met with behaviour as follows, basically handling paths as containers of sub-paths: >>> root = Path('/usr') >>> print([n for n in root]) ['bin', 'local', 'lib', 'share'] >>> print([n for n in root.dirs]) ['bin', 'local', 'lib', 'share'] >>> print([n for n in root.files]) [] >>> root['local/share'] Path('/usr/local/share') >>> root['/usr/local/share'] Path('/usr/local/share') >>> share = Path('/usr/local/share') >>> share in root True The second can be met by allowing Path to be subclassed and defining factory functions for the creation of sub-paths: >>> root = Path('/usr') >>> root.factory = my_factory_function >>> root['local/share'] MyPath('/usr/local/share') The third can be met be allowing all disk calls to be asynchonous: >>> path = Path('/home/david/newfile', async=True) >>> path.touch() >>> path.exists() False >>> time.sleep(2) >>> path.exists() True This could all be implemented in pure python using os and os.path, and threading for asynchonous calls. I haven't yet thought through a complete specification for Path, but I image it would need to contain functions such as exists(), isfile(), isdir(), stat(), walk(), and allow iterator access to children. Does anyone else see a usefulness for this? Regards David -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas.lueg at googlemail.com Tue Sep 13 17:05:10 2011 From: lukas.lueg at googlemail.com (Lukas Lueg) Date: Tue, 13 Sep 2011 17:05:10 +0200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <1315880035.30361.3.camel@Gutsy> References: <1315880035.30361.3.camel@Gutsy> Message-ID: 2011/9/13 Ron Adam : > On Mon, 2011-09-12 at 21:40 -0400, Devin Jeanpierre wrote: >> Can we not allow things like `a < b` to return non-boolean values, >> without altering the behaviour of existing Python types? > > Would that return 'a' or 'b', or something else? That depends on the object at hand. 'a < b' could return 'b - a' for set-like objects. Read 'a < b' as the answer to the question 'how much bigger is b than a?'. From jeanpierreda at gmail.com Tue Sep 13 17:04:51 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 13 Sep 2011 11:04:51 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <1315880035.30361.3.camel@Gutsy> References: <1315880035.30361.3.camel@Gutsy> Message-ID: > Would that return 'a' or 'b', or something else? Usually it'd return True or False. Same with things like `foo in bar`. I just mean that we can extend these to return other objects without insisting that builtin types (floats, sets, etc.) do weird things with them. Devin On Mon, Sep 12, 2011 at 10:13 PM, Ron Adam wrote: > On Mon, 2011-09-12 at 21:40 -0400, Devin Jeanpierre wrote: >> Can we not allow things like `a < b` to return non-boolean values, >> without altering the behaviour of existing Python types? > > Would that return 'a' or 'b', or something else? > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From jeanpierreda at gmail.com Tue Sep 13 17:16:10 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 13 Sep 2011 11:16:10 -0400 Subject: [Python-ideas] Object interface to path names In-Reply-To: References: Message-ID: There are existing implementations of this sort of OOP approach, I think it'd be worth trying to adopt one of them instead of creating a new ad-hoc interface. Here are the two that come to mind: http://twistedmatrix.com/documents/10.1.0/api/twisted.python.filepath.FilePath.html https://bitbucket.org/birkenfeld/sphinx/src/cf794ec8a096/tests/path.py I support including such an API (+1), but I don't believe you mentioned my preferred use-case: Having an object-oriented / polymorphic file API means that one can provide file objects that aren't backed by the filesystem, but look the same. This is cool in that you can do things like treat zip files as directory, or mock out the filesystem for a unit test, without worrying about monkeypatching builtins or using a nonstandard wrapper API etc. Devin On Tue, Sep 13, 2011 at 11:03 AM, David Townshend wrote: > I've recently been developing a tool to track changes to a fairly large > structured file system, and in the process got to thinking about how working > with path names could be improved. The problems I've had with using just os > and os.path have lead to three objectives of any new implementation: > 1. ?Cleaner handling of paths names, specifically constructing path names > without the need for a lot of nested os.path.join() and os.split() > functions. > 2. ?Allow a validation of paths names based on predefined rules. (Although > this requirement might be very specific to my use case) > 3. ?Allow caching of file attribute data so that queries do not have to wait > the disk or network to respond (although at the cost of accuracy). > The first can be met with behaviour as follows, basically handling paths as > containers of sub-paths: >>>> root = Path('/usr') >>>> print([n for n in root]) > ['bin', 'local', 'lib', 'share'] >>>> print([n for n in root.dirs]) > ['bin', 'local', 'lib', 'share'] >>>> print([n for n in root.files]) > [] >>>> root['local/share'] > Path('/usr/local/share') >>>> root['/usr/local/share'] > Path('/usr/local/share') >>>> share = Path('/usr/local/share') >>>> share in root > True > The second can be met by allowing Path to be subclassed and defining factory > functions for the creation of sub-paths: >>>> root = Path('/usr') >>>> root.factory = my_factory_function >>>> root['local/share'] > MyPath('/usr/local/share') > The third can be met be allowing all disk calls to be asynchonous: >>>> path = Path('/home/david/newfile', async=True) >>>> path.touch() >>>> path.exists() > False >>>> time.sleep(2) >>>> path.exists() > True > This could all be implemented in pure python using os and os.path, and > threading for asynchonous calls. I haven't yet thought through a complete > specification for Path, but I image it would need to contain functions such > as exists(), isfile(), isdir(), stat(), walk(), and allow iterator access to > children. > Does anyone else see a usefulness for this? > Regards > David > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > From dirkjan at ochtman.nl Tue Sep 13 17:16:41 2011 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Tue, 13 Sep 2011 17:16:41 +0200 Subject: [Python-ideas] Object interface to path names In-Reply-To: References: Message-ID: On Tue, Sep 13, 2011 at 17:03, David Townshend wrote: > Does anyone else see a usefulness for this? Yes. IIRC there are a number of implementations on PyPI, and it probably has come up on this mailing list before (search the archives for the outcome!). http://pypi.python.org/pypi/Unipath/0.2.1 http://pypi.python.org/pypi/fpath/0.6 http://pypi.python.org/pypi/forked-path/0.2 http://pypi.python.org/pypi/path.py Cheers, Dirkjan From hetchkay at gmail.com Tue Sep 13 17:09:34 2011 From: hetchkay at gmail.com (H. Krishnan) Date: Tue, 13 Sep 2011 08:09:34 -0700 (PDT) Subject: [Python-ideas] return value of yield expressions In-Reply-To: <4E6F69A3.5030602@improva.dk> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <24129449.2208.1315919030106.JavaMail.geo-discussion-forums@yqap15> <4E6F69A3.5030602@improva.dk> Message-ID: <17098410.132.1315926574320.JavaMail.geo-discussion-forums@yqmv39> > You need to be able to use "var1 = yield var2" in the generator > independent on what send is called with, or you can't wrap a generator > using another generator (different ways of doing that was a large part > of the PEP380 discussion). > > So what should "var1" be in each of the following cases? > > g.send() # currently illegal > g.send(None) # None > g.send((), {}) # currently illegal > g.send(((), {})) # ((), {}) > > I suppose you could change it so passing exactly one argument did > something different from passing zero or multiple arguments, but then > you have a problem distinguishing when you actually want to use the > value in the other end. > Let us consider from a function point of view: If suppose there was no support for *args and **kwds and you wrote: def func(a): do_something_with_a and this was called with: func((1,2)) and subsequently, *args, **kwds support was added to functions, will anything related to 'func' need to change? Thus, if var = yield is used, send() needs to be called with only one argument (or as send(var = x) ). If var1, var2 = yield is used, send() needs to be called with two arguments. If yield is used, send() can be called with any no. of arguments/keywords etc. Krishnan -------------- next part -------------- An HTML attachment was scrubbed... URL: From hetchkay at gmail.com Tue Sep 13 17:30:50 2011 From: hetchkay at gmail.com (H. Krishnan) Date: Tue, 13 Sep 2011 08:30:50 -0700 (PDT) Subject: [Python-ideas] return value of yield expressions In-Reply-To: <17098410.132.1315926574320.JavaMail.geo-discussion-forums@yqmv39> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <24129449.2208.1315919030106.JavaMail.geo-discussion-forums@yqap15> <4E6F69A3.5030602@improva.dk> <17098410.132.1315926574320.JavaMail.geo-discussion-forums@yqmv39> Message-ID: <22927316.193.1315927850430.JavaMail.geo-discussion-forums@yqmv39> > If > var1, var2 = yield > is used, send() needs to be called with two arguments. > > Sorry, this should be *(var1, var2) = yield if I use Jacob's syntax. var1, var2 = yield will require send be called with a single tuple as argument. -------------- next part -------------- An HTML attachment was scrubbed... URL: From aquavitae69 at gmail.com Tue Sep 13 17:33:06 2011 From: aquavitae69 at gmail.com (David Townshend) Date: Tue, 13 Sep 2011 17:33:06 +0200 Subject: [Python-ideas] Object interface to path names In-Reply-To: References: Message-ID: But none of these seem to allow asynchronous calls, which make a huge difference when dealing with a large structure. Nothing I've found really does what I need and I'm trying to keep my dependency list short so I'll end up writing something myself anyway, but my real question was whether this is something that could usefully be included in the stdlib. Having an object-oriented / polymorphic file API means that one can > provide file objects that aren't backed by the filesystem, but look > the same. This is cool in that you can do things like treat zip files > as directory, or mock out the filesystem for a unit test, without > worrying about monkeypatching builtins or using a nonstandard wrapper > API etc. > Great use case! And using factories to create create the objects would make this especially powerful. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Sep 13 17:56:12 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Sep 2011 08:56:12 -0700 Subject: [Python-ideas] return value of yield expressions In-Reply-To: <1694141.1337.1315918859671.JavaMail.geo-discussion-forums@vbac9> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> <1694141.1337.1315918859671.JavaMail.geo-discussion-forums@vbac9> Message-ID: On Tue, Sep 13, 2011 at 6:00 AM, H. Krishnan wrote: > In the particular situation of send() --> yield, one could do something like > (forgeting the backward compatibility with respect to the return value): > gen.send(2, a2=3, e=5) > and in the gen: > (a1, a2, a3 = 3, *args, **kwds) = (yield ) What on earth is this syntax supposed to mean? Never mind that there's a yield on the RHS; it is just a parenthesized expression so it could could be any other function. What on earth do you expect to happen with the syntax on the left, i.e. with the part (a1, a2, a3 = 3, *args, **kwds) = ........whatever........ ??????? It seems to me that changes to send() could be proposed that allow .send() without argument or with multiple (positional) arguments, but this would just make the yield expression return a tuple. It is not unprecedented that foo() == foo(None), nor is it inconceivable that bar(x, y) == bar((x, y)). This is not so different from the equivalence between "x = 1, 2" and "x = (1, 2)". Do note that there's perhaps a bit of an odd case where g.send((1,)) should yield a 1-tuple, whereas g.send(1,) would be equivalent to g.send(1) and hence just send the value 1, not wrapped in a tuple. That this wasn't proposed in the original PEP 342 was probably just a matter of keeping things simple -- which I am still in favor of, and barring a lot more evidence of how incredibly useful the proposed enhancement would be (with *real* use cases, not made-up examples!) I will remain -0 on the proposal of allowing different argument counts to .send(). But this business with argument unpacking syntax is different. It seems poorly thought through. Syntax proposals like this often fail because the proposer does not actually understand how Python's parser works and how it is constrained, intentionally, to limited look-ahead and no compile-time knowledge of the types of the values being passed around at runtime. Also keep in mind orthogonality -- (yield ) should be usable anywhere: in an expression, in an if-statement, in a return statement, in a function argument, in an index, as an operand of a built-in operator, etc. You can't have a grammar where the syntax for the left-hand side of the assignment symbol has a different syntax depending on whether the righ-hand side contains a (yield ) or not. -- --Guido van Rossum (python.org/~guido) From p.f.moore at gmail.com Tue Sep 13 18:02:01 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 13 Sep 2011 17:02:01 +0100 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <1315880035.30361.3.camel@Gutsy> Message-ID: On 13 September 2011 16:05, Lukas Lueg wrote: > 2011/9/13 Ron Adam : >> On Mon, 2011-09-12 at 21:40 -0400, Devin Jeanpierre wrote: >>> Can we not allow things like `a < b` to return non-boolean values, >>> without altering the behaviour of existing Python types? >> >> Would that return 'a' or 'b', or something else? > > That depends on the object at hand. 'a < b' could return 'b - a' for > set-like objects. Read 'a < b' as the answer to the question 'how much > bigger is b than a?'. Maybe I'm being dense, but why would I read it like that? The operation a < b means "is a less than b?" That's not a matter of opinion, as far as I can see, that's basically the definition of "<". You could certainly overload < to mean something else for particular types, but that's not something to do lightly (look at C++'s use of << for IO, for a relatively benign example of both the benefits and problems with such an approach...) Paul. From guido at python.org Tue Sep 13 18:04:03 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Sep 2011 09:04:03 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: On Tue, Sep 13, 2011 at 7:01 AM, Lukas Lueg wrote: > The question is if we want to treat bools more special than any other > type. Right now bools are the only place where this is done. Do we > really want to specialize the language in order to favour control flow > in the implementation? Forever? That's a bold statement. I have no problem with this statement and don't see it as particularly bold or controversial. Control flow just *is* special. FWIW It is not bool values that are being treated special; it is certain bool operators (not, and, or). I see nothing wrong with that; you can think of them as an extension of the repertoire of 'if', 'while' etc. > Performance is always an argument. Python as a language, and most > especially CPython itself, has a history of favouring design over > application; the same should be true for bools. Concerning De Morgan's > Law, I think in Python this should be treated as a convention in the > implementation, not a rule of the language. After all, we also have > [1,2,3] + [4,5,6] = [1,2,3,4,5,6] not by rule but by simple reasoning. De Morgan's law (and similar transformation) seem to me out of scope. The would violate gut feelings about the "natural" execution order of Python expressions. > Also: If an object does not override a certain boolean operation, the > compiler can still take advantage of optimisation. Progress on the AST > may pave the road for that. No, because the compiler does not have the runtime types of the object available. If you don't know how Python really works you shouldn't be making language change proposals. -- --Guido van Rossum (python.org/~guido) From jh at improva.dk Tue Sep 13 17:57:47 2011 From: jh at improva.dk (Jacob Holm) Date: Tue, 13 Sep 2011 17:57:47 +0200 Subject: [Python-ideas] return value of yield expressions In-Reply-To: <17098410.132.1315926574320.JavaMail.geo-discussion-forums@yqmv39> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <24129449.2208.1315919030106.JavaMail.geo-discussion-forums@yqap15> <4E6F69A3.5030602@improva.dk> <17098410.132.1315926574320.JavaMail.geo-discussion-forums@yqmv39> Message-ID: <4E6F7D7B.6070103@improva.dk> On 2011-09-13 17:09, H. Krishnan wrote: > > >> You need to be able to use "var1 = yield var2" in the generator >> independent on what send is called with, or you can't wrap a generator >> using another generator (different ways of doing that was a large part >> of the PEP380 discussion). >> >> So what should "var1" be in each of the following cases? >> >> g.send() # currently illegal >> g.send(None) # None >> g.send((), {}) # currently illegal >> g.send(((), {})) # ((), {}) >> >> I suppose you could change it so passing exactly one argument did >> something different from passing zero or multiple arguments, but then >> you have a problem distinguishing when you actually want to use the >> value in the other end. >> > > Let us consider from a function point of view: > If suppose there was no support for *args and **kwds and you wrote: > def func(a): > do_something_with_a > and this was called with: > func((1,2)) > > and subsequently, *args, **kwds support was added to functions, will > anything related to 'func' need to change? I think I get your point. I don't think you are getting mine. Consider this example: def foo(): *(a, b) = yield f = foo() next(f) f.send(1, 2) # Ok def wrapper(g): r = next(g) while 1: args = (yield r) r = g.send(args) # BOOM! f = wrapper(foo()) next(f) f.send(1, 2) # BOOM! That wrapper works today, and does nothing (except messing up throw handling, but that is a different story). With your suggested change, such a wrapper would break unless it was changed to use *(*args, **kwargs) = (yield r) instead. IOW a backwards incompatible change. - Jacob From guido at python.org Tue Sep 13 18:51:03 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Sep 2011 09:51:03 -0700 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> Message-ID: On Mon, Sep 12, 2011 at 9:26 PM, Nick Coghlan wrote: > On Tue, Sep 13, 2011 at 1:22 PM, Ben Finney wrote: >> Massimo Di Pierro writes: >> >>> Just a question. What is the reason the win32 extensions are shipped >>> separately? >> >> This might better be asked on the developers's forum; this forum is for >> discussing changes to Python. > > 'Why is it so?' questions are perfectly reasonable fodder for > python-ideas (since they're usually a precursor to "let's do it > differently"). Part of the reason this list exists is so > misunderstandings of why Python is the way it is can be clarified here > without generating traffic on the main development list. (Of course, for some folks python-ideas is the only list they read. :-) Regarding the original question, it's definitely not a licensing issue. Mark originally intended for pywin32 to be merged back into core Python, and I recall looking at this option seriously. I believe it was not a single issue that held it back, but a combination of factors: sheer size, coding style, robustness of the code (you can definitely cause crashes by passing bad arguments to pywin32 code), the dependency on a specific platform and compiler, the fact that no other core developer knew the codebase, and the different development speeds of core Python and pywin32. There may also have been some qualms about who was to build the installer. In the end we compromised: the core Python distro does not include pywin32all, but the download instructions contain a recommendation. For example, the Python 2.2 download page (http://www.python.org/download/releases/2.2/) says: "Windows users may also be interested in Mark Hammond's _win32all_, a collection of Windows-specific extensions including COM support and Pythonwin, an IDE built using Windows components." I don't know why that was dropped from the main release page for 2.3 and later; the same recommendation is still found on http://www.python.org/download/windows/ which is linked from a tab labeled "Windows" on all release pages. I assume that the development speed on pywin32 has slowed down to the point where we could consider including it in the core Windows installer. But I'm not sure if the logistics of building and testing the installer easily support this. There's also the impression (possibly wrong) that pywin32/win32all is somewhat dated, and hasn't changed much since the glory days of Windows 98. But actual developers using Python on Windows should be the judge of that. -- --Guido van Rossum (python.org/~guido) From hetchkay at gmail.com Tue Sep 13 19:04:57 2011 From: hetchkay at gmail.com (H. Krishnan) Date: Tue, 13 Sep 2011 10:04:57 -0700 (PDT) Subject: [Python-ideas] return value of yield expressions In-Reply-To: <4E6F7D7B.6070103@improva.dk> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <24129449.2208.1315919030106.JavaMail.geo-discussion-forums@yqap15> <4E6F69A3.5030602@improva.dk> <17098410.132.1315926574320.JavaMail.geo-discussion-forums@yqmv39> <4E6F7D7B.6070103@improva.dk> Message-ID: <1395408.894.1315933497697.JavaMail.geo-discussion-forums@vbeo1> > > Consider this example: > > def foo(): > *(a, b) = yield > > f = foo() > next(f) > f.send(1, 2) # Ok > > def wrapper(g): > r = next(g) > while 1: > args = (yield r) > r = g.send(args) # BOOM! > > f = wrapper(foo()) > next(f) > f.send(1, 2) # BOOM! > > > That wrapper works today, and does nothing (except messing up throw > handling, but that is a different story). With your suggested change, > such a wrapper would break unless it was changed to use > > *(*args, **kwargs) = (yield r) > > instead. IOW a backwards incompatible change. > I still don't get it. f.send(1, 2) is not allowed now. The wrapper will still work for send() with single argument and so there is no backward compatibility issue. The wrapper itself could be updated to support multiple arguments to send if and when it is allowed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hetchkay at gmail.com Tue Sep 13 19:21:22 2011 From: hetchkay at gmail.com (H. Krishnan) Date: Tue, 13 Sep 2011 10:21:22 -0700 (PDT) Subject: [Python-ideas] return value of yield expressions In-Reply-To: References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> <1694141.1337.1315918859671.JavaMail.geo-discussion-forums@vbac9> Message-ID: <32726776.646.1315934482698.JavaMail.geo-discussion-forums@vbkm14> > > > (a1, a2, a3 = 3, *args, **kwds) = (yield ) > > What on earth is this syntax supposed to mean? Never mind that there's > a yield on the RHS; it is just a parenthesized expression so it could > could be any other function. What on earth do you expect to happen > with the syntax on the left, i.e. with the part > > > (a1, a2, a3 = 3, *args, **kwds) = ........whatever........ > > As I said "forgeting the backward compatibility issue". Suppose we use Jacob's syntax (for argument's sake) *(a1, a2, a3 = 3, *args, **kwds) = (yield expr) and with g.send(*a, **k) being called, a1, a2, a3, args, kwds can be inferred using the same semantics that is used to decipher a1, a2, a3, args, kwds in a call to the following function with *a and **k as arguments: def func(a1, a2, a3 = 3, *args, **kwds): ... That (yield expr) can be used in expressions does not (I feel) affect this (particularly if we go with Jacob's suggestion of this being a general unpacking option), just in the same way that "a, b, *args, c = " does not affect using tuples in expressions. But after reading your comment in another thread about those who don't know how python works should keep quiet, I guess it is best if I end this here :-) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Sep 13 19:26:34 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Sep 2011 10:26:34 -0700 Subject: [Python-ideas] return value of yield expressions In-Reply-To: <4E6F7D7B.6070103@improva.dk> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <24129449.2208.1315919030106.JavaMail.geo-discussion-forums@yqap15> <4E6F69A3.5030602@improva.dk> <17098410.132.1315926574320.JavaMail.geo-discussion-forums@yqmv39> <4E6F7D7B.6070103@improva.dk> Message-ID: On Tue, Sep 13, 2011 at 8:57 AM, Jacob Holm wrote: > On 2011-09-13 17:09, H. Krishnan wrote: >> >> >>> You need to be able to use "var1 = yield var2" in the generator >>> independent on what send is called with, or you can't wrap a generator >>> using another generator (different ways of doing that was a large part >>> of the PEP380 discussion). >>> >>> So what should "var1" be in each of the following cases? >>> >>> g.send() ? ? ? ? ? # currently illegal >>> g.send(None) ? ? ? # None >>> g.send((), {}) ? ? # currently illegal >>> g.send(((), {})) ? # ((), {}) >>> >>> I suppose you could change it so passing exactly one argument did >>> something different from passing zero or multiple arguments, but then >>> you have a problem distinguishing when you actually want to use the >>> value in the other end. >>> >> >> Let us consider from a function point of view: >> If suppose there was no support for *args and **kwds and you wrote: >> ? def func(a): >> ? ? ? do_something_with_a >> and this was called with: >> func((1,2)) >> >> and subsequently, *args, **kwds support was added to functions, will >> anything related to 'func' need to change? > > I think I get your point. ?I don't think you are getting mine. > > Consider this example: > > def foo(): > ? ?*(a, b) = yield What did you mean here? It is not valid syntax. It says SyntaxError: starred assignment target must be in a list or tuple > f = foo() > next(f) > f.send(1, 2) ?# Ok > > def wrapper(g): > ? ?r = next(g) > ? ?while 1: > ? ? ? ?args = (yield r) > ? ? ? ?r = g.send(args) # BOOM! > > f = wrapper(foo()) > next(f) > f.send(1, 2) # BOOM! > > > That wrapper works today, and does nothing (except messing up throw > handling, but that is a different story). ?With your suggested change, > such a wrapper would break unless it was changed to use > > ?*(*args, **kwargs) = (yield r) > > instead. ?IOW a backwards incompatible change. What on earth would **kwds in a list/tuple unpack mean? -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue Sep 13 19:34:35 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Sep 2011 10:34:35 -0700 Subject: [Python-ideas] return value of yield expressions In-Reply-To: <32726776.646.1315934482698.JavaMail.geo-discussion-forums@vbkm14> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> <1694141.1337.1315918859671.JavaMail.geo-discussion-forums@vbac9> <32726776.646.1315934482698.JavaMail.geo-discussion-forums@vbkm14> Message-ID: On Tue, Sep 13, 2011 at 10:21 AM, H. Krishnan wrote: >> > (a1, a2, a3 = 3, *args, **kwds) = (yield ) >> >> What on earth is this syntax supposed to mean? Never mind that there's >> a yield on the RHS; it is just a parenthesized expression so it could >> could be any other function. What on earth do you expect to happen >> with the syntax on the left, i.e. with the part > > >> >> (a1, a2, a3 = 3, *args, **kwds) = ........whatever........ > > As I said "forgeting the backward compatibility issue". It's not a backwards compatibility issue. Maybe I'm dense, but *I do not understand what it means in your proposal.* > Suppose we use > Jacob's syntax (for argument's sake) > *(a1, a2, a3 = 3, *args, **kwds) = (yield expr) > and with g.send(*a, **k) being called, a1, a2, a3, args, kwds can be > inferred using the same semantics that is used to decipher a1, a2, a3, args, > kwds in a call to the following function with *a and **k as arguments: > def func(a1, a2, a3 = 3, *args, **kwds): > ? ?... I still don't follow. Can you show a specific argument list to send, e.g. g.send(1, 2, 3, foo='a', bar='b') and then tell me what the values of a1, a2, a3, args and kwds will be? And, since X = Y should always be equivalent to tmp = Y X = tmp can you also tell me what value is supposed to be produced by (yield expr)? I.e. if I did this: tmp = (yield expr) *(a1, a2, a3 = 3, *args, **kwds) = tmp what would the value of tmp be? > That (yield expr) can be used in expressions does not (I feel) affect this > (particularly if we go with Jacob's suggestion of this being a general > unpacking option), just in the same way that "a, b, *args, c = " does > not affect using tuples in expressions. But after reading your comment in > another thread about those who don't know how python works should keep > quiet, I guess it is best if I end this here :-) I understand "a, b, *c = ". I do not understand three things in your example: *(....) = # What is the meaning of the prefix *? *(.... = .....) = # What does an = inside parentheses mean? *(.... = ......, **kwds) = # What does **kwds mean in this context? -- --Guido van Rossum (python.org/~guido) From cesare.di.mauro at gmail.com Tue Sep 13 19:54:16 2011 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Tue, 13 Sep 2011 19:54:16 +0200 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> Message-ID: 2011/9/13 Guido van Rossum > > I assume that the development speed on pywin32 has slowed down to the > point where we could consider including it in the core Windows > installer. But I'm not sure if the logistics of building and testing > the installer easily support this. > > There's also the impression (possibly wrong) that pywin32/win32all is > somewhat dated, and hasn't changed much since the glory days of > Windows 98. But actual developers using Python on Windows should be > the judge of that. > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > It seems that Windows 8 will bring a lot of news even for Win32/64 APIs. ;) Regards, Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Sep 13 19:56:23 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Sep 2011 10:56:23 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: On Mon, Sep 12, 2011 at 9:51 PM, Nick Coghlan wrote: > On Tue, Sep 13, 2011 at 2:09 PM, Guido van Rossum wrote: >> On Mon, Sep 12, 2011 at 8:39 PM, Nick Coghlan wrote: >>> On Tue, Sep 13, 2011 at 11:40 AM, Devin Jeanpierre >>> wrote: >>>> Can we not allow things like `a < b` to return non-boolean values, >>>> without altering the behaviour of existing Python types? >>> >>> We already do. As far as I am aware, the only holdouts are: >>> >>> x in a (coerces to bool) >>> not a (coerces to bool) >>> a and b (doesn't coerce as such, but roughly equivalent to "a if not a else b") >>> a or b (doesn't coerce as such, but roughly equivalent to "a if a else b") >>> >>> The first case can already be overridden (via __contains__) >> >> But does the C code wrapping it preserve the non-bool value? (IIRC >> originally many of these were constrained by the C implementation more >> than by any philosophical desire to keep bool pure.) >> >>> and I >>> don't believe there's any specific reason for retaining the coercion >>> to bool (aside from nobody writing and championing a patch that >>> eliminates the restriction). >> >> Oh I see, you are saying that there is no need *in principle*. Agreed >> for __contains__. (BTW there is also a need for the reverse operator. >> Maybe it could be called __in__?) > > The two issues seem somewhat orthogonal to me, but yes, the general > idea would be to make 'in' behave more like a rich comparison operator > rather than an explicitly boolean operation as it does now. > > It occurs to me that adding __in__ could also address a slight > performance oddity with 3.2 range objects: the __contains__ check in > 3.2 has an O(1) fast path for containment checks on actual Python > integers, but falls back to the O(n) sequential search for objects > that only implement __index__(). If __in__() was available, such > objects could conceivably map containment tests to checks against the > corresponding real Python integer (although doing that carelessly > would do weird things to other containers, such as identity-keyed > dictionaries. That would be a quality of implementation issue on > __in__ methods, though). I guess I might understand this paragraph if you pointed me to the code. :-( >> But I'm not sure I want to pay the price for this flexibility >> everywhere where those operators are used in their traditional >> meanings. Admittedly I haven't read PEP 335 thoroughly. > > I think that's a pretty good explanation for why PEP 335 hasn't been > pushed too hard - there's a lot of value in having not/and/or behave > more like control flow structures rather than ordinary operations. PEP > 335 does show that it's theoretically possible to make them behave > more like operators, but that doesn't make it a good idea. > > To be honest, I don't think anyone would cry too much if you decided > to explicitly reject it on the basis of continuing to allow control > flow optimisations for code involving not/and/or. While CPython > doesn't do it, I believe there *are* control flow transformations that > the current semantics permit that PEP 335 would disallow, such as > automatically applying De Morgan's Law (I don't actually have a use > case for doing that, I'm just mentioning it as a consequence of the > semantics change proposed by the PEP). I think I just mentioned one (turning 'if not' into a jump). Anyway, I'm glad to reject the PEP for the reason that I like the status quo just fine. (But relaxing __contains__ and adding __in__ as its reverse have my blessing.) Also, after reading the PEP from beginning to end, and downloading and skimming the patch (but failing to actually compile a patched version of Python 2.3), I think the offered API is too complicated to be of much use. Certainly the NumPy folks have repeatedly claimed that they are fine with the status quo. -- --Guido van Rossum (python.org/~guido) From g.brandl at gmx.net Tue Sep 13 19:57:59 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 13 Sep 2011 19:57:59 +0200 Subject: [Python-ideas] return value of yield expressions In-Reply-To: References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> <1694141.1337.1315918859671.JavaMail.geo-discussion-forums@vbac9> <32726776.646.1315934482698.JavaMail.geo-discussion-forums@vbkm14> Message-ID: Am 13.09.2011 19:34, schrieb Guido van Rossum: > On Tue, Sep 13, 2011 at 10:21 AM, H. Krishnan wrote: >>> > (a1, a2, a3 = 3, *args, **kwds) = (yield ) >>> >>> What on earth is this syntax supposed to mean? Never mind that there's >>> a yield on the RHS; it is just a parenthesized expression so it could >>> could be any other function. What on earth do you expect to happen >>> with the syntax on the left, i.e. with the part >> >> >>> >>> (a1, a2, a3 = 3, *args, **kwds) = ........whatever........ >> >> As I said "forgeting the backward compatibility issue". > > It's not a backwards compatibility issue. Maybe I'm dense, but *I do > not understand what it means in your proposal.* > >> Suppose we use >> Jacob's syntax (for argument's sake) > >> *(a1, a2, a3 = 3, *args, **kwds) = (yield expr) > >> and with g.send(*a, **k) being called, a1, a2, a3, args, kwds can be >> inferred using the same semantics that is used to decipher a1, a2, a3, args, >> kwds in a call to the following function with *a and **k as arguments: >> def func(a1, a2, a3 = 3, *args, **kwds): >> ... > > I still don't follow. Can you show a specific argument list to send, > e.g. g.send(1, 2, 3, foo='a', bar='b') and then tell me what the > values of a1, a2, a3, args and kwds will be? And, since > > X = Y > > should always be equivalent to > > tmp = Y > X = tmp > > can you also tell me what value is supposed to be produced by (yield > expr)? I.e. if I did this: > > tmp = (yield expr) > *(a1, a2, a3 = 3, *args, **kwds) = tmp > > what would the value of tmp be? > >> That (yield expr) can be used in expressions does not (I feel) affect this >> (particularly if we go with Jacob's suggestion of this being a general >> unpacking option), just in the same way that "a, b, *args, c = " does >> not affect using tuples in expressions. But after reading your comment in >> another thread about those who don't know how python works should keep >> quiet, I guess it is best if I end this here :-) > > I understand "a, b, *c = ". I do not understand three > things in your example: > > *(....) = # What is the meaning of the prefix *? > > *(.... = .....) = # What does an = inside parentheses mean? > > *(.... = ......, **kwds) = # What does **kwds mean in this context? > Apparently that syntax would be exclusive for "= yield", which then "unpacks" the arguments given to generator.send() as if the LHS was the argument list of a function. Georg From guido at python.org Tue Sep 13 20:07:28 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Sep 2011 11:07:28 -0700 Subject: [Python-ideas] return value of yield expressions In-Reply-To: References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> <1694141.1337.1315918859671.JavaMail.geo-discussion-forums@vbac9> <32726776.646.1315934482698.JavaMail.geo-discussion-forums@vbkm14> Message-ID: On Tue, Sep 13, 2011 at 10:57 AM, Georg Brandl wrote: > Am 13.09.2011 19:34, schrieb Guido van Rossum: >> On Tue, Sep 13, 2011 at 10:21 AM, H. Krishnan wrote: >>>> > (a1, a2, a3 = 3, *args, **kwds) = (yield ) >>>> >>>> What on earth is this syntax supposed to mean? Never mind that there's >>>> a yield on the RHS; it is just a parenthesized expression so it could >>>> could be any other function. What on earth do you expect to happen >>>> with the syntax on the left, i.e. with the part >>> >>> >>>> >>>> (a1, a2, a3 = 3, *args, **kwds) = ........whatever........ >>> >>> As I said "forgeting the backward compatibility issue". >> >> It's not a backwards compatibility issue. Maybe I'm dense, but *I do >> not understand what it means in your proposal.* >> >>> Suppose we use >>> Jacob's syntax (for argument's sake) >> >>> *(a1, a2, a3 = 3, *args, **kwds) = (yield expr) >> >>> and with g.send(*a, **k) being called, a1, a2, a3, args, kwds can be >>> inferred using the same semantics that is used to decipher a1, a2, a3, args, >>> kwds in a call to the following function with *a and **k as arguments: >>> def func(a1, a2, a3 = 3, *args, **kwds): >>> ? ?... >> >> I still don't follow. Can you show a specific argument list to send, >> e.g. g.send(1, 2, 3, foo='a', bar='b') and then tell me what the >> values of a1, a2, a3, args and kwds will be? And, since >> >> X = Y >> >> should always be equivalent to >> >> tmp = Y >> X = tmp >> >> can you also tell me what value is supposed to be produced by (yield >> expr)? I.e. if I did this: >> >> tmp = (yield expr) >> *(a1, a2, a3 = 3, *args, **kwds) = tmp >> >> what would the value of tmp be? >> >>> That (yield expr) can be used in expressions does not (I feel) affect this >>> (particularly if we go with Jacob's suggestion of this being a general >>> unpacking option), just in the same way that "a, b, *args, c = " does >>> not affect using tuples in expressions. But after reading your comment in >>> another thread about those who don't know how python works should keep >>> quiet, I guess it is best if I end this here :-) >> >> I understand "a, b, *c = ". I do not understand three >> things in your example: >> >> *(....) = ?# What is the meaning of the prefix *? >> >> *(.... = .....) = ?# What does an = inside parentheses mean? >> >> *(.... = ......, **kwds) = ?# What does **kwds mean in this context? >> > > Apparently that syntax would be exclusive for "= yield", which then "unpacks" > the arguments given to generator.send() as if the LHS was the argument list > of a function. That sounds like impossible to implement. If there should be special-case treatment it should use some other form of syntax (e.g. "yield as "). But I really don't see why this is so important to change so much of the syntax of the language. -- --Guido van Rossum (python.org/~guido) From jh at improva.dk Tue Sep 13 20:35:27 2011 From: jh at improva.dk (Jacob Holm) Date: Tue, 13 Sep 2011 20:35:27 +0200 Subject: [Python-ideas] return value of yield expressions In-Reply-To: References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> <1694141.1337.1315918859671.JavaMail.geo-discussion-forums@vbac9> <32726776.646.1315934482698.JavaMail.geo-discussion-forums@vbkm14> Message-ID: <4E6FA26F.6060108@improva.dk> On 2011-09-13 20:07, Guido van Rossum wrote: > On Tue, Sep 13, 2011 at 10:57 AM, Georg Brandl wrote: >> Am 13.09.2011 19:34, schrieb Guido van Rossum: >>> On Tue, Sep 13, 2011 at 10:21 AM, H. Krishnan wrote: >>> >>>> That (yield expr) can be used in expressions does not (I feel) affect this >>>> (particularly if we go with Jacob's suggestion of this being a general >>>> unpacking option), just in the same way that "a, b, *args, c = " does >>>> not affect using tuples in expressions. But after reading your comment in >>>> another thread about those who don't know how python works should keep >>>> quiet, I guess it is best if I end this here :-) >>> >>> I understand "a, b, *c = ". I do not understand three >>> things in your example: >>> >>> *(....) = # What is the meaning of the prefix *? >>> >>> *(.... = .....) = # What does an = inside parentheses mean? >>> >>> *(.... = ......, **kwds) = # What does **kwds mean in this context? >>> >> >> Apparently that syntax would be exclusive for "= yield", which then "unpacks" >> the arguments given to generator.send() as if the LHS was the argument list >> of a function. > > That sounds like impossible to implement. If there should be > special-case treatment it should use some other form of syntax (e.g. > "yield as "). But I really don't see why this is so > important to change so much of the syntax of the language. > The '*(argument list) = ' syntax was my attempt at fixing some of the problems in the original proposal. I.e. to remove the restriction that it only should work with "= yield", and distinguish the "function argument unpacking" from the regular tuple unpacking. My idea was to allow *(argument list) = (args, kwds) where the LHS follows all the rules of a function argument list (the part of a function definition between the name and the colon), and the RHS may be any expression that returns a 2-tuple consisting of a sequence and a dictionary. The intended semantics was to compute the arguments that a function with the given argument list would see when called with (*args, **kwds), and make the necessary assignments in the local scope. Any way I can think of that works today violates DRY. A simple example: *(a, b, c=42, *args, **kwds) = ((1,), {'b':2, 'd':3}) assigns a=1, b=2, c=42, args=(), kwds={'d':3} And my best hack so far to get the same effect today is: args, kwds = ((1,), {'b':2, 'd':3}) a, b, c, args, kwds = \ (lambda a, b, c=42, *args, **kwds:a, b, c, args, kwds)(*args, **kwds) I am not wed to the particular suggested syntax, but I would like to see the functionality available *somehow* in a way where you don't have to repeat the sequence of variable names. And I don't think that is possible without *some* syntax change. - Jacob From guido at python.org Tue Sep 13 20:53:14 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Sep 2011 11:53:14 -0700 Subject: [Python-ideas] return value of yield expressions In-Reply-To: <4E6FA26F.6060108@improva.dk> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> <1694141.1337.1315918859671.JavaMail.geo-discussion-forums@vbac9> <32726776.646.1315934482698.JavaMail.geo-discussion-forums@vbkm14> <4E6FA26F.6060108@improva.dk> Message-ID: On Tue, Sep 13, 2011 at 11:35 AM, Jacob Holm wrote: > The '*(argument list) = ' syntax was my attempt at fixing > some of the problems in the original proposal. ?I.e. to remove the > restriction that it only should work with "= yield", and distinguish the > "function argument unpacking" from the regular tuple unpacking. > > My idea was to allow > > ?*(argument list) = (args, kwds) > > where the LHS follows all the rules of a function argument list (the > part of a function definition between the name and the colon), and > the RHS may be any expression that returns a 2-tuple consisting of a > sequence and a dictionary. > > The intended semantics was to compute the arguments that a function with > the given argument list would see when called with (*args, **kwds), and > make the necessary assignments in the local scope. ?Any way I can think > of that works today violates DRY. > > > A simple example: > > *(a, b, c=42, *args, **kwds) = ((1,), {'b':2, 'd':3}) > > assigns a=1, b=2, c=42, args=(), kwds={'d':3} Thanks, I finally understand. Part of the reason why it was unclear to me was that I was reading the form inside *(...) as having the same priority rules as a regular assignment, where "," binds tighter than "="; whereas in a function argument list "=" binds tighter than ",". > And my best hack so far to get the same effect today is: > > args, kwds = ((1,), {'b':2, 'd':3}) > a, b, c, args, kwds = \ > ?(lambda a, b, c=42, *args, **kwds:a, b, c, args, kwds)(*args, **kwds) > > > I am not wed to the particular suggested syntax, but I would like to see > the functionality available *somehow* in a way where you don't have to > repeat the sequence of variable names. ?And I don't think that is > possible without *some* syntax change. I agree that you can't do this today. But so what? I am not so convinced that DRY is always the most important rule to invoke. Apart from the extremely esoteric example of wanting to call g.send() with a mix of positional and keyword arguments that are somehow interpreted as a completely general function parameter list by the receiving yield expression, what is the use case? -- --Guido van Rossum (python.org/~guido) From jh at improva.dk Tue Sep 13 21:17:57 2011 From: jh at improva.dk (Jacob Holm) Date: Tue, 13 Sep 2011 21:17:57 +0200 Subject: [Python-ideas] return value of yield expressions In-Reply-To: References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> <1694141.1337.1315918859671.JavaMail.geo-discussion-forums@vbac9> <32726776.646.1315934482698.JavaMail.geo-discussion-forums@vbkm14> <4E6FA26F.6060108@improva.dk> Message-ID: <4E6FAC65.6030206@improva.dk> On 2011-09-13 20:53, Guido van Rossum wrote: > On Tue, Sep 13, 2011 at 11:35 AM, Jacob Holm wrote: >> A simple example: >> >> *(a, b, c=42, *args, **kwds) = ((1,), {'b':2, 'd':3}) >> >> assigns a=1, b=2, c=42, args=(), kwds={'d':3} > > Thanks, I finally understand. Part of the reason why it was unclear to > me was that I was reading the form inside *(...) as having the same > priority rules as a regular assignment, where "," binds tighter than > "="; whereas in a function argument list "=" binds tighter than ",". > I knew it had to be something like that. :) >> And my best hack so far to get the same effect today is: >> >> args, kwds = ((1,), {'b':2, 'd':3}) >> a, b, c, args, kwds = \ >> (lambda a, b, c=42, *args, **kwds:a, b, c, args, kwds)(*args, **kwds) >> >> >> I am not wed to the particular suggested syntax, but I would like to see >> the functionality available *somehow* in a way where you don't have to >> repeat the sequence of variable names. And I don't think that is >> possible without *some* syntax change. > > I agree that you can't do this today. > > But so what? > > I am not so convinced that DRY is always the most important rule to invoke. > I am sure it isn't. I would also have used "readability counts", but your earlier comments have shown that that argument is probably not carrying much weight either in this case. :) > Apart from the extremely esoteric example of wanting to call g.send() > with a mix of positional and keyword arguments that are somehow > interpreted as a completely general function parameter list by the > receiving yield expression, what is the use case? > See that's my main problem. I know I have run into this need several times in the past, but for the life of me I can't remember the details. I remember it had something to do with decorators, but nothing beyond that. At the moment I just like the idea because it seems like this is a case where something should be really easy, but isn't. I'll let it rest until and unless I remember what the actual use was. - Jacob From ncoghlan at gmail.com Wed Sep 14 00:26:14 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 14 Sep 2011 08:26:14 +1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: On Wed, Sep 14, 2011 at 3:56 AM, Guido van Rossum wrote: > On Mon, Sep 12, 2011 at 9:51 PM, Nick Coghlan wrote: >> The two issues seem somewhat orthogonal to me, but yes, the general >> idea would be to make 'in' behave more like a rich comparison operator >> rather than an explicitly boolean operation as it does now. >> >> It occurs to me that adding __in__ could also address a slight >> performance oddity with 3.2 range objects: the __contains__ check in >> 3.2 has an O(1) fast path for containment checks on actual Python >> integers, but falls back to the O(n) sequential search for objects >> that only implement __index__(). If __in__() was available, such >> objects could conceivably map containment tests to checks against the >> corresponding real Python integer (although doing that carelessly >> would do weird things to other containers, such as identity-keyed >> dictionaries. That would be a quality of implementation issue on >> __in__ methods, though). > > I guess I might understand this paragraph if you pointed me to the code. :-( http://hg.python.org/cpython/file/default/Objects/rangeobject.c#l603 (scroll up a bit from where that link lands for the definition of the O(1) check) For the bit about breaking identity-keyed dictionaries, consider a hypothetical naive containment implementation like this: def __in__(self, container): return int(self) in container That code involves an implicit assumption that *all* containers are equivalence based, and that simply isn't true - it is sometimes useful to create a container that relies on object identity rather than value (e.g. to store additional metadata about arbitrary objects, even mutable ones). So a more correct implementation would have to look something like: def __in__(self, container): if isinstance(container, range): # We know range containment is equivalence based return int(self) in container return NotImplemented One additional complication that Alex Gaynor pointed out is that there are potentially *two* distinct operations that would need to be covered if "in" were to become a rich comparison operation - 'in' and 'not in'. Currently "not x in y" and "x not in y" generate identical bytecode, and the control flow optimisations that apply to the former can also be applied to the latter. Making 'in' a rich comparison operation would thus require choosing one of the following behaviours: - making 'not' itself avoid coercing to a boolean value (which you've already said you don't want to do) - retaining the equivalence between "x not in y" and "not x in y" and simply accepting that "x in y" is a rich comparison while "x not in y" is not - adding two additional special methods to cover the 'not in' case, defining a method precedence that covers the various combinations of methods on the operands for both 'in' and 'not in' and disentangling all the parts of the code generator that assume the equivalence of "x not in y" and "not x in y" None of those options sound particularly appealing. A rich 'not' would probably be the cleanest solution, but you've already given valid reasons for not wanting to do that. >> To be honest, I don't think anyone would cry too much if you decided >> to explicitly reject it on the basis of continuing to allow control >> flow optimisations for code involving not/and/or. While CPython >> doesn't do it, I believe there *are* control flow transformations that >> the current semantics permit that PEP 335 would disallow, such as >> automatically applying De Morgan's Law (I don't actually have a use >> case for doing that, I'm just mentioning it as a consequence of the >> semantics change proposed by the PEP). > > I think I just mentioned one (turning 'if not' into a jump). Anyway, > I'm glad to reject the PEP for the reason that I like the status quo > just fine. (But relaxing __contains__ and adding __in__ as its reverse > have my blessing.) Also, after reading the PEP from beginning to end, > and downloading and skimming the patch (but failing to actually > compile a patched version of Python 2.3), I think the offered API is > too complicated to be of much use. Certainly the NumPy folks have > repeatedly claimed that they are fine with the status quo. OK, I'll add a rejection notice to PEP 335 with a link to this thread. Given the point above regarding the problems with "x not in y" vs "not x in y", do you want me to include something saying that rich containment checks are also rejected? Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From aquavitae69 at gmail.com Wed Sep 14 00:26:49 2011 From: aquavitae69 at gmail.com (David Townshend) Date: Wed, 14 Sep 2011 00:26:49 +0200 Subject: [Python-ideas] return value of yield expressions In-Reply-To: References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> <1694141.1337.1315918859671.JavaMail.geo-discussion-forums@vbac9> <32726776.646.1315934482698.JavaMail.geo-discussion-forums@vbkm14> <4E6FA26F.6060108@improva.dk> Message-ID: On Sep 13, 2011 8:53 PM, "Guido van Rossum" wrote: > > On Tue, Sep 13, 2011 at 11:35 AM, Jacob Holm wrote: > > The '*(argument list) = ' syntax was my attempt at fixing > > some of the problems in the original proposal. I.e. to remove the > > restriction that it only should work with "= yield", and distinguish the > > "function argument unpacking" from the regular tuple unpacking. > > > > My idea was to allow > > > > *(argument list) = (args, kwds) > > > > where the LHS follows all the rules of a function argument list (the > > part of a function definition between the name and the colon), and > > the RHS may be any expression that returns a 2-tuple consisting of a > > sequence and a dictionary. > > > > The intended semantics was to compute the arguments that a function with > > the given argument list would see when called with (*args, **kwds), and > > make the necessary assignments in the local scope. Any way I can think > > of that works today violates DRY. > > > > > > A simple example: > > > > *(a, b, c=42, *args, **kwds) = ((1,), {'b':2, 'd':3}) > > > > assigns a=1, b=2, c=42, args=(), kwds={'d':3} > > Thanks, I finally understand. Part of the reason why it was unclear to > me was that I was reading the form inside *(...) as having the same > priority rules as a regular assignment, where "," binds tighter than > "="; whereas in a function argument list "=" binds tighter than ",". > > > And my best hack so far to get the same effect today is: > > > > args, kwds = ((1,), {'b':2, 'd':3}) > > a, b, c, args, kwds = \ > > (lambda a, b, c=42, *args, **kwds:a, b, c, args, kwds)(*args, **kwds) > > > > > > I am not wed to the particular suggested syntax, but I would like to see > > the functionality available *somehow* in a way where you don't have to > > repeat the sequence of variable names. And I don't think that is > > possible without *some* syntax change. > > I agree that you can't do this today. I've only skimmed this so I might be way off track, but isn't this capability (although not the syntax) already covered to an extent by PEP 362? > > But so what? > > I am not so convinced that DRY is always the most important rule to invoke. > > Apart from the extremely esoteric example of wanting to call g.send() > with a mix of positional and keyword arguments that are somehow > interpreted as a completely general function parameter list by the > receiving yield expression, what is the use case? > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Sep 14 00:49:01 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 14 Sep 2011 08:49:01 +1000 Subject: [Python-ideas] Object interface to path names In-Reply-To: References: Message-ID: On Wed, Sep 14, 2011 at 1:33 AM, David Townshend wrote: >> Having an object-oriented / polymorphic file API means that one can >> provide file objects that aren't backed by the filesystem, but look >> the same. This is cool in that you can do things like treat zip files >> as directory, or mock out the filesystem for a unit test, without >> worrying about monkeypatching builtins or using a nonstandard wrapper >> API etc. > > Great use case! ?And using factories to create create the objects would make > this especially powerful. The thing is, the "smart path" abstraction level isn't adequate for that task - you need to do more to ensure various primitives (like listing directory contents) are handled properly, along the lines of what PyFileSystem provides (https://code.google.com/p/pyfilesystem/). As far as a smart path object goes, the previous major effort on this front was PEP 355, which focused on the API offered by Jason Orendorff's path module. While the PEP was ultimately rejected due to the "one class to rule them all" nature of that particular interface, it's still an excellent reference on why improving the standard library's filesystem abstraction is an area worth exploring further. FWIW, PyFileSystem is the only package I've seen that I think comes close to getting the abstraction level right (I've never actually needed to use it for anything though). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 14 00:58:06 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 14 Sep 2011 08:58:06 +1000 Subject: [Python-ideas] return value of yield expressions In-Reply-To: References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> <1694141.1337.1315918859671.JavaMail.geo-discussion-forums@vbac9> <32726776.646.1315934482698.JavaMail.geo-discussion-forums@vbkm14> <4E6FA26F.6060108@improva.dk> Message-ID: On Wed, Sep 14, 2011 at 8:26 AM, David Townshend wrote: > > On Sep 13, 2011 8:53 PM, "Guido van Rossum" wrote: >> >> On Tue, Sep 13, 2011 at 11:35 AM, Jacob Holm wrote: >> > I am not wed to the particular suggested syntax, but I would like to see >> > the functionality available *somehow* in a way where you don't have to >> > repeat the sequence of variable names. ?And I don't think that is >> > possible without *some* syntax change. >> >> I agree that you can't do this today. > > I've only skimmed this so I might be way off track, but isn't this > capability (although not the syntax) already covered to an extent by PEP > 362? Yep, I was going to mention PEP 362's Signature.bind() as well - the function signature objects in that PEP are closely tied in to the issues described in that thread. Sure, you can't easily unpack them into local variables, but you could fairly easily use them to pass formatted data into a coroutine. As far as non-coroutine use cases go, the main benefit (explicitly mentioned in PEP 362) lies in generating signature details for wrapper functions that go beyond the current typical "*args, **kwds". A tool like functools.partial(), for instance, could take the signature object for the function being wrapped, remove the values already supplied, and then set the result as the signature for the created object. Another Signature.bind() in particular is useful for is in prevalidating arguments for delayed calls - you can check that the parameters at least match the function being called immediately, while deferring the actual invocation of the function until later. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From anacrolix at gmail.com Wed Sep 14 01:22:11 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Wed, 14 Sep 2011 09:22:11 +1000 Subject: [Python-ideas] Object interface to path names In-Reply-To: References: Message-ID: That library, pyfilesystem has some serious firepower. On Sep 14, 2011 1:33 AM, "David Townshend" wrote: > > But none of these seem to allow asynchronous calls, which make a huge difference when dealing with a large structure. Nothing I've found really does what I need and I'm trying to keep my dependency list short so I'll end up writing something myself anyway, but my real question was whether this is something that could usefully be included in the stdlib. > >> Having an object-oriented / polymorphic file API means that one can >> provide file objects that aren't backed by the filesystem, but look >> the same. This is cool in that you can do things like treat zip files >> as directory, or mock out the filesystem for a unit test, without >> worrying about monkeypatching builtins or using a nonstandard wrapper >> API etc. > > > Great use case! And using factories to create create the objects would make this especially powerful. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Sep 14 01:35:08 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Sep 2011 11:35:08 +1200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: <4E6FE8AC.3080000@canterbury.ac.nz> Guido van Rossum wrote: > It would mean that the compiler no longer has the > option to translate "if not" into a reversal of the control flow; This only seems to happen when the expression is used directly as the subject of an if-test. For the use cases I had in mind when I wrote PEP 335, this wouldn't happen -- you're building a parse tree in order to perform some other processing on it, not to immediately use it in an if-branch. So I think it would be fine for the compiler to reserve the right to apply the laws of boolean algebra to simplify an expression that appears in a boolean context. -- Greg From guido at python.org Wed Sep 14 01:55:42 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Sep 2011 16:55:42 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: On Tue, Sep 13, 2011 at 3:26 PM, Nick Coghlan wrote: > On Wed, Sep 14, 2011 at 3:56 AM, Guido van Rossum wrote: >> On Mon, Sep 12, 2011 at 9:51 PM, Nick Coghlan wrote: >>> The two issues seem somewhat orthogonal to me, but yes, the general >>> idea would be to make 'in' behave more like a rich comparison operator >>> rather than an explicitly boolean operation as it does now. >>> >>> It occurs to me that adding __in__ could also address a slight >>> performance oddity with 3.2 range objects: the __contains__ check in >>> 3.2 has an O(1) fast path for containment checks on actual Python >>> integers, but falls back to the O(n) sequential search for objects >>> that only implement __index__(). If __in__() was available, such >>> objects could conceivably map containment tests to checks against the >>> corresponding real Python integer (although doing that carelessly >>> would do weird things to other containers, such as identity-keyed >>> dictionaries. That would be a quality of implementation issue on >>> __in__ methods, though). >> >> I guess I might understand this paragraph if you pointed me to the code. :-( > > http://hg.python.org/cpython/file/default/Objects/rangeobject.c#l603 > (scroll up a bit from where that link lands for the definition of the > O(1) check) Gotcha. I missed that this was just about range objects. :-) > For the bit about breaking identity-keyed dictionaries, consider a > hypothetical naive containment implementation like this: > > ?def __in__(self, container): > ? ?return int(self) in container > > That code involves an implicit assumption that *all* containers are > equivalence based, and that simply isn't true - it is sometimes useful > to create a container that relies on object identity rather than value > (e.g. to store additional metadata about arbitrary objects, even > mutable ones). So a more correct implementation would have to look > something like: > > ?def __in__(self, container): > ? ?if isinstance(container, range): # We know range containment is equivalence based > ? ? ?return int(self) in container > ? ?return NotImplemented Hm, this reminds me of the thorny issue about 'is' implying '==' for certain container checks... But really, there are many other ways of making similar mistakes with other operators, and IMO a good __in__ method should always check if the RHS type is a known and supported type, and otherwise return NotImplemented like a good binary operator should. > One additional complication that Alex Gaynor pointed out is that there > are potentially *two* distinct operations that would need to be > covered if "in" were to become a rich comparison operation - 'in' and > 'not in'. Currently "not x in y" and "x not in y" generate identical > bytecode, and the control flow optimisations that apply to the former > can also be applied to the latter. Making 'in' a rich comparison > operation would thus require choosing one of the following behaviours: > ?- making 'not' itself avoid coercing to a boolean value (which > you've already said you don't want to do) Right. > ?- retaining the equivalence between "x not in y" and "not x in y" > and simply accepting that "x in y" is a rich comparison while "x not > in y" is not I think this one is fine, actually. > ?- adding two additional special methods to cover the 'not in' case, > defining a method precedence that covers the various combinations of > methods on the operands for both 'in' and 'not in' and disentangling > all the parts of the code generator that assume the equivalence of "x > not in y" and "not x in y" Basically this would elevate 'not in' to a separate operator, just like '==' and '!=' are separate operators. Now, *if* we were to adopt PEP 335, this would be a reasonable approach. But since we're not, I think it's fine. If NumPy ever implements 'in' as returning an array, NumPy users might have to be warned that 'not in' doesn't work that way, and 'not (x in y)' doesn't work either. So they'll have to write something like '1 - (x in y)', assuming 'x in y' returns an array of bools. Big deal. > None of those options sound particularly appealing. A rich 'not' would > probably be the cleanest solution, but you've already given valid > reasons for not wanting to do that. And I stick to them. >>> To be honest, I don't think anyone would cry too much if you decided >>> to explicitly reject it on the basis of continuing to allow control >>> flow optimisations for code involving not/and/or. While CPython >>> doesn't do it, I believe there *are* control flow transformations that >>> the current semantics permit that PEP 335 would disallow, such as >>> automatically applying De Morgan's Law (I don't actually have a use >>> case for doing that, I'm just mentioning it as a consequence of the >>> semantics change proposed by the PEP). >> >> I think I just mentioned one (turning 'if not' into a jump). Anyway, >> I'm glad to reject the PEP for the reason that I like the status quo >> just fine. (But relaxing __contains__ and adding __in__ as its reverse >> have my blessing.) Also, after reading the PEP from beginning to end, >> and downloading and skimming the patch (but failing to actually >> compile a patched version of Python 2.3), I think the offered API is >> too complicated to be of much use. Certainly the NumPy folks have >> repeatedly claimed that they are fine with the status quo. > > OK, I'll add a rejection notice to PEP 335 with a link to this thread. > Given the point above regarding the problems with "x not in y" vs "not > x in y", do you want me to include something saying that rich > containment checks are also rejected? Let's mull that one over for a bit longer. It's not mentioned in PEP 335, is it? -- --Guido van Rossum (python.org/~guido) From greg.ewing at canterbury.ac.nz Wed Sep 14 02:08:04 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Sep 2011 12:08:04 +1200 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> Message-ID: <4E6FF064.5020109@canterbury.ac.nz> Brian Curtin wrote: > I may be missing something due to the fragmented web presence, but > documentation could be a challenge, although I remember some form of > documentation being available on the ActiveState. The ActiveState docs seem to be just html versions of the docs distributed with the package, from what I've seen. IMO, the docs would need considerable work to bring them up to stdlib standards -- they're rather sketchy and incomplete in many areas. The package itself could also do with an overhaul. A lot of stuff seems to be wrapped twice in different ways, once in win32api and again in win32gui. -- Greg From greg.ewing at canterbury.ac.nz Wed Sep 14 02:09:48 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Sep 2011 12:09:48 +1200 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> Message-ID: <4E6FF0CC.7090901@canterbury.ac.nz> Brian Curtin wrote: > I should finish this thought: I think "windows" should be a top-level > package, and the sub-packages like "win32com" or "win32api" should > drop the "win32" prefix. Sounds good to me. I've always thought they should have been under a top-level package from the beginning. -- Greg From greg.ewing at canterbury.ac.nz Wed Sep 14 02:46:41 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Sep 2011 12:46:41 +1200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: <4E6FF971.7070504@canterbury.ac.nz> Guido van Rossum wrote: > Control flow just *is* special. FWIW It is not > bool values that are being treated special; it is certain bool > operators (not, and, or). I see nothing wrong with that; you can think > of them as an extension of the repertoire of 'if', 'while' etc. Seems to me they're something of a mixture of computation and control flow. The computation consists of calling the __nonzero__ methods of the operands to find out whether they should be considered true or false. PEP 335 doesn't change that, it just shifts the emphasis slightly. It provides hooks allowing you to do more computation before deciding on the control flow. > De Morgan's law (and similar transformation) seem to me out of scope. > The would violate gut feelings about the "natural" execution order of > Python expressions. I don't see how. If e.g. you transform if not (a and b): into if not a or not b: then the evaluation order doesn't change -- you still evaluate a first, and then evaluate b if needed. It's just an extension of the existing rewriting of 'not' deeper into the expression. -- Greg From ned at nedbatchelder.com Wed Sep 14 02:59:07 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Tue, 13 Sep 2011 20:59:07 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: <4E6FFC5B.8010704@nedbatchelder.com> On 9/12/2011 4:20 PM, Lukas Lueg wrote: > Summing all up, I really think that we should break the dominance of > bool() and take a look at how we can implement boolean contexts > without relying on boolean values all the time. I was surprised to find that any() returns True or False, regardless of the values it encounters. Conceptually, these two are the same (where S is a sequence S0, S1, S2, ..., Sn): any(S) S0 or S1 or S2 or ... or Sn They are equivalent except that if Sx is the first true-ish value, the first will return True while the second returns Sx. Why shouldn't any() also return Sx? --Ned. From greg.ewing at canterbury.ac.nz Wed Sep 14 03:08:34 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Sep 2011 13:08:34 +1200 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> Message-ID: <4E6FFE92.8080307@canterbury.ac.nz> Guido van Rossum wrote: > There's also the impression (possibly wrong) that pywin32/win32all is > somewhat dated, and hasn't changed much since the glory days of > Windows 98. I think that's mainly because the thing it's wrapping -- the win32 api -- is somewhat dated, and hasn't changed much since the glory days of Windows 98. :-) By today's standards, the traditional win32 gui api is hopelessly cruddy. Unfortunately, the only efforts on Microsoft's part to replace it with something better seem to be taking place in the .NET world. As things stand, this places them out of the easy reach of Python. Adding something to the stdlib enabling Python to call .NET libraries would open up considerable new opportunities. Maybe it could be based on this: http://pythonnet.sourceforge.net/ -- Greg From bruce at leapyear.org Wed Sep 14 03:08:27 2011 From: bruce at leapyear.org (Bruce Leban) Date: Tue, 13 Sep 2011 18:08:27 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <1315880035.30361.3.camel@Gutsy> Message-ID: On Tue, Sep 13, 2011 at 8:05 AM, Lukas Lueg wrote: > 2011/9/13 Ron Adam : > > On Mon, 2011-09-12 at 21:40 -0400, Devin Jeanpierre wrote: > >> Can we not allow things like `a < b` to return non-boolean values, > >> without altering the behaviour of existing Python types? > > > > Would that return 'a' or 'b', or something else? > > That depends on the object at hand. 'a < b' could return 'b - a' for > set-like objects. Read 'a < b' as the answer to the question 'how much > bigger is b than a?'. > Then what would a <= b or a == b or a != b return? This idea of returning 'how far apart are these objects' doesn't generalize for integers much less more complex types. You can't design a new feature by giving one example ... and no use cases. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.curtin at gmail.com Wed Sep 14 03:22:16 2011 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 13 Sep 2011 20:22:16 -0500 Subject: [Python-ideas] win32 extensions In-Reply-To: <4E6FFE92.8080307@canterbury.ac.nz> References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> Message-ID: On Tue, Sep 13, 2011 at 20:08, Greg Ewing wrote: > Guido van Rossum wrote: > >> There's also the impression (possibly wrong) that pywin32/win32all is >> somewhat dated, and hasn't changed much since the glory days of >> Windows 98. > > I think that's mainly because the thing it's wrapping -- > the win32 api -- is somewhat dated, and hasn't changed much > since the glory days of Windows 98. :-) > > By today's standards, the traditional win32 gui api is > hopelessly cruddy. Unfortunately, the only efforts on > Microsoft's part to replace it with something better > seem to be taking place in the .NET world. Could we skip inclusion the win32gui stuff? We'd be including something which has been surpassed by WinForms, which itself has (mostly?) been surpassed by WPF, and soon enough we'll have Metro...on top of the fact that we'd be introducing a second way to do GUI work alongside Tkinter. From anacrolix at gmail.com Wed Sep 14 04:28:09 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Wed, 14 Sep 2011 12:28:09 +1000 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> Message-ID: Do we really need to be encouraging use of Windows APIs in Python? Given the greater user base of Windows, is there a danger this could encourage the littering of Windows-only "workarounds" in many projects? "Well Python doesn't ship with a usable GUI toolkit (hi tk), but it does ship with Win32"->"Everyone uses Windows anyway, lets make a Win32 interface". Just a thought. On Wed, Sep 14, 2011 at 11:22 AM, Brian Curtin wrote: > On Tue, Sep 13, 2011 at 20:08, Greg Ewing wrote: >> Guido van Rossum wrote: >> >>> There's also the impression (possibly wrong) that pywin32/win32all is >>> somewhat dated, and hasn't changed much since the glory days of >>> Windows 98. >> >> I think that's mainly because the thing it's wrapping -- >> the win32 api -- is somewhat dated, and hasn't changed much >> since the glory days of Windows 98. :-) >> >> By today's standards, the traditional win32 gui api is >> hopelessly cruddy. Unfortunately, the only efforts on >> Microsoft's part to replace it with something better >> seem to be taking place in the .NET world. > > Could we skip inclusion the win32gui stuff? We'd be including > something which has been surpassed by WinForms, which itself has > (mostly?) been surpassed by WPF, and soon enough we'll have Metro...on > top of the fact that we'd be introducing a second way to do GUI work > alongside Tkinter. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From python at mrabarnett.plus.com Wed Sep 14 04:39:47 2011 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 14 Sep 2011 03:39:47 +0100 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E6FFC5B.8010704@nedbatchelder.com> References: <4E6FFC5B.8010704@nedbatchelder.com> Message-ID: <4E7013F3.5080907@mrabarnett.plus.com> On 14/09/2011 01:59, Ned Batchelder wrote: > On 9/12/2011 4:20 PM, Lukas Lueg wrote: >> Summing all up, I really think that we should break the dominance of >> bool() and take a look at how we can implement boolean contexts >> without relying on boolean values all the time. > I was surprised to find that any() returns True or False, regardless of > the values it encounters. Conceptually, these two are the same (where S > is a sequence S0, S1, S2, ..., Sn): > > any(S) > > S0 or S1 or S2 or ... or Sn > > They are equivalent except that if Sx is the first true-ish value, the > first will return True while the second returns Sx. Why shouldn't any() > also return Sx? > If none are true-ish, should it return the final (false-ish) value? What should any([]) return? From ron3200 at gmail.com Tue Sep 13 19:23:26 2011 From: ron3200 at gmail.com (Ron Adam) Date: Tue, 13 Sep 2011 12:23:26 -0500 Subject: [Python-ideas] return value of yield expressions In-Reply-To: <4E6F3838.5030701@improva.dk> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> Message-ID: <1315934606.814.116.camel@Gutsy> On Tue, 2011-09-13 at 13:02 +0200, Jacob Holm wrote: > Your other proposal is really independent of generators I think. I > too > would like to see a way to do "function argument unpacking". > > args, kwds = (yield ret) # any expression really > (a1, a2, a3, *args), kwds = (lambda a1,a2,a3=default,*args, **kwds: > (a1,a2,a3)+args, kwds > )(*args, **kwds) > > is about the shortest way I can come up with that works today, and > that > is way too much repetition for my taste. If locals() were writable (a > change that is unlikely to happen btw), this could be reduced to: > > args, kwds = (yield ret) # any expression really > locals().update( > (lambda a1,a2,a3=default,*args,**kwds:locals())(*args, **kwds) > ) > > which would avoid some of the duplication but is really not that more > readable. Instead of extending tuple unpacking, I think I'd prefer to go the other way and improve function data sharing. If we could get the functions arguments when a function is done instead of getting the return value. (The modified function arguments object is the return value in this case.) fargs = &foo(a, b, c, d=4) Then someplace else use that function argument object directly. bar.__with_args__(fargs) # fargs becomes bar's argument object It could avoid unpacking and repacking data when it's passed between compatible functions. Cheers, Ron From brian.curtin at gmail.com Wed Sep 14 05:00:23 2011 From: brian.curtin at gmail.com (Brian Curtin) Date: Tue, 13 Sep 2011 22:00:23 -0500 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> Message-ID: On Tue, Sep 13, 2011 at 21:28, Matt Joiner wrote: > Do we really need to be encouraging use of Windows APIs in Python? I don't think we're encouraging anything, nor do I think we need to. People have already been doing so for quite some time - it's a fact of life. If I want to write a tool which uses Windows' ReadDirectoryChangesW API, how do I do that today? I either write a C extension (which I've done), or I use pywin32. I must be missing something, because your question comes off as saying "don't solve that problem". > Given the greater user base of Windows, is there a danger this could > encourage the littering of Windows-only "workarounds" in many > projects? I don't think the inclusion of this package is going to enlighten people on the ability to make platform specific workarounds in their code. They've already been doing this with pywin32 or extension modules where necessary, and I don't think that's a such bad thing. Obviously it's great when Python provides a way to solve a cross-platform problem with one solution, e.g., listing directory contents, but it's not all sunshine and rainbows. Sometimes working with those contents requires a platform specific operation. From massimo.dipierro at gmail.com Wed Sep 14 05:19:55 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Tue, 13 Sep 2011 22:19:55 -0500 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> Message-ID: <8DCBD1E1-2F47-4ACA-B04A-056D121AF045@gmail.com> For me the only reason win32 is important is to implement a cross-platform file locking. I did not mean to propose web32 should be included but a portable file locking api would be useful. Massimo On Sep 13, 2011, at 10:00 PM, Brian Curtin wrote: > On Tue, Sep 13, 2011 at 21:28, Matt Joiner wrote: >> Do we really need to be encouraging use of Windows APIs in Python? > > I don't think we're encouraging anything, nor do I think we need to. > People have already been doing so for quite some time - it's a fact of > life. > > If I want to write a tool which uses Windows' ReadDirectoryChangesW > API, how do I do that today? I either write a C extension (which I've > done), or I use pywin32. I must be missing something, because your > question comes off as saying "don't solve that problem". > >> Given the greater user base of Windows, is there a danger this could >> encourage the littering of Windows-only "workarounds" in many >> projects? > > I don't think the inclusion of this package is going to enlighten > people on the ability to make platform specific workarounds in their > code. They've already been doing this with pywin32 or extension > modules where necessary, and I don't think that's a such bad thing. > Obviously it's great when Python provides a way to solve a > cross-platform problem with one solution, e.g., listing directory > contents, but it's not all sunshine and rainbows. Sometimes working > with those contents requires a platform specific operation. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From nevesagar at gmail.com Wed Sep 14 05:33:31 2011 From: nevesagar at gmail.com (Sagar Neve) Date: Wed, 14 Sep 2011 09:03:31 +0530 Subject: [Python-ideas] threading : AttributeE???rror: 'NoneType' object has no attribute 'start' In-Reply-To: <20110912030012.GG20463@idyll.org> References: <20110912030012.GG20463@idyll.org> Message-ID: Thanks. This is been resolved. Will take care of this in future. On Mon, Sep 12, 2011 at 8:30 AM, C. Titus Brown wrote: > On Sun, Sep 11, 2011 at 07:54:28PM -0700, Sagar Neve wrote: > > I am new to the python and I need a small script to be written using > > threads which reads the specific files present in a directory and does > > some selections of lines and then write those lines into different > > files. > > > > However the error I am getting is: > > AttributeError: 'NoneType' object has no attribute 'start' > > [ ... ] > > Hi Sagar, > > this isn't the right place to post questions about how to use Python -- > this is > python-ideas, a list for discussing future changes to the language. Please > go to > > http://mail.python.org/mailman/listinfo/python-list > > with future requests for help. > > > Here is the code. Can somebody please help me. This is very urgent. > > ...and note that your urgency is not ours, since you're not paying us! > > That having been said, at least one source of your problems is that > 'getFiles' doesn't return a value. It's kind of hard to figure out what > you want it to do; 'thr.start()' followed by 'thr.join()' immediately isn't > going to result in much threading, unless there's more going on here than > meets the eye. > > You might want to use something like > > thr = threading.Thread(target=getFiles) > > instead. See > > http://docs.python.org/library/threading.html#thread-objects > > And please ask future questions elsewhere. > > thanks! > > cheers, > --titus > > > def getFiles(fname): > > fname=preprocessLogFileDir + "/" + fname > > print fname > > fin = open(fname, "r") > > line=fin.readline() > > print line > > .... # some extraction logic. > > fout = open("out-single-file/1xx_2xx_total", "w") > > fout.write(line) > > fout.close() > > > > files=os.listdir(preprocessLogFileDir) > > cnt=1; > > for fname in files: > > print fname > > if fnmatch.fnmatch(fname,'*.log'): #and os.path.isfile(fname): > > thr = "thr" + str(cnt) > > print thr > > thr=getFiles(fname) > > thr.start() > > thr.join > > cnt = cnt + 1 > > else: > > print "\npreprocessLogFileDir/fname is either a > > directory or does > > not end with 'log'." > > print "Probabally not a pre-process file. Ignoring the > > file.\n\n" > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > http://mail.python.org/mailman/listinfo/python-ideas > > -- > C. Titus Brown, ctb at msu.edu > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Sep 14 05:38:39 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 14 Sep 2011 13:38:39 +1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E7013F3.5080907@mrabarnett.plus.com> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> Message-ID: On Wed, Sep 14, 2011 at 12:39 PM, MRAB wrote: > If none are true-ish, should it return the final (false-ish) value? By analogy with 'or', yes. (and ditto for 'all' vs 'and') > What should any([]) return? That's the more valid rationale - any()/all() need to work with an empty iterable, so the most obvious solution is to limit the range to True/False rather than having the result be data dependent. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 14 05:41:54 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 14 Sep 2011 13:41:54 +1000 Subject: [Python-ideas] return value of yield expressions In-Reply-To: <1315934606.814.116.camel@Gutsy> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <1315934606.814.116.camel@Gutsy> Message-ID: On Wed, Sep 14, 2011 at 3:23 AM, Ron Adam wrote: > If we could get the functions arguments when a function is done instead > of getting the return value. ?(The modified function arguments object is > the return value in this case.) > > ? ? ?fargs = &foo(a, b, c, d=4) This use case is covered by the bind() method in PEP 362 [1]: foo_signature = inspect.signature(foo) fargs = foo_signature.bind(a, b, c, d=4) [1] http://www.python.org/dev/peps/pep-0362/#signature-object Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 14 06:12:18 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 14 Sep 2011 14:12:18 +1000 Subject: [Python-ideas] win32 extensions In-Reply-To: <8DCBD1E1-2F47-4ACA-B04A-056D121AF045@gmail.com> References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <8DCBD1E1-2F47-4ACA-B04A-056D121AF045@gmail.com> Message-ID: On Wed, Sep 14, 2011 at 1:19 PM, Massimo Di Pierro wrote: > For me the only reason win32 is important is to implement a cross-platform file locking. > I did not mean to propose web32 should be included but a portable file locking api would be useful. If Windows developers wanted to work with Mark Hammond to figure out and systematically extract a core subset of stable pywin32 functionality potentially suitable for inclusion in the standard library, that may not be a bad thing (there's currently already duplication between the two, since the stdlib needs aspects of the win32 API to implement modules like subprocess and multiprocessing, but in those cases the pywin32 version is almost always more feature complete). The stdlib version could evolve as the standard library does, while pywin32 could move a bit quicker in responding to releases of new versions of Windows. However, CPython+pywin32 suffers from the curse of being a "good enough" solution most of the time (especially if you start with a bundled installer that takes care of finding a suitable version of pywin32 for the version of CPython being installed). Personally, I do think this idea has potential to be a valuable project, but I also think the same could be said just for helping pywin32 consolidate its web presence and promote itself more clearly. Currently, a Google search for pywin32 gives you: ~4 hits on the SourceForge project. This is *correct*, but SF in general (and currently pywin32 in particular) is organised for existing developers, not for users or potential new developers (finding the latest release and being sure it *is* the latest release is not trivial - more on this below). The next hit is on Mark's old starship page - that's ancient and hasn't been used for pywin32 downloads in years Next we have the PyPI page, which appears to be a couple of releases out of date (SF has 216, PyPI has 214) Then we have two copies of the PyPI documentation, one hosted by ActiveState (as part of the ActivePython 2.7 docs), the other by Tim Golden. Assuming Mark was happy to go along with the idea, it would be great to see some real content at http://pywin32.sourceforge.net/ instead of the current placeholder page. For example: - quick description of the project and how it can really help with Windows-specific development (and general Windows compatibility) - link directly to latest download file on Sourceforge - link to PyPI entry - link to the user mailing list (python-win32 at python.org, I believe) - basic instructions on how to get involved as a developer As an even larger task, it would be great to see some Sphinx docs for Pywin32 that provided an overview of the various modules included in the package, and references out to the relevant MS documentation (as well as to the automatically generated documentation for pywin32 itself). Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From anacrolix at gmail.com Wed Sep 14 06:13:58 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Wed, 14 Sep 2011 14:13:58 +1000 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> Message-ID: Well... ReadDirectoryChangesW is present in pyfilesystem, that Nick mentioned in another thread. In fairness you shouldn't expose this without exposing inotify for Linux, and whatever nasty equivalents are available on other systems. pywin32 gives you full access to the Windows kernel interface. The same functionality is not bundled in the standard library for Linux. To use Massimo's example regarding file locking, just this morning I had to write a ctypes wrapper around struct flock for this purpose because the fcntl module is only a partial implementation. If the full Windows API is made available, you might as well complete some of those half-done modules now that the single biggest sore thumb (Windows) for OS interfaces will be laid bare. Python is well respected for its cross platform support. Making bindings available in both cross platform and native forms will greatly improve uptake of Python. My thoughts are that more effort should be made to completely expose Unix/Linux/Mac interfaces before (or simultaneously) with making the entire Windows API available. On Wed, Sep 14, 2011 at 1:00 PM, Brian Curtin wrote: > On Tue, Sep 13, 2011 at 21:28, Matt Joiner wrote: >> Do we really need to be encouraging use of Windows APIs in Python? > > I don't think we're encouraging anything, nor do I think we need to. > People have already been doing so for quite some time - it's a fact of > life. > > If I want to write a tool which uses Windows' ReadDirectoryChangesW > API, how do I do that today? I either write a C extension (which I've > done), or I use pywin32. I must be missing something, because your > question comes off as saying "don't solve that problem". > >> Given the greater user base of Windows, is there a danger this could >> encourage the littering of Windows-only "workarounds" in many >> projects? > > I don't think the inclusion of this package is going to enlighten > people on the ability to make platform specific workarounds in their > code. They've already been doing this with pywin32 or extension > modules where necessary, and I don't think that's a such bad thing. > Obviously it's great when Python provides a way to solve a > cross-platform problem with one solution, e.g., listing directory > contents, but it's not all sunshine and rainbows. Sometimes working > with those contents requires a platform specific operation. > From massimo.dipierro at gmail.com Wed Sep 14 06:26:44 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Tue, 13 Sep 2011 23:26:44 -0500 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <8DCBD1E1-2F47-4ACA-B04A-056D121AF045@gmail.com> Message-ID: <76337FF3-9D30-4D42-BC5C-D4594B99285F@gmail.com> I would help but I do not feel competent as I am not a windows user. For portability reasons I have had a need for: win32con.LOCKFILE_EXCLUSIVE_LOCK win32con.LOCKFILE_FAIL_IMMEDIATELY win32file._get_osfhandle win32file.LockFileEx pywintypes.OVERLAPPED() Although I have not used this much, the ability to talk to windows app is also useful: import win32com.client xl = win32com.client.Dispatch("Excel.Application") Massimo On Sep 13, 2011, at 11:12 PM, Nick Coghlan wrote: > On Wed, Sep 14, 2011 at 1:19 PM, Massimo Di Pierro > wrote: >> For me the only reason win32 is important is to implement a cross-platform file locking. >> I did not mean to propose web32 should be included but a portable file locking api would be useful. > > If Windows developers wanted to work with Mark Hammond to figure out > and systematically extract a core subset of stable pywin32 > functionality potentially suitable for inclusion in the standard > library, that may not be a bad thing (there's currently already > duplication between the two, since the stdlib needs aspects of the > win32 API to implement modules like subprocess and multiprocessing, > but in those cases the pywin32 version is almost always more feature > complete). > > The stdlib version could evolve as the standard library does, while > pywin32 could move a bit quicker in responding to releases of new > versions of Windows. > > However, CPython+pywin32 suffers from the curse of being a "good > enough" solution most of the time (especially if you start with a > bundled installer that takes care of finding a suitable version of > pywin32 for the version of CPython being installed). > > Personally, I do think this idea has potential to be a valuable > project, but I also think the same could be said just for helping > pywin32 consolidate its web presence and promote itself more clearly. > Currently, a Google search for pywin32 gives you: > > ~4 hits on the SourceForge project. This is *correct*, but SF in > general (and currently pywin32 in particular) is organised for > existing developers, not for users or potential new developers > (finding the latest release and being sure it *is* the latest release > is not trivial - more on this below). > > The next hit is on Mark's old starship page - that's ancient and > hasn't been used for pywin32 downloads in years > > Next we have the PyPI page, which appears to be a couple of releases > out of date (SF has 216, PyPI has 214) > > Then we have two copies of the PyPI documentation, one hosted by > ActiveState (as part of the ActivePython 2.7 docs), the other by Tim > Golden. > > Assuming Mark was happy to go along with the idea, it would be great > to see some real content at http://pywin32.sourceforge.net/ instead of > the current placeholder page. > > For example: > - quick description of the project and how it can really help with > Windows-specific development (and general Windows compatibility) > - link directly to latest download file on Sourceforge > - link to PyPI entry > - link to the user mailing list (python-win32 at python.org, I believe) > - basic instructions on how to get involved as a developer > > As an even larger task, it would be great to see some Sphinx docs for > Pywin32 that provided an overview of the various modules included in > the package, and references out to the relevant MS documentation (as > well as to the automatically generated documentation for pywin32 > itself). > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From massimo.dipierro at gmail.com Wed Sep 14 06:28:33 2011 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Tue, 13 Sep 2011 23:28:33 -0500 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> Message-ID: <31B2BD07-807D-4307-8263-33A4FE17ED83@gmail.com> I agree. On Sep 13, 2011, at 11:13 PM, Matt Joiner wrote: > Well... > > ReadDirectoryChangesW is present in pyfilesystem, that Nick mentioned > in another thread. In fairness you shouldn't expose this without > exposing inotify for Linux, and whatever nasty equivalents are > available on other systems. > > pywin32 gives you full access to the Windows kernel interface. The > same functionality is not bundled in the standard library for Linux. > To use Massimo's example regarding file locking, just this morning I > had to write a ctypes wrapper around struct flock for this purpose > because the fcntl module is only a partial implementation. If the full > Windows API is made available, you might as well complete some of > those half-done modules now that the single biggest sore thumb > (Windows) for OS interfaces will be laid bare. > > Python is well respected for its cross platform support. Making > bindings available in both cross platform and native forms will > greatly improve uptake of Python. My thoughts are that more effort > should be made to completely expose Unix/Linux/Mac interfaces before > (or simultaneously) with making the entire Windows API available. > > On Wed, Sep 14, 2011 at 1:00 PM, Brian Curtin wrote: >> On Tue, Sep 13, 2011 at 21:28, Matt Joiner wrote: >>> Do we really need to be encouraging use of Windows APIs in Python? >> >> I don't think we're encouraging anything, nor do I think we need to. >> People have already been doing so for quite some time - it's a fact of >> life. >> >> If I want to write a tool which uses Windows' ReadDirectoryChangesW >> API, how do I do that today? I either write a C extension (which I've >> done), or I use pywin32. I must be missing something, because your >> question comes off as saying "don't solve that problem". >> >>> Given the greater user base of Windows, is there a danger this could >>> encourage the littering of Windows-only "workarounds" in many >>> projects? >> >> I don't think the inclusion of this package is going to enlighten >> people on the ability to make platform specific workarounds in their >> code. They've already been doing this with pywin32 or extension >> modules where necessary, and I don't think that's a such bad thing. >> Obviously it's great when Python provides a way to solve a >> cross-platform problem with one solution, e.g., listing directory >> contents, but it's not all sunshine and rainbows. Sometimes working >> with those contents requires a platform specific operation. >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From hetchkay at gmail.com Wed Sep 14 06:53:24 2011 From: hetchkay at gmail.com (H. Krishnan) Date: Tue, 13 Sep 2011 21:53:24 -0700 (PDT) Subject: [Python-ideas] return value of yield expressions In-Reply-To: <4E6FA26F.6060108@improva.dk> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> <1694141.1337.1315918859671.JavaMail.geo-discussion-forums@vbac9> <32726776.646.1315934482698.JavaMail.geo-discussion-forums@vbkm14> <4E6FA26F.6060108@improva.dk> Message-ID: <29042347.388.1315976004752.JavaMail.geo-discussion-forums@preb19> > My idea was to allow > > *(argument list) = (args, kwds) > May be tuple unpacking can be extended to support: (a,b,c,*args,d,**kwds) = p,q,*a1,r,s,**k1 where "p,q,*a,r,s" is equivalent to itertools.chain((p,q),a,(r,s)) (something like scheme's ",@") And in parallel, function argument unpacking could be extended to support def func(a,b,c,*args,d,**kwds): pass But I guess this would not fly: 1. I am not sure about how default values in function arguments would be handled 2. PEP 3113 disallows tuple unpacking as part of function arguments 3. Currently, *args is populated with a list (and not tuple) in tuple unpacking and possibly a dozen other issues. Krishnan -------------- next part -------------- An HTML attachment was scrubbed... URL: From aquavitae69 at gmail.com Wed Sep 14 07:49:20 2011 From: aquavitae69 at gmail.com (David Townshend) Date: Wed, 14 Sep 2011 07:49:20 +0200 Subject: [Python-ideas] Object interface to path names In-Reply-To: References: Message-ID: Thanks for the pointer - PyFilesystem looks perfect for my requirements! I've had a look at PEP 355 and can see the drawbacks to the proposed implementation (especially subclassing str), but I'm not clear on what the problems are with the concept. Would something like PyFilesystem be any more palatable? On Wed, Sep 14, 2011 at 1:22 AM, Matt Joiner wrote: > That library, pyfilesystem has some serious firepower. > > On Sep 14, 2011 1:33 AM, "David Townshend" wrote: > > > > But none of these seem to allow asynchronous calls, which make a huge > difference when dealing with a large structure. Nothing I've found really > does what I need and I'm trying to keep my dependency list short so I'll end > up writing something myself anyway, but my real question was whether this is > something that could usefully be included in the stdlib. > > > >> Having an object-oriented / polymorphic file API means that one can > >> provide file objects that aren't backed by the filesystem, but look > >> the same. This is cool in that you can do things like treat zip files > >> as directory, or mock out the filesystem for a unit test, without > >> worrying about monkeypatching builtins or using a nonstandard wrapper > >> API etc. > > > > > > Great use case! And using factories to create create the objects would > make this especially powerful. > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > http://mail.python.org/mailman/listinfo/python-ideas > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hetchkay at gmail.com Wed Sep 14 07:52:23 2011 From: hetchkay at gmail.com (H. Krishnan) Date: Tue, 13 Sep 2011 22:52:23 -0700 (PDT) Subject: [Python-ideas] return value of yield expressions In-Reply-To: <29042347.388.1315976004752.JavaMail.geo-discussion-forums@preb19> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <4E6F4D41.1080904@improva.dk> <1694141.1337.1315918859671.JavaMail.geo-discussion-forums@vbac9> <32726776.646.1315934482698.JavaMail.geo-discussion-forums@vbkm14> <4E6FA26F.6060108@improva.dk> <29042347.388.1315976004752.JavaMail.geo-discussion-forums@preb19> Message-ID: <11937563.443.1315979543072.JavaMail.geo-discussion-forums@prfh23> In general, I wonder if a, b, *args, c, *args2, d could be syntax sugar for itertools.chain((a,b), args, (c,), args2, (d,)) Function calls could then support multiple *args and interperse *args with other positional arguments. Krishnan -------------- next part -------------- An HTML attachment was scrubbed... URL: From cesare.di.mauro at gmail.com Wed Sep 14 07:59:09 2011 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Wed, 14 Sep 2011 07:59:09 +0200 Subject: [Python-ideas] win32 extensions In-Reply-To: <4E6FFE92.8080307@canterbury.ac.nz> References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> Message-ID: 2011/9/14 Greg Ewing > Guido van Rossum wrote: > > There's also the impression (possibly wrong) that pywin32/win32all is >> somewhat dated, and hasn't changed much since the glory days of >> Windows 98. >> > > I think that's mainly because the thing it's wrapping -- > the win32 api -- is somewhat dated, and hasn't changed much > since the glory days of Windows 98. :-) > > By today's standards, the traditional win32 gui api is > hopelessly cruddy. Unfortunately, the only efforts on > Microsoft's part to replace it with something better > seem to be taking place in the .NET world. > http://arstechnica.com/microsoft/news/2011/06/windows-8-for-software-developers-the-longhorn-dream-reborn.ars/2 ;) > > As things stand, this places them out of the easy reach > of Python. Adding something to the stdlib enabling > Python to call .NET libraries would open up considerable > new opportunities. > > Maybe it could be based on this: > > http://pythonnet.sourceforge.net/ > > -- > Greg Python for .NET is a GREAT project which opens up a new world for Windows developers which want to start playing around with things like Silverlight, WPF, WCF, etc.. It's a tiny, but very useful project . Regards, Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From cesare.di.mauro at gmail.com Wed Sep 14 08:02:52 2011 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Wed, 14 Sep 2011 08:02:52 +0200 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> Message-ID: 2011/9/14 Brian Curtin > > Could we skip inclusion the win32gui stuff? We'd be including > something which has been surpassed by WinForms, which itself has > (mostly?) been surpassed by WPF, and soon enough we'll have Metro...on > top of the fact that we'd be introducing a second way to do GUI work > alongside Tkinter. > I totally agree. WPF is a masterpiece at producing rich and visually appealing UI in very little time; something that Windows programmers should definitely look at. Regards, Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From cesare.di.mauro at gmail.com Wed Sep 14 08:13:08 2011 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Wed, 14 Sep 2011 08:13:08 +0200 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> Message-ID: 2011/9/14 Matt Joiner > Do we really need to be encouraging use of Windows APIs in Python? > Given the greater user base of Windows, is there a danger this could > encourage the littering of Windows-only "workarounds" in many > projects? > > "Well Python doesn't ship with a usable GUI toolkit (hi tk), but it > does ship with Win32"->"Everyone uses Windows anyway, lets make a > Win32 interface". > > Just a thought. > > We already have platform-specific modules on stdlib. ;) Have you written any interactive command-line interpreter in Python? It's great if you are using an Unix-like o.s., since the readline module can be used, but Windows programmers can't do the same. -_- As a programmer, I use Python to help on solving my problems, whatever the platform I was using to develop for. If I need to make a Windows-only project, there's nothing that Python can do to stop me. But I'll definitely appreciate it, if it can make my life easier on doing this. Regards, Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Sep 14 09:17:23 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 14 Sep 2011 17:17:23 +1000 Subject: [Python-ideas] Object interface to path names In-Reply-To: References: Message-ID: On Wed, Sep 14, 2011 at 3:49 PM, David Townshend wrote: > Thanks for the pointer - PyFilesystem looks perfect for my requirements! > I've had a look at PEP 355 and can see the drawbacks to the proposed > implementation (especially subclassing str), but I'm not clear on what the > problems are with the concept. There's nothing wrong with the concept of a more object-oriented interface to the filesystem - PEP 355 was rejected due to the specifics of the proposed API rather than the idea in general being unacceptable. However, designing a nice OO filesystem API, getting feedback on it, getting it to a point where it is evolving slowly enough to be a suitable for inclusion in the stdlib, then getting it through the gauntlet of python-dev's design and implementation critique for standard library inclusion isn't exactly a task for the faint-hearted :) >?Would something like PyFilesystem be any > more palatable? I don't know the PyFilesystem API well enough to really say. However, from my quick review of the docs, it has potential. It's API oriented approach definitely aligns well with the role of the standard library in other areas (such as the file-like API itself, as well as more formal interfaces like the DB API and the crypto component APIs). I've cc'ed Ryan on a couple of messages in this thread, so hopefully he'll be inclined to chime in. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 14 09:30:00 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 14 Sep 2011 17:30:00 +1000 Subject: [Python-ideas] return value of yield expressions In-Reply-To: <1315976466.2986.56.camel@Gutsy> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <1315934606.814.116.camel@Gutsy> <1315976466.2986.56.camel@Gutsy> Message-ID: On Wed, Sep 14, 2011 at 3:01 PM, Ron Adam wrote: > On Wed, 2011-09-14 at 13:41 +1000, Nick Coghlan wrote: >> On Wed, Sep 14, 2011 at 3:23 AM, Ron Adam wrote: >> > If we could get the functions arguments when a function is done instead >> > of getting the return value. ?(The modified function arguments object is >> > the return value in this case.) >> > >> > ? ? ?fargs = &foo(a, b, c, d=4) >> >> This use case is covered by the bind() method in PEP 362 [1]: >> >> ? ? foo_signature = inspect.signature(foo) >> ? ? fargs = foo_signature.bind(a, b, c, d=4) >> >> [1] http://www.python.org/dev/peps/pep-0362/#signature-object > > It doesn't quite do the same thing, and it's not nearly as easy to use. And magic syntax is better? This is an incredibly niche use case, so there is zero justification for giving it special syntax when functions, methods and classes will do the job just fine (it's also worth keeping in mind that Guido hasn't even officially given PEP 362 itself the nod at this point). > Regarding PEP 362: > > The signature function gets the signature object from the target > function, ok, ?The ?use case in the pep describes binding and > introspecting the signature object for various purposes, which is a nice > thing to be able to do. ? But after that, it just calls the functions > with the same *args and **kwds that was used to bind the signature. ? So > it's unpacking args and kwds a second time. > > Will there be a way to reuse the bound signature object with a function > directly without unpacking it? No such micro-optimisation is planned (it would require a complete redesign of the way arguments are passed to functions). If the performance hit of explicit validation is unacceptable, just don't bother with it and accept that you won't get an error until the function is actually used. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From anacrolix at gmail.com Wed Sep 14 10:20:43 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Wed, 14 Sep 2011 18:20:43 +1000 Subject: [Python-ideas] win32 extensions In-Reply-To: <76337FF3-9D30-4D42-BC5C-D4594B99285F@gmail.com> References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <8DCBD1E1-2F47-4ACA-B04A-056D121AF045@gmail.com> <76337FF3-9D30-4D42-BC5C-D4594B99285F@gmail.com> Message-ID: I think it's clear that file locking and asynchronous IO (aio and OVERLAPPED) are sore points and should be better exposed. On Sep 14, 2011 2:27 PM, "Massimo Di Pierro" wrote: > I would help but I do not feel competent as I am not a windows user. > For portability reasons I have had a need for: > > win32con.LOCKFILE_EXCLUSIVE_LOCK > win32con.LOCKFILE_FAIL_IMMEDIATELY > win32file._get_osfhandle > win32file.LockFileEx > pywintypes.OVERLAPPED() > > Although I have not used this much, the ability to talk to windows app is also useful: > > import win32com.client > xl = win32com.client.Dispatch("Excel.Application") > > Massimo > > On Sep 13, 2011, at 11:12 PM, Nick Coghlan wrote: > >> On Wed, Sep 14, 2011 at 1:19 PM, Massimo Di Pierro >> wrote: >>> For me the only reason win32 is important is to implement a cross-platform file locking. >>> I did not mean to propose web32 should be included but a portable file locking api would be useful. >> >> If Windows developers wanted to work with Mark Hammond to figure out >> and systematically extract a core subset of stable pywin32 >> functionality potentially suitable for inclusion in the standard >> library, that may not be a bad thing (there's currently already >> duplication between the two, since the stdlib needs aspects of the >> win32 API to implement modules like subprocess and multiprocessing, >> but in those cases the pywin32 version is almost always more feature >> complete). >> >> The stdlib version could evolve as the standard library does, while >> pywin32 could move a bit quicker in responding to releases of new >> versions of Windows. >> >> However, CPython+pywin32 suffers from the curse of being a "good >> enough" solution most of the time (especially if you start with a >> bundled installer that takes care of finding a suitable version of >> pywin32 for the version of CPython being installed). >> >> Personally, I do think this idea has potential to be a valuable >> project, but I also think the same could be said just for helping >> pywin32 consolidate its web presence and promote itself more clearly. >> Currently, a Google search for pywin32 gives you: >> >> ~4 hits on the SourceForge project. This is *correct*, but SF in >> general (and currently pywin32 in particular) is organised for >> existing developers, not for users or potential new developers >> (finding the latest release and being sure it *is* the latest release >> is not trivial - more on this below). >> >> The next hit is on Mark's old starship page - that's ancient and >> hasn't been used for pywin32 downloads in years >> >> Next we have the PyPI page, which appears to be a couple of releases >> out of date (SF has 216, PyPI has 214) >> >> Then we have two copies of the PyPI documentation, one hosted by >> ActiveState (as part of the ActivePython 2.7 docs), the other by Tim >> Golden. >> >> Assuming Mark was happy to go along with the idea, it would be great >> to see some real content at http://pywin32.sourceforge.net/ instead of >> the current placeholder page. >> >> For example: >> - quick description of the project and how it can really help with >> Windows-specific development (and general Windows compatibility) >> - link directly to latest download file on Sourceforge >> - link to PyPI entry >> - link to the user mailing list (python-win32 at python.org, I believe) >> - basic instructions on how to get involved as a developer >> >> As an even larger task, it would be great to see some Sphinx docs for >> Pywin32 that provided an overview of the various modules included in >> the package, and references out to the relevant MS documentation (as >> well as to the automatically generated documentation for pywin32 >> itself). >> >> Regards, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From jh at improva.dk Wed Sep 14 10:30:26 2011 From: jh at improva.dk (Jacob Holm) Date: Wed, 14 Sep 2011 10:30:26 +0200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> Message-ID: <4E706622.2070808@improva.dk> On 2011-09-14 05:38, Nick Coghlan wrote: > On Wed, Sep 14, 2011 at 12:39 PM, MRAB wrote: >> If none are true-ish, should it return the final (false-ish) value? > > By analogy with 'or', yes. (and ditto for 'all' vs 'and') > >> What should any([]) return? > > That's the more valid rationale - any()/all() need to work with an > empty iterable, so the most obvious solution is to limit the range to > True/False rather than having the result be data dependent. > The alternative would be to include a "start" parameter similar to sum(). It would default to False for any(), and True for all(). - Jacob From ron3200 at gmail.com Wed Sep 14 07:01:06 2011 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 14 Sep 2011 00:01:06 -0500 Subject: [Python-ideas] return value of yield expressions In-Reply-To: References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <1315934606.814.116.camel@Gutsy> Message-ID: <1315976466.2986.56.camel@Gutsy> On Wed, 2011-09-14 at 13:41 +1000, Nick Coghlan wrote: > On Wed, Sep 14, 2011 at 3:23 AM, Ron Adam wrote: > > If we could get the functions arguments when a function is done instead > > of getting the return value. (The modified function arguments object is > > the return value in this case.) > > > > fargs = &foo(a, b, c, d=4) > > This use case is covered by the bind() method in PEP 362 [1]: > > foo_signature = inspect.signature(foo) > fargs = foo_signature.bind(a, b, c, d=4) > > [1] http://www.python.org/dev/peps/pep-0362/#signature-object It doesn't quite do the same thing, and it's not nearly as easy to use. Regarding PEP 362: The signature function gets the signature object from the target function, ok, The use case in the pep describes binding and introspecting the signature object for various purposes, which is a nice thing to be able to do. But after that, it just calls the functions with the same *args and **kwds that was used to bind the signature. So it's unpacking args and kwds a second time. Will there be a way to reuse the bound signature object with a function directly without unpacking it? Cheers, Ron (My replies are still taking more than 6 to 8 hours to show up on the list? I changed my email address recently, The old one no longer works as I chose not to renew the domain name.) From ned at nedbatchelder.com Wed Sep 14 13:30:04 2011 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 14 Sep 2011 07:30:04 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E706622.2070808@improva.dk> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E706622.2070808@improva.dk> Message-ID: <4E70903C.5030301@nedbatchelder.com> On 9/14/2011 4:30 AM, Jacob Holm wrote: > On 2011-09-14 05:38, Nick Coghlan wrote: >> On Wed, Sep 14, 2011 at 12:39 PM, MRAB wrote: >>> If none are true-ish, should it return the final (false-ish) value? >> By analogy with 'or', yes. (and ditto for 'all' vs 'and') >> >>> What should any([]) return? >> That's the more valid rationale - any()/all() need to work with an >> empty iterable, so the most obvious solution is to limit the range to >> True/False rather than having the result be data dependent. >> > The alternative would be to include a "start" parameter similar to > sum(). It would default to False for any(), and True for all(). I don't see any need for extra parameters. any([]) returns False, and all([]) returns True, just as they do now. If you think that's an odd special case, I think it already is. For example, max([]) raises a ValueError, so why doesn't any([])? Don't answer that, it's just an illustration of how edge cases like this can be unusual. --Ned. > > - Jacob > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From greg.ewing at canterbury.ac.nz Wed Sep 14 14:59:01 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Sep 2011 00:59:01 +1200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: <4E70A515.1070602@canterbury.ac.nz> Nick Coghlan wrote: > def __in__(self, container): > return int(self) in container > > That code involves an implicit assumption that *all* containers are > equivalence based, and that simply isn't true - it is sometimes useful > to create a container that relies on object identity rather than value Maybe the right operand should be given the first chance for this particular operator? -- Greg From greg.ewing at canterbury.ac.nz Wed Sep 14 15:04:55 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Sep 2011 01:04:55 +1200 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> Message-ID: <4E70A677.3070002@canterbury.ac.nz> Matt Joiner wrote: > Do we really need to be encouraging use of Windows APIs in Python? No, but we should at least be making it *possible* to use all of the native facilities available on Windows, including the more modern ones such as WPF, IMO. -- Greg From ethan at stoneleaf.us Wed Sep 14 15:12:01 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 14 Sep 2011 06:12:01 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E70903C.5030301@nedbatchelder.com> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> Message-ID: <4E70A821.5010007@stoneleaf.us> Ned Batchelder wrote: > On 9/14/2011 4:30 AM, Jacob Holm wrote: >> On 2011-09-14 05:38, Nick Coghlan wrote: >>> On Wed, Sep 14, 2011 at 12:39 PM, MRAB wrote: >>>> If none are true-ish, should it return the final (false-ish) value? >>> >>> By analogy with 'or', yes. (and ditto for 'all' vs 'and') >>> >>>> What should any([]) return? >>> >>> That's the more valid rationale - any()/all() need to work with an >>> empty iterable, so the most obvious solution is to limit the range to >>> True/False rather than having the result be data dependent. >>> >> The alternative would be to include a "start" parameter similar to >> sum(). It would default to False for any(), and True for all(). > > I don't see any need for extra parameters. any([]) returns False, and > all([]) returns True, just as they do now. Perhaps 'need' is putting it too strongly, but if any()/all() were changed to return the first true-ish/last false-ish value, then the extra parameter would be convenient in specifying the value desired if it wasn't True/False. Sort of like {}.get('nothere', '5'). ~Ethan~ From mikegraham at gmail.com Wed Sep 14 15:18:07 2011 From: mikegraham at gmail.com (Mike Graham) Date: Wed, 14 Sep 2011 09:18:07 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E70A821.5010007@stoneleaf.us> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> <4E70A821.5010007@stoneleaf.us> Message-ID: On Wed, Sep 14, 2011 at 9:12 AM, Ethan Furman wrote: > Perhaps 'need' is putting it too strongly, but if any()/all() were changed > to return the first true-ish/last false-ish value, then the extra parameter > would be convenient in specifying the value desired if it wasn't True/False. > ?Sort of like {}.get('nothere', '5'). > > ~Ethan~ It's worth noting that this implementation of any would make it suspiciously similar to next. From brian.curtin at gmail.com Wed Sep 14 16:54:38 2011 From: brian.curtin at gmail.com (Brian Curtin) Date: Wed, 14 Sep 2011 09:54:38 -0500 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> Message-ID: On Tue, Sep 13, 2011 at 23:13, Matt Joiner wrote: > pywin32 gives you full access to the Windows kernel interface. The > same functionality is not bundled in the standard library for Linux. Correct me if I'm wrong, but it doesn't exist. > To use Massimo's example regarding file locking, just this morning I > had to write a ctypes wrapper around struct flock for this purpose > because the fcntl module is only a partial implementation. If the full > Windows API is made available, you might as well complete some of > those half-done modules now that the single biggest sore thumb > (Windows) for OS interfaces will be laid bare. > > Python is well respected for its cross platform support. Making > bindings available in both cross platform and native forms will > greatly improve uptake of Python. My thoughts are that more effort > should be made to completely expose Unix/Linux/Mac interfaces before > (or simultaneously) with making the entire Windows API available. Mark or someone could probably answer this question in a second, but I've tried to find a date for when pywin32 was first released, which is kind of tough. win32all build 135 added Python 2.0 support, which was released in October 2000, so I suspect that build occurred close to that time. There are 29 releases listed before that one on http://python.net/crew/mhammond/win32/RecentChangeHistory.html (and potentially more elsewhere, as the list only goes to 106), all going back to the Python 1.x times. Do we really need to hold back a 10+ year old project because no one thought to do the same thing for other platforms? I don't disagree that having bindings for all popular platforms is a good thing, but we'll be on Python 5 by the time we can wrangle up people to write all of that code, maintain it, and watch it mature while helping it find wide reaching adoption. Mark released the book "Python Programming on Win32" in January 2000, which I still have on my desk despite it barely surviving a flooded basement, and it's almost all based on this package. It was mature enough even then to write most of a 600+ page book with, which puts it in a very small camp of projects. From tjreedy at udel.edu Wed Sep 14 18:01:00 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Sep 2011 12:01:00 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E70903C.5030301@nedbatchelder.com> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> Message-ID: On 9/14/2011 7:30 AM, Ned Batchelder wrote: > For example, max([]) raises a ValueError, so why doesn't any([])? You should turn the question around, any([]) returns the identity element for any(), so why does't max([])? Because Python does not have an artificial universal minimum object. Guido has rejected the idea and it makes even less sense in Python where cross-type comparisons are generally discouraged. If max and min were restricted to totally ordered numbers, then fhi=float('inf') and flo=-fhi would work. But they are not. -- Terry Jan Reedy From alexander.belopolsky at gmail.com Wed Sep 14 18:17:13 2011 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 14 Sep 2011 12:17:13 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E6FFC5B.8010704@nedbatchelder.com> References: <4E6FFC5B.8010704@nedbatchelder.com> Message-ID: On Tue, Sep 13, 2011 at 8:59 PM, Ned Batchelder wrote: > .. Why shouldn't any() also return Sx? What is the use case for returning the first "trueish" value? In most cases, any() is used in boolean context, say if any(S): do_something() In this case, returning Sx would result in Sx.__bool__() being called twice. It is also rare that any() is called on the raw sequence of objects that are of interest. More often it is used to express predicates such as any(x < 0 for x in S). From python at mrabarnett.plus.com Wed Sep 14 19:19:45 2011 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 14 Sep 2011 18:19:45 +0100 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FFC5B.8010704@nedbatchelder.com> Message-ID: <4E70E231.2050607@mrabarnett.plus.com> On 14/09/2011 17:17, Alexander Belopolsky wrote: > On Tue, Sep 13, 2011 at 8:59 PM, Ned Batchelder wrote: >> .. Why shouldn't any() also return Sx? > > What is the use case for returning the first "trueish" value? In most > cases, any() is used in boolean context, say > > if any(S): > do_something() > > In this case, returning Sx would result in Sx.__bool__() being called > twice. It is also rare that any() is called on the raw sequence of > objects that are of interest. More often it is used to express > predicates such as any(x< 0 for x in S). > I had a use-case recently. I was looking for an entry in a pseudo-dict, where the key didn't have to be an exact match, so I called the .get method on the pseudo-dict until it returned a true-ish value (the value would never be false-ish). I just wrote a short function to do what I wanted. From cesare.di.mauro at gmail.com Wed Sep 14 21:33:36 2011 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Wed, 14 Sep 2011 21:33:36 +0200 Subject: [Python-ideas] win32 extensions In-Reply-To: <4E70A677.3070002@canterbury.ac.nz> References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> Message-ID: 2011/9/14 Greg Ewing > Matt Joiner wrote: > >> Do we really need to be encouraging use of Windows APIs in Python? >> > > No, but we should at least be making it *possible* to use all of > the native facilities available on Windows, including the more > modern ones such as WPF, IMO. > > -- > Greg I agree, and it'll be interesting to know if encouraging the use of Linux or OS X APIs raises such "questions" as well... Regards, Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Sep 15 00:44:13 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 15 Sep 2011 08:44:13 +1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E7013F3.5080907@mrabarnett.plus.com> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> Message-ID: <4E712E3D.1060301@pearwood.info> MRAB wrote: > What should any([]) return? I'm not sure I understand the question, because any([]) already has an answer, and changing it would be a backwards-incompatible change. So I don't quite understand the point of the question: the answer is "exactly what it already returns". >>> any([]) False Likewise for all: >>> all([]) True This is known as "vacuous truth": https://secure.wikimedia.org/wikipedia/en/wiki/Vacuous_truth As far as justifying the results returned by any() and all(), the decision to return False and True respectively seems to work well for me. I have rarely, if ever, needed the opposite behaviour: if not mylist or any(mylist): ... if mylist and all(mylist): .... so, at least for me, the behaviour of all() and any() seems like the right behaviour. However, I point out that if the argument is an iterator instead of a sequence, it's significantly harder: try: flag = bool(next(it)) except StopIteration: flag = False # instead of True else: flag = all(it) Perhaps all() and any() should take an optional argument to return if the iterable is empty? -- Steven From greg.ewing at canterbury.ac.nz Thu Sep 15 00:45:23 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Sep 2011 10:45:23 +1200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> Message-ID: <4E712E83.3050709@canterbury.ac.nz> Terry Reedy wrote: > You should turn the question around, any([]) returns the identity > element for any(), so why does't max([])? Because Python does not have > an artificial universal minimum object. Even if it did, it wouldn't necessarily make sense for max() to return it. max(s) means "the largest member of set s", and for it to return something that wasn't a member of s would be perverse. In every case I've encountered where there's a possibility of min() or max() being applied to an empty collection, the empty state had to be treated as a special case anyway. So it's just as easy to test for the empty case before calling min() or max(). -- Greg From greg.ewing at canterbury.ac.nz Thu Sep 15 01:00:17 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Sep 2011 11:00:17 +1200 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> Message-ID: <4E713201.4090105@canterbury.ac.nz> Cesare Di Mauro wrote: > I agree, and it'll be interesting to know if encouraging the use of > Linux or OS X APIs raises such "questions" as well... We already have reasonably good third-party wrappers for GUI facilities on the other platforms -- PyObjC on MacOSX, and pygtk on Linux. We don't have anything equivalent for recent Windows technologies, though. -- Greg From steve at pearwood.info Thu Sep 15 01:04:12 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 15 Sep 2011 09:04:12 +1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> <4E70A821.5010007@stoneleaf.us> Message-ID: <4E7132EC.70506@pearwood.info> Mike Graham wrote: > On Wed, Sep 14, 2011 at 9:12 AM, Ethan Furman wrote: >> Perhaps 'need' is putting it too strongly, but if any()/all() were changed >> to return the first true-ish/last false-ish value, then the extra parameter >> would be convenient in specifying the value desired if it wasn't True/False. >> Sort of like {}.get('nothere', '5'). >> >> ~Ethan~ > > It's worth noting that this implementation of any would make it > suspiciously similar to next. I don't see that. next() returns the next item from an iterator regardless of its boolean state. The actual and proposed implementations of any() skip over false items. The proposal might be implemented something like this: def any(iterable): item = False for item in iterable: if item: return item # currently return True return item The practical difference should be clear if you compare calling: next(iter([0]*1000 + [1])) versus: any([0]*1000 + [1]) -- Steven From jeanpierreda at gmail.com Thu Sep 15 01:07:03 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 14 Sep 2011 19:07:03 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E712E83.3050709@canterbury.ac.nz> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> <4E712E83.3050709@canterbury.ac.nz> Message-ID: > Even if it did, it wouldn't necessarily make sense for max() to > return it. max(s) means "the largest member of set s", and for it > to return something that wasn't a member of s would be perverse. Perverse and also wrong. Perhaps you want an infimum/supremum function? Devin On Wed, Sep 14, 2011 at 6:45 PM, Greg Ewing wrote: > Terry Reedy wrote: > >> You should turn the question around, any([]) returns the identity element >> for any(), so why does't max([])? ?Because Python does not have an >> artificial universal minimum object. > > Even if it did, it wouldn't necessarily make sense for max() to > return it. max(s) means "the largest member of set s", and for it > to return something that wasn't a member of s would be perverse. > > In every case I've encountered where there's a possibility of > min() or max() being applied to an empty collection, the empty > state had to be treated as a special case anyway. So it's just > as easy to test for the empty case before calling min() or max(). > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From greg.ewing at canterbury.ac.nz Thu Sep 15 01:15:05 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Sep 2011 11:15:05 +1200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E7132EC.70506@pearwood.info> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> <4E70A821.5010007@stoneleaf.us> <4E7132EC.70506@pearwood.info> Message-ID: <4E713579.2030403@canterbury.ac.nz> Steven D'Aprano wrote: > I don't see that. next() returns the next item from an iterator > regardless of its boolean state. The actual and proposed implementations > of any() skip over false items. While such a function might be useful, I think it would be better provided as a new function called something like first(), rather than changing any(). -- Greg From python at mrabarnett.plus.com Thu Sep 15 01:25:09 2011 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 15 Sep 2011 00:25:09 +0100 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E713579.2030403@canterbury.ac.nz> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> <4E70A821.5010007@stoneleaf.us> <4E7132EC.70506@pearwood.info> <4E713579.2030403@canterbury.ac.nz> Message-ID: <4E7137D5.50206@mrabarnett.plus.com> On 15/09/2011 00:15, Greg Ewing wrote: > Steven D'Aprano wrote: > >> I don't see that. next() returns the next item from an iterator >> regardless of its boolean state. The actual and proposed >> implementations of any() skip over false items. > > While such a function might be useful, I think it would be > better provided as a new function called something like > first(), rather than changing any(). > Could it also have something like a 'key' argument, by default __bool__? It would return the first item for which key(item) returns True or something true-ish. There's also the option of a 'default' argument for when there are no true-ish items. From python at mrabarnett.plus.com Thu Sep 15 01:27:12 2011 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 15 Sep 2011 00:27:12 +0100 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E712E3D.1060301@pearwood.info> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E712E3D.1060301@pearwood.info> Message-ID: <4E713850.9000706@mrabarnett.plus.com> On 14/09/2011 23:44, Steven D'Aprano wrote: > MRAB wrote: > >> What should any([]) return? > > I'm not sure I understand the question, because any([]) already has an > answer, and changing it would be a backwards-incompatible change. So I > don't quite understand the point of the question: the answer is "exactly > what it already returns". > I'm just checking that the corner cases have been considered. After all, explicit is better than implicit... From ncoghlan at gmail.com Thu Sep 15 01:49:18 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Sep 2011 09:49:18 +1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E70E231.2050607@mrabarnett.plus.com> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E70E231.2050607@mrabarnett.plus.com> Message-ID: On Thu, Sep 15, 2011 at 3:19 AM, MRAB wrote: > I just wrote a short function to do what I wanted. +1 QoTW :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From pyideas at rebertia.com Thu Sep 15 02:20:48 2011 From: pyideas at rebertia.com (Chris Rebert) Date: Wed, 14 Sep 2011 17:20:48 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E713579.2030403@canterbury.ac.nz> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> <4E70A821.5010007@stoneleaf.us> <4E7132EC.70506@pearwood.info> <4E713579.2030403@canterbury.ac.nz> Message-ID: On Wed, Sep 14, 2011 at 4:15 PM, Greg Ewing wrote: > Steven D'Aprano wrote: > >> I don't see that. next() returns the next item from an iterator regardless >> of its boolean state. The actual and proposed implementations of any() skip >> over false items. > > While such a function might be useful, I think it would be > better provided as a new function called something like > first(), rather than changing any(). Ruby calls it "detect": http://www.ruby-doc.org/core/classes/Enumerable.html#M001485 Cheers, Chris From ncoghlan at gmail.com Thu Sep 15 03:38:13 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Sep 2011 11:38:13 +1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> <4E70A821.5010007@stoneleaf.us> <4E7132EC.70506@pearwood.info> <4E713579.2030403@canterbury.ac.nz> Message-ID: On Thu, Sep 15, 2011 at 10:20 AM, Chris Rebert wrote: >> While such a function might be useful, I think it would be >> better provided as a new function called something like >> first(), rather than changing any(). > > Ruby calls it "detect": > http://www.ruby-doc.org/core/classes/Enumerable.html#M001485 from itertools import dropwhile def _not(x): return not x def first(iterable): return next(dropwhile(_not, iterable)) Adjust implementation to taste/use case (e.g. make the predicate configurable or accept a default value rather than throwing StopIteration for empy sequences). There comes a point where it is better to just let people compose existing tools according to their specific needs rather than providing yet another special purpose tool. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tjreedy at udel.edu Thu Sep 15 04:52:47 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Sep 2011 22:52:47 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> <4E70A821.5010007@stoneleaf.us> <4E7132EC.70506@pearwood.info> <4E713579.2030403@canterbury.ac.nz> Message-ID: On 9/14/2011 9:38 PM, Nick Coghlan wrote: > On Thu, Sep 15, 2011 at 10:20 AM, Chris Rebert wrote: >>> While such a function might be useful, I think it would be >>> better provided as a new function called something like >>> first(), rather than changing any(). >> >> Ruby calls it "detect": >> http://www.ruby-doc.org/core/classes/Enumerable.html#M001485 > > from itertools import dropwhile > def _not(x): return not x > > def first(iterable): > return next(dropwhile(_not, iterable)) > > Adjust implementation to taste/use case (e.g. make the predicate > configurable or accept a default value rather than throwing > StopIteration for empy sequences). There comes a point where it is > better to just let people compose existing tools according to their > specific needs rather than providing yet another special purpose tool. That is what itertool is designed for. I believe chain(dropwhile(...), [default]) will add a default. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Thu Sep 15 05:41:35 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Sep 2011 15:41:35 +1200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E7137D5.50206@mrabarnett.plus.com> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> <4E70A821.5010007@stoneleaf.us> <4E7132EC.70506@pearwood.info> <4E713579.2030403@canterbury.ac.nz> <4E7137D5.50206@mrabarnett.plus.com> Message-ID: <4E7173EF.4020701@canterbury.ac.nz> On 15/09/11 11:25, MRAB wrote: > Could it also have something like a 'key' argument, by default > __bool__? Or more generally, a function. Although that's not strictly necessary. If there were a first() function that simply returned the first item from an iterator, whether true or not, one could write first(itertools.ifilter(func, something)) -- Greg From greg.ewing at canterbury.ac.nz Thu Sep 15 05:55:03 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Sep 2011 15:55:03 +1200 Subject: [Python-ideas] Object interface to path names In-Reply-To: References: Message-ID: <4E717717.5090401@canterbury.ac.nz> On 14/09/11 03:03, David Townshend wrote: > 3. Allow caching of file attribute data so that queries do not have to wait > the disk or network to respond (although at the cost of accuracy). > ... > The third can be met be allowing all disk calls to be asynchonous: You're mixing up two completely different concepts here. Cacheing has nothing to do with asynchronous calls; it's storing the result so that you don't have to wait the *next* time you want the information. Both could be useful, but only for certain applications, and they should both be off by default in any general-purpose impelentation. -- Greg From cmjohnson.mailinglist at gmail.com Thu Sep 15 06:01:20 2011 From: cmjohnson.mailinglist at gmail.com (Carl Matthew Johnson) Date: Wed, 14 Sep 2011 18:01:20 -1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E7173EF.4020701@canterbury.ac.nz> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> <4E70A821.5010007@stoneleaf.us> <4E7132EC.70506@pearwood.info> <4E713579.2030403@canterbury.ac.nz> <4E7137D5.50206@mrabarnett.plus.com> <4E7173EF.4020701@canterbury.ac.nz> Message-ID: <71D90DC7-71D6-400D-9ECD-AEF7ACFDBAF2@gmail.com> On Sep 14, 2011, at 5:41 PM, Greg Ewing wrote: > On 15/09/11 11:25, MRAB wrote: > >> Could it also have something like a 'key' argument, by default >> __bool__? > > Or more generally, a function. > > Although that's not strictly necessary. If there were a first() > function that simply returned the first item from an iterator, > whether true or not, one could write > > first(itertools.ifilter(func, something)) > Time machine: next(filter(func, something)) #in Python 3, filter is ifilter! :-D From aquavitae69 at gmail.com Thu Sep 15 06:15:32 2011 From: aquavitae69 at gmail.com (David Townshend) Date: Thu, 15 Sep 2011 06:15:32 +0200 Subject: [Python-ideas] win32 extensions In-Reply-To: <4E713201.4090105@canterbury.ac.nz> References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> <4E713201.4090105@canterbury.ac.nz> Message-ID: On Sep 15, 2011 1:00 AM, "Greg Ewing" wrote: > > Cesare Di Mauro wrote: > >> I agree, and it'll be interesting to know if encouraging the use of Linux or OS X APIs raises such "questions" as well... > > > We already have reasonably good third-party wrappers for > GUI facilities on the other platforms -- PyObjC on MacOSX, > and pygtk on Linux. We don't have anything equivalent for > recent Windows technologies, though. PyQt? -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Sep 15 06:22:38 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Sep 2011 16:22:38 +1200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: Message-ID: <4E717D8E.60103@canterbury.ac.nz> On 14/09/11 05:56, Guido van Rossum wrote: > I think I just mentioned one (turning 'if not' into a jump). Anyway, > I'm glad to reject the PEP for the reason that I like the status quo > just fine. Did you see the comment I made concerning jump optimizations? (Briefly, I don't think it would do any harm to keep performing this optimization, since it wouldn't affect any of my intended use cases.) > Also, after reading the PEP from beginning to end ... I think the > offered API is too complicated to be of much use. There's a bit more machinery involved than there is with the other operators, but I believe that any given use cases will usually only *use* part of it. In particular, a system for building parse trees only needs to implement the 2-operand methods, which is no more complicated than overriding any of the existing operators. If it would help, I could provide a couple of fully-worked examples, so you can see how complicated (or not) it would be to use in practice. There's also the possibility of simplifying the API if it's considered that some of it will *never* be used, e.g. by eliminating the machinery for custom short-circuiting. > Certainly the NumPy folks have > repeatedly claimed that they are fine with the status quo. NumPy is not the only use case by a long shot. The main reason I want it is for expressing database queries. You only have to look at the contortions that things like Django have to go through to wonder whether there's a better way. Apologies if I appear to be flogging a resting equine here. If you really don't want to think about it any more, I'll shut up. But if it's going to be rejected, I'd prefer it to be rejected for genuine reasons rather than FUD. -- Greg From ncoghlan at gmail.com Thu Sep 15 06:36:42 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Sep 2011 14:36:42 +1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E717D8E.60103@canterbury.ac.nz> References: <4E717D8E.60103@canterbury.ac.nz> Message-ID: On Thu, Sep 15, 2011 at 2:22 PM, Greg Ewing wrote: > On 14/09/11 05:56, Guido van Rossum wrote: >> Certainly the NumPy folks have >> repeatedly claimed that they are fine with the status quo. > > NumPy is not the only use case by a long shot. The main reason > I want it is for expressing database queries. You only have to > look at the contortions that things like Django have to go > through to wonder whether there's a better way. > > Apologies if I appear to be flogging a resting equine here. If > you really don't want to think about it any more, I'll shut up. > But if it's going to be rejected, I'd prefer it to be rejected > for genuine reasons rather than FUD. Some concrete use cases might help a lot. While such use cases can also be addressed via AST transformations, that's a rather large sledgehammer to break out just to customise some boolean logic. I'll hold off on marking the PEP as rejected until you've add a chance to add (and Guido has had a chance to read) those examples. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From greg.ewing at canterbury.ac.nz Thu Sep 15 06:37:05 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Sep 2011 16:37:05 +1200 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> <4E713201.4090105@canterbury.ac.nz> Message-ID: <4E7180F1.6080107@canterbury.ac.nz> On 15/09/11 16:15, David Townshend wrote: > > On Sep 15, 2011 1:00 AM, "Greg Ewing" > wrote: > > > > We already have reasonably good third-party wrappers for > > GUI facilities on the other platforms -- PyObjC on MacOSX, > > and pygtk on Linux. We don't have anything equivalent for > > recent Windows technologies, though. > > PyQt? That's not the same kind of thing, since it doesn't directly expose the platform facilities. -- Greg From greg.ewing at canterbury.ac.nz Thu Sep 15 06:39:11 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Sep 2011 16:39:11 +1200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E717D8E.60103@canterbury.ac.nz> Message-ID: <4E71816F.2070107@canterbury.ac.nz> On 15/09/11 16:36, Nick Coghlan wrote: > I'll hold off on marking the PEP as rejected until you've add a chance > to add (and Guido has had a chance to read) those examples. Thanks very much. -- Greg From pyideas at rebertia.com Thu Sep 15 07:30:58 2011 From: pyideas at rebertia.com (Chris Rebert) Date: Wed, 14 Sep 2011 22:30:58 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E717D8E.60103@canterbury.ac.nz> References: <4E717D8E.60103@canterbury.ac.nz> Message-ID: On Wed, Sep 14, 2011 at 9:22 PM, Greg Ewing wrote: > NumPy is not the only use case by a long shot. The main reason > I want it is for expressing database queries. You only have to > look at the contortions that things like Django have to go > through to wonder whether there's a better way. So, you want this in order to avoid (e.g.) `X & Y` and `not_(Z)`, in favor of `X and Y` and `not Z`? Doesn't seem like that big of a win in the `and` and `or` cases. Cheers, Chris From ncoghlan at gmail.com Thu Sep 15 08:00:09 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Sep 2011 16:00:09 +1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E717D8E.60103@canterbury.ac.nz> Message-ID: On Thu, Sep 15, 2011 at 3:30 PM, Chris Rebert wrote: > On Wed, Sep 14, 2011 at 9:22 PM, Greg Ewing wrote: > >> NumPy is not the only use case by a long shot. The main reason >> I want it is for expressing database queries. You only have to >> look at the contortions that things like Django have to go >> through to wonder whether there's a better way. > > So, you want this in order to avoid (e.g.) `X & Y` and `not_(Z)`, in > favor of `X and Y` and `not Z`? Doesn't seem like that big of a win in > the `and` and `or` cases. It's actually: 'x and y' vs 'x & y' 'x or y' vs 'x ^ y' 'not x' vs '~x' (assuming you aren't already using the bitwise versions for bitwise operations) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From pyideas at rebertia.com Thu Sep 15 08:06:22 2011 From: pyideas at rebertia.com (Chris Rebert) Date: Wed, 14 Sep 2011 23:06:22 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E717D8E.60103@canterbury.ac.nz> Message-ID: On Wed, Sep 14, 2011 at 11:00 PM, Nick Coghlan wrote: > On Thu, Sep 15, 2011 at 3:30 PM, Chris Rebert wrote: >> So, you want this in order to avoid (e.g.) `X & Y` and `not_(Z)`, in >> favor of `X and Y` and `not Z`? Doesn't seem like that big of a win in >> the `and` and `or` cases. > > It's actually: > > 'x and y' vs 'x & y' > 'x or y' vs 'x ^ y' Er, you mean 'x | y', right? > 'not x' vs '~x' Right, I forgot we had that operator! Cheers, Chris From g.brandl at gmx.net Thu Sep 15 08:56:21 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 15 Sep 2011 08:56:21 +0200 Subject: [Python-ideas] win32 extensions In-Reply-To: <4E7180F1.6080107@canterbury.ac.nz> References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> <4E713201.4090105@canterbury.ac.nz> <4E7180F1.6080107@canterbury.ac.nz> Message-ID: Am 15.09.2011 06:37, schrieb Greg Ewing: > On 15/09/11 16:15, David Townshend wrote: >> >> On Sep 15, 2011 1:00 AM, "Greg Ewing" > > wrote: >> > >> > We already have reasonably good third-party wrappers for >> > GUI facilities on the other platforms -- PyObjC on MacOSX, >> > and pygtk on Linux. We don't have anything equivalent for >> > recent Windows technologies, though. >> >> PyQt? > > That's not the same kind of thing, since it doesn't directly > expose the platform facilities. Gtk isn't any more a platform facility than Qt is. Georg From ncoghlan at gmail.com Thu Sep 15 09:12:42 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Sep 2011 17:12:42 +1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E717D8E.60103@canterbury.ac.nz> Message-ID: On Thu, Sep 15, 2011 at 4:06 PM, Chris Rebert wrote: > On Wed, Sep 14, 2011 at 11:00 PM, Nick Coghlan wrote: >> 'x or y' vs 'x ^ y' > > Er, you mean 'x | y', right? Uh, yeah. Oops :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From anacrolix at gmail.com Thu Sep 15 09:43:12 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Thu, 15 Sep 2011 17:43:12 +1000 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> <4E713201.4090105@canterbury.ac.nz> <4E7180F1.6080107@canterbury.ac.nz> Message-ID: It's a damn site better tho. On Sep 15, 2011 4:55 PM, "Georg Brandl" wrote: > > Am 15.09.2011 06:37, schrieb Greg Ewing: > > On 15/09/11 16:15, David Townshend wrote: > >> > >> On Sep 15, 2011 1:00 AM, "Greg Ewing" >> > wrote: > >> > > >> > We already have reasonably good third-party wrappers for > >> > GUI facilities on the other platforms -- PyObjC on MacOSX, > >> > and pygtk on Linux. We don't have anything equivalent for > >> > recent Windows technologies, though. > >> > >> PyQt? > > > > That's not the same kind of thing, since it doesn't directly > > expose the platform facilities. > > Gtk isn't any more a platform facility than Qt is. > > Georg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From sven at marnach.net Thu Sep 15 14:30:44 2011 From: sven at marnach.net (Sven Marnach) Date: Thu, 15 Sep 2011 13:30:44 +0100 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E70E231.2050607@mrabarnett.plus.com> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E70E231.2050607@mrabarnett.plus.com> Message-ID: <20110915123044.GA5268@pantoffel-wg.de> MRAB schrieb am Mi, 14. Sep 2011, um 18:19:45 +0100: > I had a use-case recently. > > I was looking for an entry in a pseudo-dict, where the key didn't have > to be an exact match, so I called the .get method on the pseudo-dict > until it returned a true-ish value (the value would never be false-ish). > > I just wrote a short function to do what I wanted. FWIW, I just came to python-ideas to propose the same change to any() for the same use case (look-up in a pseudo dict). I needed this several times now, and I would consider the change to any() worthwhile. (For consistency, all() would also have to be changed, though I didn't have a use case yet.) -- Sven From sven at marnach.net Thu Sep 15 14:40:22 2011 From: sven at marnach.net (Sven Marnach) Date: Thu, 15 Sep 2011 13:40:22 +0100 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> <4E70A821.5010007@stoneleaf.us> <4E7132EC.70506@pearwood.info> <4E713579.2030403@canterbury.ac.nz> Message-ID: <20110915124022.GB5268@pantoffel-wg.de> Terry Reedy schrieb am Mi, 14. Sep 2011, um 22:52:47 -0400: > That is what itertool is designed for. I believe > chain(dropwhile(...), [default]) will add a default. ... unless the default is falsy. -- Sven From alexander.belopolsky at gmail.com Thu Sep 15 15:40:50 2011 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 15 Sep 2011 09:40:50 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <20110915123044.GA5268@pantoffel-wg.de> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E70E231.2050607@mrabarnett.plus.com> <20110915123044.GA5268@pantoffel-wg.de> Message-ID: On Thu, Sep 15, 2011 at 8:30 AM, Sven Marnach wrote: > MRAB schrieb am Mi, 14. Sep 2011, um 18:19:45 +0100: >> I had a use-case recently. >> >> I was looking for an entry in a pseudo-dict, where the key didn't have >> to be an exact match, so I called the .get method on the pseudo-dict >> until it returned a true-ish value (the value would never be false-ish). >> >> I just wrote a short function to do what I wanted. > > FWIW, I just came to python-ideas to propose the same change to any() > for the same use case (look-up in a pseudo dict). ?I needed this > several times now, and I would consider the change to any() > worthwhile. ?(For consistency, all() would also have to be changed, > though I didn't have a use case yet.) As someone suggested earlier in this thread, this can be achieved with a very simple expression (not even a function!): next(filter(None, S)) will return the first item x from S with bool(x) evaluating to True. Further simplifying this is not worth slowing down all current uses of any(). From p.f.moore at gmail.com Thu Sep 15 15:56:30 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 15 Sep 2011 14:56:30 +0100 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FFC5B.8010704@nedbatchelder.com> <4E70E231.2050607@mrabarnett.plus.com> <20110915123044.GA5268@pantoffel-wg.de> Message-ID: On 15 September 2011 14:40, Alexander Belopolsky wrote: >> FWIW, I just came to python-ideas to propose the same change to any() >> for the same use case (look-up in a pseudo dict). ?I needed this >> several times now, and I would consider the change to any() >> worthwhile. ?(For consistency, all() would also have to be changed, >> though I didn't have a use case yet.) > > As someone suggested earlier in this thread, this can be achieved with > a very simple expression (not even a function!): > > next(filter(None, S)) > > will return the first item x from S with bool(x) evaluating to True. > Further simplifying this is not worth slowing down all current uses of > any(). And if the special-case behaviour of filter(None, ...) upsets you, a generator expression is just as good: next(x for x in S if x) That's simple and readable enough (to my mind). In fact, it's more obvious to me than any(S) returning a non-boolean result would be... Paul. From aquavitae69 at gmail.com Thu Sep 15 15:59:24 2011 From: aquavitae69 at gmail.com (David Townshend) Date: Thu, 15 Sep 2011 15:59:24 +0200 Subject: [Python-ideas] Object interface to path names In-Reply-To: <4E717717.5090401@canterbury.ac.nz> References: <4E717717.5090401@canterbury.ac.nz> Message-ID: On Thu, Sep 15, 2011 at 5:55 AM, Greg Ewing wrote: > On 14/09/11 03:03, David Townshend wrote: > > 3. Allow caching of file attribute data so that queries do not have to >> wait >> the disk or network to respond (although at the cost of accuracy). >> ... >> The third can be met be allowing all disk calls to be asynchonous: >> > > You're mixing up two completely different concepts here. Cacheing has > nothing to do with asynchronous calls; it's storing the result so that > you don't have to wait the *next* time you want the information. > > I'm not sure if I'm mixing up concepts or terminology. My meaning is, for example, if a method Path.files() is used to obtain a list of files in a directory, it would call os.listdir() in another thread and store the result to a cache. At the same time, the current contents of the cache are returned by Path.files(). Most of the time, the cache would be written to after its contents are returned by Path.files(), so the actual value returned would be inaccurate, but would be more accurate on the next call. To me, this means caching the result of an asynchronous call to os.listdir(). I suspect, however, that I'm not using the term "asynchronous" as it normally refers to disk operations, so that's probably the confusion. > Both could be useful, but only for certain applications, and they > should both be off by default in any general-purpose impelentation. > > Agreed! -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Sep 15 16:43:52 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Sep 2011 07:43:52 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E6FF971.7070504@canterbury.ac.nz> References: <4E6FF971.7070504@canterbury.ac.nz> Message-ID: On Tue, Sep 13, 2011 at 5:46 PM, Greg Ewing wrote: > Guido van Rossum wrote: >> >> Control flow just *is* special. FWIW It is not >> bool values that are being treated special; it is certain bool >> operators (not, and, or). I see nothing wrong with that; you can think >> of them as an extension of the repertoire of 'if', 'while' etc. > > Seems to me they're something of a mixture of computation > and control flow. The computation consists of calling the > __nonzero__ methods of the operands to find out whether > they should be considered true or false. > > PEP 335 doesn't change that, it just shifts the emphasis > slightly. It provides hooks allowing you to do more > computation before deciding on the control flow. Yeah, and the extra computation (however slight) bothers me. >> De Morgan's law (and similar transformation) seem to me out of scope. >> The would violate gut feelings about the "natural" execution order of >> Python expressions. > > I don't see how. If e.g. you transform > > ?if not (a and b): > > into > > ?if not a or not b: > > then the evaluation order doesn't change -- you still > evaluate a first, and then evaluate b if needed. It's > just an extension of the existing rewriting of 'not' > deeper into the expression. Fair enough, especially since the bytecode compiler already gladly interchanges JUMP_IF_FALSE and JUMP_IF_TRUE (swapping the targets). -- --Guido van Rossum (python.org/~guido) From guido at python.org Thu Sep 15 16:59:58 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Sep 2011 07:59:58 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E717D8E.60103@canterbury.ac.nz> References: <4E717D8E.60103@canterbury.ac.nz> Message-ID: On Wed, Sep 14, 2011 at 9:22 PM, Greg Ewing wrote: > On 14/09/11 05:56, Guido van Rossum wrote: > >> I think I just mentioned one (turning 'if not' into a jump). Anyway, >> I'm glad to reject the PEP for the reason that I like the status quo >> just fine. > > Did you see the comment I made concerning jump optimizations? > (Briefly, I don't think it would do any harm to keep performing > this optimization, since it wouldn't affect any of my intended > use cases.) > >> Also, after reading the PEP from beginning to end ... I think the >> offered API is too complicated to be of much use. > > There's a bit more machinery involved than there is with the > other operators, but I believe that any given use cases will > usually only *use* part of it. In particular, a system for > building parse trees only needs to implement the 2-operand > methods, which is no more complicated than overriding any of > the existing operators. > > If it would help, I could provide a couple of fully-worked > examples, so you can see how complicated (or not) it would be > to use in practice. > > There's also the possibility of simplifying the API if it's > considered that some of it will *never* be used, e.g. by > eliminating the machinery for custom short-circuiting. > >> Certainly the NumPy folks have >> repeatedly claimed that they are fine with the status quo. > > NumPy is not the only use case by a long shot. The main reason > I want it is for expressing database queries. You only have to > look at the contortions that things like Django have to go > through to wonder whether there's a better way. > > Apologies if I appear to be flogging a resting equine here. If > you really don't want to think about it any more, I'll shut up. > But if it's going to be rejected, I'd prefer it to be rejected > for genuine reasons rather than FUD. Actually, accepting or rejecting a PEP is always a matter of gut feelings. Usually there's no hard argument that can be made about the benefits for the language (a few use cases are easier to write, some bugs are avoided) vs. the downsides (some users will be confused, some bugs will occur, docs have to be updated, the compiler and runtime will be a little more complex, the generated code may be a little bit bulkier, etc.). It's hard to weigh these, so please don't call it FUD, and be glad it's not your responsibility to decide. That said, having just implemented a query engine (http://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/query.py) I appreciate the pain for that particular use case and will hold off until I've seen your examples. -- --Guido van Rossum (python.org/~guido) From janssen at parc.com Thu Sep 15 19:13:46 2011 From: janssen at parc.com (Bill Janssen) Date: Thu, 15 Sep 2011 10:13:46 PDT Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FFC5B.8010704@nedbatchelder.com> <4E70E231.2050607@mrabarnett.plus.com> <20110915123044.GA5268@pantoffel-wg.de> Message-ID: <32523.1316106826@parc.com> Alexander Belopolsky wrote: > As someone suggested earlier in this thread, this can be achieved with > a very simple expression (not even a function!): > > next(filter(None, S)) > > will return the first item x from S with bool(x) evaluating to True. > Further simplifying this is not worth slowing down all current uses of > any(). Yeah, but you've got to think up that expression. I agree that slowing down any() is not a good thing, but there's surely a place for this functionality -- probably more useful than any() itself. I've wished several times that any() just returned the first hit. Personally, I'd add a function some() that either returns a matching value or raises an exception. Bill From g.brandl at gmx.net Thu Sep 15 19:19:22 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 15 Sep 2011 19:19:22 +0200 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> <4E713201.4090105@canterbury.ac.nz> <4E7180F1.6080107@canterbury.ac.nz> Message-ID: Not sure I understand what you mean by that... Georg Am 15.09.2011 09:43, schrieb Matt Joiner: > It's a damn site better tho. > On Sep 15, 2011 4:55 PM, "Georg Brandl" > wrote: >> >> Am 15.09.2011 06:37, schrieb Greg Ewing: >> > On 15/09/11 16:15, David Townshend wrote: >> >> >> >> On Sep 15, 2011 1:00 AM, "Greg Ewing" > >> >> >> wrote: >> >> > >> >> > We already have reasonably good third-party wrappers for >> >> > GUI facilities on the other platforms -- PyObjC on MacOSX, >> >> > and pygtk on Linux. We don't have anything equivalent for >> >> > recent Windows technologies, though. >> >> >> >> PyQt? >> > >> > That's not the same kind of thing, since it doesn't directly >> > expose the platform facilities. >> >> Gtk isn't any more a platform facility than Qt is. >> >> Georg >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org > >> http://mail.python.org/mailman/listinfo/python-ideas > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From guido at python.org Thu Sep 15 19:45:40 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Sep 2011 10:45:40 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <32523.1316106826@parc.com> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E70E231.2050607@mrabarnett.plus.com> <20110915123044.GA5268@pantoffel-wg.de> <32523.1316106826@parc.com> Message-ID: On Thu, Sep 15, 2011 at 10:13 AM, Bill Janssen wrote: > Alexander Belopolsky wrote: > >> As someone suggested earlier in this thread, this can be achieved with >> a very simple expression (not even a function!): >> >> next(filter(None, S)) >> >> will return the first item x from S with bool(x) evaluating to True. >> Further simplifying this is not worth slowing down all current uses of >> any(). > > Yeah, but you've got to think up that expression. ?I agree that slowing > down any() is not a good thing, but there's surely a place for this > functionality -- probably more useful than any() itself. ?I've wished > several times that any() just returned the first hit. ?Personally, I'd > add a function some() that either returns a matching value or raises an > exception. I recall we discussed this, but I don't recall the exact reasons why we (I?) decided that any() and all() should always return True/False. I can guess that it was because of consistency with the return values for any([]) and all([]), which have to be False and True, respectively. Considering the use cases, and Python's pervasive use of any-object-as-Boolean, I think it would have been fine if they had been defined as follows: def any(xs): x = False for x in xs: if x: break return x def all(xs): x = True for x in xs: if not x: break return x (Relying on two Pythonic properties of the for-loop: if the sequence xs is empty, x is not set by the for-loop at all, and keeps its previous value; if xs is not empty and the for-loop runs until xs is exhausted, x keeps the last value assigned to it in the loop.) I am not so sure that it is safe to change them now, however. Maybe these upgraded versions can be added to some other module, e.g. functools or itertools? -- --Guido van Rossum (python.org/~guido) From alexander.belopolsky at gmail.com Thu Sep 15 19:55:31 2011 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 15 Sep 2011 13:55:31 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FFC5B.8010704@nedbatchelder.com> <4E70E231.2050607@mrabarnett.plus.com> <20110915123044.GA5268@pantoffel-wg.de> <32523.1316106826@parc.com> Message-ID: On Thu, Sep 15, 2011 at 1:45 PM, Guido van Rossum wrote: .. > Considering the use cases, and Python's pervasive use of > any-object-as-Boolean, I think it would have been fine if they had > been defined as follows: > > def any(xs): > ?x = False > ?for x in xs: > ? ?if x: break > ?return x Did you consider that in a typical "if any(S):" construct, x.__bool__() will be called twice on the found object? It is not unheard of to have expensive __bool__(). For example in a vector library a vector may be considered "false" if all its components are zero. From guido at python.org Thu Sep 15 19:58:48 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Sep 2011 10:58:48 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FFC5B.8010704@nedbatchelder.com> <4E70E231.2050607@mrabarnett.plus.com> <20110915123044.GA5268@pantoffel-wg.de> <32523.1316106826@parc.com> Message-ID: On Thu, Sep 15, 2011 at 10:55 AM, Alexander Belopolsky wrote: > On Thu, Sep 15, 2011 at 1:45 PM, Guido van Rossum wrote: > .. >> Considering the use cases, and Python's pervasive use of >> any-object-as-Boolean, I think it would have been fine if they had >> been defined as follows: >> >> def any(xs): >> ?x = False >> ?for x in xs: >> ? ?if x: break >> ?return x > > Did you consider that in a typical "if any(S):" construct, > x.__bool__() will be called twice on the found object? ?It is not > unheard of to have expensive __bool__(). ?For example in a vector > library a vector may be considered "false" if all its components are > zero. I hadn't just now, but it is possible that this was part of the original reasoning for always returning True or False. -- --Guido van Rossum (python.org/~guido) From ethan at stoneleaf.us Thu Sep 15 20:40:40 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 15 Sep 2011 11:40:40 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FFC5B.8010704@nedbatchelder.com> <4E70E231.2050607@mrabarnett.plus.com> <20110915123044.GA5268@pantoffel-wg.de> <32523.1316106826@parc.com> Message-ID: <4E7246A8.7090507@stoneleaf.us> Alexander Belopolsky wrote: > On Thu, Sep 15, 2011 at 1:45 PM, Guido van Rossum wrote: > .. >> Considering the use cases, and Python's pervasive use of >> any-object-as-Boolean, I think it would have been fine if they had >> been defined as follows: >> >> def any(xs): >> x = False >> for x in xs: >> if x: break >> return x > > Did you consider that in a typical "if any(S):" construct, > x.__bool__() will be called twice on the found object? It is not > unheard of to have expensive __bool__(). For example in a vector > library a vector may be considered "false" if all its components are > zero. If the calculation is expensive, shouldn't the vector cache the result for subsequent access? ~Ethan~ From tjreedy at udel.edu Thu Sep 15 21:40:02 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 15 Sep 2011 15:40:02 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <20110915124022.GB5268@pantoffel-wg.de> References: <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> <4E70A821.5010007@stoneleaf.us> <4E7132EC.70506@pearwood.info> <4E713579.2030403@canterbury.ac.nz> <20110915124022.GB5268@pantoffel-wg.de> Message-ID: On 9/15/2011 8:40 AM, Sven Marnach wrote: > Terry Reedy schrieb am Mi, 14. Sep 2011, um 22:52:47 -0400: >> That is what itertool is designed for. I believe >> chain(dropwhile(...), [default]) will add a default. > > ... unless the default is falsy. Out of context nonsense. In from itertools import dropwhile def _not(x): return not x def first(iterable): return next(dropwhile(_not, iterable)) replacing 'dropwhile(_not, iterable)' with 'chain(dropwhile(_not,iterable),[default]' will *always* return default if dropwhile is empty *and* guarantee that there is a first to be returned instead of raising StopIteration, which should otherwise be trapped, as StopIteration should only be passed out by iterators. Actually, I would probably not bother with chain and instead write def first(iterable,default): try: return next(dropwhile(_not, iterable)) except StopIteration: return default -- Terry Jan Reedy From ncoghlan at gmail.com Fri Sep 16 00:25:13 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Sep 2011 08:25:13 +1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E725D98.2090809@trueblade.com> References: <4E717D8E.60103@canterbury.ac.nz> <4E725D98.2090809@trueblade.com> Message-ID: On Fri, Sep 16, 2011 at 6:18 AM, Eric V. Smith wrote: > On 9/15/2011 2:00 AM, Nick Coghlan wrote: >> On Thu, Sep 15, 2011 at 3:30 PM, Chris Rebert wrote: >>> On Wed, Sep 14, 2011 at 9:22 PM, Greg Ewing wrote: >>> >>>> NumPy is not the only use case by a long shot. The main reason >>>> I want it is for expressing database queries. You only have to >>>> look at the contortions that things like Django have to go >>>> through to wonder whether there's a better way. >>> >>> So, you want this in order to avoid (e.g.) `X & Y` and `not_(Z)`, in >>> favor of `X and Y` and `not Z`? Doesn't seem like that big of a win in >>> the `and` and `or` cases. >> >> It's actually: >> >> 'x and y' vs 'x & y' >> 'x or y' vs 'x ^ y' >> 'not x' vs '~x' > > The problem with this is the precedence difference between '&' and > 'and', etc. > > sqlalchemy uses this approach, and my code contains many extra sets of > unnatural looking parens to deal with this issue. It's not a huge deal, > but I see newbies (and me, and oldie) leaving off the parens. It's not > an easy bug to track down. Yeah, Greg does mention that in the PEP. Some concrete examples showing: - buggy code with bitwise operator overloading - correct code with parens used to correct bitwise precedence disparity - correct code with boolean logic overloading (as proposed by the PEP) would help make it crystal clear. Those would also help decide how important it is to preserve the short-circuiting semantics when overloading the operations - element-wise and DSL type use cases are always going to need the RHS, so there may be an opportunity there to simplify the proposed semantics to match those of other binary operations. And, of course, the final option is to go the C/C++ route: leave and/or/not as short-circuiting flow control statements and introduce && and || as low precedence binary logical operators. (I can't see Guido ever going for that idea, but it *does* sidestep the concerns about currently legal control flow transformations) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From greg.ewing at canterbury.ac.nz Fri Sep 16 01:57:26 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Sep 2011 11:57:26 +1200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E717D8E.60103@canterbury.ac.nz> Message-ID: <4E7290E6.2090903@canterbury.ac.nz> Chris Rebert wrote: > So, you want this in order to avoid (e.g.) `X & Y` and `not_(Z)`, in > favor of `X and Y` and `not Z`? Yes. The problem with & and | is that their precedence is all wrong in relation to the comparison operators. So instead of a == 17 and b == 42 you have to write (a == 17) & (b == 42) And not_(x) is just plain ugly. -- Greg From greg.ewing at canterbury.ac.nz Fri Sep 16 02:06:45 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Sep 2011 12:06:45 +1200 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> <4E713201.4090105@canterbury.ac.nz> <4E7180F1.6080107@canterbury.ac.nz> Message-ID: <4E729315.2060303@canterbury.ac.nz> Georg Brandl wrote: > Gtk isn't any more a platform facility than Qt is. It's the closest thing we have to a platform-native GUI facility in the Linux world, though. It's not the *only* one -- there are others, such as KDE. But if you work in a Gnome environment, all the system stuff is built on it, and it defines the look and feel that you expect apps to have. So it occupies much the same place that Cocoa does on MacOSX. -- Greg From greg.ewing at canterbury.ac.nz Fri Sep 16 02:34:59 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Sep 2011 12:34:59 +1200 Subject: [Python-ideas] Object interface to path names In-Reply-To: References: <4E717717.5090401@canterbury.ac.nz> Message-ID: <4E7299B3.7080008@canterbury.ac.nz> David Townshend wrote: > I suspect, however, that I'm not using the term > "asynchronous" as it normally refers to disk operations, It sounds like you're using a rather application-dependent combination of asynchronous I/O and cacheing. The way asynchronous I/O is normally used is that you start the operation, go away and do something else, and either come back later to check whether it's finished or arrange some kind of callback when it finishes. Most applications would not be tolerant of inaccurate results. -- Greg From greg.ewing at canterbury.ac.nz Fri Sep 16 02:39:10 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Sep 2011 12:39:10 +1200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FF971.7070504@canterbury.ac.nz> Message-ID: <4E729AAE.2040204@canterbury.ac.nz> Guido van Rossum wrote: > On Tue, Sep 13, 2011 at 5:46 PM, Greg Ewing wrote: > >>PEP 335 doesn't change that, it just shifts the emphasis >>slightly. It provides hooks allowing you to do more >>computation before deciding on the control flow. > > Yeah, and the extra computation (however slight) bothers me. If the objects don't do anything special, the extra computation reduces to noticing that a particular type slot is NULL at the C level. -- Greg From ncoghlan at gmail.com Fri Sep 16 03:38:35 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Sep 2011 11:38:35 +1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E729AAE.2040204@canterbury.ac.nz> References: <4E6FF971.7070504@canterbury.ac.nz> <4E729AAE.2040204@canterbury.ac.nz> Message-ID: On Fri, Sep 16, 2011 at 10:39 AM, Greg Ewing wrote: > If the objects don't do anything special, the extra > computation reduces to noticing that a particular type > slot is NULL at the C level. abstract.c says "Hi!". The memory layout of type objects means the dance to handle binary operators correctly is generally a little more complicated than that. The short circuiting support in PEP 335 will only make it worse, since there would be two slots to check on each operand. In this particular case though, the additional problem is that the interpreter currently treats 'and' and 'or' as flow control structures, not operators: # python 3.2 >>> dis.dis("x & y") 1 0 LOAD_NAME 0 (x) 3 LOAD_NAME 1 (y) 6 BINARY_AND 7 RETURN_VALUE >>> dis.dis("x and y") 1 0 LOAD_NAME 0 (x) 3 JUMP_IF_FALSE_OR_POP 9 6 LOAD_NAME 1 (y) >> 9 RETURN_VALUE That means making it possible to overload them is actually a fairly deep change to the expression semantics as far as the compiler is concerned. The control flow would need to be something like: LOAD_NAME x JUMP_IF_LOGICAL_AND (<-- handle shortcircuiting cases) LOAD_NAME y LOGICAL_AND (<-- handle binary version, potentially doing stack twiddling to ditch LH operand if there are no slots to call) RETURN_VALUE That's why my suggestion to consider '&&' and '||' as alternate, lower precedence, spellings of '&' and '|' wasn't entirely in jest. Their overloading semantics could be identical to the existing bitwise operators, but their precedence would be just above that of the control flow operations. Having operators that are equivalent aside from subtle details of precedence would be odd, but it strikes me as being less problematic overall than messing with the control flow semantics of 'and' and 'or'. Reusing the '&' and '|' typeslots would also mean that existing libraries like SQLAlchemy that already use '&' and '|' for boolean logic would just work with the new operators. C/C++ programmers would get rather confused when they found out that '&&' and '||' on numeric types still resulted in bitwise operations, though. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Fri Sep 16 03:53:55 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Sep 2011 18:53:55 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FF971.7070504@canterbury.ac.nz> <4E729AAE.2040204@canterbury.ac.nz> Message-ID: But didn't Greg have a working patch? (For Python 2.3, but still...) On Thu, Sep 15, 2011 at 6:38 PM, Nick Coghlan wrote: > On Fri, Sep 16, 2011 at 10:39 AM, Greg Ewing > wrote: >> If the objects don't do anything special, the extra >> computation reduces to noticing that a particular type >> slot is NULL at the C level. > > abstract.c says "Hi!". The memory layout of type objects means the > dance to handle binary operators correctly is generally a little more > complicated than that. The short circuiting support in PEP 335 will > only make it worse, since there would be two slots to check on each > operand. > > In this particular case though, the additional problem is that the > interpreter currently treats 'and' and 'or' as flow control > structures, not operators: > > # python 3.2 >>>> dis.dis("x & y") > ?1 ? ? ? ? ? 0 LOAD_NAME ? ? ? ? ? ? ? ?0 (x) > ? ? ? ? ? ? ?3 LOAD_NAME ? ? ? ? ? ? ? ?1 (y) > ? ? ? ? ? ? ?6 BINARY_AND > ? ? ? ? ? ? ?7 RETURN_VALUE >>>> dis.dis("x and y") > ?1 ? ? ? ? ? 0 LOAD_NAME ? ? ? ? ? ? ? ?0 (x) > ? ? ? ? ? ? ?3 JUMP_IF_FALSE_OR_POP ? ? 9 > ? ? ? ? ? ? ?6 LOAD_NAME ? ? ? ? ? ? ? ?1 (y) > ? ? ? ?>> ? ?9 RETURN_VALUE > > That means making it possible to overload them is actually a fairly > deep change to the expression semantics as far as the compiler is > concerned. The control flow would need to be something like: > > ?LOAD_NAME x > ?JUMP_IF_LOGICAL_AND (<-- handle shortcircuiting cases) > ?LOAD_NAME y > ?LOGICAL_AND (<-- handle binary version, potentially doing stack > twiddling to ditch LH operand if there are no slots to call) > ? RETURN_VALUE > > That's why my suggestion to consider '&&' and '||' as alternate, lower > precedence, spellings of '&' and '|' wasn't entirely in jest. Their > overloading semantics could be identical to the existing bitwise > operators, but their precedence would be just above that of the > control flow operations. > > Having operators that are equivalent aside from subtle details of > precedence would be odd, but it strikes me as being less problematic > overall than messing with the control flow semantics of 'and' and > 'or'. Reusing the '&' and '|' typeslots would also mean that existing > libraries like SQLAlchemy that already use '&' and '|' for boolean > logic would just work with the new operators. > > C/C++ programmers would get rather confused when they found out that > '&&' and '||' on numeric types still resulted in bitwise operations, > though. > > Cheers, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Fri Sep 16 05:14:24 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Sep 2011 13:14:24 +1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FF971.7070504@canterbury.ac.nz> <4E729AAE.2040204@canterbury.ac.nz> Message-ID: On Fri, Sep 16, 2011 at 11:53 AM, Guido van Rossum wrote: > But didn't Greg have a working patch? (For Python 2.3, but still...) It isn't that I think the code changes are going to be huge, just that even in the basic case they're not as trivial as "reduces to noticing that a particular type slot is NULL at the C level" makes them sound. The shift from 'control flow operation' to 'binary operator' is fairly substantial, even if the changes are isolated to a couple of new opcodes and the code generation for a couple of AST nodes. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From eric at trueblade.com Thu Sep 15 22:18:32 2011 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 15 Sep 2011 16:18:32 -0400 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E717D8E.60103@canterbury.ac.nz> Message-ID: <4E725D98.2090809@trueblade.com> On 9/15/2011 2:00 AM, Nick Coghlan wrote: > On Thu, Sep 15, 2011 at 3:30 PM, Chris Rebert wrote: >> On Wed, Sep 14, 2011 at 9:22 PM, Greg Ewing wrote: >> >>> NumPy is not the only use case by a long shot. The main reason >>> I want it is for expressing database queries. You only have to >>> look at the contortions that things like Django have to go >>> through to wonder whether there's a better way. >> >> So, you want this in order to avoid (e.g.) `X & Y` and `not_(Z)`, in >> favor of `X and Y` and `not Z`? Doesn't seem like that big of a win in >> the `and` and `or` cases. > > It's actually: > > 'x and y' vs 'x & y' > 'x or y' vs 'x ^ y' > 'not x' vs '~x' The problem with this is the precedence difference between '&' and 'and', etc. sqlalchemy uses this approach, and my code contains many extra sets of unnatural looking parens to deal with this issue. It's not a huge deal, but I see newbies (and me, and oldie) leaving off the parens. It's not an easy bug to track down. Eric. From guido at python.org Fri Sep 16 05:46:01 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Sep 2011 20:46:01 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FF971.7070504@canterbury.ac.nz> <4E729AAE.2040204@canterbury.ac.nz> Message-ID: On Thu, Sep 15, 2011 at 8:14 PM, Nick Coghlan wrote: > On Fri, Sep 16, 2011 at 11:53 AM, Guido van Rossum wrote: >> But didn't Greg have a working patch? (For Python 2.3, but still...) > > It isn't that I think the code changes are going to be huge, just that > even in the basic case they're not as trivial as "reduces to noticing > that a particular type slot is NULL at the C level" makes them sound. > The shift from 'control flow operation' to 'binary operator' is fairly > substantial, even if the changes are isolated to a couple of new > opcodes and the code generation for a couple of AST nodes. Yeah, I'd like to see the new generated code. If it's even a single opcode longer than the old code, that's a big cost that everybody pays -- decoding opcodes is a lot more expensive than checking a slot at the C level. I also worry that those 5 extra slots cost a lot of memory (and cache lines), given that a typical program has 1000s of type objects lying around. (If you don't believe me, instrument the type initialization function.) -- --Guido van Rossum (python.org/~guido) From steve at pearwood.info Fri Sep 16 05:57:55 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 16 Sep 2011 13:57:55 +1000 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E7173EF.4020701@canterbury.ac.nz> References: <4E6FFC5B.8010704@nedbatchelder.com> <4E7013F3.5080907@mrabarnett.plus.com> <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> <4E70A821.5010007@stoneleaf.us> <4E7132EC.70506@pearwood.info> <4E713579.2030403@canterbury.ac.nz> <4E7137D5.50206@mrabarnett.plus.com> <4E7173EF.4020701@canterbury.ac.nz> Message-ID: <4E72C943.7030307@pearwood.info> Greg Ewing wrote: > On 15/09/11 11:25, MRAB wrote: > >> Could it also have something like a 'key' argument, by default >> __bool__? > > Or more generally, a function. > > Although that's not strictly necessary. If there were a first() > function that simply returned the first item from an iterator, > whether true or not, one could write > > first(itertools.ifilter(func, something)) Surely the way to return the first item from an iterator, whether true or not, is next(iterator)? -- Steven From g.brandl at gmx.net Fri Sep 16 06:12:01 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 16 Sep 2011 06:12:01 +0200 Subject: [Python-ideas] win32 extensions In-Reply-To: <4E729315.2060303@canterbury.ac.nz> References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> <4E713201.4090105@canterbury.ac.nz> <4E7180F1.6080107@canterbury.ac.nz> <4E729315.2060303@canterbury.ac.nz> Message-ID: Am 16.09.2011 02:06, schrieb Greg Ewing: > Georg Brandl wrote: > >> Gtk isn't any more a platform facility than Qt is. > > It's the closest thing we have to a platform-native GUI > facility in the Linux world, though. It's not the *only* > one -- there are others, such as KDE. But if you work > in a Gnome environment, all the system stuff is built on > it, and it defines the look and feel that you expect apps > to have. So it occupies much the same place that Cocoa > does on MacOSX. And if you work in a KDE environment, all its stuff is built on Qt. I really don't see how the two are on different levels. Georg From g.brandl at gmx.net Fri Sep 16 06:15:19 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 16 Sep 2011 06:15:19 +0200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E706622.2070808@improva.dk> <4E70903C.5030301@nedbatchelder.com> <4E70A821.5010007@stoneleaf.us> <4E7132EC.70506@pearwood.info> <4E713579.2030403@canterbury.ac.nz> <20110915124022.GB5268@pantoffel-wg.de> Message-ID: Am 15.09.2011 21:40, schrieb Terry Reedy: > On 9/15/2011 8:40 AM, Sven Marnach wrote: >> Terry Reedy schrieb am Mi, 14. Sep 2011, um 22:52:47 -0400: >>> That is what itertool is designed for. I believe >>> chain(dropwhile(...), [default]) will add a default. >> >> ... unless the default is falsy. > > Out of context nonsense. In > > from itertools import dropwhile > def _not(x): return not x > > def first(iterable): > return next(dropwhile(_not, iterable)) > > replacing 'dropwhile(_not, iterable)' with > 'chain(dropwhile(_not,iterable),[default]' > will *always* return default if dropwhile is empty > *and* guarantee that there is a first to be returned > instead of raising StopIteration, which should otherwise be trapped, as > StopIteration should only be passed out by iterators. > Actually, I would probably not bother with chain and instead write > > def first(iterable,default): > try: > return next(dropwhile(_not, iterable)) > except StopIteration: > return default This is spelled next(dropwhile(_not, iterable), default) Georg From ron3200 at gmail.com Fri Sep 16 06:26:40 2011 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 15 Sep 2011 23:26:40 -0500 Subject: [Python-ideas] return value of yield expressions In-Reply-To: References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <1315934606.814.116.camel@Gutsy> <1315976466.2986.56.camel@Gutsy> Message-ID: <1316147200.11460.164.camel@Gutsy> On Wed, 2011-09-14 at 17:30 +1000, Nick Coghlan wrote: > On Wed, Sep 14, 2011 at 3:01 PM, Ron Adam wrote: > > On Wed, 2011-09-14 at 13:41 +1000, Nick Coghlan wrote: > >> On Wed, Sep 14, 2011 at 3:23 AM, Ron Adam wrote: > >> > If we could get the functions arguments when a function is done instead > >> > of getting the return value. (The modified function arguments object is > >> > the return value in this case.) > >> > > >> > fargs = &foo(a, b, c, d=4) > >> > >> This use case is covered by the bind() method in PEP 362 [1]: > >> > >> foo_signature = inspect.signature(foo) > >> fargs = foo_signature.bind(a, b, c, d=4) > >> > >> [1] http://www.python.org/dev/peps/pep-0362/#signature-object > > > > It doesn't quite do the same thing, and it's not nearly as easy to use. > > And magic syntax is better? > This is an incredibly niche use case, so > there is zero justification for giving it special syntax when > functions, methods and classes will do the job just fine (it's also > worth keeping in mind that Guido hasn't even officially given PEP 362 > itself the nod at this point) I don't know about it being a niche use case. How many time is (*args, **kwds) used to pass the signature data forward or back? I think this might be a lateral issue that could compliment Pep 362. Thinking about it, I am able (with a bit of work) get the final signature data mapping from within a function and return it. def xupdate_map(map1, map2): """ Only update values in map1 from map2. """ for k in map1.keys(): map1[k] = map2[k] def foo(v, count=2): sig_vals = locals() # Do stuff that may change v and/or count. for n = range(count): v += 1 return xupdate_map(sig_vals, locals()) That's pretty limiting and I can't call foo again with the result. It's also not very efficient either. I need to play with this a bit more before I can really explain what I seem to be thinking of. If I get it pinned down with some real world examples of how it would help, then maybe I'll post it here later. The general idea is to be able to use the signature mapping "such as what bind returns" in a more direct way with more freedom than *args, and **kwds allows. Cheers, Ron From amauryfa at gmail.com Fri Sep 16 08:16:58 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Fri, 16 Sep 2011 08:16:58 +0200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FFC5B.8010704@nedbatchelder.com> <4E70E231.2050607@mrabarnett.plus.com> <20110915123044.GA5268@pantoffel-wg.de> <32523.1316106826@parc.com> Message-ID: Hi, Le 15 sept. 2011 19:55, "Alexander Belopolsky" < alexander.belopolsky at gmail.com> a ?crit : > Did you consider that in a typical "if any(S):" construct, > x.__bool__() will be called twice on the found object? It is not > unheard of to have expensive __bool__(). For example in a vector > library a vector may be considered "false" if all its components are > zero It is the same for "a or b", even when a is true. -- Amaury -------------- next part -------------- An HTML attachment was scrubbed... URL: From anacrolix at gmail.com Fri Sep 16 10:25:12 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Fri, 16 Sep 2011 18:25:12 +1000 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> <4E713201.4090105@canterbury.ac.nz> <4E7180F1.6080107@canterbury.ac.nz> <4E729315.2060303@canterbury.ac.nz> Message-ID: Well technically not. KDE mostly supplements Qt, A few objects here and there make use of some Qt features. The gobject, glib combination are bound up in numerous low level system services and supplementary libraries that have nothing to do do with GUIs, even I think on a KDE system. That said, if system GUI capable APIs are to be exposed on the standard library, those should be Win32, Cocoa, and GTK+. On Sep 16, 2011 2:11 PM, "Georg Brandl" wrote: > Am 16.09.2011 02:06, schrieb Greg Ewing: >> Georg Brandl wrote: >> >>> Gtk isn't any more a platform facility than Qt is. >> >> It's the closest thing we have to a platform-native GUI >> facility in the Linux world, though. It's not the *only* >> one -- there are others, such as KDE. But if you work >> in a Gnome environment, all the system stuff is built on >> it, and it defines the look and feel that you expect apps >> to have. So it occupies much the same place that Cocoa >> does on MacOSX. > > And if you work in a KDE environment, all its stuff is built on > Qt. I really don't see how the two are on different levels. > > Georg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Sep 16 12:32:09 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Sep 2011 20:32:09 +1000 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> <4E713201.4090105@canterbury.ac.nz> <4E7180F1.6080107@canterbury.ac.nz> <4E729315.2060303@canterbury.ac.nz> Message-ID: On Fri, Sep 16, 2011 at 6:25 PM, Matt Joiner wrote: > Well technically not. KDE mostly supplements Qt, A few objects here and > there make use of some Qt features. The gobject, glib combination are bound > up in numerous low level system services and supplementary libraries that > have nothing to do do with GUIs, even I think on a KDE system. The freedesktop.org standard APIs are a mix of services that originated in various parts of the Linux software stack. Some originate from x.org, some from Gnome/Gtk, some from KDE/Qt. I've yet to see a Gtk based app that feels native on KDE, mainly because the Gtk file dialogs are so atrocious and fd.org has yet to standardise things to the point where the apps designed for each environment can invoke each other's system dialogs. Regardless, there are going to be limits to what the standard library can realistically keep up to date with when it comes to platform-specific support. The cutting edge for the various platforms is always going to require third party libraries. It may be that there are stable subsets of functionality that *could* be included in the standard library (there are certainly plenty of *nix specific wrappers for system APIs), but the existence of those are predicated on interested parties stepping up to maintain them in accordance with python-dev policies. And really, it doesn't gain all that much - if you're targeting a specific platform, you're probably going to want the up-to-date platform specific libraries *anyway*. If you don't, then the cross-platform stuff in the core will likely suffice. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From greg.ewing at canterbury.ac.nz Fri Sep 16 15:05:58 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 17 Sep 2011 01:05:58 +1200 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> <4E713201.4090105@canterbury.ac.nz> <4E7180F1.6080107@canterbury.ac.nz> <4E729315.2060303@canterbury.ac.nz> Message-ID: <4E7349B6.2080602@canterbury.ac.nz> Georg Brandl wrote: > And if you work in a KDE environment, all its stuff is built on > Qt. I really don't see how the two are on different levels. I'm not necessarily saying that they are, only that gtk as it exists on Linux is at no *higher* a level than Cocoa or the win32 api are on their respective platforms, and that there is nothing else on Linux that is any more native. -- Greg From g.brandl at gmx.net Fri Sep 16 15:23:03 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 16 Sep 2011 15:23:03 +0200 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> <4E713201.4090105@canterbury.ac.nz> <4E7180F1.6080107@canterbury.ac.nz> <4E729315.2060303@canterbury.ac.nz> Message-ID: Am 16.09.2011 10:25, schrieb Matt Joiner: > Well technically not. KDE mostly supplements Qt, A few objects here and there > make use of some Qt features. This is not quite true: just grep for "Q" through the KDE headers. Most objects there are QObjects, and the basic signal-slot mechanism comes from Qt, as well. But I will stop here, since it's not the point of the discussion. It's just not true that Gtk is "the" blessed toolkit for X11. (Just like Vim is not the one true editor :) Georg From sven at marnach.net Fri Sep 16 15:30:23 2011 From: sven at marnach.net (Sven Marnach) Date: Fri, 16 Sep 2011 14:30:23 +0100 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FFC5B.8010704@nedbatchelder.com> <4E70E231.2050607@mrabarnett.plus.com> <20110915123044.GA5268@pantoffel-wg.de> <32523.1316106826@parc.com> Message-ID: <20110916133023.GB5335@pantoffel-wg.de> Amaury Forgeot d'Arc schrieb am Fr, 16. Sep 2011, um 08:16:58 +0200: > Le 15 sept. 2011 19:55, "Alexander Belopolsky" > <[1]alexander.belopolsky at gmail.com> a ?crit : > > Did you consider that in a typical "if any(S):" construct, > > x.__bool__() will be called twice on the found object? It is not > > unheard of to have expensive __bool__(). For example in a vector > > library a vector may be considered "false" if all its components are > > zero > > It is the same for "a or b", even when a is true. It's not the same -- the statement "if a or b:" will check the truth values of "a" and "b" at most once in CPython, while an updated version of "any()" would check the truth value of the found object twice -- since "any()" is an ordinary function, it can't easily be special-cased in the compiler. This seems like a good reason to make "any()" behave differently than "or" (apart from this behaviour being the current status quo anyway). -- Sven From guido at python.org Fri Sep 16 16:40:03 2011 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Sep 2011 07:40:03 -0700 Subject: [Python-ideas] return value of yield expressions In-Reply-To: <1316147200.11460.164.camel@Gutsy> References: <512ddde5-1fda-4acb-b752-6adddbd3ffe5@m3g2000pri.googlegroups.com> <4E6F3838.5030701@improva.dk> <1315934606.814.116.camel@Gutsy> <1315976466.2986.56.camel@Gutsy> <1316147200.11460.164.camel@Gutsy> Message-ID: On Thu, Sep 15, 2011 at 9:26 PM, Ron Adam wrote: > On Wed, 2011-09-14 at 17:30 +1000, Nick Coghlan wrote: >> On Wed, Sep 14, 2011 at 3:01 PM, Ron Adam wrote: >> > On Wed, 2011-09-14 at 13:41 +1000, Nick Coghlan wrote: >> >> On Wed, Sep 14, 2011 at 3:23 AM, Ron Adam wrote: >> >> > If we could get the functions arguments when a function is done instead >> >> > of getting the return value. ?(The modified function arguments object is >> >> > the return value in this case.) >> >> > >> >> > ? ? ?fargs = &foo(a, b, c, d=4) >> >> >> >> This use case is covered by the bind() method in PEP 362 [1]: >> >> >> >> ? ? foo_signature = inspect.signature(foo) >> >> ? ? fargs = foo_signature.bind(a, b, c, d=4) >> >> >> >> [1] http://www.python.org/dev/peps/pep-0362/#signature-object >> > >> > It doesn't quite do the same thing, and it's not nearly as easy to use. >> >> And magic syntax is better? >> This is an incredibly niche use case, so >> there is zero justification for giving it special syntax when >> functions, methods and classes will do the job just fine (it's also >> worth keeping in mind that Guido hasn't even officially given PEP 362 >> itself the nod at this point) > > I don't know about it being a niche use case. ? How many time is (*args, > **kwds) used to pass the signature data forward or back? > > I think this might be a lateral issue that could compliment Pep 362. > > > Thinking about it, I am able (with a bit of work) get the final > signature data mapping from within a function and return it. > > def xupdate_map(map1, map2): > ? ?""" Only update values in map1 from map2. """ > ? ?for k in map1.keys(): > ? ? ? ?map1[k] = map2[k] > > > def foo(v, count=2): > ? ?sig_vals = locals() > > ? ?# Do stuff that may change v and/or count. > ? ?for n = range(count): > ? ? ? ?v += 1 > > ? ?return xupdate_map(sig_vals, locals()) > > > That's pretty limiting and I can't call foo again with the result. ?It's > also not very efficient either. > > > I need to play with this a bit more before I can really explain what I > seem to be thinking of. ?If I get it pinned down with some real world > examples of how it would help, then maybe I'll post it here later. > > The general idea is to be able to use the signature mapping "such as > what bind returns" in a more direct way with more freedom than *args, > and **kwds allows. I think you're on the wrong path. Structured data ought to be represented as a class instance, or some other suitable data structure, not as an argument list to a function. Otherwise before you know it you will have reinvented namedtuple. I feel pretty strongly that adding syntax so that you can unpack an expression the way function arguments are unpacked is a bad way to evolve Python. -- --Guido van Rossum (python.org/~guido) From ron3200 at gmail.com Fri Sep 16 23:30:47 2011 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 16 Sep 2011 16:30:47 -0500 Subject: [Python-ideas] Before and after the colon in funciton defs. Message-ID: <1316208647.15431.105.camel@Gutsy> (This idea may have been suggested before, because it seems too obvious to me.) How about if we remove the requirement that the colon be on the same line as the function name. And, what if the colon is then used to split the difference between the definition time, and call time code in function definitions. So that every thing before the colon is done at definition time. Everything after the colon is done at call time. def foo(...): """ doc string """ Then could become ... def foo(...) """ doc string """ # foo.__doc__ = """ doc string """ : I think this represents what is actually happening a bit better. One possibility for define time code is to have decorators listed that read in the order they are applied instead of bottom up. def foo(n) """ function to be decorated. """ @deco1 # foo = deco1(foo) The '@' notation still works. @deco2 # foo = deco2(foo) : Note, that putting the doc string after the decorators may be better as it would put the doc string on the decorated function instead of the original. I think there may be more things possible with this idea than the simple cases above. Cheers, Ron From ncoghlan at gmail.com Sat Sep 17 03:01:13 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 17 Sep 2011 11:01:13 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <1316208647.15431.105.camel@Gutsy> References: <1316208647.15431.105.camel@Gutsy> Message-ID: On Sat, Sep 17, 2011 at 7:30 AM, Ron Adam wrote: > (This idea may have been suggested before, because it seems too obvious > to me.) > > How about if we remove the requirement that the colon be on the same > line as the function name. I don't see any benefit in offering a second way to spell docstrings or decorators (the existing spellings are easy enough to remember), but this suggestion did prompt an idea for potentially replacing uses of the default argument hack: allow a sequence of 'implicit locals' to be defined within square brackets after the parameter list. (This idea may have come up before, but if it has, I don't recall the arguments against it). Running through the 3 at least arguably legitimate use cases for the default argument hack: Micro-optimisation: # Current Python (internal to functools.lru_cache with default argument hack) def decorating_function(user_function, tuple=tuple, sorted=sorted, len=len, KeyError=KeyError): ... # 60 line function # Proposal - explicitly implicit locals to clarify real calling signature def decorating_function(user_function) [tuple=tuple, sorted=sorted, len=len, KeyError=KeyError)]: ... # 60 line function (Note: in practice, the lru_cache usage of the default argument hack doesn't really impact introspection, since the function is short-lived - it is the decorating function returned by the lru_cache decorator factory and hence only exists briefly as part of the function definition. The signatures of lru_cache itself and of the decorated function are not affected. I still like it as an example, since it isn't hypothetical - the standard library really uses the default argument feature this way). Early binding in a loop: # Current Python adders = [] for i in range(10): def f(x, _i=i): return x + _i adders.append(f) # Proposal adders = [] for i in range(10): def f(x) [i=i]: # Real calling signature is clear return x + i adders.append(f) Algorithmic shared state (without a class): # Current Python def f(*, _cache=[]): # Consenting adults, please don't override _cache when calling! # Proposal def f() [_cache=[]]: # Real calling signature is clear # _cache is an implicit local, not a keyword-only argument, so its safe To prevent grammar ambiguity, the list of implicit locals would most likely need to appear before the annotation for the function return type. The syntax within the list would be the same as those for arguments with default values (while I don't have a use case for allowing annotations, I don't have any reason to prevent them either, so reusing the same grammar fragment seems like reasonable idea) The implementation would also be basically the same as for arguments with default values (a list of names stored as an attribute on the code object and a dictionary on the function object to populate them) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ron3200 at gmail.com Sat Sep 17 07:47:26 2011 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 17 Sep 2011 00:47:26 -0500 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> Message-ID: <1316238446.17476.90.camel@Gutsy> On Sat, 2011-09-17 at 11:01 +1000, Nick Coghlan wrote: > On Sat, Sep 17, 2011 at 7:30 AM, Ron Adam wrote: > > (This idea may have been suggested before, because it seems too obvious > > to me.) > > > > How about if we remove the requirement that the colon be on the same > > line as the function name. > > I don't see any benefit in offering a second way to spell docstrings > or decorators (the existing spellings are easy enough to remember), > but this suggestion did prompt an idea for potentially replacing uses > of the default argument hack: allow a sequence of 'implicit locals' to > be defined within square brackets after the parameter list. (This idea > may have come up before, but if it has, I don't recall the arguments > against it). I almost included an example similar to this. Using your adder example, it would have looked something like this. adders = [] for i in range(10): def f(x) # Real calling signature is clear f.initial_values(i=i) # done at def time : return x + i adders.append(f) Of course there is no way to preinitialize a functions name space. Your [i=i] notation would do something like that behind the scenes. I was thinking it would be good if the parts in front of the colon were valid python statements except in the case of the already special docstrings and decorators. It could be limited to only valid python commands. If it turns out that a particular command becomes very common, a special syntax could be considered. Having the colon on a separate line would be optional, so it wouldn't change existing code at all. I tried to find a way to use a decorator to do this and didn't find anything that worked nicely. It can be done nicely with class's, but the point of using a function is it would be more efficient. I think this also starts to get into the area of meta programming. In python library there are a number of functions of the form. def foo(*args, **kwds): return _foo(*args, **kwds) I presume there is a reason why they didn't just do foo = _foo. currently you can swap out the __code__ part of a function, but not the signature part. If you could, then this might be possible. def foo() # Using _foo's signature. foo.signature = _foo.signature foo.__code__ = _foo.__code__ : pass # using _foo.__code__ Cheers, Ron From ncoghlan at gmail.com Sat Sep 17 08:16:19 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 17 Sep 2011 16:16:19 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <1316238446.17476.90.camel@Gutsy> References: <1316208647.15431.105.camel@Gutsy> <1316238446.17476.90.camel@Gutsy> Message-ID: On Sat, Sep 17, 2011 at 3:47 PM, Ron Adam wrote: > I was thinking it would be good if the parts in front of the colon were > valid python statements except in the case of the already special > docstrings and decorators. You did say that, but advanced no arguments in favour of breaking with the language aesthetic that way. Indented suites are introduced by a colon on the preceding header line and you'd need a really compelling argument to change that. "It might be nice" doesn't cut it. Specifically targeting existing uses of the default argument hack, on the other hand, comes with at least 3 already known use cases (i.e. name lookup micro-optimisation, early binding semantics and shared state for algorithms). People already tolerate messing with the function signature to obtain those behaviours, so it's reasonable to wonder if it is possible to provide a cleaner way to obtain the same effect. A simple set of definition time name bindings would likely suffice without greatly increasing the language complexity or the runtime overhead of function calls. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From cesare.di.mauro at gmail.com Sat Sep 17 08:30:58 2011 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Sat, 17 Sep 2011 08:30:58 +0200 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> <4E713201.4090105@canterbury.ac.nz> <4E7180F1.6080107@canterbury.ac.nz> <4E729315.2060303@canterbury.ac.nz> Message-ID: 2011/9/16 Matt Joiner > Well technically not. KDE mostly supplements Qt, A few objects here and > there make use of some Qt features. The gobject, glib combination are bound > up in numerous low level system services and supplementary libraries that > have nothing to do do with GUIs, even I think on a KDE system. > > That said, if system GUI capable APIs are to be exposed on the standard > library, those should be Win32, Cocoa, and GTK+ > And .NET too, since it's a Windows technology which you'll find on any Windows >= Vista (but also installable on XP). It doesn't mean that we necessarily need to wrap every single win32 and/or .NET API into a Python package on the stdlib. ctypes is useful to access whatever you want (win32 and any DLL, OS X dynamic libraries, Linux .so). Python for .NET can do the same for .NET, and is small enough to be made available on the stdlib. I think that something similar can be provided for OS X and Linux through Mono. Regards, Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at gmail.com Sat Sep 17 13:26:10 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Sat, 17 Sep 2011 12:26:10 +0100 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> <4E713201.4090105@canterbury.ac.nz> <4E7180F1.6080107@canterbury.ac.nz> <4E729315.2060303@canterbury.ac.nz> Message-ID: On 17 September 2011 07:30, Cesare Di Mauro wrote: > 2011/9/16 Matt Joiner > >> Well technically not. KDE mostly supplements Qt, A few objects here and >> there make use of some Qt features. The gobject, glib combination are bound >> up in numerous low level system services and supplementary libraries that >> have nothing to do do with GUIs, even I think on a KDE system. >> >> That said, if system GUI capable APIs are to be exposed on the standard >> library, those should be Win32, Cocoa, and GTK+ >> > And .NET too, since it's a Windows technology which you'll find on any > Windows >= Vista (but also installable on XP). > > It doesn't mean that we necessarily need to wrap every single win32 and/or > .NET API into a Python package on the stdlib. > > ctypes is useful to access whatever you want (win32 and any DLL, OS X > dynamic libraries, Linux .so). > > Python for .NET can do the same for .NET, and is small enough to be made > available on the stdlib. I think that something similar can be provided for > OS X and Linux through Mono. > > Python.NET is basically unmaintained, and requires very specialised knowledge to develop. I don't think it's in a position to move it into the standard library. For accessing .NET from Python IronPython is a much better alternative (with caveats about the use of C extensions that apply to any of the alternative implementations). All the best, Michael > Regards, > Cesare > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Sep 17 17:17:38 2011 From: guido at python.org (Guido van Rossum) Date: Sat, 17 Sep 2011 08:17:38 -0700 Subject: [Python-ideas] [Python-Dev] Persistent Python - a la Smalltalk In-Reply-To: References: Message-ID: [BCC python-dev, +python-ideas] Funny you should mention this. ABC, Python's predecessor, worked like this. However, it didn't work out very well. So, I'd say you're about 30 years too late with your idea... :-( --Guido On Sat, Sep 17, 2011 at 8:05 AM, Albert Zeyer wrote: > Hi, > > I was thinking about a persistent Python interpreter system. I.e. you > start a Python interpreter instance and you load and create all your > objects, classes and code in there (or load it in there from other > files). > > The basic idea is that you wont restart your Python script, you would > always modify it on-the-fly. Or a bit less extreme: You would at least > have the possibility with this to do this (like just doing minor > changes). Also, if your PC halts for whatever reason, you can continue > your Python script after a restart. > > This goes along my other recent proposal to store the AST of > statements in the related code objects > (http://thread.gmane.org/gmane.comp.python.devel/126754). An internal > editor could then edit this AST and recompile the code object. > > For the persistance, there would be an image file containing all the > Python objects. > > All in all, much like most Smalltalk systems. > > --- > > Has anyone done something like this already? > > --- > > There are a few implementation details which are not trivial and there > doesn't seem to be straight forward solutions, e.g. most generally: > > * How to implement the persistance? > * How to handle image compatibility between CPython updates? Even possible? > > Regards, > Albert > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Sat Sep 17 17:22:44 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 18 Sep 2011 01:22:44 +1000 Subject: [Python-ideas] [Python-Dev] Persistent Python - a la Smalltalk In-Reply-To: References: Message-ID: On Sun, Sep 18, 2011 at 1:17 AM, Guido van Rossum wrote: > [BCC python-dev, +python-ideas] > > Funny you should mention this. ABC, Python's predecessor, worked like > this. However, it didn't work out very well. So, I'd say you're about > 30 years too late with your idea... :-( Given the challenges of just getting pickle to work properly for explicit state transfer, I shudder at the prospect of trying to implicitly freeze the entire interpreter state... (and that's even before we get into the problem of programmers responding with "What do you mean restarting my script doesn't reinitialise all the state?") Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From sturla at molden.no Sat Sep 17 20:36:35 2011 From: sturla at molden.no (Sturla Molden) Date: Sat, 17 Sep 2011 20:36:35 +0200 Subject: [Python-ideas] [Python-Dev] Persistent Python - a la Smalltalk In-Reply-To: References: Message-ID: <4E74E8B3.40008@molden.no> Den 17.09.2011 17:22, skrev Nick Coghlan: > Given the challenges of just getting pickle to work properly for > explicit state transfer, I shudder at the prospect of trying to > implicitly freeze the entire interpreter state... (and that's even > before we get into the problem of programmers responding with "What do > you mean restarting my script doesn't reinitialise all the state?") > Cheers, Nick. I am not sure what he means. But if he e.g. wants to save an interactive session with the command prompt, a la Matlab, then ipython will do what he wants. Suspending and restarting a process on Unix is of course as easy as $ kill -s STOP [pid] $ kill -s CONT [pid] I don't know of any Windows equivalent. But virtual computers (e.g. VirtualBox, VMware) can be paused and restarted. Sturla From phd at phdru.name Sat Sep 17 21:35:56 2011 From: phd at phdru.name (Oleg Broytman) Date: Sat, 17 Sep 2011 23:35:56 +0400 Subject: [Python-ideas] [Python-Dev] Persistent Python - a la Smalltalk In-Reply-To: <4E74E8B3.40008@molden.no> References: <4E74E8B3.40008@molden.no> Message-ID: <20110917193556.GA25867@iskra.aviel.ru> On Sat, Sep 17, 2011 at 08:36:35PM +0200, Sturla Molden wrote: > Den 17.09.2011 17:22, skrev Nick Coghlan: > >Given the challenges of just getting pickle to work properly for > >explicit state transfer, I shudder at the prospect of trying to > >implicitly freeze the entire interpreter state... (and that's even > >before we get into the problem of programmers responding with > >"What do you mean restarting my script doesn't reinitialise all > >the state?") Cheers, Nick. > > I am not sure what he means. But if he e.g. wants to save > an interactive session with the command prompt, a la > Matlab, then ipython will do what he wants. > > Suspending and restarting a process on Unix is of course > as easy as > > $ kill -s STOP [pid] > $ kill -s CONT [pid] > > I don't know of any Windows equivalent. But virtual > computers (e.g. VirtualBox, VMware) can be paused and > restarted. The OP wants something like Emacs/TeX/Perl undump/unexec. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From arnodel at gmail.com Sat Sep 17 23:18:56 2011 From: arnodel at gmail.com (Arnaud Delobelle) Date: Sat, 17 Sep 2011 22:18:56 +0100 Subject: [Python-ideas] allow line break at operators In-Reply-To: <87fwkcortg.fsf@benfinney.id.au> References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <1313031175.38817.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> <1315092177.54288.YahooMailNeo@web121504.mail.ne1.yahoo.com> <4E62DCC6.8000906@mrabarnett.plus.com> <1315130288.86666.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E63BC83.9000401@mrabarnett.plus.com> <87liu4oseu.fsf@benfinney.id.au> <87fwkcortg.fsf@benfinney.id.au> Message-ID: On 4 September 2011 23:39, Ben Finney wrote: > Ben Finney writes: > >> MRAB writes: >> >> > As well as still limiting a comment to a line, I'd also still limit >> > a string literal (except a triple-quoted string literal) to a line. >> >> How many string literals do you count in the following statement? I >> count one: >> >> ? ? raise HoustonWeHaveAProblemError( >> ? ? ? ? "Lorem ipsum dolor sit amet," >> ? ? ? ? " consectetur adipiscing elit.") > > The Python compiler agrees with me: > > ? ?>>> import dis > ? ?>>> def foo(): > ? ?... ? ? raise ValueError( > ? ?... ? ? ? ? "Lorem ipsum dolor sit amet," > ? ?... ? ? ? ? " consectetur adipiscing elit.") > ? ?... > ? ?>>> dis.dis(foo) > ? ? ?2 ? ? ? ? ? 0 LOAD_GLOBAL ? ? ? ? ? ? ?0 (ValueError) > > ? ? ?3 ? ? ? ? ? 3 LOAD_CONST ? ? ? ? ? ? ? 1 ('Lorem ipsum dolor sit amet, consectetur adipiscing elit.') > ? ? ? ? ? ? ? ? ?6 CALL_FUNCTION ? ? ? ? ? ?1 > ? ? ? ? ? ? ? ? ?9 RAISE_VARARGS ? ? ? ? ? ?1 > ? ? ? ? ? ? ? ? 12 LOAD_CONST ? ? ? ? ? ? ? 0 (None) > ? ? ? ? ? ? ? ? 15 RETURN_VALUE The code object says that there's one string constant in the compiled function. It says nothing (and knows nothing) about the number of string literals that made up this string. In the following, how many string literals can you see? >>> def bar(): return "a" + "b" ... Now let's look at the code object: >>> dis.dis(bar) 1 0 LOAD_CONST 3 ('ab') 3 RETURN_VALUE -- Arnaud From grosser.meister.morti at gmx.net Sun Sep 18 02:12:34 2011 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Sun, 18 Sep 2011 02:12:34 +0200 Subject: [Python-ideas] [Python-Dev] Persistent Python - a la Smalltalk In-Reply-To: <20110917193556.GA25867@iskra.aviel.ru> References: <4E74E8B3.40008@molden.no> <20110917193556.GA25867@iskra.aviel.ru> Message-ID: <4E753772.1050003@gmx.net> On 09/17/2011 09:35 PM, Oleg Broytman wrote: > On Sat, Sep 17, 2011 at 08:36:35PM +0200, Sturla Molden wrote: >> Den 17.09.2011 17:22, skrev Nick Coghlan: >>> Given the challenges of just getting pickle to work properly for >>> explicit state transfer, I shudder at the prospect of trying to >>> implicitly freeze the entire interpreter state... (and that's even >>> before we get into the problem of programmers responding with >>> "What do you mean restarting my script doesn't reinitialise all >>> the state?") Cheers, Nick. >> >> I am not sure what he means. But if he e.g. wants to save >> an interactive session with the command prompt, a la >> Matlab, then ipython will do what he wants. >> >> Suspending and restarting a process on Unix is of course >> as easy as >> >> $ kill -s STOP [pid] >> $ kill -s CONT [pid] >> >> I don't know of any Windows equivalent. But virtual >> computers (e.g. VirtualBox, VMware) can be paused and >> restarted. > > The OP wants something like Emacs/TeX/Perl undump/unexec. > > Oleg. I can't imagine how it would work with any bindings (e.g. PyQt) or with opened handles to files/devices or even network sockets. It would only make sense for a strictly controlled environment where these things just don't exist and not for a full python installation. -panzi From steve at pearwood.info Sun Sep 18 04:18:59 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 18 Sep 2011 12:18:59 +1000 Subject: [Python-ideas] allow line break at operators In-Reply-To: References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> <1315092177.54288.YahooMailNeo@web121504.mail.ne1.yahoo.com> <4E62DCC6.8000906@mrabarnett.plus.com> <1315130288.86666.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E63BC83.9000401@mrabarnett.plus.com> <87liu4oseu.fsf@benfinney.id.au> <87fwkcortg.fsf@benfinney.id.au> Message-ID: <4E755513.9050700@pearwood.info> Arnaud Delobelle wrote: > On 4 September 2011 23:39, Ben Finney wrote: >> Ben Finney writes: >> >>> MRAB writes: >>> >>>> As well as still limiting a comment to a line, I'd also still limit >>>> a string literal (except a triple-quoted string literal) to a line. >>> How many string literals do you count in the following statement? I >>> count one: >>> >>> raise HoustonWeHaveAProblemError( >>> "Lorem ipsum dolor sit amet," >>> " consectetur adipiscing elit.") >> The Python compiler agrees with me: >> >> >>> import dis >> >>> def foo(): >> ... raise ValueError( >> ... "Lorem ipsum dolor sit amet," >> ... " consectetur adipiscing elit.") >> ... >> >>> dis.dis(foo) >> 2 0 LOAD_GLOBAL 0 (ValueError) >> >> 3 3 LOAD_CONST 1 ('Lorem ipsum dolor sit amet, consectetur adipiscing elit.') >> 6 CALL_FUNCTION 1 >> 9 RAISE_VARARGS 1 >> 12 LOAD_CONST 0 (None) >> 15 RETURN_VALUE Compile-time implicit concatenation of string literals is a guarantee of the language. Any Python implementation must do that, going back to at least CPython 1.5 and possibly older. > The code object says that there's one string constant in the compiled > function. It says nothing (and knows nothing) about the number of > string literals that made up this string. In the following, how many > string literals can you see? > > >>>> def bar(): return "a" + "b" > ... > > Now let's look at the code object: > > >>>> dis.dis(bar) > 1 0 LOAD_CONST 3 ('ab') > 3 RETURN_VALUE I'm not entirely sure I understand your point there. That's the keyhole optimizer at work. It does the same thing here: >>> dis.dis(compile("1+1", "", "single")) 1 0 LOAD_CONST 2 (2) 3 PRINT_EXPR 4 LOAD_CONST 1 (None) 7 RETURN_VALUE and it is an implementation feature, not a language feature. The oldest version I can find that does this is CPython 2.5, and there's no guarantee that either other implementations or future versions will do the same thing. -- Steven From steven.samuel.cole at gmail.com Sun Sep 18 06:59:14 2011 From: steven.samuel.cole at gmail.com (Steven Samuel Cole) Date: Sun, 18 Sep 2011 14:59:14 +1000 Subject: [Python-ideas] truncate sequences in pretty-print ? Message-ID: hello, i use pprint quite a bit during development to give me quick insight into what is going on inside my application. however, when there's any sequences involved, the output becomes less useful the longer these sequences are - i often find myself scrolling around (or even searching) in the terminal window, trying to find the bit of output i am interested in. imo, it would be great if pprint had a parameter 'max_len' or so that reduces output of every sequence to a maximum and inserts something like '...' to indicate truncation, e.g. {'my key': ['my list item 01', 'my list item 02', 'my list item 03', 'my list item 04', 'my list item 05', 'my list item 06', 'my list item 07', 'my list item 08', '...', 'my list item 10',]} somewhat comparable to the '...' already printed when a structure is more deeply nested than you want to know right now. what do you think ? cheers, ssc From arnodel at gmail.com Sun Sep 18 08:26:42 2011 From: arnodel at gmail.com (Arnaud Delobelle) Date: Sun, 18 Sep 2011 07:26:42 +0100 Subject: [Python-ideas] allow line break at operators In-Reply-To: <4E755513.9050700@pearwood.info> References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> <1315092177.54288.YahooMailNeo@web121504.mail.ne1.yahoo.com> <4E62DCC6.8000906@mrabarnett.plus.com> <1315130288.86666.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E63BC83.9000401@mrabarnett.plus.com> <87liu4oseu.fsf@benfinney.id.au> <87fwkcortg.fsf@benfinney.id.au> <4E755513.9050700@pearwood.info> Message-ID: On 18 September 2011 03:18, Steven D'Aprano wrote: > Arnaud Delobelle wrote: >> >> On 4 September 2011 23:39, Ben Finney wrote: >>> >>> Ben Finney writes: >>> >>>> MRAB writes: >>>> >>>>> As well as still limiting a comment to a line, I'd also still limit >>>>> a string literal (except a triple-quoted string literal) to a line. >>>> >>>> How many string literals do you count in the following statement? I >>>> count one: >>>> >>>> ? ?raise HoustonWeHaveAProblemError( >>>> ? ? ? ?"Lorem ipsum dolor sit amet," >>>> ? ? ? ?" consectetur adipiscing elit.") >>> >>> The Python compiler agrees with me: >>> >>> ? >>> import dis >>> ? >>> def foo(): >>> ? ... ? ? raise ValueError( >>> ? ... ? ? ? ? "Lorem ipsum dolor sit amet," >>> ? ... ? ? ? ? " consectetur adipiscing elit.") >>> ? ... >>> ? >>> dis.dis(foo) >>> ? ? 2 ? ? ? ? ? 0 LOAD_GLOBAL ? ? ? ? ? ? ?0 (ValueError) >>> >>> ? ? 3 ? ? ? ? ? 3 LOAD_CONST ? ? ? ? ? ? ? 1 ('Lorem ipsum dolor sit >>> amet, consectetur adipiscing elit.') >>> ? ? ? ? ? ? ? ? 6 CALL_FUNCTION ? ? ? ? ? ?1 >>> ? ? ? ? ? ? ? ? 9 RAISE_VARARGS ? ? ? ? ? ?1 >>> ? ? ? ? ? ? ? ?12 LOAD_CONST ? ? ? ? ? ? ? 0 (None) >>> ? ? ? ? ? ? ? ?15 RETURN_VALUE > > > Compile-time implicit concatenation of string literals is a guarantee of the > language. Any Python implementation must do that, going back to at least > CPython 1.5 and possibly older. > > >> The code object says that there's one string constant in the compiled >> function. ?It says nothing (and knows nothing) about the number of >> string literals that made up this string. ?In the following, how many >> string literals can you see? >> >> >>>>> def bar(): return "a" + "b" >> >> ... >> >> Now let's look at the code object: >> >> >>>>> dis.dis(bar) >> >> ?1 ? ? ? ? ? 0 LOAD_CONST ? ? ? ? ? ? ? 3 ('ab') >> ? ? ? ? ? ? ?3 RETURN_VALUE > > > I'm not entirely sure I understand your point there. My point is that looking at the number of string constants in compiled code does not tell you the number of string literals in the source code. I thought this was clear from the context of my reply to Ben (see quoted messages above). -- Arnaud From ben+python at benfinney.id.au Sun Sep 18 12:22:50 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 18 Sep 2011 20:22:50 +1000 Subject: [Python-ideas] allow line break at operators References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> <1315092177.54288.YahooMailNeo@web121504.mail.ne1.yahoo.com> <4E62DCC6.8000906@mrabarnett.plus.com> <1315130288.86666.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E63BC83.9000401@mrabarnett.plus.com> <87liu4oseu.fsf@benfinney.id.au> <87fwkcortg.fsf@benfinney.id.au> Message-ID: <87litm2lqd.fsf@benfinney.id.au> Arnaud Delobelle writes: > On 4 September 2011 23:39, Ben Finney wrote: > > Ben Finney writes: > > > >> How many string literals do you count in the following statement? I > >> count one: > >> > >> ? ? raise HoustonWeHaveAProblemError( > >> ? ? ? ? "Lorem ipsum dolor sit amet," > >> ? ? ? ? " consectetur adipiscing elit.") > > The code object says that there's one string constant in the compiled > function. It says nothing (and knows nothing) about the number of > string literals that made up this string. Hmm. So how should I be looking to answer the question? My AST-fu is weak. -- \ ?Science doesn't work by vote and it doesn't work by | `\ authority.? ?Richard Dawkins, _Big Mistake_ (The Guardian, | _o__) 2006-12-27) | Ben Finney From g.brandl at gmx.net Sun Sep 18 12:46:21 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 18 Sep 2011 12:46:21 +0200 Subject: [Python-ideas] allow line break at operators In-Reply-To: <87litm2lqd.fsf@benfinney.id.au> References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> <1315092177.54288.YahooMailNeo@web121504.mail.ne1.yahoo.com> <4E62DCC6.8000906@mrabarnett.plus.com> <1315130288.86666.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E63BC83.9000401@mrabarnett.plus.com> <87liu4oseu.fsf@benfinney.id.au> <87fwkcortg.fsf@benfinney.id.au> <87litm2lqd.fsf@benfinney.id.au> Message-ID: Am 18.09.2011 12:22, schrieb Ben Finney: > Arnaud Delobelle > writes: > >> On 4 September 2011 23:39, Ben Finney wrote: >> > Ben Finney writes: >> > >> >> How many string literals do you count in the following statement? I >> >> count one: >> >> >> >> raise HoustonWeHaveAProblemError( >> >> "Lorem ipsum dolor sit amet," >> >> " consectetur adipiscing elit.") >> >> The code object says that there's one string constant in the compiled >> function. It says nothing (and knows nothing) about the number of >> string literals that made up this string. > > Hmm. So how should I be looking to answer the question? My AST-fu is > weak. It is sufficient to look into Grammar/Grammar: atom: ('(' [yield_expr|testlist_comp] ')' | '[' [listmaker] ']' | '{' [dictorsetmaker] '}' | '`' testlist1 '`' | NAME | NUMBER | STRING+) The little "+" after STRING tells you taht the multiple string literals survive the tokenizer and are concatenated during AST creation (look for parsestrplus() in Python/ast.c). Georg From ben+python at benfinney.id.au Sun Sep 18 13:38:33 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 18 Sep 2011 21:38:33 +1000 Subject: [Python-ideas] allow line break at operators References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> <1315092177.54288.YahooMailNeo@web121504.mail.ne1.yahoo.com> <4E62DCC6.8000906@mrabarnett.plus.com> <1315130288.86666.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E63BC83.9000401@mrabarnett.plus.com> <87liu4oseu.fsf@benfinney.id.au> <87fwkcortg.fsf@benfinney.id.au> <87litm2lqd.fsf@benfinney.id.au> Message-ID: <87hb4a2i86.fsf@benfinney.id.au> Georg Brandl writes: > Am 18.09.2011 12:22, schrieb Ben Finney: > > Arnaud Delobelle > > writes: > > > >> On 4 September 2011 23:39, Ben Finney wrote: > >> > Ben Finney writes: > >> > > >> >> How many string literals do you count in the following > >> >> statement? I count one: > >> >> > >> >> raise HoustonWeHaveAProblemError( > >> >> "Lorem ipsum dolor sit amet," > >> >> " consectetur adipiscing elit.") > >> > >> The code object says that there's one string constant in the > >> compiled function. It says nothing (and knows nothing) about the > >> number of string literals that made up this string. > > > > Hmm. So how should I be looking to answer the question? My AST-fu is > > weak. > > It is sufficient to look into Grammar/Grammar: No, my question is how can I introspect the Python runtime state to compare two statements: foo = "spam" "eggs" bar = "spam" + "eggs" and show that the *result as produced by the parser* is that the first statement has a single string literal on the right hand side, while the second statement has a concatenation expression between two string literals. In other words: once I've come to a hypothesis from my reading of the grammar, how can I *verify* that against the actual running Python language parser? -- \ ?Broken promises don't upset me. I just think, why did they | `\ believe me?? ?Jack Handey | _o__) | Ben Finney From ncoghlan at gmail.com Sun Sep 18 13:56:49 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 18 Sep 2011 21:56:49 +1000 Subject: [Python-ideas] allow line break at operators In-Reply-To: <87hb4a2i86.fsf@benfinney.id.au> References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> <1315092177.54288.YahooMailNeo@web121504.mail.ne1.yahoo.com> <4E62DCC6.8000906@mrabarnett.plus.com> <1315130288.86666.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E63BC83.9000401@mrabarnett.plus.com> <87liu4oseu.fsf@benfinney.id.au> <87fwkcortg.fsf@benfinney.id.au> <87litm2lqd.fsf@benfinney.id.au> <87hb4a2i86.fsf@benfinney.id.au> Message-ID: On Sun, Sep 18, 2011 at 9:38 PM, Ben Finney wrote: > In other words: once I've come to a hypothesis from my reading of the > grammar, how can I *verify* that against the actual running Python > language parser? To an arbitrary level, you can't. The ast module gets you pretty close, though, since optimisations aren't applied when you request the AST as the output of compilation: # 3.2 >>> ast.dump(compile("'Hello ' 'World!'", "", "eval", flags=ast.PyCF_ONLY_AST)) "Expression(body=Str(s='Hello World!'))" >>> ast.dump(compile("'Hello ' + 'World!'", "", "eval", flags=ast.PyCF_ONLY_AST)) "Expression(body=BinOp(left=Str(s='Hello '), op=Add(), right=Str(s='World!')))" Not that even the AST is post the implicit string concatenation step, though. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Sun Sep 18 13:57:16 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 18 Sep 2011 21:57:16 +1000 Subject: [Python-ideas] allow line break at operators In-Reply-To: References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> <1315092177.54288.YahooMailNeo@web121504.mail.ne1.yahoo.com> <4E62DCC6.8000906@mrabarnett.plus.com> <1315130288.86666.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E63BC83.9000401@mrabarnett.plus.com> <87liu4oseu.fsf@benfinney.id.au> <87fwkcortg.fsf@benfinney.id.au> <87litm2lqd.fsf@benfinney.id.au> <87hb4a2i86.fsf@benfinney.id.au> Message-ID: On Sun, Sep 18, 2011 at 9:56 PM, Nick Coghlan wrote: > Not that even the AST is post the implicit string concatenation step, though. s/Not that/Note that/ Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Sun Sep 18 16:54:35 2011 From: guido at python.org (Guido van Rossum) Date: Sun, 18 Sep 2011 07:54:35 -0700 Subject: [Python-ideas] allow line break at operators In-Reply-To: <87hb4a2i86.fsf@benfinney.id.au> References: <1312951356.77394.YahooMailNeo@web121518.mail.ne1.yahoo.com> <4E43D2F2.1090004@mrabarnett.plus.com> <1314884634.78252.YahooMailNeo@web121507.mail.ne1.yahoo.com> <877h5rjtdb.fsf@uwakimon.sk.tsukuba.ac.jp> <1315023501.31777.YahooMailNeo@web121514.mail.ne1.yahoo.com> <87aaamjgvh.fsf@uwakimon.sk.tsukuba.ac.jp> <1315036271.77996.YahooMailNeo@web121504.mail.ne1.yahoo.com> <1315092177.54288.YahooMailNeo@web121504.mail.ne1.yahoo.com> <4E62DCC6.8000906@mrabarnett.plus.com> <1315130288.86666.YahooMailNeo@web121515.mail.ne1.yahoo.com> <4E63BC83.9000401@mrabarnett.plus.com> <87liu4oseu.fsf@benfinney.id.au> <87fwkcortg.fsf@benfinney.id.au> <87litm2lqd.fsf@benfinney.id.au> <87hb4a2i86.fsf@benfinney.id.au> Message-ID: On Sun, Sep 18, 2011 at 4:38 AM, Ben Finney wrote: > No, my question is how can I introspect the Python runtime state to > compare two statements: > > ? ?foo = "spam" "eggs" > ? ?bar = "spam" + "eggs" > > and show that the *result as produced by the parser* is that the first > statement has a single string literal on the right hand side, while the > second statement has a concatenation expression between two string > literals. > > In other words: once I've come to a hypothesis from my reading of the > grammar, how can I *verify* that against the actual running Python > language parser? I'm confused why anybody would care about the answer. Is this just to settle an argument where someone claimed that "foo" "bar" was two strings whereas someone else claimed that it was one? You can't answer that by looking at the AST -- you have to think about what the proper definition of string should be. Personally I think of it as two string literals. Maybe that will stop this thread? -- --Guido van Rossum (python.org/~guido) From guido at python.org Sun Sep 18 16:59:39 2011 From: guido at python.org (Guido van Rossum) Date: Sun, 18 Sep 2011 07:59:39 -0700 Subject: [Python-ideas] truncate sequences in pretty-print ? In-Reply-To: References: Message-ID: On Sat, Sep 17, 2011 at 9:59 PM, Steven Samuel Cole wrote: > hello, > > i use pprint quite a bit during development to give me quick insight > into what is going on inside my application. > however, when there's any sequences involved, the output becomes less > useful the longer these sequences are - i often find myself scrolling > around (or even searching) in the terminal window, trying to find the > bit of output i am interested in. > > imo, it would be great if pprint had a parameter 'max_len' or so that > reduces output of every sequence to a maximum and inserts something > like '...' to indicate truncation, e.g. > {'my key': ['my list item 01', > ? ? ? ? ? ?'my list item 02', > ? ? ? ? ? ?'my list item 03', > ? ? ? ? ? ?'my list item 04', > ? ? ? ? ? ?'my list item 05', > ? ? ? ? ? ?'my list item 06', > ? ? ? ? ? ?'my list item 07', > ? ? ? ? ? ?'my list item 08', > ? ? ? ? ? ?'...', > ? ? ? ? ? ?'my list item 10',]} > > somewhat comparable to the '...' already printed when a structure is > more deeply nested than you want to know right now. Agreed, this would be a useful feature. I've reimplemented something like pprint a few times and always had to implement this truncation feature. If you or someone can contribute a patch that would be much appreciated! -- --Guido van Rossum (python.org/~guido) From tjreedy at udel.edu Sun Sep 18 19:41:32 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 18 Sep 2011 13:41:32 -0400 Subject: [Python-ideas] truncate sequences in pretty-print ? In-Reply-To: References: Message-ID: On 9/18/2011 10:59 AM, Guido van Rossum wrote: > On Sat, Sep 17, 2011 at 9:59 PM, Steven Samuel Cole > wrote: >> > hello, >> > >> > i use pprint quite a bit during development to give me quick insight >> > into what is going on inside my application. >> > however, when there's any sequences involved, the output becomes less >> > useful the longer these sequences are - i often find myself scrolling >> > around (or even searching) in the terminal window, trying to find the >> > bit of output i am interested in. >> > >> > imo, it would be great if pprint had a parameter 'max_len' or so that >> > reduces output of every sequence to a maximum and inserts something >> > like '...' to indicate truncation, e.g. >> > {'my key': ['my list item 01', >> > 'my list item 02', >> > 'my list item 03', >> > 'my list item 04', >> > 'my list item 05', >> > 'my list item 06', >> > 'my list item 07', >> > 'my list item 08', >> > '...', >> > 'my list item 10',]} >> > >> > somewhat comparable to the '...' already printed when a structure is >> > more deeply nested than you want to know right now. > Agreed, this would be a useful feature. I've reimplemented something > like pprint a few times and always had to implement this truncation > feature. If you or someone can contribute a patch that would be much > appreciated! http://bugs.python.org/issue13004 -- Terry Jan Reedy From glyph at twistedmatrix.com Sun Sep 18 20:33:26 2011 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Sun, 18 Sep 2011 14:33:26 -0400 Subject: [Python-ideas] [Python-Dev] Persistent Python - a la Smalltalk In-Reply-To: References: Message-ID: <9DB5E185-7173-4856-BAA5-FAEB4DAC344F@twistedmatrix.com> On Sep 18, 2011, at 4:55 AM, Godson Gera wrote: > Twisted has some feature like that implemented using pickles or some thing. It meant to save the state of the program during restart. I am not sure if that's what you are after. http://twistedmatrix.com > Actually, the Twisted developers also found this to be a terrible idea, and doesn't really do it any more. I definitely understand the appeal of such a system, but it has far too many problems to work well in Python, and even in Smalltalk (which was built from the start to work like this) it was the source of a number of long-term systematic problems with the ecosystem. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimjjewett at gmail.com Sun Sep 18 21:32:34 2011 From: jimjjewett at gmail.com (Jim Jewett) Date: Sun, 18 Sep 2011 15:32:34 -0400 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <1316238446.17476.90.camel@Gutsy> Message-ID: On Sat, Sep 17, 2011 at 2:16 AM, Nick Coghlan wrote: > Specifically targeting existing uses of the default argument hack, on > the other hand, comes with at least 3 already known use cases (i.e. > name lookup micro-optimisation, early binding semantics and shared > state for algorithms). People already tolerate messing with the > function signature to obtain those behaviours, so it's reasonable to > wonder if it is possible to provide a cleaner way to obtain the same > effect. A simple set of definition time name bindings would likely > suffice without greatly increasing the language complexity or the > runtime overhead of function calls. Since the goal is to add some context, like a wrapper, it seems reasonable to use a specialization of @. @+ suggests adding something to the environment, and won't be ambiguous unless numbers become callable. @+tuple=tuple @+sorted=sorted @+len=len @+KeyError=KeyError def decorating_function(user_function): I would personally be fine with a restriction to @+name= but I suppose others might prefer a tuple, like @+(tuple, sorted, len, KeyError)=(tuple, sorted, len, KeyError) def decorating_function(user_function): -jJ From ron3200 at gmail.com Sun Sep 18 22:58:48 2011 From: ron3200 at gmail.com (Ron Adam) Date: Sun, 18 Sep 2011 15:58:48 -0500 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <1316238446.17476.90.camel@Gutsy> Message-ID: <1316379528.8673.57.camel@Gutsy> On Sun, 2011-09-18 at 15:32 -0400, Jim Jewett wrote: > On Sat, Sep 17, 2011 at 2:16 AM, Nick Coghlan wrote: > > > Specifically targeting existing uses of the default argument hack, on > > the other hand, comes with at least 3 already known use cases (i.e. > > name lookup micro-optimisation, early binding semantics and shared > > state for algorithms). People already tolerate messing with the > > function signature to obtain those behaviours, so it's reasonable to > > wonder if it is possible to provide a cleaner way to obtain the same > > effect. A simple set of definition time name bindings would likely > > suffice without greatly increasing the language complexity or the > > runtime overhead of function calls. > > Since the goal is to add some context, like a wrapper, it seems > reasonable to use a specialization of @. @+ suggests adding something > to the environment, and won't be ambiguous unless numbers become > callable. > > @+tuple=tuple > @+sorted=sorted > @+len=len > @+KeyError=KeyError > def decorating_function(user_function): > > I would personally be fine with a restriction to @+name= but I > suppose others might prefer a tuple, like > > > @+(tuple, sorted, len, KeyError)=(tuple, sorted, len, KeyError) > def decorating_function(user_function): If there was a method on a function object to set default values, then you could use a regular decorator. # Most likely to live in functools def preset(**kwds): def apply(f): f.set_default_values(kwds) return f return apply #--------------------------------- from functools import preset @preset(tuple=tuple, sorted=sorted, len=len, keyerror=keyerror) def foo(...): ... I was hoping there might be a more direct way to do it. My previous idea breaks the white space consistency, and Nicks idea adds special syntax. Decorators with long arguments make the function name stand out less. Is there a more recent patch for Pep 362? Pep 362 seems to be targeted at only introspection and really isn't a part of the function object. Is that correct? Cheers, Ron From adam.jorgensen.za at gmail.com Mon Sep 19 06:45:11 2011 From: adam.jorgensen.za at gmail.com (Adam Jorgensen) Date: Mon, 19 Sep 2011 06:45:11 +0200 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <1316379528.8673.57.camel@Gutsy> References: <1316208647.15431.105.camel@Gutsy> <1316238446.17476.90.camel@Gutsy> <1316379528.8673.57.camel@Gutsy> Message-ID: It looks ugly and it's not so obvious what the : is associated with -1 On 18 September 2011 22:58, Ron Adam wrote: > On Sun, 2011-09-18 at 15:32 -0400, Jim Jewett wrote: > > On Sat, Sep 17, 2011 at 2:16 AM, Nick Coghlan > wrote: > > > > > Specifically targeting existing uses of the default argument hack, on > > > the other hand, comes with at least 3 already known use cases (i.e. > > > name lookup micro-optimisation, early binding semantics and shared > > > state for algorithms). People already tolerate messing with the > > > function signature to obtain those behaviours, so it's reasonable to > > > wonder if it is possible to provide a cleaner way to obtain the same > > > effect. A simple set of definition time name bindings would likely > > > suffice without greatly increasing the language complexity or the > > > runtime overhead of function calls. > > > > Since the goal is to add some context, like a wrapper, it seems > > reasonable to use a specialization of @. @+ suggests adding something > > to the environment, and won't be ambiguous unless numbers become > > callable. > > > > @+tuple=tuple > > @+sorted=sorted > > @+len=len > > @+KeyError=KeyError > > def decorating_function(user_function): > > > > I would personally be fine with a restriction to @+name= but I > > suppose others might prefer a tuple, like > > > > > > @+(tuple, sorted, len, KeyError)=(tuple, sorted, len, KeyError) > > def decorating_function(user_function): > > If there was a method on a function object to set default values, then > you could use a regular decorator. > > # Most likely to live in functools > > def preset(**kwds): > def apply(f): > f.set_default_values(kwds) > return f > return apply > > #--------------------------------- > from functools import preset > > @preset(tuple=tuple, sorted=sorted, len=len, keyerror=keyerror) > def foo(...): > ... > > I was hoping there might be a more direct way to do it. My previous > idea breaks the white space consistency, and Nicks idea adds special > syntax. Decorators with long arguments make the function name stand out > less. > > Is there a more recent patch for Pep 362? > > Pep 362 seems to be targeted at only introspection and really isn't a > part of the function object. Is that correct? > > Cheers, > Ron > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.jorgensen.za at gmail.com Mon Sep 19 06:47:20 2011 From: adam.jorgensen.za at gmail.com (Adam Jorgensen) Date: Mon, 19 Sep 2011 06:47:20 +0200 Subject: [Python-ideas] [Python-Dev] Persistent Python - a la Smalltalk In-Reply-To: <9DB5E185-7173-4856-BAA5-FAEB4DAC344F@twistedmatrix.com> References: <9DB5E185-7173-4856-BAA5-FAEB4DAC344F@twistedmatrix.com> Message-ID: Isn't this kind of thing (Like everything) possible today with Common Lisp? I seem to recall reading about a CL app a while back that was basically a persisted run-time state. The downside from what I read was that memory usage was high and could get very high with inconsistent GC intervals. On 18 September 2011 20:33, Glyph Lefkowitz wrote: > > On Sep 18, 2011, at 4:55 AM, Godson Gera wrote: > > Twisted has some feature like that implemented using pickles or some > thing. It meant to save the state of the program during restart. I am not > sure if that's what you are after. http://twistedmatrix.com > > Actually, the Twisted developers also found this to be a terrible idea, and > doesn't really do it any more. I definitely understand the appeal of such a > system, but it has far too many problems to work well in Python, and even in > Smalltalk (which was built from the start to work like this) it was the > source of a number of long-term systematic problems with the ecosystem. > > -glyph > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aquavitae69 at gmail.com Mon Sep 19 09:44:47 2011 From: aquavitae69 at gmail.com (David Townshend) Date: Mon, 19 Sep 2011 09:44:47 +0200 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <1316379528.8673.57.camel@Gutsy> References: <1316208647.15431.105.camel@Gutsy> <1316238446.17476.90.camel@Gutsy> <1316379528.8673.57.camel@Gutsy> Message-ID: I think something similar could be achieved with annotations, as in the following example. This is incomplete, and probably full of errors, but it gives the idea. def setlocals(func): ? ? def wrapper(*args, **kwargs): ? ? ? ? call_kwargs = [] ? ? ? ? for k, v in kwargs.items(): ? ? ? ? ? ? if k in func.__annotations__ and func.__annotations__[k] == 'local': ? ? ? ? ? ? ? ? raise AttributeError("Can't set argument") ? ? ? ? ? ? else: ? ? ? ? ? ? ? ? call_kwargs[k] = v ? ? ? ? return func(*args, **call_kwargs) >>> @setlocals ... def myfunc(a, b, loc=4:'local'): ... ? ? print(loc) >>> myfunc(2, 3) 4 >>> myfunc(2, 3, loc=5) AttributeError On Sep 18, 2011 10:59 PM, "Ron Adam" wrote: > On Sun, 2011-09-18 at 15:32 -0400, Jim Jewett wrote: >> On Sat, Sep 17, 2011 at 2:16 AM, Nick Coghlan wrote: >> >> > Specifically targeting existing uses of the default argument hack, on >> > the other hand, comes with at least 3 already known use cases (i.e. >> > name lookup micro-optimisation, early binding semantics and shared >> > state for algorithms). People already tolerate messing with the >> > function signature to obtain those behaviours, so it's reasonable to >> > wonder if it is possible to provide a cleaner way to obtain the same >> > effect. A simple set of definition time name bindings would likely >> > suffice without greatly increasing the language complexity or the >> > runtime overhead of function calls. >> >> Since the goal is to add some context, like a wrapper, it seems >> reasonable to use a specialization of @. @+ suggests adding something >> to the environment, and won't be ambiguous unless numbers become >> callable. >> >> @+tuple=tuple >> @+sorted=sorted >> @+len=len >> @+KeyError=KeyError >> def decorating_function(user_function): >> >> I would personally be fine with a restriction to @+name= but I >> suppose others might prefer a tuple, like >> >> >> @+(tuple, sorted, len, KeyError)=(tuple, sorted, len, KeyError) >> def decorating_function(user_function): > > If there was a method on a function object to set default values, then > you could use a regular decorator. > > # Most likely to live in functools > > def preset(**kwds): > def apply(f): > f.set_default_values(kwds) > return f > return apply > > #--------------------------------- > from functools import preset > > @preset(tuple=tuple, sorted=sorted, len=len, keyerror=keyerror) > def foo(...): > ... > > I was hoping there might be a more direct way to do it. My previous > idea breaks the white space consistency, and Nicks idea adds special > syntax. Decorators with long arguments make the function name stand out > less. > > Is there a more recent patch for Pep 362? > > Pep 362 seems to be targeted at only introspection and really isn't a > part of the function object. Is that correct? > > Cheers, > Ron > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From ncoghlan at gmail.com Mon Sep 19 11:35:56 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 19 Sep 2011 19:35:56 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <1316238446.17476.90.camel@Gutsy> <1316379528.8673.57.camel@Gutsy> Message-ID: On Mon, Sep 19, 2011 at 5:44 PM, David Townshend wrote: > I think something similar could be achieved with annotations, as in > the following example. > > This is incomplete, and probably full of errors, but it gives the idea. Proposals that are uglier than the default argument hack itself (and I put both annotations+decorator and the '@+' idea in that category) just aren't worth the hassle. The *only* change even potentially worth making in this area would be one that: - looks good (i.e. is easy to read) - is easy to remember and write - doesn't hurt call time performance - addresses all the default argument hack use cases - doesn't have the downside of altering the function's externally visible signature My initialised locals idea *might* qualify (although I've yet to see anyone make a serious attempt at poking holes in it, so I'm not convinced of that), but the other suggestions in this thread haven't really offered anything to recommend them over the existing approach. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From aquavitae69 at gmail.com Mon Sep 19 14:19:54 2011 From: aquavitae69 at gmail.com (David Townshend) Date: Mon, 19 Sep 2011 14:19:54 +0200 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <1316238446.17476.90.camel@Gutsy> <1316379528.8673.57.camel@Gutsy> Message-ID: I agree, both are ugly (btw, mine wasn't a proposal, just an illustration that it is possible with the status quo). One question I have is would the locals be local to a specific call or the the function object? I don't know enough about the python language internals to know the correct terminology to explain what I mean, but from a users point of view, would the locals be reset at every call? The fact that they are locals implies that they don't exist outside a function call, but assigning them in the function definition implies that they exist within the function object in a similar way to the value of keyword arguments. David On Mon, Sep 19, 2011 at 11:35 AM, Nick Coghlan wrote: > On Mon, Sep 19, 2011 at 5:44 PM, David Townshend wrote: >> I think something similar could be achieved with annotations, as in >> the following example. >> >> This is incomplete, and probably full of errors, but it gives the idea. > > Proposals that are uglier than the default argument hack itself (and I > put both annotations+decorator and the '@+' idea in that category) > just aren't worth the hassle. The *only* change even potentially worth > making in this area would be one that: > - looks good (i.e. is easy to read) > - is easy to remember and write > - doesn't hurt call time performance > - addresses all the default argument hack use cases > - doesn't have the downside of altering the function's externally > visible signature > > My initialised locals idea *might* qualify (although I've yet to see > anyone make a serious attempt at poking holes in it, so I'm not > convinced of that), but the other suggestions in this thread haven't > really offered anything to recommend them over the existing approach. > > Cheers, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > From sven at marnach.net Mon Sep 19 15:22:18 2011 From: sven at marnach.net (Sven Marnach) Date: Mon, 19 Sep 2011 14:22:18 +0100 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> Message-ID: <20110919132218.GC5335@pantoffel-wg.de> Nick Coghlan wrote: > [...] allow a sequence of 'implicit locals' to be defined within > square brackets after the parameter list. I'm not quite sure whether the benefits of this proposal are worth the further rococoization of the language. As far as I can see, the only benefit is keeping function signatures clean. Looking at the use cases one by one: > Micro-optimisation: > # Current Python (internal to functools.lru_cache with default > argument hack) > def decorating_function(user_function, tuple=tuple, sorted=sorted, > len=len, KeyError=KeyError): > ... # 60 line function > > # Proposal - explicitly implicit locals to clarify real calling signature > def decorating_function(user_function) [tuple=tuple, > sorted=sorted, len=len, KeyError=KeyError)]: > ... # 60 line function The proposed syntax isn't explicit enough for my taste to justify the change. What the programmer really wants to tell the compiler is: Bind the given names at compile time rather than at run time. I'd prefer if the syntax would reflect this intention, similar to the "global" and "nonlocal" declarations: def decorating_function(user_function): earlybind tuple, sorted, len, KeyError (Not being a native speaker of English, I don't even try to find a less dull name for this than "earlybind". And I'm quite aware that the bar for the introduction of a new keyword is quite high.) > Early binding in a loop: > > # Current Python > adders = [] > for i in range(10): > def f(x, _i=i): > return x + _i > adders.append(f) > > # Proposal > adders = [] > for i in range(10): > def f(x) [i=i]: # Real calling signature is clear > return x + i > adders.append(f) I don't see too much benefit of the proposed syntax for this use case. If f() is a local throw-away function, I wouldn't worry about its signature. If f() is a longer-lived object and I do care about its signature, I'd uses a class: class Adder: def __init__(self, i): self.i = i def __call__(self, x): return x + self.i [...] adders.append(Adder(i)) I still think classes are the Python way to hide state, not closures. That said, this case would also be covered by the "earlybind" porposal above. > Algorithmic shared state (without a class): > > # Current Python > def f(*, _cache=[]): > # Consenting adults, please don't override _cache when calling! > > # Proposal > def f() [_cache=[]]: # Real calling signature is clear > # _cache is an implicit local, not a keyword-only argument, so its safe Again, I'd use a class to hide state in the first place. If someone really wants to avoid using a class for some reason, using function attributes would also be a viable way: def f(): try: cache = f._cache except AttributeError: cache = f._cache = [] Cheers, Sven From glyph at twistedmatrix.com Mon Sep 19 17:50:29 2011 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 19 Sep 2011 11:50:29 -0400 Subject: [Python-ideas] [Python-Dev] Persistent Python - a la Smalltalk In-Reply-To: References: <9DB5E185-7173-4856-BAA5-FAEB4DAC344F@twistedmatrix.com> Message-ID: <92DDC5CA-9632-43B4-80B4-98A90EF8926B@twistedmatrix.com> On Sep 19, 2011, at 3:12 AM, Godson Gera wrote: > On Mon, Sep 19, 2011 at 12:03 AM, Glyph Lefkowitz wrote: > > Actually, the Twisted developers also found this to be a terrible idea, and doesn't really do it any more. I definitely understand the appeal of such a system, but it has far too many problems to work well in Python, and even in Smalltalk (which was built from the start to work like this) it was the source of a number of long-term systematic problems with the ecosystem. > > I willfully leftout the detail that Twisted is not doing it any more :) . So that the person who asked this won't get discouraged, if he has time and energy to pursue it and make some thing new, I thought all the Python community may have another battery called persistance included. Yes, and I put it in in the hopes that they are discouraged :-). There's a staggering number of really good ideas out there for people to work on; why get stuck re-hashing the bad ones? Of course, maybe they've got a really fantastic idea about how a persistent python might work, but they'll have to face more obstacles than this one comment in order to get it done, so it shouldn't make that much of a difference. I think it's my responsibility, as someone who has tried this idea, to make sure that others attempting it know that it's problematic. As Clay Shirky put it: "... learning from experience is the worst possible way to learn something. Learning from experience is one up from remembering. That's not great. The best way to learn something is when someone else figures it out and tells you: 'Don't go in that swamp. There are alligators in there.'" So, trust me: there are alligators in that pickle. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Sep 19 17:59:01 2011 From: guido at python.org (Guido van Rossum) Date: Mon, 19 Sep 2011 08:59:01 -0700 Subject: [Python-ideas] [Python-Dev] Persistent Python - a la Smalltalk In-Reply-To: <92DDC5CA-9632-43B4-80B4-98A90EF8926B@twistedmatrix.com> References: <9DB5E185-7173-4856-BAA5-FAEB4DAC344F@twistedmatrix.com> <92DDC5CA-9632-43B4-80B4-98A90EF8926B@twistedmatrix.com> Message-ID: On Mon, Sep 19, 2011 at 8:50 AM, Glyph Lefkowitz wrote: > So, trust me: there are alligators in that pickle. +1 QOTW -- --Guido van Rossum (python.org/~guido) From anacrolix at gmail.com Mon Sep 19 21:05:00 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Tue, 20 Sep 2011 05:05:00 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <20110919132218.GC5335@pantoffel-wg.de> References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> Message-ID: -1 plx On Mon, Sep 19, 2011 at 11:22 PM, Sven Marnach wrote: > Nick Coghlan wrote: >> [...] allow a sequence of 'implicit locals' to be defined within >> square brackets after the parameter list. > > I'm not quite sure whether the benefits of this proposal are worth the > further rococoization of the language. ?As far as I can see, the only > benefit is keeping function signatures clean. > > Looking at the use cases one by one: > >> Micro-optimisation: >> ? ? # Current Python (internal to functools.lru_cache with default >> argument hack) >> ? ? def decorating_function(user_function, tuple=tuple, sorted=sorted, >> len=len, KeyError=KeyError): >> ? ? ? ? ?... # 60 line function >> >> ? ? # Proposal - explicitly implicit locals to clarify real calling signature >> ? ? def decorating_function(user_function) [tuple=tuple, >> sorted=sorted, len=len, KeyError=KeyError)]: >> ? ? ? ? ?... # 60 line function > > The proposed syntax isn't explicit enough for my taste to justify the > change. ?What the programmer really wants to tell the compiler is: > Bind the given names at compile time rather than at run time. ?I'd > prefer if the syntax would reflect this intention, similar to the > "global" and "nonlocal" declarations: > > ? ?def decorating_function(user_function): > ? ? ? ?earlybind tuple, sorted, len, KeyError > > (Not being a native speaker of English, I don't even try to find a > less dull name for this than "earlybind". ?And I'm quite aware that > the bar for the introduction of a new keyword is quite high.) > >> Early binding in a loop: >> >> ? ?# Current Python >> ? ?adders = [] >> ? ?for i in range(10): >> ? ? ? def f(x, _i=i): >> ? ? ? ? ? return x + _i >> ? ? ? adders.append(f) >> >> ? ?# Proposal >> ? ?adders = [] >> ? ?for i in range(10): >> ? ? ? def f(x) [i=i]: ?# Real calling signature is clear >> ? ? ? ? ? return x + i >> ? ? ? adders.append(f) > > I don't see too much benefit of the proposed syntax for this use > case. ?If f() is a local throw-away function, I wouldn't worry about > its signature. ?If f() is a longer-lived object and I do care about > its signature, I'd uses a class: > > ? ?class Adder: > ? ? ? ?def __init__(self, i): > ? ? ? ? ? ?self.i = i > ? ? ? ?def __call__(self, x): > ? ? ? ? ? ?return x + self.i > > ? ?[...] adders.append(Adder(i)) > > I still think classes are the Python way to hide state, not closures. > That said, this case would also be covered by the "earlybind" porposal > above. > >> Algorithmic shared state (without a class): >> >> ? ? # Current Python >> ? ? def f(*, _cache=[]): >> ? ? ? ? # Consenting adults, please don't override _cache when calling! >> >> ? ? # Proposal >> ? ? def f() [_cache=[]]: ?# Real calling signature is clear >> ? ? ? ? # _cache is an implicit local, not a keyword-only argument, so its safe > > Again, I'd use a class to hide state in the first place. ?If someone > really wants to avoid using a class for some reason, using function > attributes would also be a viable way: > > ? ?def f(): > ? ? ? ?try: > ? ? ? ? ? ?cache = f._cache > ? ? ? ?except AttributeError: > ? ? ? ? ? ?cache = f._cache = [] > > Cheers, > ? ?Sven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From ncoghlan at gmail.com Tue Sep 20 00:23:14 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 20 Sep 2011 08:23:14 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <20110919132218.GC5335@pantoffel-wg.de> References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> Message-ID: On Mon, Sep 19, 2011 at 11:22 PM, Sven Marnach wrote: > I don't see too much benefit of the proposed syntax for this use > case. ?If f() is a local throw-away function, I wouldn't worry about > its signature. ?If f() is a longer-lived object and I do care about > its signature, I'd uses a class: > > ? ?class Adder: > ? ? ? ?def __init__(self, i): > ? ? ? ? ? ?self.i = i > ? ? ? ?def __call__(self, x): > ? ? ? ? ? ?return x + self.i > > ? ?[...] adders.append(Adder(i)) > > I still think classes are the Python way to hide state, not closures. The thing is, real function objects genuinely *are* special. They have a privileged place in the interpreter, the inspect module and other introspection tools know more about how to deal with them, they have instance method descriptor behaviour built in, etc. Switching from a real function object to a custom class with a __call__ method is genuinely expensive in terms of the difficulty of writing the code in the first place, as well as in being able to read it later. Your 'Adder' example above, for instance, doesn't implement the descriptor protocol, so will behave like a staticmethod when placed in a class. That may be what you want, but it isn't easy to get instance method behaviour in the case where you would prefer that. $ python3 -m timeit -s "from call import Adder; f = Adder(1)" "f(5)" 1000000 loops, best of 3: 0.362 usec per loop $ python3 -m timeit -s "from call import Adder; f = Adder(1).__call__" "f(5)" 1000000 loops, best of 3: 0.222 usec per loop $ python3 -m timeit -s "from closure import adder; f = adder(1)" "f(5)" 10000000 loops, best of 3: 0.174 usec per loop $ python3 -m timeit -s "from default import adder; f = adder(1)" "f(5)" 10000000 loops, best of 3: 0.166 usec per loop When what you're trying to express is a single short algorithm, overriding __call__ isn't even in the contest - we aren't talking minor percentage differences in call overhead, we're talking more than double. You can gain a fair bit of that back by retrieving __call__ as a bound method in advance, but really, your algorithm needs to be complex enough for the difference in call overhead to be trivial before implementing __call__ becomes an attractive alternative to using a closure Now, if there are *multiple* algorithms operating on the same data, then obviously you want a class with multiple methods. But in that case, you're less likely to bless any one of them with privilege of occupying the '__call__' slot. Basically, classes make sense when the state is the most important thing, while functions focus on a specific algorithm. For the special case of "single algorithm with some associated state", a closure (or the default argument hack) will often be a better modelling tool than a class. (Obviously, anyone that disagrees with me on this point will consider this whole thread silly - however, the popularity of closures for the implementation of factory functions, including decorator factories, shows that there is plenty of code out there that happily follows this approach) With closures vs the default argument hack, the performance and introspection arguments don't apply - in both of these cases you have a real function, so the only trade-off is between forcing readers to understand how closures work and forcing them to ignore additional arguments that are there just to prepopulate the local namespace with some data. However, the closure approach has some genuine downsides from a readability point of view. The real function when using a closure is the inner one. The outer function definition, the trailing return statement and the invocation of the outer function are all boilerplate that obscures the actual purpose of the code. You can tidy some of that up with a decorator, but you can't avoid the need for the nested function in order to create the closure. And that's why people use the default argument hack today - they weigh those downsides up against the consequences of having a bit of noise in the function signature (as seen by introspection tools) and decide they're happy to accept that trade-off in return for a simple and straightforward expression of a single algorithm with some associated state. Various ideas for addressing this have been proposed in the past. PEP 3150's statement local namespaces are one, as are the assorted incarnations of the proposal for flagging arbitrary expressions within a function for evaluation at definition time rather than runtime (search the python-dev and python-ideas archives for phrase like 'once statement' and 'atdef' - alas, nobody has stepped forward to codify those ideas into a PEP). Where those proposals all come unstuck is that they try to do more than the default argument hack allows, *without compelling use cases to guide the additional semantics*. The pre-initialised locals concept deliberately avoids that problem by targeting exactly the use cases that are *already* supported via the default argument hack, just in a way that tries to avoid the negative effects on introspection and readability. Cheers, Nick. P.S. Code for the timeit runs: $ cat > call.py class Adder(object): def __init__(self, i): self.i = i def __call__(self, x): return x + self.i $ cat > closure.py def adder(i): def f(x): return x + i return f $ cat > default.py def adder(i): def f(x, _i=i): return x + _i return f -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ghostwriter402 at gmail.com Tue Sep 20 01:52:47 2011 From: ghostwriter402 at gmail.com (Spectral One) Date: Mon, 19 Sep 2011 18:52:47 -0500 Subject: [Python-ideas] [Python-Dev] Persistent Python - a la Smalltalk In-Reply-To: References: <9DB5E185-7173-4856-BAA5-FAEB4DAC344F@twistedmatrix.com> <92DDC5CA-9632-43B4-80B4-98A90EF8926B@twistedmatrix.com> Message-ID: <4E77D5CF.1030306@gmail.com> On 9/19/2011 10:59 AM, Guido van Rossum wrote: > On Mon, Sep 19, 2011 at 8:50 AM, Glyph Lefkowitz > wrote: >> So, trust me: there are alligators in that pickle. > +1 QOTW > Now I want to try a spicy gator pickle... (that _is_ a great quote.) The only way I could see something like this working is on a closed system, s.t. there are no assumptions or interactions (files open/locked, external window managers/states, etc.) that cannot be fully accounted for by the values of the system itself. (You could theoretically attempt to record the information not available form interrogating the current state as such are declared, but there would be lots of room for errors, and, even then, no guarantees.) I don't think Python is the correct place to attempt such a thing. This sounds more like a VM- or OS-level affair, to me. I CAN see some value in adding functionality to ease interaction with such features, such as invoking or being aware when it's invoked. I guess I want to know what problem this is intended to solve. If it's purely theoretical, that's cool, but I'd like to know. If not, can you explain why it would help you? That would help me understand this idea better. -Nate From ron3200 at gmail.com Tue Sep 20 05:40:02 2011 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 19 Sep 2011 22:40:02 -0500 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> Message-ID: <1316490002.13636.107.camel@Gutsy> On Tue, 2011-09-20 at 08:23 +1000, Nick Coghlan wrote: > On Mon, Sep 19, 2011 at 11:22 PM, Sven Marnach wrote: > > I don't see too much benefit of the proposed syntax for this use > > case. If f() is a local throw-away function, I wouldn't worry about > > its signature. If f() is a longer-lived object and I do care about > > its signature, I'd uses a class: > > > > class Adder: > > def __init__(self, i): > > self.i = i > > def __call__(self, x): > > return x + self.i > > > > [...] adders.append(Adder(i)) > > > > I still think classes are the Python way to hide state, not closures. > > The thing is, real function objects genuinely *are* special. They have > a privileged place in the interpreter, the inspect module and other > introspection tools know more about how to deal with them, they have > instance method descriptor behaviour built in, etc. Heh, On an abstract level, I do know there is something special about function objects. Putting my finger on it is a bit harder. Trying a few things to see what happens... At some point there does need to be a base object that can't be taken apart into it's smaller parts. That normally would be the object class. But with functions, we are referring to the base *executable* object. A function is is a union of other basic objects that can then be executed. (I think there can be some improvements in how a function object is constructed and organized.) I guess the real *special* magic is the byte code, CALL_FUNCTION, which knows how to use a function object. Along with the other Function related bytecodes. > Where those proposals all come unstuck is that they try to do more > than the default argument hack allows, *without compelling use cases > to guide the additional semantics*. The pre-initialised locals concept > deliberately avoids that problem by targeting exactly the use cases > that are *already* supported via the default argument hack, just in a > way that tries to avoid the negative effects on introspection and > readability. Umm... "...default arguments hack allows,..."? Not well said, I think you meant, "...unstuck, by doing more than is needed...". Even so, I do understand your point you are trying to get across. Would your suggestion add any additional methods, or attributes to a function object? Would it need any additional byte codes? An alternative proposal, I came across the other day... don't remember exactly where, is to be able to put default arguments after the **kwds, parameter. >>> def boo(a, *args, b=1, **kwds, c=3): File "", line 1 def boo(a, *args, b=1, **kwds, c=3): ^ SyntaxError: invalid syntax It's currently a syntax error. But that requires using an ** someplace. I don't really like this one, as I think function arguments are already hard enough to understand. For a language that is suppose to be easy to understand, that's not a good thing. Cheers, Ron From greg.ewing at canterbury.ac.nz Tue Sep 20 07:44:10 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 20 Sep 2011 17:44:10 +1200 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <1316490002.13636.107.camel@Gutsy> References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <1316490002.13636.107.camel@Gutsy> Message-ID: <4E78282A.7030503@canterbury.ac.nz> Ron Adam wrote: > I guess the real *special* magic is the byte code, CALL_FUNCTION, which > knows how to use a function object. Along with the other Function > related bytecodes. I'd say it's more in the function object's __call__ method, which invokes the C code that knows how to interpret bytecodes. The CALL_FUNCTION bytecode just calls the __call__method of the object being called, whatever it happens to be. The function object's __get__ method also plays a part when a method call is involved, as does the __getattribute__ method of the type object. The magic tends to be spread around a fair bit in Python. -- Greg From sven at marnach.net Tue Sep 20 16:00:45 2011 From: sven at marnach.net (Sven Marnach) Date: Tue, 20 Sep 2011 15:00:45 +0100 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> Message-ID: <20110920140045.GD5335@pantoffel-wg.de> Nick Coghlan schrieb am Di, 20. Sep 2011, um 08:23:14 +1000: > [stripped well-reasoned case for using closures against classes] > > And that's why people use the default argument hack today - they weigh > those downsides up against the consequences of having a bit of noise > in the function signature (as seen by introspection tools) and decide > they're happy to accept that trade-off in return for a simple and > straightforward expression of a single algorithm with some associated > state. I actually *do* use the default argument hack for early binding myself. My main points were that the current status quo isn't too bad, there are alternatives if the function signature is really important, and that the cases which require both using closures *and* having a clean signature are too rare to justify yet another confusing-for-beginners syntax. (When I started learning Python in 1998, the language was praised for having a small language core. I didn't hear this very often in more recent times.) Assuming a new syntax *is* necessary -- there is no objective way to decide this after all -- I don't particularly like the proposed square bracket syntax because a) it puts the information at the wrong place. The first line of the function definition is for the outside appearance of the function, including signature and annotations. Separating the internal information on early bound variables from the signature is the primary goal of the proposal after all. b) it does not make the intention of early binding very clear. c) it consists of a list of the form [i=i] or [tuple=tuple, len=len] in most use cases, making it necessary to type every name that shall use early binding twice. That's why I would prefer a syntax that explicitly declares the names that should use early binding, similar to "global" or "nonlocal" statements, if such a syntax is deemed necessary at all. > Various ideas for addressing this have been proposed in the past. PEP > 3150's statement local namespaces are one, as are the assorted > incarnations of the proposal for flagging arbitrary expressions within > a function for evaluation at definition time rather than runtime > (search the python-dev and python-ideas archives for phrase like 'once > statement' and 'atdef' - alas, nobody has stepped forward to codify > those ideas into a PEP). How does PEP 3150 target the use cases of the default argument hack given in your original mail? Cheers, Sven From jimjjewett at gmail.com Tue Sep 20 17:43:49 2011 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 20 Sep 2011 11:43:49 -0400 Subject: [Python-ideas] [Python-Dev] Persistent Python - a la Smalltalk In-Reply-To: <4E77D5CF.1030306@gmail.com> References: <9DB5E185-7173-4856-BAA5-FAEB4DAC344F@twistedmatrix.com> <92DDC5CA-9632-43B4-80B4-98A90EF8926B@twistedmatrix.com> <4E77D5CF.1030306@gmail.com> Message-ID: [Regarding a proposal to persist and resume the current application state] On Mon, Sep 19, 2011 at 7:52 PM, Spectral One wrote: > The only way I could see something like this working is on a closed system, Neither Lisp nor Smalltalk are entirely closed, but ... they are certainly more closed-world in practice than other languages are in practice. > s.t. there are no assumptions or interactions (files open/locked, external > window managers/states, etc.) that cannot be fully accounted for by the > values of the system itself. That was in fact the case for everything I ever actually cared to persist. The primary use case was an extended interactive-shell exploration, rather than a server process. (A secondary use case was checkpointing, but that already took a bit of work to ensure your program would pause and give you a chance to take that checkpoint.) A third use case was faster testing of updated code, but I'm not certain that was a net gain against the possibility of having multiple versions of a function alive simultaneously. > (You could theoretically attempt to record the > information not available form interrogating the current state as such are > declared, but there would be lots of room for errors, and, even then, no > guarantees.) Some information is already recorded for garbage collection, and for good error reports; it really doesn't take all that much more. > I don't think Python is the correct place to attempt such a thing. This > sounds more like a VM- or OS-level affair, to me. -jJ From tjreedy at udel.edu Tue Sep 20 18:59:27 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 Sep 2011 12:59:27 -0400 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <20110920140045.GD5335@pantoffel-wg.de> References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> Message-ID: On 9/20/2011 10:00 AM, Sven Marnach wrote: > I actually *do* use the default argument hack for early binding > myself. My main points were that the current status quo isn't too > bad, there are alternatives if the function signature is really > important, and that the cases which require both using closures *and* > having a clean signature are too rare to justify yet another > confusing-for-beginners syntax. (When I started learning Python in > 1998, the language was praised for having a small language core. I > didn't hear this very often in more recent times.) I have noticed the same. The big syntax additions are generators, comprehensions, decorators, and the with statement. The switch to unicode doesn't change syntax but can complicate text processing. > > Assuming a new syntax *is* necessary -- there is no objective way to > decide this after all -- I don't particularly like the proposed square > bracket syntax because > > a) it puts the information at the wrong place. The first line of the > function definition is for the outside appearance of the function, > including signature and annotations. Separating the internal > information on early bound variables from the signature is the > primary goal of the proposal after all. > > b) it does not make the intention of early binding very clear. > > c) it consists of a list of the form [i=i] or [tuple=tuple, len=len] > in most use cases, making it necessary to type every name that > shall use early binding twice. > > That's why I would prefer a syntax that explicitly declares the names > that should use early binding, similar to "global" or "nonlocal" > statements, if such a syntax is deemed necessary at all. The problem with an earlybind statement in the body is that it is in the body, which should be runtime stuff. I would prefer something in the header. A semi-colon or other char would be sufficient syntactically: def f(a; len, alist=[]): alist.append(a); return len(alist) where a bare identifier like 'len' *means* 'len=len'. -- Terry Jan Reedy From guido at python.org Tue Sep 20 19:09:21 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Sep 2011 10:09:21 -0700 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> Message-ID: On Tue, Sep 20, 2011 at 9:59 AM, Terry Reedy wrote: > On 9/20/2011 10:00 AM, Sven Marnach wrote: > >> I actually *do* use the default argument hack for early binding >> myself. ?My main points were that the current status quo isn't too >> bad, there are alternatives if the function signature is really >> important, and that the cases which require both using closures *and* >> having a clean signature are too rare to justify yet another >> confusing-for-beginners syntax. ?(When I started learning Python in >> 1998, the language was praised for having a small language core. ?I >> didn't hear this very often in more recent times.) > > I have noticed the same. The big syntax additions are generators, > comprehensions, decorators, and the with statement. The switch to unicode > doesn't change syntax but can complicate text processing. >> >> Assuming a new syntax *is* necessary -- there is no objective way to >> decide this after all -- I don't particularly like the proposed square >> bracket syntax because >> >> ?a) it puts the information at the wrong place. The first line of the >> ? ? function definition is for the outside appearance of the function, >> ? ? including signature and annotations. ?Separating the internal >> ? ? information on early bound variables from the signature is the >> ? ? primary goal of the proposal after all. >> >> ?b) it does not make the intention of early binding very clear. >> >> ?c) it consists of a list of the form [i=i] or [tuple=tuple, len=len] >> ? ? in most use cases, making it necessary to type every name that >> ? ? shall use early binding twice. >> >> That's why I would prefer a syntax that explicitly declares the names >> that should use early binding, similar to "global" or "nonlocal" >> statements, if such a syntax is deemed necessary at all. > > The problem with an earlybind statement in the body is that it is in the > body, which should be runtime stuff. I would prefer something in the header. > A semi-colon or other char would be sufficient syntactically: > > def f(a; len, alist=[]): alist.append(a); return len(alist) > > where a bare identifier like 'len' *means* 'len=len'. I would be much more interested in a tweak to the language semantics where the compiler is allowed to assume that "len means len" if there is no global assignment to it in a module. There are many past threads about this topic. It should make the feature proposed here unnecessary -- at least its use for manual micro-optimizations, which I mostly find an offense to the reader (even in the shortened form proposed here). -- --Guido van Rossum (python.org/~guido) From python at mrabarnett.plus.com Tue Sep 20 19:43:07 2011 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 20 Sep 2011 18:43:07 +0100 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> Message-ID: <4E78D0AB.1060700@mrabarnett.plus.com> On 20/09/2011 17:59, Terry Reedy wrote: > On 9/20/2011 10:00 AM, Sven Marnach wrote: > >> I actually *do* use the default argument hack for early binding >> myself. My main points were that the current status quo isn't too >> bad, there are alternatives if the function signature is really >> important, and that the cases which require both using closures *and* >> having a clean signature are too rare to justify yet another >> confusing-for-beginners syntax. (When I started learning Python in >> 1998, the language was praised for having a small language core. I >> didn't hear this very often in more recent times.) > > I have noticed the same. The big syntax additions are generators, > comprehensions, decorators, and the with statement. The switch to > unicode doesn't change syntax but can complicate text processing. >> >> Assuming a new syntax *is* necessary -- there is no objective way to >> decide this after all -- I don't particularly like the proposed square >> bracket syntax because >> >> a) it puts the information at the wrong place. The first line of the >> function definition is for the outside appearance of the function, >> including signature and annotations. Separating the internal >> information on early bound variables from the signature is the >> primary goal of the proposal after all. >> >> b) it does not make the intention of early binding very clear. >> >> c) it consists of a list of the form [i=i] or [tuple=tuple, len=len] >> in most use cases, making it necessary to type every name that >> shall use early binding twice. >> >> That's why I would prefer a syntax that explicitly declares the names >> that should use early binding, similar to "global" or "nonlocal" >> statements, if such a syntax is deemed necessary at all. > > The problem with an earlybind statement in the body is that it is in the > body, which should be runtime stuff. I would prefer something in the > header. A semi-colon or other char would be sufficient syntactically: > > def f(a; len, alist=[]): alist.append(a); return len(alist) > > where a bare identifier like 'len' *means* 'len=len'. > To me that's like a 'static' declaration in C++, which in Python might be something like: def f(a): static len=len static alist=[] alist.append(a) return len(alist) What follows 'static' would be an assignment which is executed when the function is defined, much like a default argument, and would be shared between invocations in the same way. From tjreedy at udel.edu Wed Sep 21 05:36:35 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 Sep 2011 23:36:35 -0400 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> Message-ID: On 9/20/2011 1:09 PM, Guido van Rossum wrote: > On Tue, Sep 20, 2011 at 9:59 AM, Terry Reedy wrote: >> The problem with an earlybind statement in the body is that it is in the >> body, which should be runtime stuff. I would prefer something in the header. >> A semi-colon or other char would be sufficient syntactically: >> >> def f(a; len, alist=[]): alist.append(a); return len(alist) >> >> where a bare identifier like 'len' *means* 'len=len'. > > I would be much more interested in a tweak to the language semantics > where the compiler is allowed to assume that "len means len" if there > is no global assignment to it in a module. There are many past threads > about this topic. It should make the feature proposed here unnecessary > -- at least its use for manual micro-optimizations, which I mostly > find an offense to the reader (even in the shortened form proposed > here). For those not party to the earlier threads, this alternate proposal would bread any code that monkeypatches an imported-module-only override of a builtin, something like: def mylen(o): return import mod; mod.len = mylen I do not mind the semantic change itself, and it would benefit nearly all code, including module-level code would not be affected by def gimmicks. Some issues I see that perhaps were discussed, but I do not remember: 1) module.__setattr__ should then raise an exception at such attempts; 2) it makes the language more context sensitive than it currently is; 3) compile() should apply the rule even for single statements or even expressions. -- Terry Jan Reedy From ncoghlan at gmail.com Wed Sep 21 07:19:18 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Sep 2011 15:19:18 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> Message-ID: On Wed, Sep 21, 2011 at 1:36 PM, Terry Reedy wrote: > On 9/20/2011 1:09 PM, Guido van Rossum wrote: >> I would be much more interested in a tweak to the language semantics >> where the compiler is allowed to assume that "len means len" if there >> is no global assignment to it in a module. There are many past threads >> about this topic. It should make the feature proposed here unnecessary >> -- at least its use for manual micro-optimizations, which I mostly >> find an offense to the reader (even in the shortened form proposed >> here). > > For those not party to the earlier threads, this alternate proposal would > bread any code that monkeypatches an imported-module-only override of a > builtin, something like: > > def mylen(o): return > import mod; mod.len = mylen > > I do not mind the semantic change itself, and it would benefit nearly all > code, including module-level code would not be affected by def gimmicks. > Some issues I see that perhaps were discussed, but I do not remember: 1) > module.__setattr__ should then raise an exception at such attempts; 2) it > makes the language more context sensitive than it currently is; 3) compile() > should apply the rule even for single statements or even expressions. The two sticking points tend to be open() and print(). Overriding those externally can be *incredibly* useful for testing code that uses them, as well as in interacting with code that wasn't designed in a flexible way. Regardless, I'm going to stop mentioning the manual micro-optimisation use case for the default argument hack. Alex Gaynor pointed out that that aspect is completely irrelevant on PyPy, and as Guido notes, there are likely other ways to tackle name lookup optimisation even in CPython. That still leaves two arguably valid use cases: - early binding for name lookups - an algorithm with shared state (kind of like an inverted approach to creating a class with a custom __call__ method) Perhaps a non-syntax way to approach both of these would be to add a new decorator to 'functools': def closure(f): """Invokes the decorated function and returns the result after transcribing essential function metadata This can be used to easily share algorithm state and get early binding semantics for names. """ impl = f() impl.__name__ = f.__name__ doc = f.__doc__ if doc is not None: impl.__doc__ = doc impl.__dict__.update(f.__dict__) return impl This would be used as follows: @functools.closure def adder(i=i): # 'impl' defines call time signature "Increments 'x' by adder.value" def impl(x): impl.call_count += 1 return x + i impl.value = i impl.call_count = 0 return impl >>> adder.value 10 >>> adder(1) 11 >>> adder(5) 15 >>> adder(10) 20 >>> adder.call_count 3 Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From hetchkay at gmail.com Wed Sep 21 09:37:40 2011 From: hetchkay at gmail.com (H. Krishnan) Date: Wed, 21 Sep 2011 00:37:40 -0700 (PDT) Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <4E78D0AB.1060700@mrabarnett.plus.com> References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> <4E78D0AB.1060700@mrabarnett.plus.com> Message-ID: <8792534.1382.1316590661016.JavaMail.geo-discussion-forums@prfp37> To me that's like a 'static' declaration in C++, which in Python might > be something like: > > def f(a): > static len=len > static alist=[] > alist.append(a) > return len(alist) > > What follows 'static' would be an assignment which is executed when the > function is defined, much like a default argument, and would be shared > between invocations in the same way. > > How about def f(a) with len as len, [] as alist: alist.append(a) return len(alist) Or if "len as len" is redundant: def f(a) with len, [] as alist: alist.append(a) return len(alist) Krishnan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Wed Sep 21 20:12:27 2011 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 21 Sep 2011 13:12:27 -0500 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> Message-ID: <1316628747.20601.77.camel@Gutsy> On Wed, 2011-09-21 at 15:19 +1000, Nick Coghlan wrote: > Perhaps a non-syntax way to approach both of these would be to add a > new decorator to 'functools': > > def closure(f): > """Invokes the decorated function and returns the result after > transcribing essential function metadata > > This can be used to easily share algorithm state and get > early binding semantics for names. > """ > impl = f() > impl.__name__ = f.__name__ > doc = f.__doc__ > if doc is not None: > impl.__doc__ = doc > impl.__dict__.update(f.__dict__) > return impl > > This would be used as follows: > > @functools.closure > def adder(i=i): # 'impl' defines call time signature > "Increments 'x' by adder.value" > def impl(x): > impl.call_count += 1 > return x + i > impl.value = i > impl.call_count = 0 > return impl > > >>> adder.value > 10 > >>> adder(1) > 11 > >>> adder(5) > 15 > >>> adder(10) > 20 > >>> adder.call_count > 3 Simplifying things like this is one of the use cases of allowing define time statements. That's a lot of work to just avoid putting a keyword in the signature. And it's not easy to understand. Decorators could be a good way to do this, but the problem in these cases, is the function object doesn't have the needed support to make things like this easy. Probably the easiest and most direct way, would to be to add a new keyword 'static' as MRAB suggested, but have it be an expression instead of a command. value = (static ) def adder(x): return x + (static i) # evaluate (static i) at compile time. The parentheses would be optional. The (static i) expression, could be spelled (i=i). I think that was what Guido was suggesting, but not as an expression. As an expression, you would then see things like.. i = (i=i). But that may only be poor style, because it's easy enough to just not do that. I think the expression form is better myself, it allows you to get both the compile time value, and the current value of an identifier. def value elapsed_frames(): """ Where f is the frame counter in the parent scope. """ return f - (static f) Cheers, Ron From guido at python.org Wed Sep 21 20:34:17 2011 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Sep 2011 11:34:17 -0700 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <1316628747.20601.77.camel@Gutsy> References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> <1316628747.20601.77.camel@Gutsy> Message-ID: When you say "compile time", do you literally mean "when the compiler creates the bytecode" or do you really intend this to be computed at function definition time (the same time when default values are evaluated)? The latter seems to make more sense. --Guido On Wed, Sep 21, 2011 at 11:12 AM, Ron Adam wrote: > On Wed, 2011-09-21 at 15:19 +1000, Nick Coghlan wrote: > >> Perhaps a non-syntax way to approach both of these would be to add a >> new decorator to 'functools': >> >> ? ? def closure(f): >> ? ? ? ? """Invokes the decorated function and returns the result after >> transcribing essential function metadata >> >> ? ? ? ? ? ?This can be used to easily share algorithm state and get >> early binding semantics for names. >> ? ? ? ? """ >> ? ? ? ? impl = f() >> ? ? ? ? impl.__name__ = f.__name__ >> ? ? ? ? doc = f.__doc__ >> ? ? ? ? if doc is not None: >> ? ? ? ? ? ? impl.__doc__ = doc >> ? ? ? ? impl.__dict__.update(f.__dict__) >> ? ? ? ? return impl >> >> This would be used as follows: >> >> ? ? @functools.closure >> ? ? def adder(i=i): # 'impl' defines call time signature >> ? ? ? ? "Increments 'x' by adder.value" >> ? ? ? ? def impl(x): >> ? ? ? ? ? ? impl.call_count += 1 >> ? ? ? ? ? ? return x + i >> ? ? ? ? impl.value = i >> ? ? ? ? impl.call_count = 0 >> ? ? ? ? return impl >> >> >>> adder.value >> 10 >> >>> adder(1) >> 11 >> >>> adder(5) >> 15 >> >>> adder(10) >> 20 >> >>> adder.call_count >> 3 > > Simplifying things like this is one of the use cases of allowing define > time statements. ?That's a lot of work to just avoid putting a keyword > in the signature. ?And it's not easy to understand. > > Decorators could be a good way to do this, but the problem in these > cases, is the function object doesn't have the needed support to make > things like this easy. > > > Probably the easiest and most direct way, would to be to add a new > keyword 'static' as MRAB suggested, but have it be an expression instead > of a command. > > value = (static ) > > > def adder(x): > ? ?return x + (static i) ? ?# evaluate (static i) at compile time. > > The parentheses would be optional. > > The (static i) expression, could be spelled (i=i). ?I think that was > what Guido was suggesting, but not as an expression. ?As an expression, > you would then see things like.. i = (i=i). ?But that may only be poor > style, because it's easy enough to just not do that. > > I think the expression form is better myself, it allows you to get both > the compile time value, and the current value of an identifier. > > > def value elapsed_frames(): > ? ?""" Where f is the frame counter in the parent scope. """ > ? ?return f - (static f) > > > Cheers, > ? Ron > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (python.org/~guido) From python at mrabarnett.plus.com Wed Sep 21 21:30:41 2011 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 21 Sep 2011 20:30:41 +0100 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> <1316628747.20601.77.camel@Gutsy> Message-ID: <4E7A3B61.9020003@mrabarnett.plus.com> On 21/09/2011 19:34, Guido van Rossum wrote: > When you say "compile time", do you literally mean "when the compiler > creates the bytecode" or do you really intend this to be computed at > function definition time (the same time when default values are > evaluated)? The latter seems to make more sense. > It would be at function definition time. > --Guido > > On Wed, Sep 21, 2011 at 11:12 AM, Ron Adam wrote: >> On Wed, 2011-09-21 at 15:19 +1000, Nick Coghlan wrote: >> >>> Perhaps a non-syntax way to approach both of these would be to add a >>> new decorator to 'functools': >>> >>> def closure(f): >>> """Invokes the decorated function and returns the result after >>> transcribing essential function metadata >>> >>> This can be used to easily share algorithm state and get >>> early binding semantics for names. >>> """ >>> impl = f() >>> impl.__name__ = f.__name__ >>> doc = f.__doc__ >>> if doc is not None: >>> impl.__doc__ = doc >>> impl.__dict__.update(f.__dict__) >>> return impl >>> >>> This would be used as follows: >>> >>> @functools.closure >>> def adder(i=i): # 'impl' defines call time signature >>> "Increments 'x' by adder.value" >>> def impl(x): >>> impl.call_count += 1 >>> return x + i >>> impl.value = i >>> impl.call_count = 0 >>> return impl >>> >>>>>> adder.value >>> 10 >>>>>> adder(1) >>> 11 >>>>>> adder(5) >>> 15 >>>>>> adder(10) >>> 20 >>>>>> adder.call_count >>> 3 >> >> Simplifying things like this is one of the use cases of allowing define >> time statements. That's a lot of work to just avoid putting a keyword >> in the signature. And it's not easy to understand. >> >> Decorators could be a good way to do this, but the problem in these >> cases, is the function object doesn't have the needed support to make >> things like this easy. >> >> >> Probably the easiest and most direct way, would to be to add a new >> keyword 'static' as MRAB suggested, but have it be an expression instead >> of a command. >> >> value = (static) >> >> >> def adder(x): >> return x + (static i) # evaluate (static i) at compile time. >> >> The parentheses would be optional. >> >> The (static i) expression, could be spelled (i=i). I think that was >> what Guido was suggesting, but not as an expression. As an expression, >> you would then see things like.. i = (i=i). But that may only be poor >> style, because it's easy enough to just not do that. >> >> I think the expression form is better myself, it allows you to get both >> the compile time value, and the current value of an identifier. >> >> >> def value elapsed_frames(): >> """ Where f is the frame counter in the parent scope. """ >> return f - (static f) >> A function to calculate a Fibonacci number with memoisation would change from: def fib(n, known={}): if n in known: return known[n] if n < 2: f = n else: f = fib(n - 1) + fib(n - 2) known[n] = f return f to: def fib(n): known = static {} if n in known: return known[n] if n < 2: f = n else: f = fib(n - 1) + fib(n - 2) known[n] = f return f This: known = {} creates a dict and binds a name to it at execution time, whereas this: known = static {} creates a dict at definition time, storing a reference to it in the function object, and binds a name to it at execution time. I suppose it's like a class variable: class Foo: known = {} which is shared by its instances. From ron3200 at gmail.com Wed Sep 21 21:31:02 2011 From: ron3200 at gmail.com (ron adam) Date: Wed, 21 Sep 2011 14:31:02 -0500 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> <1316628747.20601.77.camel@Gutsy> Message-ID: <1316633462.21193.5.camel@Gutsy> On Wed, 2011-09-21 at 11:34 -0700, Guido van Rossum wrote: > When you say "compile time", do you literally mean "when the compiler > creates the bytecode" or do you really intend this to be computed at > function definition time (the same time when default values are > evaluated)? The latter seems to make more sense. > > --Guido Yes, I meant at definition time. :-) > On Wed, Sep 21, 2011 at 11:12 AM, Ron Adam wrote: > > On Wed, 2011-09-21 at 15:19 +1000, Nick Coghlan wrote: > > > >> Perhaps a non-syntax way to approach both of these would be to add a > >> new decorator to 'functools': > >> > >> def closure(f): > >> """Invokes the decorated function and returns the result after > >> transcribing essential function metadata > >> > >> This can be used to easily share algorithm state and get > >> early binding semantics for names. > >> """ > >> impl = f() > >> impl.__name__ = f.__name__ > >> doc = f.__doc__ > >> if doc is not None: > >> impl.__doc__ = doc > >> impl.__dict__.update(f.__dict__) > >> return impl > >> > >> This would be used as follows: > >> > >> @functools.closure > >> def adder(i=i): # 'impl' defines call time signature > >> "Increments 'x' by adder.value" > >> def impl(x): > >> impl.call_count += 1 > >> return x + i > >> impl.value = i > >> impl.call_count = 0 > >> return impl > >> > >> >>> adder.value > >> 10 > >> >>> adder(1) > >> 11 > >> >>> adder(5) > >> 15 > >> >>> adder(10) > >> 20 > >> >>> adder.call_count > >> 3 > > > > Simplifying things like this is one of the use cases of allowing define > > time statements. That's a lot of work to just avoid putting a keyword > > in the signature. And it's not easy to understand. > > > > Decorators could be a good way to do this, but the problem in these > > cases, is the function object doesn't have the needed support to make > > things like this easy. > > > > > > Probably the easiest and most direct way, would to be to add a new > > keyword 'static' as MRAB suggested, but have it be an expression instead > > of a command. > > > > value = (static ) > > > > > > def adder(x): > > return x + (static i) # evaluate (static i) at compile time. > > > > The parentheses would be optional. > > > > The (static i) expression, could be spelled (i=i). I think that was > > what Guido was suggesting, but not as an expression. As an expression, > > you would then see things like.. i = (i=i). But that may only be poor > > style, because it's easy enough to just not do that. > > > > I think the expression form is better myself, it allows you to get both > > the compile time value, and the current value of an identifier. > > > > > > def value elapsed_frames(): > > """ Where f is the frame counter in the parent scope. """ > > return f - (static f) > > > > > > Cheers, > > Ron > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > http://mail.python.org/mailman/listinfo/python-ideas > > > > > From guido at python.org Wed Sep 21 23:59:29 2011 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Sep 2011 14:59:29 -0700 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <1316633462.21193.5.camel@Gutsy> References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> <1316628747.20601.77.camel@Gutsy> <1316633462.21193.5.camel@Gutsy> Message-ID: Hm. I think doing this as an expression modifier is too wacky. But as a statement modifier, it could fit in the lineage of global and nonlocal. On Wed, Sep 21, 2011 at 12:31 PM, ron adam wrote: > On Wed, 2011-09-21 at 11:34 -0700, Guido van Rossum wrote: >> When you say "compile time", do you literally mean "when the compiler >> creates the bytecode" or do you really intend this to be computed at >> function definition time (the same time when default values are >> evaluated)? The latter seems to make more sense. >> >> --Guido > > Yes, I meant at definition time. :-) > > > >> On Wed, Sep 21, 2011 at 11:12 AM, Ron Adam wrote: >> > On Wed, 2011-09-21 at 15:19 +1000, Nick Coghlan wrote: >> > >> >> Perhaps a non-syntax way to approach both of these would be to add a >> >> new decorator to 'functools': >> >> >> >> ? ? def closure(f): >> >> ? ? ? ? """Invokes the decorated function and returns the result after >> >> transcribing essential function metadata >> >> >> >> ? ? ? ? ? ?This can be used to easily share algorithm state and get >> >> early binding semantics for names. >> >> ? ? ? ? """ >> >> ? ? ? ? impl = f() >> >> ? ? ? ? impl.__name__ = f.__name__ >> >> ? ? ? ? doc = f.__doc__ >> >> ? ? ? ? if doc is not None: >> >> ? ? ? ? ? ? impl.__doc__ = doc >> >> ? ? ? ? impl.__dict__.update(f.__dict__) >> >> ? ? ? ? return impl >> >> >> >> This would be used as follows: >> >> >> >> ? ? @functools.closure >> >> ? ? def adder(i=i): # 'impl' defines call time signature >> >> ? ? ? ? "Increments 'x' by adder.value" >> >> ? ? ? ? def impl(x): >> >> ? ? ? ? ? ? impl.call_count += 1 >> >> ? ? ? ? ? ? return x + i >> >> ? ? ? ? impl.value = i >> >> ? ? ? ? impl.call_count = 0 >> >> ? ? ? ? return impl >> >> >> >> >>> adder.value >> >> 10 >> >> >>> adder(1) >> >> 11 >> >> >>> adder(5) >> >> 15 >> >> >>> adder(10) >> >> 20 >> >> >>> adder.call_count >> >> 3 >> > >> > Simplifying things like this is one of the use cases of allowing define >> > time statements. ?That's a lot of work to just avoid putting a keyword >> > in the signature. ?And it's not easy to understand. >> > >> > Decorators could be a good way to do this, but the problem in these >> > cases, is the function object doesn't have the needed support to make >> > things like this easy. >> > >> > >> > Probably the easiest and most direct way, would to be to add a new >> > keyword 'static' as MRAB suggested, but have it be an expression instead >> > of a command. >> > >> > value = (static ) >> > >> > >> > def adder(x): >> > ? ?return x + (static i) ? ?# evaluate (static i) at compile time. >> > >> > The parentheses would be optional. >> > >> > The (static i) expression, could be spelled (i=i). ?I think that was >> > what Guido was suggesting, but not as an expression. ?As an expression, >> > you would then see things like.. i = (i=i). ?But that may only be poor >> > style, because it's easy enough to just not do that. >> > >> > I think the expression form is better myself, it allows you to get both >> > the compile time value, and the current value of an identifier. >> > >> > >> > def value elapsed_frames(): >> > ? ?""" Where f is the frame counter in the parent scope. """ >> > ? ?return f - (static f) >> > >> > >> > Cheers, >> > ? Ron >> > >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas at python.org >> > http://mail.python.org/mailman/listinfo/python-ideas >> > >> >> >> > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (python.org/~guido) From ericsnowcurrently at gmail.com Thu Sep 22 00:38:39 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 21 Sep 2011 16:38:39 -0600 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> <1316628747.20601.77.camel@Gutsy> <1316633462.21193.5.camel@Gutsy> Message-ID: On Wed, Sep 21, 2011 at 3:59 PM, Guido van Rossum wrote: > Hm. I think doing this as an expression modifier is too wacky. But as > a statement modifier, it could fit in the lineage of global and > nonlocal. Agreed, though just for assignment statements. Also, it may be confusing to allow more than one static assignment to the same name in the function body. Perhaps that should be a SyntaxError. -eric > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From greg.ewing at canterbury.ac.nz Thu Sep 22 02:06:10 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Sep 2011 12:06:10 +1200 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: References: <4E6FF971.7070504@canterbury.ac.nz> <4E729AAE.2040204@canterbury.ac.nz> Message-ID: <4E7A7BF2.1040306@canterbury.ac.nz> Guido van Rossum wrote: > Yeah, I'd like to see the new generated code. If it's even a single > opcode longer than the old code, that's a big cost that everybody pays In the prototype implementation it is slightly longer in some cases, but if we're willing to introduce a few more bytecode instructions it could be made the same length. I'll see about adding some discussion on this to the PEP. Concerning examples, there's already one included with the patch file, but it's not very visible anywhere. Would you like the examples included in-line in the PEP (they will be about 2-3 screenfuls each or so) or would it be better to host them elsewhere and just put links in the PEP? -- Greg From ncoghlan at gmail.com Thu Sep 22 02:09:03 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 22 Sep 2011 10:09:03 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> <1316628747.20601.77.camel@Gutsy> <1316633462.21193.5.camel@Gutsy> Message-ID: On Thu, Sep 22, 2011 at 8:38 AM, Eric Snow wrote: > On Wed, Sep 21, 2011 at 3:59 PM, Guido van Rossum wrote: >> Hm. I think doing this as an expression modifier is too wacky. But as >> a statement modifier, it could fit in the lineage of global and >> nonlocal. > > Agreed, though just for assignment statements. ? Also, it may be > confusing to allow more than one static assignment to the same name in > the function body. ?Perhaps that should be a SyntaxError. And so we come full circle... it looks like Jan didn't have the time to write up a PEP following the last discussion of this topic [2], so perhaps you'd be interested in taking up the task? FWIW, I still quite like the explicit 'atdef' keyword I suggested last time [1] (and most of the arguments from that thread against the 'after-the-**' syntax are equally valid arguments against the '[] in the function header' syntax from this iteration). [1] http://mail.python.org/pipermail/python-ideas/2011-June/010565.html [2] http://mail.python.org/pipermail/python-ideas/2011-June/010552.html Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Thu Sep 22 02:08:54 2011 From: guido at python.org (Guido van Rossum) Date: Wed, 21 Sep 2011 17:08:54 -0700 Subject: [Python-ideas] Break the dominance of boolean values in boolean context In-Reply-To: <4E7A7BF2.1040306@canterbury.ac.nz> References: <4E6FF971.7070504@canterbury.ac.nz> <4E729AAE.2040204@canterbury.ac.nz> <4E7A7BF2.1040306@canterbury.ac.nz> Message-ID: Just put it all in the PEP -- that way it's all together. On Wed, Sep 21, 2011 at 5:06 PM, Greg Ewing wrote: > Guido van Rossum wrote: > >> Yeah, I'd like to see the new generated code. If it's even a single >> opcode longer than the old code, that's a big cost that everybody pays > > In the prototype implementation it is slightly longer > in some cases, but if we're willing to introduce a few > more bytecode instructions it could be made the same > length. > > I'll see about adding some discussion on this to the > PEP. > > Concerning examples, there's already one included with > the patch file, but it's not very visible anywhere. > > Would you like the examples included in-line in the PEP > (they will be about 2-3 screenfuls each or so) or would > it be better to host them elsewhere and just put links > in the PEP? > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (python.org/~guido) From greg.ewing at canterbury.ac.nz Thu Sep 22 02:12:41 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Sep 2011 12:12:41 +1200 Subject: [Python-ideas] win32 extensions In-Reply-To: References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> <4E713201.4090105@canterbury.ac.nz> <4E7180F1.6080107@canterbury.ac.nz> <4E729315.2060303@canterbury.ac.nz> Message-ID: <4E7A7D79.9080206@canterbury.ac.nz> Michael Foord wrote: > For accessing .NET from Python IronPython is a > much better alternative (with caveats about the use of C extensions that > apply to any of the alternative implementations). Well, those caveats are rather important to someone who doesn't want to live entirely in the .NET world, but just wants to use some of the things that it provides. -- Greg From fuzzyman at gmail.com Thu Sep 22 02:21:39 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Thu, 22 Sep 2011 01:21:39 +0100 Subject: [Python-ideas] win32 extensions In-Reply-To: <4E7A7D79.9080206@canterbury.ac.nz> References: <93DDD7EB-72FA-454C-9C56-7C8125B16652@gmail.com> <878vptkty3.fsf@benfinney.id.au> <4E6FFE92.8080307@canterbury.ac.nz> <4E70A677.3070002@canterbury.ac.nz> <4E713201.4090105@canterbury.ac.nz> <4E7180F1.6080107@canterbury.ac.nz> <4E729315.2060303@canterbury.ac.nz> <4E7A7D79.9080206@canterbury.ac.nz> Message-ID: On 22 September 2011 01:12, Greg Ewing wrote: > Michael Foord wrote: > >> For accessing .NET from Python IronPython is a much better alternative >> (with caveats about the use of C extensions that apply to any of the >> alternative implementations). >> > > Well, those caveats are rather important to someone who doesn't > want to live entirely in the .NET world, but just wants to > use some of the things that it provides. > That in itself doesn't make adding Python.NET to the standard library feasible... Michael > > -- > Greg > > ______________________________**_________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/**mailman/listinfo/python-ideas > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Thu Sep 22 02:37:05 2011 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 22 Sep 2011 01:37:05 +0100 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> <1316628747.20601.77.camel@Gutsy> <1316633462.21193.5.camel@Gutsy> Message-ID: <4E7A8331.8060904@mrabarnett.plus.com> On 21/09/2011 23:38, Eric Snow wrote: > On Wed, Sep 21, 2011 at 3:59 PM, Guido van Rossum wrote: >> Hm. I think doing this as an expression modifier is too wacky. But as >> a statement modifier, it could fit in the lineage of global and >> nonlocal. > > Agreed, though just for assignment statements. Also, it may be > confusing to allow more than one static assignment to the same name in > the function body. Perhaps that should be a SyntaxError. > I'm not sure that SyntaxError would be right (it's not the syntax that's the problem!). NameError, perhaps? From zuo at chopin.edu.pl Thu Sep 22 03:01:40 2011 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Thu, 22 Sep 2011 03:01:40 +0200 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> Message-ID: <20110922010140.GA4237@chopin.edu.pl> I'm adding both new propositions: 1. the `def foo(...) [len=len]:` syntax, 2. the `len = (static len)` expression syntax -- to the PEP-draft I'm preparing -- which Nick suggested in June [http://mail.python.org/pipermail/python-ideas/2011-June/010569.html] (I'm sorry that that preparing lasts so much time, but my everyday- -activity-CPU has been overloaded a bit for a few months...). Ad 1: I think it's better than the `after-**` proposition from June, though still has some its drawbacks (Sven just mentioned some of them). Ad 2: I must admit that this one becomes my favorite syntax for early binding (though I don't like the abbreviated form '(i=i)'). IMHO it's not only clear (no all that questions about assignment semantics) but also elegant, explicit and consistent with some existing syntax constructs (especially with `yield EXPR`). Note that (as you can use any expression) it makes possible to use e.g. tuples of several names: spam, print, open = static (len, print, open) Big +1 from me. 3. Another variant could be with a colon (a bit similar to the lambda syntax): len = static: len spam, print, open = static: (spam, print, open) or even (reusing the existing keyword): len = def: len spam, print, open = def: (spam, print, open) (here "def" means: "at definition time") But I'd rather prefer the #2. Regards. *j From ericsnowcurrently at gmail.com Thu Sep 22 04:31:32 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 21 Sep 2011 20:31:32 -0600 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <20110922010140.GA4237@chopin.edu.pl> References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> Message-ID: On Wed, Sep 21, 2011 at 7:01 PM, Jan Kaliszewski wrote: > I'm adding both new propositions: > > 1. the `def foo(...) [len=len]:` syntax, > 2. the `len = (static len)` expression syntax > > -- to the PEP-draft I'm preparing -- which Nick suggested in June > [http://mail.python.org/pipermail/python-ideas/2011-June/010569.html] > (I'm sorry that that preparing lasts so much time, but my everyday- > -activity-CPU has been overloaded a bit for a few months...). > > > Ad 1: I think it's better than the `after-**` proposition from June, > though still has some its drawbacks (Sven just mentioned some of them). > > > Ad 2: I must admit that this one becomes my favorite syntax for > early binding (though I don't like the abbreviated form '(i=i)'). > > IMHO it's not only clear (no all that questions about assignment > semantics) but also elegant, explicit and consistent with some existing > syntax constructs (especially with `yield EXPR`). > > Note that (as you can use any expression) it makes possible to use e.g. > tuples of several names: > > ? ?spam, print, open = static (len, print, open) > > Big +1 from me. Agreed, though the the keyword should be on the LHS: static spam, print, open = len, print, open It's more consistent with other keywords that way and easier to spot when reading code. Also, "static" as a keyword is in use as an identifier in a number of places per google's codesearch (on the order of 100) including in Django. It's not like it's in super broad use, but it is in use. -eric > > > 3. Another variant could be with a colon (a bit similar to the lambda > syntax): > > ? ?len = static: len > ? ?spam, print, open = static: (spam, print, open) > > or even (reusing the existing keyword): > > ? ?len = def: len > ? ?spam, print, open = def: (spam, print, open) > > (here "def" means: "at definition time") > > But I'd rather prefer the #2. > > > Regards. > *j > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From pyideas at rebertia.com Thu Sep 22 04:46:59 2011 From: pyideas at rebertia.com (Chris Rebert) Date: Wed, 21 Sep 2011 19:46:59 -0700 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> Message-ID: On Wed, Sep 21, 2011 at 7:31 PM, Eric Snow wrote: > On Wed, Sep 21, 2011 at 7:01 PM, Jan Kaliszewski wrote: >> I'm adding both new propositions: >> >> 1. the `def foo(...) [len=len]:` syntax, >> 2. the `len = (static len)` expression syntax >> >> -- to the PEP-draft I'm preparing -- which Nick suggested in June >> Ad 2: I must admit that this one becomes my favorite syntax for >> early binding (though I don't like the abbreviated form '(i=i)'). >> >> IMHO it's not only clear (no all that questions about assignment >> semantics) but also elegant, explicit and consistent with some existing >> syntax constructs (especially with `yield EXPR`). >> >> Note that (as you can use any expression) it makes possible to use e.g. >> tuples of several names: >> >> ? ?spam, print, open = static (len, print, open) >> >> Big +1 from me. > > Agreed, though the the keyword should be on the LHS: > > ?static spam, print, open = len, print, open > > It's more consistent with other keywords that way and easier to spot > when reading code. +1 (FWIW) on this "assignment statement with leading keyword" variant. `static` is an excellent choice of keyword for this. Cheers, Chris From ron3200 at gmail.com Thu Sep 22 05:44:20 2011 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 21 Sep 2011 22:44:20 -0500 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> Message-ID: <1316663060.31837.26.camel@Gutsy> On Wed, 2011-09-21 at 20:31 -0600, Eric Snow wrote: > On Wed, Sep 21, 2011 at 7:01 PM, Jan Kaliszewski wrote: > > I'm adding both new propositions: > > > > 1. the `def foo(...) [len=len]:` syntax, > > 2. the `len = (static len)` expression syntax > > > > -- to the PEP-draft I'm preparing -- which Nick suggested in June > > [http://mail.python.org/pipermail/python-ideas/2011-June/010569.html] > > (I'm sorry that that preparing lasts so much time, but my everyday- > > -activity-CPU has been overloaded a bit for a few months...). > > > > > > Ad 1: I think it's better than the `after-**` proposition from June, > > though still has some its drawbacks (Sven just mentioned some of them). > > > > > > Ad 2: I must admit that this one becomes my favorite syntax for > > early binding (though I don't like the abbreviated form '(i=i)'). > > > > IMHO it's not only clear (no all that questions about assignment > > semantics) but also elegant, explicit and consistent with some existing > > syntax constructs (especially with `yield EXPR`). > > > > Note that (as you can use any expression) it makes possible to use e.g. > > tuples of several names: > > > > spam, print, open = static (len, print, open) > > > > Big +1 from me. > > Agreed, though the the keyword should be on the LHS: > > static spam, print, open = len, print, open The above could also be ... spam, print, open = (static len, print, open) As Jan pointed out, it would be very similar to how the 'yield' is used. It doesn't do anything to the identifiers on the left side. The only thing the static expression does is save a reference to what ever the expression part on the right evaluates to at the time the function is defined. Later, when the function is called, is when it's actually used. At define time... spam, print, open = static (len, print, open) Becomes this at call time... spam, print, open = _tuple #tuple created by static expression Cheers, Ron (Apologies for any double or delayed posts, I'm still trying to get some email glitches worked out.) > It's more consistent with other keywords that way and easier to spot > when reading code. > > Also, "static" as a keyword is in use as an identifier in a number of > places per google's codesearch (on the order of 100) including in > Django. It's not like it's in super broad use, but it is in use. From cmjohnson.mailinglist at gmail.com Thu Sep 22 06:21:45 2011 From: cmjohnson.mailinglist at gmail.com (Carl Matthew Johnson) Date: Wed, 21 Sep 2011 18:21:45 -1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <1316663060.31837.26.camel@Gutsy> References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <1316663060.31837.26.camel@Gutsy> Message-ID: Well, now that we're getting down to picking bike shed colors, time to chime in: blah = static blah is worse than static blah = blah because it's easier to visually pick out something on the LHS and it's analogous to global/nonlocal. Anything having to do with parens is visually ugly and should be avoided unless absolutely needed for explicitness. One question I have is how this will work with control flow. This example is works with Python 2.7 and 3.2: >>> x = 0 >>> def f(): ... if False: global x ... x = 7 ... return x ... >>> f() 7 >>> x 7 Presumably static would work the same way? Still, I find this sort of distasteful, since the behavior is a bit surprising to those not familiar with the in's and out's of the compilation model. Why not put the static declaration in the argument list, something like: def f(static l=[]): And maybe as a shortcut for static NAME=NAME, use just static NAME. fs = [] for i in range(10): def f(static i): return i fs.append(f) Looks OK to me. From ncoghlan at gmail.com Thu Sep 22 06:34:24 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 22 Sep 2011 14:34:24 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> Message-ID: On Thu, Sep 22, 2011 at 12:46 PM, Chris Rebert wrote: > +1 (FWIW) on this "assignment statement with leading keyword" variant. > `static` is an excellent choice of keyword for this. Care to elaborate? This has always struck me as one of the more nonsensical keyword choices in C, so I'd like to see some solid justification before perpetuating it in Python (particular when C/C++ intuitions about statics being process global will hinder rather than help in understanding the scope of this new namespace). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Thu Sep 22 06:40:33 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 22 Sep 2011 14:40:33 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <1316663060.31837.26.camel@Gutsy> Message-ID: On Thu, Sep 22, 2011 at 2:21 PM, Carl Matthew Johnson wrote: > Presumably static would work the same way? Still, I find this sort of distasteful, since the behavior is a bit surprising to those not familiar with the in's and out's of the compilation model. There's a reason style guides say not to do that with global and nonlocal, even though the language technically allows it. Definition time variable declarations would work the same way. > Why not put the static declaration in the argument list, something like: > > def f(static l=[]): For all the same reasons people don't like the 'after **' and '[] in the function header' suggestions. Most notably, the fact that the evaluation of certain expressions at definition time is an implementation detail rather than part of the public API, so it doesn't really belong in the header. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ron3200 at gmail.com Thu Sep 22 09:47:27 2011 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 22 Sep 2011 02:47:27 -0500 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <1316663060.31837.26.camel@Gutsy> Message-ID: <1316677647.634.104.camel@Gutsy> On Wed, 2011-09-21 at 18:21 -1000, Carl Matthew Johnson wrote: > Well, now that we're getting down to picking bike shed colors, time to chime in: > > blah = static blah > > is worse than > > static blah = blah Which way, will depend on what kind of behavior is decided on. Statement? or Expression? As a statement, it would be first and not have the assignment. static blah (if it does, then working out how that works will be tricky.) As an expression, the only thing that matters is that the names used in the expression, are findable at define time. No assignment is needed. The value takes the place of the static expression when the function it is in is called. Here the expression "(static y)" is replaced with the object y was bound to at the time the function was defined. y = ['red', 'blue', 'green'] ... def f() ... for x in (static y): ... The parentheses are optional if the expression only has single value. But I think it makes it easier to see what is being done. An expression you can be used in lambdas. A statement can't. Static statement: def get_adders(count): adders = [] for n in range(count): def add_it(x): static n return n + x adders.append(add_it) return adders Static expression: def get_adders(count): adders = [] for n in range(count): adders.append(lambda x: return (static n) + x) return adders > One question I have is how this will work with control flow. This example is works with Python 2.7 and 3.2: > > >>> x = 0 > >>> def f(): > ... if False: global x > ... x = 7 > ... return x > ... > >>> f() > 7 > >>> x > 7 > > Presumably static would work the same way? Still, I find this sort of distasteful, > since the behavior is a bit surprising to those not familiar with the in's and out's > of the compilation model. I can see why that is surprising. I don't think the static would have this issue in either form. Cheers, Ron From ericsnowcurrently at gmail.com Thu Sep 22 09:57:06 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 22 Sep 2011 01:57:06 -0600 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <1316663060.31837.26.camel@Gutsy> References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <1316663060.31837.26.camel@Gutsy> Message-ID: On Wed, Sep 21, 2011 at 9:44 PM, Ron Adam wrote: > On Wed, 2011-09-21 at 20:31 -0600, Eric Snow wrote: >> Agreed, though the the keyword should be on the LHS: >> >> ? static spam, print, open = len, print, open > > > The above could also be ... > > ? ? spam, print, open = (static len, print, open) > > > As Jan pointed out, it would be very similar to how the 'yield' is used. > > It doesn't do anything to the identifiers on the left side. Yeah, "static" on the LHS does imply a different meaning. Perhaps "static" is the wrong keyword, for all the baggage it carries. And a real "static" statement may not even be the right solution[1]. We're looking at two approaches here: do it in the function "header" or in the body. Here're some pros and cons: In-header Pros: - implies definition time In-header Cons: - limited space so more info adds clutter - could be mistaken as part of function's signature In-body Pros: - less cluttered - more closely associated with function locals? In-body Cons: - easy to miss that it's a definition-time directive (particularly if on RHS of assignment) - evaluated expression must not reference any of the function's locals We definitely want the solution to be explicit that it's a definition-time behavior. If the new syntax is in the body then it'll have to be especially obvious. Either way, the new syntax should clearly isolate the expression and associate it with a name to which it will be bound in the locals. All things considered, I don't think the that RHS static expression fits the bill. The right solution seems to be pretty elusive here... As an aside, what would take precedence in the case of a name collision: default arguments or the new syntax? -eric [1] If a static statement had an assignment clause for the initial value, then it would work. However, it doesn't seem explicit enough that the value is evaluated at definition time. [2] Ideally we would just use attributes on the function object. However, this has a number of problems including performance and the ambiguity of to which object the function name points. (I have a patch in issue 12857 that addresses exactly that.) > > > The only thing the static expression does is save a reference to what > ever the expression part on the right evaluates to at the time the > function is defined. ?Later, when the function is called, is when it's > actually used. > > At define time... > > ? ? spam, print, open = static (len, print, open) > > > Becomes this at call time... > > ? ? spam, print, open = _tuple ?#tuple created by static expression > > > Cheers, > ? Ron > > (Apologies for any double or delayed posts, I'm still trying to get some > email glitches worked out.) > > >> It's more consistent with other keywords that way and easier to spot >> when reading code. >> >> Also, "static" as a keyword is in use as an identifier in a number of >> places per google's codesearch (on the order of 100) including in >> Django. ?It's not like it's in super broad use, but it is in use. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From zuo at chopin.edu.pl Thu Sep 22 10:03:37 2011 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Thu, 22 Sep 2011 10:03:37 +0200 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> Message-ID: <20110922080337.GA2356@chopin.edu.pl> Nick Coghlan dixit (2011-09-22, 14:34): > On Thu, Sep 22, 2011 at 12:46 PM, Chris Rebert wrote: > > +1 (FWIW) on this "assignment statement with leading keyword" variant. > > `static` is an excellent choice of keyword for this. > > Care to elaborate? This has always struck me as one of the more > nonsensical keyword choices in C, so I'd like to see some solid > justification before perpetuating it in Python (particular when C/C++ > intuitions about statics being process global will hinder rather than > help in understanding the scope of this new namespace). I'd rather suggest one of: * ondef * atdef * deftime * defbound * early * earlybound Cheers. *j From zuo at chopin.edu.pl Thu Sep 22 10:11:35 2011 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Thu, 22 Sep 2011 10:11:35 +0200 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> Message-ID: <20110922081135.GB2356@chopin.edu.pl> Eric Snow dixit (2011-09-21, 20:31): > On Wed, Sep 21, 2011 at 7:01 PM, Jan Kaliszewski > wrote: [snip] > > ? ?spam, print, open = static (len, print, open) > > > > Big +1 from me. > > Agreed, though the the keyword should be on the LHS: > > static spam, print, open = len, print, open But then you lose important advantages of the expression variant -- especially that assignment is not a part of the syntax construct so it not only more powerfull but also more explicit and clear for newcomers: any assignment is at run-time and normal assignment rules apply to it. A bit like in case of `yield EXPR`. Cheers. *j From p.f.moore at gmail.com Thu Sep 22 10:21:12 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 22 Sep 2011 09:21:12 +0100 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <20110922081135.GB2356@chopin.edu.pl> References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> Message-ID: On 22 September 2011 09:11, Jan Kaliszewski wrote: > But then you lose important advantages of the expression variant -- > especially that assignment is not a part of the syntax construct so it > not only more powerfull but also more explicit and clear for newcomers: > any assignment is at run-time and normal assignment rules apply to it. > > A bit like in case of `yield EXPR`. A concern I have with the expression variant is that I don't understand what it would mean except in the restricted contexts it's been discussed in. Can you describe the semantics of static EXPR in isolation, so that we're not just interpreting it in terms of its use in an assignment, and can understand the wider implications? Thanks, Paul. From ncoghlan at gmail.com Thu Sep 22 15:58:41 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 22 Sep 2011 23:58:41 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> Message-ID: On Thu, Sep 22, 2011 at 6:21 PM, Paul Moore wrote: > A concern I have with the expression variant is that I don't > understand what it would mean except in the restricted contexts it's > been discussed in. Can you describe the semantics of > > ? ?static EXPR > > in isolation, so that we're not just interpreting it in terms of its > use in an assignment, and can understand the wider implications? It isn't really that different from the statement version - a definition time expression would be calculated at definition time by the interpreter and the resulting value cached on the function object. At execution time, the cached value would be used in place of the original expression. (As an implementation detail, this would likely work by assigning such expressions an index in the function's closure list and populating them at definition time as cells) To use the classic "adder in a loop" example: # default argument hack adders = [] for i in range(1): def adder(x, i=i): return x + i adders.append(adder) # default argument hack (lambda) adders = [(lambda x, i=i: x + i) for i in range(10)] # Definition time statement (lambda variant not supported) adders = [] for i in range(1): def adder(x): atdef i=i # Could conceivably shorten this to just 'atdef i' return x + i adders.append(adder) # Definition time expression (using same parenthesis rules as yield expressions) adders = [] for i in range(1): def adder(x): return x + (atdef i) adders.append(adder) # Definition time expression (lambda) adders = [(lambda x: x + (atdef i)) for i in range(10)] I think there's a case to be made for the expression version - it's likely to require less boilerplate than the statement version in simple cases and is compatible with comprehension syntax (since it can be embedded inside a lambda expression). The analogy with yield expressions is a reasonable one - but instead of coming from send(), the result of the expression is retrieved from the cache on the function object. The other advantage of the expression version is that it avoids the issue of definition a new namespace where names can be looked up. Instead, it's just a mechanism for caching the values of certain expressions at definition time - at execution time, those values can then either be used directly or else assigned to an ordinary local variable. I wasn't initially a fan of the idea, but it's growing on me. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From p.f.moore at gmail.com Thu Sep 22 16:34:23 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 22 Sep 2011 15:34:23 +0100 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> Message-ID: On 22 September 2011 14:58, Nick Coghlan wrote: > On Thu, Sep 22, 2011 at 6:21 PM, Paul Moore wrote: >> A concern I have with the expression variant is that I don't >> understand what it would mean except in the restricted contexts it's >> been discussed in. Can you describe the semantics of >> >> ? ?static EXPR >> >> in isolation, so that we're not just interpreting it in terms of its >> use in an assignment, and can understand the wider implications? > > It isn't really that different from the statement version - a > definition time expression would be calculated at definition time by > the interpreter and the resulting value cached on the function object. > At execution time, the cached value would be used in place of the > original expression. (As an implementation detail, this would likely > work by assigning such expressions an index in the function's closure > list and populating them at definition time as cells) OK, I think I get it now. So the example a, b, c = atdef ([], {}, set()) would create and cache a tuple at definition-time, then retrieve and unpack it at runtime. Which means that it has the same effect as a = atdef [] b = atdef {} c = atdef set() with some slightly subtle differences in detail (there's only one cached "value" rather than 3, but it'd be hard to detect that in practice. I think that based on your definition, all my "but what about X" pathological examples have reasonably sane behaviours (you need a good understanding of what "definition time" actually means, but once you internalise the fact that def is an executable statement, that isn't too hard). I can see the attractions of the idea but: * I'm sorry but the keyword atdef is ugly. The only proposal I like is static, but I agree that it only makes sense to someone with a C-type background. * I remain unconvinced that the problem is severe enough to warrant a new keyword. * The semantics, although clear and easy to state, are probably one more step away from Python being a simple language. I can imagine beginners' heads exploding...[1] Paul. [1] Although anyone needing the semantics is arguably no longer a beginner :-) From jeanpierreda at gmail.com Thu Sep 22 17:18:50 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 22 Sep 2011 11:18:50 -0400 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <1316628747.20601.77.camel@Gutsy> References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> <1316628747.20601.77.camel@Gutsy> Message-ID: > Decorators could be a good way to do this, but the problem in these > cases, is the function object doesn't have the needed support to make > things like this easy. > > > Probably the easiest and most direct way, would to be to add a new > keyword 'static' as MRAB suggested, but have it be an expression instead > of a command. I hope I'm not bikeshedding here, but I much prefer decorators because they let you do more than just micro-optimization. They were mentioned before, in another thread, and I find them attractive. A decorator basically lets you inject locals into already-existing functions. So if you want to, e.g., test a function that uses a global "urlopen" function to download a file, by replacing urlopen. from urllib import urlopen from StringIO import StringIO def myfunction(): return urlopen('...').read() class TestMyFunction(unittest.TestCase): def setUp(self): self.func = inject(urlopen=lambda s: StringIO(s) def test_opens(self): self.assertEqual(self.func(), '...') There's also the benefits of not altering the language etc. -- it's just a function that returns a new function that's very similar to the old one. That simplicity appeals to me as well. And it still satisfies the use-case of the new "static" keyword. Devin On Wed, Sep 21, 2011 at 2:12 PM, Ron Adam wrote: > On Wed, 2011-09-21 at 15:19 +1000, Nick Coghlan wrote: > >> Perhaps a non-syntax way to approach both of these would be to add a >> new decorator to 'functools': >> >> ? ? def closure(f): >> ? ? ? ? """Invokes the decorated function and returns the result after >> transcribing essential function metadata >> >> ? ? ? ? ? ?This can be used to easily share algorithm state and get >> early binding semantics for names. >> ? ? ? ? """ >> ? ? ? ? impl = f() >> ? ? ? ? impl.__name__ = f.__name__ >> ? ? ? ? doc = f.__doc__ >> ? ? ? ? if doc is not None: >> ? ? ? ? ? ? impl.__doc__ = doc >> ? ? ? ? impl.__dict__.update(f.__dict__) >> ? ? ? ? return impl >> >> This would be used as follows: >> >> ? ? @functools.closure >> ? ? def adder(i=i): # 'impl' defines call time signature >> ? ? ? ? "Increments 'x' by adder.value" >> ? ? ? ? def impl(x): >> ? ? ? ? ? ? impl.call_count += 1 >> ? ? ? ? ? ? return x + i >> ? ? ? ? impl.value = i >> ? ? ? ? impl.call_count = 0 >> ? ? ? ? return impl >> >> >>> adder.value >> 10 >> >>> adder(1) >> 11 >> >>> adder(5) >> 15 >> >>> adder(10) >> 20 >> >>> adder.call_count >> 3 > > Simplifying things like this is one of the use cases of allowing define > time statements. ?That's a lot of work to just avoid putting a keyword > in the signature. ?And it's not easy to understand. > > Decorators could be a good way to do this, but the problem in these > cases, is the function object doesn't have the needed support to make > things like this easy. > > > Probably the easiest and most direct way, would to be to add a new > keyword 'static' as MRAB suggested, but have it be an expression instead > of a command. > > value = (static ) > > > def adder(x): > ? ?return x + (static i) ? ?# evaluate (static i) at compile time. > > The parentheses would be optional. > > The (static i) expression, could be spelled (i=i). ?I think that was > what Guido was suggesting, but not as an expression. ?As an expression, > you would then see things like.. i = (i=i). ?But that may only be poor > style, because it's easy enough to just not do that. > > I think the expression form is better myself, it allows you to get both > the compile time value, and the current value of an identifier. > > > def value elapsed_frames(): > ? ?""" Where f is the frame counter in the parent scope. """ > ? ?return f - (static f) > > > Cheers, > ? Ron > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From python at mrabarnett.plus.com Thu Sep 22 17:55:18 2011 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 22 Sep 2011 16:55:18 +0100 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> Message-ID: <4E7B5A66.5050405@mrabarnett.plus.com> On 22/09/2011 14:58, Nick Coghlan wrote: > On Thu, Sep 22, 2011 at 6:21 PM, Paul Moore wrote: >> A concern I have with the expression variant is that I don't >> understand what it would mean except in the restricted contexts it's >> been discussed in. Can you describe the semantics of >> >> static EXPR >> >> in isolation, so that we're not just interpreting it in terms of its >> use in an assignment, and can understand the wider implications? > > It isn't really that different from the statement version - a > definition time expression would be calculated at definition time by > the interpreter and the resulting value cached on the function object. > At execution time, the cached value would be used in place of the > original expression. (As an implementation detail, this would likely > work by assigning such expressions an index in the function's closure > list and populating them at definition time as cells) > > To use the classic "adder in a loop" example: > > # default argument hack > adders = [] > for i in range(1): > def adder(x, i=i): > return x + i > adders.append(adder) > > # default argument hack (lambda) > adders = [(lambda x, i=i: x + i) for i in range(10)] > > # Definition time statement (lambda variant not supported) > adders = [] > for i in range(1): > def adder(x): > atdef i=i # Could conceivably shorten this to just 'atdef i' > return x + i > adders.append(adder) > > # Definition time expression (using same parenthesis rules as > yield expressions) > adders = [] > for i in range(1): > def adder(x): > return x + (atdef i) > adders.append(adder) > > # Definition time expression (lambda) > adders = [(lambda x: x + (atdef i)) for i in range(10)] > > I think there's a case to be made for the expression version - it's > likely to require less boilerplate than the statement version in > simple cases and is compatible with comprehension syntax (since it can > be embedded inside a lambda expression). The analogy with yield > expressions is a reasonable one - but instead of coming from send(), > the result of the expression is retrieved from the cache on the > function object. > > The other advantage of the expression version is that it avoids the > issue of definition a new namespace where names can be looked up. > Instead, it's just a mechanism for caching the values of certain > expressions at definition time - at execution time, those values can > then either be used directly or else assigned to an ordinary local > variable. > There are 2 issues here: 1. Shared variables: def accumulate(n): shared total = 0 total += n return total 2. Definition-time expressions: adders = [] for i in range(1): def adder(x): return x + deftime i adders.append(adder) From jimjjewett at gmail.com Thu Sep 22 18:06:43 2011 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 22 Sep 2011 12:06:43 -0400 Subject: [Python-ideas] static vs atdef, etc Message-ID: On Thu, Sep 22, 2011 at 10:34 AM, Paul Moore wrote: > * I'm sorry but the keyword atdef is ugly. The only proposal I like is > static, but I agree that it only makes sense to someone with a C-type > background. Even with a C background, static is a bad choice, because it has already been used for too many different meanings. (Admittedly, not all in C.) -jJ From jimjjewett at gmail.com Thu Sep 22 18:16:27 2011 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 22 Sep 2011 12:16:27 -0400 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <1316677647.634.104.camel@Gutsy> References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <1316663060.31837.26.camel@Gutsy> <1316677647.634.104.camel@Gutsy> Message-ID: On Thu, Sep 22, 2011 at 3:47 AM, Ron Adam wrote: > As an expression, the only thing that matters is that the names used in > the expression, are findable at define time. Not just that the names are findable, but that they are already bound to something. def f(a): b=a static c=b should fail, and should probably be a syntax error to prevent it apparently working (but perhaps with the wrong value) when (and only when) a more global b was set earlier. >?No assignment is needed. Agreed; in particular, static imports should reduce the cost of importing from inside a function body. (But then will people effectively start spelling "import" as "static import" even at the top of a module?) -jJ From guido at python.org Thu Sep 22 18:58:38 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Sep 2011 09:58:38 -0700 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <1316663060.31837.26.camel@Gutsy> <1316677647.634.104.camel@Gutsy> Message-ID: FWIW, I'm at best +0 on doing anything here; I'm -1 on the expression-style (static ) form but could live with the statement (whether or not combined with assignment -- IIRC that was planned for nonlocal and I don't see why we couldn't upgrade nonlocal and global at the same time). The main argument for static would be that this is what all other relevant languages call it. If we really want to do something else, I recommend reviving Algol-60's "OWN" keyword. -- --Guido van Rossum (python.org/~guido) From steve at pearwood.info Thu Sep 22 19:41:29 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Sep 2011 03:41:29 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> <1316628747.20601.77.camel@Gutsy> Message-ID: <4E7B7349.3070306@pearwood.info> Devin Jeanpierre wrote: >> Decorators could be a good way to do this, but the problem in these >> cases, is the function object doesn't have the needed support to make >> things like this easy. >> >> >> Probably the easiest and most direct way, would to be to add a new >> keyword 'static' as MRAB suggested, but have it be an expression instead >> of a command. > > I hope I'm not bikeshedding here, but I much prefer decorators because > they let you do more than just micro-optimization. They were mentioned > before, in another thread, and I find them attractive. > > A decorator basically lets you inject locals into already-existing > functions. So if you want to, e.g., test a function that uses a global > "urlopen" function to download a file, by replacing urlopen. I find this idea extremely exciting. But have I missed something? Does this inject function actually exist somewhere? If I recall correctly, I've seen a bytecode hack or two that seemed to work, but I didn't think that was officially supported. Comparing syntax: def func(a, b): static f, g, h f = ... g = ... h = ... return f(g(h(a+b))) @inject( f = ... g = ... h = ...) def func(a, b): return f(g(h(a+b))) or even: def func(a, b): return f(g(h(a+b))) another_func = @inject( f = ... g = ... h = ...)(func) As far as syntax goes, I think both variants look good, but a decorator has two major advantages: * you aren't limited to using it at function definition time, you can apply it after the event * it doesn't need to be a keyword, or even a built-in: it could go into functools -- Steven From steve at pearwood.info Thu Sep 22 19:58:32 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Sep 2011 03:58:32 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <1316663060.31837.26.camel@Gutsy> Message-ID: <4E7B7748.6070707@pearwood.info> Eric Snow wrote: > We're looking at two approaches here: do it in the function "header" > or in the body. Here're some pros and cons: There's a third approach, although it would probably take a bit more implementation effort: a decorator. # Demonstrate early-binding, micro-optimization and monkey-patching # in a single call: @inject(a=some_value(), len=len, sum=my_sum) def func(x): ... The downside is that the decorator gets called after the function is already created, therefore it would have to take an existing function and make a modified copy of it. This may be harder than doing the "static" bindings while the function is being assembled in the first place. > In-header Pros: > - implies definition time Likewise for the usual @ decorator syntax. > In-header Cons: > - limited space so more info adds clutter > - could be mistaken as part of function's signature Neither apply to decorator syntax. > In-body Pros: > - less cluttered > - more closely associated with function locals? Decorator is also less cluttered, but not so much obvious what it does. > In-body Cons: > - easy to miss that it's a definition-time directive (particularly if > on RHS of assignment) > - evaluated expression must not reference any of the function's locals Neither of these is a problem for a decorator: using the @ syntax, it is clearly and obviously definition-time. And because the inject decorator is called outside the function, it's obvious that the bindings can't access the function locals. -- Steven From cmjohnson.mailinglist at gmail.com Thu Sep 22 20:49:53 2011 From: cmjohnson.mailinglist at gmail.com (Carl Matthew Johnson) Date: Thu, 22 Sep 2011 08:49:53 -1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> Message-ID: <3C14768F-3983-4753-B65C-DEBD81488087@gmail.com> On Sep 22, 2011, at 3:58 AM, Nick Coghlan wrote: > (As an implementation detail, this would likely > work by assigning such expressions an index in the function's closure > list and populating them at definition time as cells) This reminds me that there was a proposal a few weeks back to be able to refer to the function one is in absolutely. Right now, you can't just slap an attribute onto f in a loop and then get it back at exec time, since the variable f will refer to the last f defined in the loop. But suppose we had some sort of "THIS_FUNCTION" magic variable? Then this would work: fs = [] for i in range(10): def f(): return THIS_FUNC.value f.value = i [f() for f in fs] (Again, it has to be some sort of magical variable, since "return f.value" will refer to the 9 producing function at the end.) This could then be combined with a standard inject decorator. @inject(i=i) def f(): return THIS_FUNC.inject_values.i In cases where you're not in a loop or something, you could just write "f.values.i" instead of using the new magical variable. From ericsnowcurrently at gmail.com Thu Sep 22 21:14:44 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 22 Sep 2011 13:14:44 -0600 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <3C14768F-3983-4753-B65C-DEBD81488087@gmail.com> References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <3C14768F-3983-4753-B65C-DEBD81488087@gmail.com> Message-ID: On Thu, Sep 22, 2011 at 12:49 PM, Carl Matthew Johnson wrote: > This reminds me that there was a proposal a few weeks back to be able to refer to the function one is in absolutely. Right now, you can't just slap an attribute onto f in a loop and then get it back at exec time, since the variable f will refer to the last f defined in the loop. But suppose we had some sort of "THIS_FUNCTION" magic variable? Yeah, a "THIS_FUNCTION" in the function locals didn't go very far. It would be too easy to abuse it and the implementation doesn't fit well into CPython (as far as I could see). I have a patch in for adding f_func to the frame object (you have to jump through the inspect module to get to it). However, that doesn't help so much for this. > Then this would work: > > fs = [] > for i in range(10): > ? ?def f(): > ? ? ? ?return THIS_FUNC.value > ? ?f.value = i > > [f() for f in fs] > > (Again, it has to be some sort of magical variable, since "return f.value" will refer to the 9 producing function at the end.) > > This could then be combined with a standard inject decorator. > > ? @inject(i=i) > ? ?def f(): > ? ? ? ?return THIS_FUNC.inject_values.i > > In cases where you're not in a loop or something, you could just write "f.values.i" instead of using the new magical variable. Performance is another concern here. The default argument hack is pretty efficient compared to attribute lookups on the function object. -eric > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From p.f.moore at gmail.com Thu Sep 22 21:42:33 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 22 Sep 2011 20:42:33 +0100 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> Message-ID: On 22 September 2011 15:34, Paul Moore wrote: > I think that based on your definition, all my "but what about X" > pathological examples have reasonably sane behaviours (you need a good > understanding of what "definition time" actually means, but once you > internalise the fact that def is an executable statement, that isn't > too hard). I spotted one that might still be worth mentioning, though. a = 10 def foo(): a = 20 b = static a What should that mean? Clearly b isn't 20, as a wasn't set to 20 at function define time. But b = 10 (on the assumption that "static a" takes the value of a at define time) will confusing to users, I would suggest. Not allowed might be best, but I suspect it could be hard to implement. But as Guido's -1 on the expression form, this is probably moot. Paul. From jkbbwr at gmail.com Thu Sep 22 23:07:07 2011 From: jkbbwr at gmail.com (Jakob Bowyer) Date: Thu, 22 Sep 2011 22:07:07 +0100 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> Message-ID: Sorry Im still a bit new to all this, what would be the specific advantage of having this? On 9/22/11, Paul Moore wrote: > On 22 September 2011 15:34, Paul Moore wrote: >> I think that based on your definition, all my "but what about X" >> pathological examples have reasonably sane behaviours (you need a good >> understanding of what "definition time" actually means, but once you >> internalise the fact that def is an executable statement, that isn't >> too hard). > > I spotted one that might still be worth mentioning, though. > > a = 10 > def foo(): > a = 20 > b = static a > > What should that mean? Clearly b isn't 20, as a wasn't set to 20 at > function define time. But b = 10 (on the assumption that "static a" > takes the value of a at define time) will confusing to users, I would > suggest. Not allowed might be best, but I suspect it could be hard to > implement. > > But as Guido's -1 on the expression form, this is probably moot. > > Paul. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From zuo at chopin.edu.pl Thu Sep 22 23:27:15 2011 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Thu, 22 Sep 2011 23:27:15 +0200 Subject: [Python-ideas] static vs atdef, etc In-Reply-To: References: Message-ID: <20110922212715.GA4285@chopin.edu.pl> Jim Jewett dixit (2011-09-22, 12:06): > On Thu, Sep 22, 2011 at 10:34 AM, Paul Moore wrote: > > > * I'm sorry but the keyword atdef is ugly. The only proposal I like is > > static, but I agree that it only makes sense to someone with a C-type > > background. > > Even with a C background, static is a bad choice, because it has > already been used for too many different meanings. (Admittedly, not > all in C.) Yeas, we have already staticmethod -- something *completely* different. -1 for `static` name. +1 for `atdef`, IMHO it is not ungly :). Cheers. *j From greg.ewing at canterbury.ac.nz Fri Sep 23 00:13:43 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 23 Sep 2011 10:13:43 +1200 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> Message-ID: <4E7BB317.6000906@canterbury.ac.nz> Paul Moore wrote: > a = 10 > def foo(): > a = 20 > b = static a > > Not allowed might be best, but I suspect it could be hard to > implement. Also hard to specify without excluding one of the intended use cases: def foo(): len = static len or in the statement version, def foo(): static len = len I think we'd just have to accept that the RHS is evaluated outside the function scope, despite being written within it. That's how static initialisers in C work. -- Greg From steve at pearwood.info Fri Sep 23 01:51:26 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Sep 2011 09:51:26 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> Message-ID: <4E7BC9FE.2050005@pearwood.info> Paul Moore wrote: > a = 10 > def foo(): > a = 20 > b = static a > > What should that mean? Clearly b isn't 20, as a wasn't set to 20 at > function define time. But b = 10 (on the assumption that "static a" > takes the value of a at define time) will confusing to users, I would > suggest. Not allowed might be best, but I suspect it could be hard to > implement. With decorator syntax, the scoping rules are obvious and straightforward: a = 10 @inject(b=a) def foo(): a = 20 return b+a -- Steven From ncoghlan at gmail.com Fri Sep 23 03:11:03 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Sep 2011 11:11:03 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <4E7BC9FE.2050005@pearwood.info> References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> Message-ID: On Fri, Sep 23, 2011 at 9:51 AM, Steven D'Aprano wrote: > With decorator syntax, the scoping rules are obvious and straightforward: > > a = 10 > @inject(b=a) > def foo(): > ? ?a = 20 > ? ?return b+a Please read the previous thread from June (linked earlier in this thread). Decorator syntax cannot work without deep magic, because the compiler *doesn't know* that injected names need to be given special treatment. Python's scoping relies on the compiler being able to classify names at compile time into 3 kinds of reference: - local (direct references into the local variable namespace of the executing frame) - cells (indirect references via cells stored on the function object) - unknown (looked up by name at runtime, first in the module globals and then in the builtin namespace) These 3 reference types are baked into the immutable code objects by the compiler - you *cannot* change them later without hacking the bytecode and recreating the function object. Now, we have two 'magical' names ('super' and '__cell__') that cause the compiler to spontaneously do interesting things with namespaces to make Python 3's new simplified (and incredibly convenient) super() invocation work. However, aside from that special case, the rules are very simple: - names bound in the current function are locals (unless marked with 'nonlocal' or 'global') - names bound as locals in an outer function and referenced from the current function are looked up via cells - anything else is treated as an unknown name The 'nonlocal' and 'global' keywords override the 'local by default' behaviour for bound names (forcing the second or third interpretations respectively). The default argument hack effectively creates a 4th namespace option by using the default arguments as "pre-populated locals" - the argument passing machinery is set up so that any parameter not supplied as an argument is filled in on the current frame from its default argument value. By adding additional parameters that are *never* supplied as arguments, the author of a function can create arbitrary locals from expressions that are evaluated when the function is defined rather than when it is called. That means there are four very different ways of looking at potential replacements for this technique: 1. Leave the technique alone, but improve the introspection tools and conventions associated with it def f(x, _i=i): # pydoc would, by default, display the signature as 'f(x)' return x + _i Keyword-only arguments in Py3k already help with this approach to the question, especially when the 'hidden' keyword is prefixed with an underscore to indicate it isn't meant for public consumption. This approach is also highly amenable to monkey-patching, since the default arguments can be deliberately overridden at call time, just like any other parameter. It wouldn't be hard to adjust pydoc to leave out underscore-prefixed keyword only parameters by default, requiring an explicit request to include them. In other words, this approach just involves taking the existing default argument hack, tidying it up a bit, explaining it in the docs, and blessing it as the official way to do things and a technique that experienced Python programmers should know and understand. 2. Definition time parameters This approach keeps the pre-populated locals in the function header, but tweaks the spelling and storage so they're no longer part of the function signature. Two ideas have been put forward for this approach: def f(x, **, i=i): # extending the keyword-only syntax one step further return x + i def f(x) [i=i]: # adding a dedicated set of brackets return x + i The general consensus seems to be that these don't offer enough benefit over the status quo to be worth the hassle. 3. Definition time expressions With a wide variety of proposed spellings (e.g. once, static, atdef), this proposals aims to mark individual expressions for evaluation at function definition time and caching on the function object. At function call time, the value would be inserted in place of the expression. I explained this in my previous email, and Guido has already said '-1' to this approach, so I won't elaborate any further. 4. Function scoped variables This is the approach most analogous to C's static variables - named variables that are shared across all invocations of a function, rather than being local to the current invocation. In essence, each function becomes its own closure - just as a function can share state across invocations by using an outer function for storage, this technique would allow a function to use its *own* cell array for such storage. Framing the idea that way also suggests a fairly obvious spelling: def f(x): nonlocal i=i # Use 'f' as a closure over *itself* return x + i With this spelling, the above would be roughly equivalent to: def outer(): i = i def f(x): return x + i return f f = outer() The only visible difference would be that the cell referenced by 'i' would be stored directly on 'f' rather than on an outer function. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Fri Sep 23 03:14:47 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Sep 2011 11:14:47 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> Message-ID: On Fri, Sep 23, 2011 at 11:11 AM, Nick Coghlan wrote: > Now, we have two 'magical' names ('super' and '__cell__') Oops. s/__cell__/__class__/ Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Fri Sep 23 03:31:38 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Sep 2011 11:31:38 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> Message-ID: On Fri, Sep 23, 2011 at 11:11 AM, Nick Coghlan wrote: > 4. Function scoped variables > > This is the approach most analogous to C's static variables - named > variables that are shared across all invocations of a function, rather > than being local to the current invocation. In essence, each function > becomes its own closure - just as a function can share state across > invocations by using an outer function for storage, this technique > would allow a function to use its *own* cell array for such storage. Applying this 'functions as their own closure' concept to the classic counter problem: def counter(): x = 0 def increment(): nonlocal x x += 1 return x return increment would become: def counter(): def increment(): nonlocal x=0 x += 1 return x return increment Or, if you wanted a process-global counter: def global_counter(): nonlocal x=0 x += 1 return x Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From jeanpierreda at gmail.com Fri Sep 23 03:39:39 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 22 Sep 2011 21:39:39 -0400 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> Message-ID: > Decorator syntax cannot work without deep magic, because the > compiler *doesn't know* that injected names need to be given special > treatment. This is the last time you mention the decorator solution (aside from further explanation of the problem). Is it being discarded for that reason? I was under the impression that everyone who liked that idea was fully aware that it's deep magic. I would assume it actually creates an entirely new function object with new closure cell to bind the name, similar to your last solution involving the function closing over "itself", but more general and as part of a decorator above the function header. I'm not really seeing a problem other than maybe distaste because it seems "hacky". It can work, though. Devin On Thu, Sep 22, 2011 at 9:11 PM, Nick Coghlan wrote: > On Fri, Sep 23, 2011 at 9:51 AM, Steven D'Aprano wrote: >> With decorator syntax, the scoping rules are obvious and straightforward: >> >> a = 10 >> @inject(b=a) >> def foo(): >> ? ?a = 20 >> ? ?return b+a > > Please read the previous thread from June (linked earlier in this > thread). Decorator syntax cannot work without deep magic, because the > compiler *doesn't know* that injected names need to be given special > treatment. > > Python's scoping relies on the compiler being able to classify names > at compile time into 3 kinds of reference: > - local (direct references into the local variable namespace of the > executing frame) > - cells (indirect references via cells stored on the function object) > - unknown (looked up by name at runtime, first in the module globals > and then in the builtin namespace) > > These 3 reference types are baked into the immutable code objects by > the compiler - you *cannot* change them later without hacking the > bytecode and recreating the function object. > > Now, we have two 'magical' names ('super' and '__cell__') that cause > the compiler to spontaneously do interesting things with namespaces to > make Python 3's new simplified (and incredibly convenient) super() > invocation work. However, aside from that special case, the rules are > very simple: > > - names bound in the current function are locals (unless marked with > 'nonlocal' or 'global') > - names bound as locals in an outer function and referenced from the > current function are looked up via cells > - anything else is treated as an unknown name > > The 'nonlocal' and 'global' keywords override the 'local by default' > behaviour for bound names (forcing the second or third interpretations > respectively). > > The default argument hack effectively creates a 4th namespace option > by using the default arguments as "pre-populated locals" - the > argument passing machinery is set up so that any parameter not > supplied as an argument is filled in on the current frame from its > default argument value. By adding additional parameters that are > *never* supplied as arguments, the author of a function can create > arbitrary locals from expressions that are evaluated when the function > is defined rather than when it is called. > > That means there are four very different ways of looking at potential > replacements for this technique: > > 1. Leave the technique alone, but improve the introspection tools and > conventions associated with it > > ? ?def f(x, _i=i): ?# pydoc would, by default, display the signature as 'f(x)' > ? ? ? ?return x + _i > > Keyword-only arguments in Py3k already help with this approach to the > question, especially when the 'hidden' keyword is prefixed with an > underscore to indicate it isn't meant for public consumption. This > approach is also highly amenable to monkey-patching, since the default > arguments can be deliberately overridden at call time, just like any > other parameter. It wouldn't be hard to adjust pydoc to leave out > underscore-prefixed keyword only parameters by default, requiring an > explicit request to include them. > > In other words, this approach just involves taking the existing > default argument hack, tidying it up a bit, explaining it in the docs, > and blessing it as the official way to do things and a technique that > experienced Python programmers should know and understand. > > 2. Definition time parameters > > This approach keeps the pre-populated locals in the function header, > but tweaks the spelling and storage so they're no longer part of the > function signature. > > Two ideas have been put forward for this approach: > > ? ?def f(x, **, i=i): ?# extending the keyword-only syntax one step further > ? ? ? ?return x + i > > ? ?def f(x) [i=i]: ?# adding a dedicated set of brackets > ? ? ? ?return x + i > > The general consensus seems to be that these don't offer enough > benefit over the status quo to be worth the hassle. > > 3. Definition time expressions > > With a wide variety of proposed spellings (e.g. once, static, atdef), > this proposals aims to mark individual expressions for evaluation at > function definition time and caching on the function object. At > function call time, the value would be inserted in place of the > expression. > > I explained this in my previous email, and Guido has already said '-1' > to this approach, so I won't elaborate any further. > > 4. Function scoped variables > > This is the approach most analogous to C's static variables - named > variables that are shared across all invocations of a function, rather > than being local to the current invocation. In essence, each function > becomes its own closure - just as a function can share state across > invocations by using an outer function for storage, this technique > would allow a function to use its *own* cell array for such storage. > Framing the idea that way also suggests a fairly obvious spelling: > > ? ?def f(x): > ? ? ? ?nonlocal i=i # Use 'f' as a closure over *itself* > ? ? ? ?return x + i > > With this spelling, the above would be roughly equivalent to: > > ? ?def outer(): > ? ? ? ?i = i > ? ? ? ?def f(x): > ? ? ? ? ? ?return x + i > ? ? ? ?return f > ? ?f = outer() > > The only visible difference would be that the cell referenced by 'i' > would be stored directly on 'f' rather than on an outer function. > > Regards, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From bruce at leapyear.org Fri Sep 23 04:18:12 2011 From: bruce at leapyear.org (Bruce Leban) Date: Thu, 22 Sep 2011 19:18:12 -0700 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> Message-ID: On Thu, Sep 22, 2011 at 6:11 PM, Nick Coghlan wrote: > On Fri, Sep 23, 2011 at 9:51 AM, Steven D'Aprano > wrote: > > With decorator syntax, the scoping rules are obvious and straightforward: > > > > a = 10 > > @inject(b=a) > > def foo(): > > a = 20 > > return b+a > > Please read the previous thread from June (linked earlier in this > thread). Decorator syntax cannot work without deep magic, because the > compiler *doesn't know* that injected names need to be given special > treatment. > > Yes, you're right that this can't be done with today's decorators. That doesn't mean it's impossible. Any of the other changes would also require compiler changes. Instead of @inject being a standard decorator, imagine a new kind of decorator that works as follows: a = 10 $inject(b=a) def foo() a = 20 return b+a where the effect of a $inject decorator is that it modifies the behavior of the function as follows: a = 10 _inject_ = inject_decorator(b=a) def foo(): locals().update(_inject_) a = 20 return b + a Yes, I know that update on locals() won't do the right thing but pretend it does. I think the semantics of this are pretty clear and more obvious than the suggested alternatives. Now what about these $-decorators? As defined here, it's pretty specialized to this one operation. Let's imagine that we define $-decorators to be a special class of decorators that are built into Python. We use $-decorators for cases we would like users to think of as decorators but that do some kind of magic that's beyond the capability of a regular decorator. To the user the $ is a signal to the user that this decorator is a bit more special than usual. Right now, there would only be one $-decorator but it's available next time it's desired to add something that just can't quite be a regular decorator. A good objection to adding new things to the language is that they increase the load on people trying to learn the language. If I see a syntax like: def f() [i=i]: pass I can't imagine how I would find out what that does. Search engines don't readily allow searching on special characters. On the other hand, when I see $inject, I go type "python $inject decorator" into my favorite search engine and whether it ignores the $ or not, I'm fairly likely to get good results. Whether the decorator-like syntax is spelled $inject, @@static or something else, I think it's going to be much easier to figure out and remember what's going on. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Fri Sep 23 04:27:17 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 22 Sep 2011 22:27:17 -0400 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> Message-ID: > Yes, you're right that this can't be done with today's decorators. It can work with today's decorators and today's syntax, it just requires magic. (Magic doesn't mean "impossible things", it means "things that go below the expected level of abstraction". In this case, things like directly inspecting fields of the function object and creating a new one with modified fields -- that, or else mutating a supposedly-immutable object (also possible)) That doesn't blow down the whole idea of $ special syntactic decorators, though. And maybe even this case is good for them. Definitely there might be room for a more general-case thing for compile-time annotations than new extra-special syntax. Magical decorators that get understood by the compiler have been mentioned before, and are inappropriate for obvious reasons. But if they were prefixed by $ instead of @... I still like real decorators because they can be applied at run-time instead of just compile-time, which gives them additional use-cases. Devin On Thu, Sep 22, 2011 at 10:18 PM, Bruce Leban wrote: > > On Thu, Sep 22, 2011 at 6:11 PM, Nick Coghlan wrote: >> >> On Fri, Sep 23, 2011 at 9:51 AM, Steven D'Aprano >> wrote: >> > With decorator syntax, the scoping rules are obvious and >> > straightforward: >> > >> > a = 10 >> > @inject(b=a) >> > def foo(): >> > ? ?a = 20 >> > ? ?return b+a >> >> Please read the previous thread from June (linked earlier in this >> thread). Decorator syntax cannot work without deep magic, because the >> compiler *doesn't know* that injected names need to be given special >> treatment. >> > > ?Yes, you're right that this can't be done with today's decorators. That > doesn't mean it's impossible. Any of the other changes would also require > compiler changes. Instead of @inject being a standard decorator, imagine a > new kind of decorator that works as follows: > > a = 10 > $inject(b=a) > def foo() > ? ? a = 20 > ? ? return b+a > > where the effect of a $inject decorator is that it modifies the behavior of > the function as follows: > > a = 10 > _inject_ = inject_decorator(b=a) > def foo(): > ? ? locals().update(_inject_) > ? ? a = 20 > ? ? return b + a > > Yes, I know that update on locals() won't do the right thing but pretend it > does. I think the semantics of this are pretty clear and more obvious than > the suggested alternatives. Now what about these $-decorators? As defined > here, it's pretty specialized to this one operation. Let's imagine that we > define $-decorators to be a special class of decorators that are built into > Python. We use $-decorators for cases we would like users to think of as > decorators but that do some kind of magic that's beyond the capability of a > regular decorator. To the user the $ is a signal to the user that this > decorator is a bit more special than usual. Right now, there would only be > one $-decorator but it's available next time it's desired to add something > that just can't quite be a regular decorator. > A good objection to adding new things to the language is that they increase > the load on people trying to learn the language. If I see a syntax like: > > def f() [i=i]: pass > > I can't imagine how I would find out what that does. Search engines don't > readily allow searching on special characters. On the other hand, when I see > $inject, I go type "python $inject decorator" into my favorite search engine > and whether it ignores the $ or not, I'm fairly likely to get good results. > Whether the decorator-like syntax is spelled $inject, @@static or something > else, I think it's going to be much easier to figure out and remember what's > going on. > > --- Bruce > Follow me:?http://www.twitter.com/Vroo?http://www.vroospeak.com > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > From bruce at leapyear.org Fri Sep 23 04:39:16 2011 From: bruce at leapyear.org (Bruce Leban) Date: Thu, 22 Sep 2011 19:39:16 -0700 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> Message-ID: On Thu, Sep 22, 2011 at 7:27 PM, Devin Jeanpierre wrote: > > I still like real decorators because they can be applied at run-time > instead of just compile-time, which gives them additional use-cases. > > The $inject special decorator would need to be recognized at compile time but would be applied at run time just as @decorators are. That is, I see no reason this couldn't work: def foo(a): $inject(b=a) def bar(c): return b+c return bar --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Sep 23 05:17:01 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Sep 2011 13:17:01 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> Message-ID: <4E7BFA2D.4060508@pearwood.info> Nick Coghlan wrote: > On Fri, Sep 23, 2011 at 9:51 AM, Steven D'Aprano wrote: >> With decorator syntax, the scoping rules are obvious and straightforward: >> >> a = 10 >> @inject(b=a) >> def foo(): >> a = 20 >> return b+a > > Please read the previous thread from June (linked earlier in this > thread). Decorator syntax cannot work without deep magic, because the > compiler *doesn't know* that injected names need to be given special > treatment. I have read the previous thread, and I'm fully aware that this would be special. I'm pretty sure I even said it would be special in one of my posts :) Excluding the default "do nothing" position, and the minimalist "just add conventions to introspection tools" proposal, I believe that @inject is the least magical proposal made so far. The other proposals require a new keyword, new syntax, or both. They require the compiler to do extra work. @inject requires nothing from the compiler. It all happens when the decorator is called. Comparing it to the voodoo needed for super() is completely unfair. It also offers the most benefits: * all the benefits of the proposed "static" declaration (early-binding, micro-optimisation, monkey-patching) * the ability to patch pre-existing functions * no call-time cost (it's not a wrapper, its a new function) * fewer side-effects than conventional monkey-patching * familiar syntax * obvious semantics * no new keywords * no extra "line-noise" symbols What's not to like? Yes, inject() will be special. But I don't believe it will be magic, or at least not deep magic. We can get most of the way using supported Python functionality already: we can pull a function object apart, make a copy of the pieces as needed, and reassemble them into a new function. All that is supported by Python, none of it requires messing with private attributes: it's not much more magic than what functools.wraps() already does. We can even take the bytecode from the code object, and because it's just a string, we can perform transformations on it. The only "magic" is the knowledge of what transformations to perform. Take this simple example: >>> import dis >>> def func(): ... c = 1 ... return c+d ... >>> dis.dis(func) 2 0 LOAD_CONST 1 (1) 3 STORE_FAST 0 (c) 3 6 LOAD_FAST 0 (c) 9 LOAD_GLOBAL 0 (d) 12 BINARY_ADD 13 RETURN_VALUE @inject(d=2)(func) would probably look produce something like this: 2 6 LOAD_CONST 2 (2) 9 STORE_FAST 1 (d) 3 0 LOAD_CONST 1 (1) 3 STORE_FAST 0 (c) 4 12 LOAD_FAST 0 (c) 15 LOAD_FAST 1 (d) 18 BINARY_ADD 19 RETURN_VALUE If we had a Python assembler to match the disassembler, it would probably be easy. If we are really allergic to the idea of bytecode manipulation, perhaps there are other ways to solve this. Just tossing ideas around, maybe function objects should keep a copy of their AST around, so that instead of bytecode hacking, you manipulate the AST and recompile the function. That might be less "hacky" than manipulating the bytecode, but I don't know that carrying around the AST for every function is a cost that is worth bearing. But it's an idea. -- Steven From steve at pearwood.info Fri Sep 23 05:26:31 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 23 Sep 2011 13:26:31 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> Message-ID: <4E7BFC67.2050806@pearwood.info> Devin Jeanpierre wrote: >> Yes, you're right that this can't be done with today's decorators. > > It can work with today's decorators and today's syntax, it just requires magic. > > (Magic doesn't mean "impossible things", it means "things that go > below the expected level of abstraction". In this case, things like > directly inspecting fields of the function object and creating a new > one with modified fields -- that, or else mutating a > supposedly-immutable object (also possible)) In this case, I believe that most of the work that needs to be done -- making a copy of the function and code object -- are not magic. They are fully supported in standard Python. The only "magic" is manipulating the bytecode of the code object to that it turns some globals into locals. > That doesn't blow down the whole idea of $ special syntactic > decorators, though. And maybe even this case is good for them. I don't see any reason to introduce extra syntax for a different sort of decorator. What benefit is there? I think this $ proposal actually undermines the argument I am trying to make. A big advantage of using a decorator is that it requires no new syntax and no magic symbols beyond the standard @ symbol. The argument "yes, I like decorators, but I want to use $ instead of @" doesn't really help. > I still like real decorators because they can be applied at run-time > instead of just compile-time, which gives them additional use-cases. I have negative interest in a "magic decorator" that can only work at compile time. If we're going to have that (unnecessary, in my opinion) limitation, then I prefer the static declaration proposal. -- Steven From ncoghlan at gmail.com Fri Sep 23 05:26:35 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Sep 2011 13:26:35 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> Message-ID: On Fri, Sep 23, 2011 at 11:39 AM, Devin Jeanpierre wrote: >> Decorator syntax cannot work without deep magic, because the >> compiler *doesn't know* that injected names need to be given special >> treatment. > > This is the last time you mention the decorator solution (aside from > further explanation of the problem). Is it being discarded for that > reason? Yes, it's far too hard to explain what it does in terms of existing Python semantics (the Zen has something to say on that point). Bytecode hackery and other tricks would *work*, but deep magic should only be employed when there aren't any alternatives and the problem is sufficiently common. (PEP 3135, the new super semantics for 3.x, pushes the boundaries of what's reasonable, but it's so easy to *use* that it's worth the additional under the hood complexity). Magical behaviour is also far more likely to cause problems for other implementations - it tends to rely on assumptions that aren't explicitly guaranteed by the language specification. Use of default arguments for pre-initialised function locals is nowhere near common enough to justify deep magic as a solution, so the idea just isn't worth pursuing. That's the real benefit of the "nonlocal i=i" idea: it takes an *existing* concept (i.e. closures), and just tweaks it a bit by avoiding the need for a separate outer scope when all you really want to do is share some state between invocations. Anyone that already understands closures shouldn't have much trouble grasping the idea that the 'innermost containing scope' can now be the function itself rather than the next level up. Any implementation that already correctly handles closures should also be able to cope with the self reference without too much trouble. No new keywords, no new namespace semantics, just a slight tweak to the way the compiler handles def and nonlocal statements. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Fri Sep 23 05:40:43 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Sep 2011 13:40:43 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <4E7BFA2D.4060508@pearwood.info> References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> <4E7BFA2D.4060508@pearwood.info> Message-ID: On Fri, Sep 23, 2011 at 1:17 PM, Steven D'Aprano wrote: > Yes, inject() will be special. But I don't believe it will be magic, or at > least not deep magic. We can get most of the way using supported Python > functionality already: we can pull a function object apart, make a copy of > the pieces as needed, and reassemble them into a new function. All that is > supported by Python, none of it requires messing with private attributes: > it's not much more magic than what functools.wraps() already does. > > We can even take the bytecode from the code object, and because it's just a > string, we can perform transformations on it. The only "magic" is the > knowledge of what transformations to perform. Bytecode hacks are inherently implementation dependent deep magic. Almost *nothing* about code objects is part of the language specification (IIRC, it expects them to exist, but that's about it). Jython and IronPython use JVM and CLR bytecode. PyPy uses a format at least similar to CPython bytecode (since it was convenient for them to do so), but there's absolutely no requirement for them to keep it the same as the two implementations evolve. The entire *point* of wpython is to use a word-oriented rather than byte-oriented format. Even CPython will happily change the bytecode format between releases - there's a reason a bytecode version marker is embedded in every .pyc file. Taking an existing function apart and putting it back together again in a different way is skating right along that border between language specification and implementation details. Proposals that play in that space are almost guaranteed to be a hack that pays no attention to the overall semantic concepts of the language design and they're simply *not* going to happen. Any reasonable piece of Python code *must* treat code objects as opaque blobs that the interpreter uses to define an algorithm. It can be fun, and sometimes even useful, to go further, but it would be thoroughly inappropriate for the interpreter core or standard library to indulge in that kind of thing. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Fri Sep 23 05:45:00 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Sep 2011 13:45:00 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <4E7BFC67.2050806@pearwood.info> References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> <4E7BFC67.2050806@pearwood.info> Message-ID: On Fri, Sep 23, 2011 at 1:26 PM, Steven D'Aprano wrote: > In this case, I believe that most of the work that needs to be done -- > making a copy of the function and code object -- are not magic. They are > fully supported in standard Python. The only "magic" is manipulating the > bytecode of the code object to that it turns some globals into locals. That may be the core of the confusion here. Bytecode hacking is NOT supported in standard Python. The language definition pretty much stops at "function objects have a __code__ attribute that refers to the immutable code object defining the algorithm they execute when called". Anything below that point is implementation dependent (hence the disclaimer on the output of the 'dis' module). Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From barry at python.org Fri Sep 23 05:51:38 2011 From: barry at python.org (Barry Warsaw) Date: Thu, 22 Sep 2011 23:51:38 -0400 Subject: [Python-ideas] Before and after the colon in function defs. References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> Message-ID: <20110922235138.0ada2189@resist.wooz.org> On Sep 23, 2011, at 11:11 AM, Nick Coghlan wrote: > def f(x): > nonlocal i=i # Use 'f' as a closure over *itself* > return x + i Interestingly, PEP 3104 proposes that syntax, as shorthand for nonlocal x; x = 3 though this was not adopted. I think your proposal is a very interesting, modest but useful extension of that original PEP. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Fri Sep 23 06:03:27 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Sep 2011 14:03:27 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> Message-ID: On Fri, Sep 23, 2011 at 11:11 AM, Nick Coghlan wrote: > With this spelling, the above would be roughly equivalent to: > > ? ?def outer(): > ? ? ? ?i = i > ? ? ? ?def f(x): > ? ? ? ? ? ?return x + i > ? ? ? ?return f > ? ?f = outer() Alex Gaynor pointed out the above would give an unbound local error. The actual rough equivalent would be more like: def outer(i=i): def f(x): return x + i return f f = outer() Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From cmjohnson.mailinglist at gmail.com Fri Sep 23 06:34:15 2011 From: cmjohnson.mailinglist at gmail.com (Carl Matthew Johnson) Date: Thu, 22 Sep 2011 18:34:15 -1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> Message-ID: <018BF312-031B-42BD-AE3E-3967646CC7CE@gmail.com> Instead of a $ decorator, you could let decorators have a __prepare__ method that returns a dict to use for locals, as metaclasses do today. I did a quick test in Python 3.2: >>> @print("1") ... def f(x=print("2")): pass ... 1 2 Traceback (most recent call last): File "", line 2, in TypeError: 'NoneType' object is not callable This shows that decorators are executed before the function is built. Since we're looking up the decorator before the function object has been created anyway, it wouldn't be impossible for the locals dictionary of the function to be "modified" before it's created, as we can do with metaclasses. If we did this, inject would look something like this: class inject: def __init__(self, **kwargs): self.kwargs = kwargs def __prepare__(self): return self.kwargs def __call__(self, f): return f And of course, there could be other applications for the __prepare__, such as using OrderedDict or whatever. (Not sure if we'd want to encourage that though?) From ncoghlan at gmail.com Fri Sep 23 06:52:30 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Sep 2011 14:52:30 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <018BF312-031B-42BD-AE3E-3967646CC7CE@gmail.com> References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> <018BF312-031B-42BD-AE3E-3967646CC7CE@gmail.com> Message-ID: On Fri, Sep 23, 2011 at 2:34 PM, Carl Matthew Johnson wrote: > Instead of a $ decorator, you could let decorators have a __prepare__ method that returns a dict to use for locals, as metaclasses do today. No, locals simply don't work like that. They're handled by frame objects and the eval loop, not function objects. There are a whole host of complicated interactions between the compiler's symbol table analysis and code generation, function, code and frame objects and the runtime evaluation loop going on when it comes to handling name lookups in Python. When you're looking at it purely from a Python user's point of view, there are plenty of ideas that seem potentially reasonable on the surface but simply don't fit with the underlying data model of the language. It's hard enough coming up with good proposals for semantic and syntactic tweaks when you *do* know how they all work together to achieve the current behaviour - the current setup is actually pretty good, and the number of ways we could make it worse vastly outnumbers the ways we could conceivably improve it. Try to keep this discussion in perspective: we're talking about handling a niche use case in a slightly more elegant fashion. The magnitude of language change that can be justified for this purpose is tiny. I've thrown plenty of ideas of my own at the task over the years (and seen even more ideas from others), and tweaking the syntax and semantics of nonlocal is the first of them that I've genuinely liked better than the status quo. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ericsnowcurrently at gmail.com Fri Sep 23 07:31:33 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 22 Sep 2011 23:31:33 -0600 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> <018BF312-031B-42BD-AE3E-3967646CC7CE@gmail.com> Message-ID: On Thu, Sep 22, 2011 at 10:52 PM, Nick Coghlan wrote: > On Fri, Sep 23, 2011 at 2:34 PM, Carl Matthew Johnson > wrote: >> Instead of a $ decorator, you could let decorators have a __prepare__ method that returns a dict to use for locals, as metaclasses do today. > > No, locals simply don't work like that. They're handled by frame > objects and the eval loop, not function objects. > > There are a whole host of complicated interactions between the > compiler's symbol table analysis and code generation, function, code > and frame objects and the runtime evaluation loop going on when it > comes to handling name lookups in Python. Yeah, that's what I found out when I tried to add in an implicit "__function__" in the locals. I mostly had it working, but it was a hack and a half (a lot worse than super's "@__class__" injection). >When you're looking at it > purely from a Python user's point of view, there are plenty of ideas > that seem potentially reasonable on the surface but simply don't fit > with the underlying data model of the language. > > It's hard enough coming up with good proposals for semantic and > syntactic tweaks when you *do* know how they all work together to > achieve the current behaviour - the current setup is actually pretty > good, and the number of ways we could make it worse vastly outnumbers > the ways we could conceivably improve it. > > Try to keep this discussion in perspective: we're talking about > handling a niche use case in a slightly more elegant fashion. The > magnitude of language change that can be justified for this purpose is > tiny. I've thrown plenty of ideas of my own at the task over the years > (and seen even more ideas from others), and tweaking the syntax and > semantics of nonlocal is the first of them that I've genuinely liked > better than the status quo. +1 -eric > > Regards, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From cmjohnson.mailinglist at gmail.com Fri Sep 23 07:44:19 2011 From: cmjohnson.mailinglist at gmail.com (Carl Matthew Johnson) Date: Thu, 22 Sep 2011 19:44:19 -1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> <018BF312-031B-42BD-AE3E-3967646CC7CE@gmail.com> Message-ID: <4F67F2A5-37F9-430A-ABB8-3B8571BAD173@gmail.com> On Sep 22, 2011, at 6:52 PM, Nick Coghlan wrote: > No, locals simply don't work like that. They're handled by frame > objects and the eval loop, not function objects. I feel like I have the same complaint that other people did earlier in the thread: I understand that locals don't work like that *now*. (And they understood that decorators can't do an inject *now*.) When I said "locals" I was using a shorthand, because I don't know all the correct terminology. What I'm proposing is a change to how the system works. I want the byte code for functions to be created differently after this change. Obviously, that can't be done by the function objects, but it will need to be done beforehand. There's probably much more to this sort of change than I realize--maybe even so much that it wouldn't be worth it to do it, since the benefit would be relatively small. If so, that's fine. But objecting that this isn't how it really works feels like it's missing the point to me. The more relevant objection is that it would be too huge of a project to make it work a different way. With respect, -- Carl From ncoghlan at gmail.com Fri Sep 23 08:29:24 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 23 Sep 2011 16:29:24 +1000 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <4F67F2A5-37F9-430A-ABB8-3B8571BAD173@gmail.com> References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> <018BF312-031B-42BD-AE3E-3967646CC7CE@gmail.com> <4F67F2A5-37F9-430A-ABB8-3B8571BAD173@gmail.com> Message-ID: On Fri, Sep 23, 2011 at 3:44 PM, Carl Matthew Johnson wrote: > I feel like I have the same complaint that other people did earlier in the thread: I understand that locals don't work like that *now*. (And they understood that decorators can't do an inject *now*.) When I said "locals" I was using a shorthand, because I don't know all the correct terminology. What I'm proposing is a change to how the system works. I want the byte code for functions to be created differently after this change. ?Obviously, that can't be done by the function objects, but it will need to be done beforehand. There's probably much more to this sort of change than I realize--maybe even so much that it wouldn't be worth it to do it, since the benefit would be relatively small. If so, that's fine. But objecting that this isn't how it really works feels like it's missing the point to me. The more relevant objection is that it would be too huge of a project to make it work a different way. That's exactly my point though - the default argument use cases to be addressed are *trivial*. A trivial problem requires a comparably trivial answer - there is *nothing* in the use cases being discussed that even comes close to justifying substantial changes to the way the language works. Changing the way locals works, allowing variable injection, those are huge, language changing ideas, that require powerful, compelling use cases to motivate them. As far as I am aware, those use cases don't exist (and if they do, they certainly haven't been raised in this thread). It would be really helpful if people could comment on ideas that have at least some chance of making it under that "simple enough to be worth doing" bar. Obviously, I consider adjusting 'nonlocal' so we can use it to store cell references on the function being defined to be one such idea, as it is essentially just blending PEP 3104 with the underlying implementation of PEP 3135's magic '__class__' reference in methods. As far as I can tell, the biggest potential practical problem with it is on the compiler side, where the AST nodes for the initialisation expressions would need to be lifted out for inclusion in the code generated in the surrounding scope. That's not impossible (we already do something similar in deciding whether a def statement is defining an ordinary function or a generator), but the state to be collected is a fair bit more complicated than anything the symbol analysis pass currently handles. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From hetchkay at gmail.com Fri Sep 23 13:26:31 2011 From: hetchkay at gmail.com (H. Krishnan) Date: Fri, 23 Sep 2011 04:26:31 -0700 (PDT) Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> <018BF312-031B-42BD-AE3E-3967646CC7CE@gmail.com> <4F67F2A5-37F9-430A-ABB8-3B8571BAD173@gmail.com> Message-ID: <28638055.591.1316777191248.JavaMail.geo-discussion-forums@prfp13> Is there a restriction that nonlocal can be used only inside nested functions and would that limitation have any impact. Also, would it make sense/be possible to execute nested class/function definitions at function definition time. Regards, -Krishnan -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikegraham at gmail.com Fri Sep 23 13:59:30 2011 From: mikegraham at gmail.com (Mike Graham) Date: Fri, 23 Sep 2011 07:59:30 -0400 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110919132218.GC5335@pantoffel-wg.de> <20110920140045.GD5335@pantoffel-wg.de> Message-ID: On Wed, Sep 21, 2011 at 1:19 AM, Nick Coghlan wrote: > On Wed, Sep 21, 2011 at 1:36 PM, Terry Reedy wrote: >> For those not party to the earlier threads, this alternate proposal would >> bread any code that monkeypatches an imported-module-only override of a >> builtin, something like: >> >> def mylen(o): return >> import mod; mod.len = mylen >> >> ... > > The two sticking points tend to be open() and print(). Overriding > those externally can be *incredibly* useful for testing code that uses > them, as well as in interacting with code that wasn't designed in a > flexible way. > > Regardless, I'm going to stop mentioning the manual micro-optimisation > use case for the default argument hack. Alex Gaynor pointed out that > that aspect is completely irrelevant on PyPy, and as Guido notes, > there are likely other ways to tackle name lookup optimisation even in > CPython. > > ... With the right tooling, even given these semantics, we could monkeypatch by getting in at module runtime. One system that uses an idea suitable for this is exocet, which was introduced with this blog post - http://washort.twistedmatrix.com/2011/01/introducing-exocet.html . I'm don't think exocet today is appropriate for use, but its basic ideas could make for much more controlled tests, and become necessary in the face of these optimizations. Mike From ron3200 at gmail.com Fri Sep 23 17:40:12 2011 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 23 Sep 2011 10:40:12 -0500 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> Message-ID: <1316792412.2198.2.camel@Gutsy> On Fri, 2011-09-23 at 11:11 +1000, Nick Coghlan wrote: > 4. Function scoped variables > > This is the approach most analogous to C's static variables - named > variables that are shared across all invocations of a function, rather > than being local to the current invocation. In essence, each function > becomes its own closure - just as a function can share state across > invocations by using an outer function for storage, this technique > would allow a function to use its *own* cell array for such storage. > Framing the idea that way also suggests a fairly obvious spelling: > > def f(x): > nonlocal i=i # Use 'f' as a closure over *itself* > return x + i > > With this spelling, the above would be roughly equivalent to: > > def outer(): > i = i > def f(x): > return x + i > return f > f = outer() > > The only visible difference would be that the cell referenced by 'i' > would be stored directly on 'f' rather than on an outer function. +1 I think this will work just fine. Will the same change be made to global? Cheers, Ron From tjreedy at udel.edu Fri Sep 23 23:18:13 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 23 Sep 2011 17:18:13 -0400 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <28638055.591.1316777191248.JavaMail.geo-discussion-forums@prfp13> References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> <018BF312-031B-42BD-AE3E-3967646CC7CE@gmail.com> <4F67F2A5-37F9-430A-ABB8-3B8571BAD173@gmail.com> <28638055.591.1316777191248.JavaMail.geo-discussion-forums@prfp13> Message-ID: On 9/23/2011 7:26 AM, H. Krishnan wrote: > Is there a restriction that nonlocal can be used only inside nested > functions. >>> a=3 >>> def f(): nonlocal a; return a SyntaxError: no binding for nonlocal 'a' found Yes, althought the current 3.2.2 manual does not make that clear enough. I believe there is a tracker issue to clarify this. > > Also, would it make sense/be possible to execute nested class/function > definitions at function definition time. No. The body of a function is executed when the function is called. However, it is already the case that code objects for nested functions *are* compiled just once. So just a nested function object has to be assembled with each call. Default args and cell vars are part of each new function object. -- Terry Jan Reedy From tjreedy at udel.edu Fri Sep 23 23:32:21 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 23 Sep 2011 17:32:21 -0400 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> Message-ID: On 9/22/2011 9:11 PM, Nick Coghlan wrote: > def f(x, _i=i): # pydoc would, by default, display the signature as 'f(x)' > return x + _i I presume you meant def f(x, *, _i=i): return x+_i so that _i will be keyword only. > Keyword-only arguments in Py3k already help with this approach to the > question, especially when the 'hidden' keyword is prefixed with an > underscore to indicate it isn't meant for public consumption. It certainly makes it impossible to 'accidentally' override with an extra positional parameter. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Sat Sep 24 01:12:34 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 24 Sep 2011 11:12:34 +1200 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <1316792412.2198.2.camel@Gutsy> References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> <1316792412.2198.2.camel@Gutsy> Message-ID: <4E7D1262.4020606@canterbury.ac.nz> Ron Adam wrote: > On Fri, 2011-09-23 at 11:11 +1000, Nick Coghlan wrote: > >>Framing the idea that way also suggests a fairly obvious spelling: >> >> def f(x): >> nonlocal i=i # Use 'f' as a closure over *itself* >> return x + i Sorry, but that spelling is very far from obvious to me. According to the current meaning of 'nonlocal', it looks like a no-op. I don't understand the reasoning that leads from there to your proposed semantics. You'll also have to explain how that reasoning applies to the following variations: nonlocal i = i + i nonlocal i = i + j nonlocal i = j + k -- Greg From ron3200 at gmail.com Sat Sep 24 02:35:52 2011 From: ron3200 at gmail.com (ron adam) Date: Fri, 23 Sep 2011 19:35:52 -0500 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <4E7D1262.4020606@canterbury.ac.nz> References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> <1316792412.2198.2.camel@Gutsy> <4E7D1262.4020606@canterbury.ac.nz> Message-ID: <1316824552.4154.14.camel@Gutsy> On Sat, 2011-09-24 at 11:12 +1200, Greg Ewing wrote: > Ron Adam wrote: > > On Fri, 2011-09-23 at 11:11 +1000, Nick Coghlan wrote: > > > >>Framing the idea that way also suggests a fairly obvious spelling: > >> > >> def f(x): > >> nonlocal i=i # Use 'f' as a closure over *itself* > >> return x + i > > Sorry, but that spelling is very far from obvious to me. > According to the current meaning of 'nonlocal', it looks like > a no-op. I don't understand the reasoning that leads from > there to your proposed semantics. > > You'll also have to explain how that reasoning applies to > the following variations: > > nonlocal i = i + i > nonlocal i = i + j > nonlocal i = j + k Here's my view of what would happen, but you don't say weather or not those are defined together or if they are separate cases. nonlocal i Gives read write access to i in a parent frame. nonlocal i=i Creates a new 'i' in a functions own frame with the value of 'i' from a parent frame at the time the function is defined. (Note, it also makes sense to do it on the first time the function is called. I'm not sure there is any advantage to that.) Once the function is called, the access of 'i' would be in the local frame, and not effect the parent frames 'i' because the local 'i' is found first. As for the various cases, they would work the same except the initial value would be different. My 2cents for what it's worth. Cheers, Ron From guido at python.org Sat Sep 24 02:54:47 2011 From: guido at python.org (Guido van Rossum) Date: Fri, 23 Sep 2011 17:54:47 -0700 Subject: [Python-ideas] Before and after the colon in funciton defs. In-Reply-To: <1316824552.4154.14.camel@Gutsy> References: <1316208647.15431.105.camel@Gutsy> <20110922010140.GA4237@chopin.edu.pl> <20110922081135.GB2356@chopin.edu.pl> <4E7BC9FE.2050005@pearwood.info> <1316792412.2198.2.camel@Gutsy> <4E7D1262.4020606@canterbury.ac.nz> <1316824552.4154.14.camel@Gutsy> Message-ID: On Fri, Sep 23, 2011 at 5:35 PM, ron adam wrote: > On Sat, 2011-09-24 at 11:12 +1200, Greg Ewing wrote: >> Ron Adam wrote: >> > On Fri, 2011-09-23 at 11:11 +1000, Nick Coghlan wrote: >> > >> >>Framing the idea that way also suggests a fairly obvious spelling: >> >> >> >> ? ?def f(x): >> >> ? ? ? ?nonlocal i=i # Use 'f' as a closure over *itself* >> >> ? ? ? ?return x + i >> >> Sorry, but that spelling is very far from obvious to me. >> According to the current meaning of 'nonlocal', it looks like >> a no-op. I don't understand the reasoning that leads from >> there to your proposed semantics. >> >> You'll also have to explain how that reasoning applies to >> the following variations: >> >> ? ?nonlocal i = i + i >> ? ?nonlocal i = i + j >> ? ?nonlocal i = j + k > > > Here's my view of what would happen, but you don't say weather or not > those are defined together or if they are separate cases. > > > ? ?nonlocal i > > Gives read write access to i in a parent frame. > > > ? ?nonlocal i=i > > Creates a new 'i' in a functions own frame with the value of 'i' from a > parent frame at the time the function is defined. > > (Note, it also makes sense to do it on the first time the function is > called. ?I'm not sure there is any advantage to that.) > > > Once the function is called, the access of 'i' would be in the local > frame, and not effect the parent frames 'i' because the local 'i' is > found first. > > As for the various cases, they would work the same except the initial > value would be different. The only sensible meaning that could possibly assigned to "nonlocal = " is that it should be strictly equivalent to nonlocal = Anything else is too confusing for our users. So your proposal is right out. -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Mon Sep 26 03:01:15 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2011 11:01:15 +1000 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined Message-ID: The problem of the 'default argument hack' and it's use for early binding and shared state in function definitions is one that has been bugging me for years. You could rightly say that the degree to which it irritates me is all out of proportion to the significance of the use case and frequency with which it arises, and I'd agree with you. That, at least in part, is what has made it such an interesting problem for me: the status quo is suboptimal and confusing (there's a reason the term 'default argument hack' gets thrown around, including by me), but most proposed solutions have involved fairly significant changes to the semantics and syntax of the language that cannot be justified by such a niche use case. The proposed cures (including my own suggestions) have all been worse than the disease. I finally have a possible answer I actually *like* ("nonlocal VAR from EXPR"), but it involves a somewhat novel way of thinking about closures and lexical scopes for it to make sense. This post is an attempt to explain that thought process. The history outlined here will be familiar to many folks on the list, but I hope to make the case that this approach actually simplifies and unifies a few aspects of the language rather than adding anything fundamentally new. The novel aspect lies in recognising and exposing to developers as a coherent feature something that is already implicit in the operation of the language as a whole. == Default Arguments == Default arguments have been a part of Python function definitions for a very long time (since the beginning, even?), so it makes sense to start with those. At function call time, if the relevant parameters are not supplied as arguments, they're populated on the frame object based on the values stored on the function object. Their behaviour is actually quite like a closure: they define shared state that is common to all invocations of the function. == Lexical Scoping == The second step in this journey is the original introduction of lexical scoping by PEP 227 back in Python 2.1 (or 2.2 without a __future__ statement). This changed Python from its original locals->globals->builtins lookup mechanism (still used in class scope to this day), to the closure semantics for nested functions that we're familiar with. However, at this stage, there was no ability to rebind names in outer scopes - they were read-only, so you needed to use other techniques (like 'boxing' in a list) to update immutable values. == Writing to Outer Scopes == PEP 3104 added the ability to write to outer scopes by using the 'nonlocal' statement to declare that a particular variable was not a local in the current frame, but rather a local in an outer frame which is alive at the time the inner function definition statement is executed. It expects the variable to already exist in an outer lexically nested scope and complains if it can't find one. == The "__class__" cell reference == The final entrant in this game, the "__class__" cell reference was added to the language by PEP 335 in order to implement the 3.x super() shorthand. For functions defined within a class body, this effectively lets the class definition play a role in lexical scoping, as the compiler and eval loop cooperate to give the function an indirect reference to the class being defined, even though the function definition completes first. == The Status Quo == If you go look up the definition of 'closure', you'll find that it doesn't actually say anything about nested functions. Instead, it will talk about 'free variables' in the algorithm definition without placing any restrictions on how those variables are later hooked up to the appropriate values. In current Python, ordinary named references can refer to one of 4 namespaces: - locals (stored on the currently executing frame object) - closure reference (stored in a cell object by the function that defined it, kept alive after the frame is recycled by references from still living inner functions that need it) - globals (stored on the module object) - builtins (also stored on a module object, specifically the one for the builtin namespace) PEP 335 also creates a closure reference for "__class__", but in a slightly unusual way. Whereas most targets for closure references are created by the code in the outer function when it runs [1], this closure reference is populated implicitly by the type machinery [2]. The important aspect from my point of view is that this implementation technique starts to break down Python's historical correlation between "function closure" and "lexically nested scope". == Conceptual Unification == The moment of clarity for me came when I realised that default arguments, lexically nested scopes and the new super() implementation can all be seen as just special cases of the broader concept of free variables and function closures. Lexically nested scopes have always been talked about in those terms, so that aspect shouldn't surprise anyone. The new super() implementation is also fairly obviously a closure, since it uses the closure machinery to work its magic. The only difference is in the way the value gets populated in the first place (i.e. by the type machinery rather than by the execution of an outer function). Due to history, default argument *values* aren't often thought of as closure references, but they really are anonymous closures. Instead of using cells, the references are stored in dedicated attributes that are known to the argument parsing machinery, but you could quite easily dispense with that and store everything as cells in the function closure (you wouldn't, since it would be a waste of time and energy, I'm just pointing out the conceptual equivalence. A *new* Python implementation, though, could choose to go down that path). After I had that realisation, the natural follow-up question seemed to be: if I wanted to explicitly declare a closure variable, and provide it with an initial value, without introducing a nested function purely for that purpose, how should I spell that? Well, I think PEP 3104 has already given us the answer: by declaring the variable name as explicitly 'nonlocal', but also providing an initial value so the compiler knows it is a *new* closure variable, rather than one from an outer lexically nested scope. This is a far more useful and meaningful addition than the trivial syntactic sugar mentioned in the PEP (but ultimately not implemented). The other question is what scope the initialisation operation should be executed in, and I think there, default arguments have the answer: in the containing scope, before the function has been defined. == Precise Syntax == By reusing 'nonlocal', we would make it clear that we're not adding a new concept to the language, but rather generalising an existing one (i.e. closure references) to provide additional flexibility in the way they're used. So I *really* want to use that keyword rather than adding a new one just for this task. However, I'm less certain about the spelling of the rest of the statement. There are at least a few possible alternative spellings: nonlocal VAR = EXPR # My initial suggestion nonlocal VAR from EXPR # Strongly indicates there's more than a simple assignment going on here nonlocal EXPR as VAR # Parser may struggle with this one Of the three, 'nonlocal VAR from EXPR' may be the best bet - it's easy for the compiler to parse, PEP 380 set the precedent for the 'from EXPR' clause to introduce a subexpression and 'nonlocal VAR = EXPR' may be too close to 'nonlocal VAR; VAR = EXPR'. Regards, Nick. [1] Some dis module details regarding the different kinds of name reference. Most notable for my point is the correspondence between the 'cell variable' in the outer function and the 'free variable' in the inner function: >>> def outer(): ... closure_ref = 1 ... def inner(): ... local_ref = 2 ... print(local_ref, closure_ref, global_ref, len) ... >>> global_ref = 3 >>> import dis >>> dis.show_code(outer) Name: outer Filename: Argument count: 0 Kw-only arguments: 0 Number of locals: 1 Stack size: 2 Flags: OPTIMIZED, NEWLOCALS Constants: 0: None 1: 1 2: ", line 3> Variable names: 0: inner Cell variables: 0: closure_ref >>> dis.show_code(outer.__code__.co_consts[2]) Name: inner Filename: Argument count: 0 Kw-only arguments: 0 Number of locals: 1 Stack size: 5 Flags: OPTIMIZED, NEWLOCALS, NESTED Constants: 0: None 1: 2 Names: 0: print 1: global_ref 2: len Variable names: 0: local_ref Free variables: 0: closure_ref [2] Some dis module output to show that there's no corresponding '__class__' cell variable anywhere when the implicit closure entry is created by the new super() machinery. >>> def outer2(): ... class C: ... def inner(): ... print(__class__) ... >>> dis.show_code(outer2) Name: outer2 Filename: Argument count: 0 Kw-only arguments: 0 Number of locals: 1 Stack size: 3 Flags: OPTIMIZED, NEWLOCALS, NOFREE Constants: 0: None 1: ", line 2> 2: 'C' Variable names: 0: C >>> dis.show_code(outer2.__code__.co_consts[1]) Name: C Filename: Argument count: 1 Kw-only arguments: 0 Number of locals: 1 Stack size: 2 Flags: NEWLOCALS Constants: 0: ", line 3> Names: 0: __name__ 1: __module__ 2: inner Variable names: 0: __locals__ Cell variables: 0: __class__ >>> dis.show_code(outer2.__code__.co_consts[1].co_consts[0]) Name: inner Filename: Argument count: 0 Kw-only arguments: 0 Number of locals: 0 Stack size: 2 Flags: OPTIMIZED, NEWLOCALS, NESTED Constants: 0: None Names: 0: print Free variables: 0: __class__ -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From greg.ewing at canterbury.ac.nz Mon Sep 26 08:56:02 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 26 Sep 2011 19:56:02 +1300 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: Message-ID: <4E802202.4080009@canterbury.ac.nz> Nick Coghlan wrote: > In current Python, ordinary named references can refer to one of 4 namespaces: > > ... > - closure reference (stored in a cell object by the function that > defined it, kept alive after the frame is recycled by references from > still living inner functions that need it) > ... I think you're mixing up concepts and implementation here a bit. Cells are an implementation detail. What's important is which *namespace* the name is referring to: - A 'global' declaration makes it refer to the module-level namespace. - A 'nonlocal' declaration makes it refer to some namespace in between local and module-level (there may be more than one of these, so I wouldn't say that there are only 4 namespaces). Now, while the *value* of your proposed new kind of variable would be stored as part of the function's closure, its *name* would be part of the function's *local* namespace. Consider this: i = 88 def f(): nonlocal i = 17 print i def g(): nonlocal i = 42 print i f() g() print i I'm assuming you intend that the two i's here would have nothing to do with each other, or with the i in the enclosing scope, so that the output from this would be 17 42 88 rather than 42 42 42 So, your proposed use of 'nonlocal' would actually be declaring a name to be *local*. That strikes me as weird and perverse. NOBODY-expects-the-spanish-nonlocal-declaration!-ly, Greg From p.f.moore at gmail.com Mon Sep 26 13:33:52 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 26 Sep 2011 12:33:52 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E802202.4080009@canterbury.ac.nz> References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On 26 September 2011 07:56, Greg Ewing wrote: > So, your proposed use of 'nonlocal' would actually be declaring > a name to be *local*. That strikes me as weird and perverse. Aha! That's precisely the concern I had with the suggestion of "nonlocal" for this, although I had failed to understand *why* it bothered me, and so hadn't commented... Paul From ncoghlan at gmail.com Mon Sep 26 13:34:36 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2011 07:34:36 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E802202.4080009@canterbury.ac.nz> References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On Mon, Sep 26, 2011 at 2:56 AM, Greg Ewing wrote: > - A 'nonlocal' declaration makes it refer to some namespace in > ?between local and module-level (there may be more than one of > ?these, so I wouldn't say that there are only 4 namespaces). > > Now, while the *value* of your proposed new kind of variable > would be stored as part of the function's closure, its *name* > would be part of the function's *local* namespace. Consider > this: > > ? ?i = 88 > > ? ?def f(): > ? ? ? ?nonlocal i = 17 > ? ? ? ?print i > > ? ?def g(): > ? ? ? ?nonlocal i = 42 > ? ? ? ?print i > > ? ?f() > ? ?g() > ? ?print i > > I'm assuming you intend that the two i's here would have nothing > to do with each other, or with the i in the enclosing scope, so > that the output from this would be > > ? ?17 > ? ?42 > ? ?88 > > rather than > > ? ?42 > ? ?42 > ? ?42 > > So, your proposed use of 'nonlocal' would actually be declaring > a name to be *local*. That strikes me as weird and perverse. Ah, but it *wouldn't* be local, that's the point - it would be stored on the function rather than on the frame, and hence be shared across invocations. Change your example function to this so it actually modifies the name binding, and it becomes clear that this is *not* a local variable declaration: i = 88 def f(): nonlocal i from 17 print(i) i += 1 >>> f() 17 >>> f() 18 It would work exactly as if we had introduced a containing scope as a closure: def outer(): i = 17 def f(): nonlocal i print(i) i += 1 return f >>> f = outer() >>> f() 17 >>> f() 18 The *only* thing that would change is the way the closure reference would be initialised - it would happen as part of the function definition (just like default arguments) rather than needing the developer to write the outer scope explicitly just to initialise the closure references. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From p.f.moore at gmail.com Mon Sep 26 13:45:36 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 26 Sep 2011 12:45:36 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On 26 September 2011 12:34, Nick Coghlan wrote: > On Mon, Sep 26, 2011 at 2:56 AM, Greg Ewing wrote: >> So, your proposed use of 'nonlocal' would actually be declaring >> a name to be *local*. That strikes me as weird and perverse. > > Ah, but it *wouldn't* be local, that's the point - it would be stored > on the function rather than on the frame, and hence be shared across > invocations. Hmm, its lifetime is non-local, but the visibility is still local. My instincts associate the word "(non-)local" with visibility rather than lifetime. If you want a bikeshed to colour in, maybe "persistent" is a better keyword for this: def counter(): persistent n as 1 print(n) n += 1 Paul. From alex.gaynor at gmail.com Mon Sep 26 14:26:41 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 26 Sep 2011 12:26:41 +0000 (UTC) Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined References: <4E802202.4080009@canterbury.ac.nz> Message-ID: Nick Coghlan writes: > i = 88 > > def f(): > nonlocal i from 17 > print(i) > i += 1 > > def outer(): > i = 17 > def f(): > nonlocal i > print(i) > i += 1 > return f > > >>> f = outer() You had me, you really did. Right up until you showed the current equivalent. This strikes me as a few things. Most importantly, as you noted yourself, a pretty rare case, even in C static variables are probably the rarest scope of variable. This strikes me as a) not saving very much code, it's like crappy HFS instead of real sugar ;), and b) not adding fundamental value, I think both blocks of code are equally readable. Other examples of syntatic sugar, such as decorators, have code motion properties that let you think about code in the places that makes sense, and I don't think this has that. Alex From masklinn at masklinn.net Mon Sep 26 15:03:49 2011 From: masklinn at masklinn.net (Masklinn) Date: Mon, 26 Sep 2011 15:03:49 +0200 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On 2011-09-26, at 14:26 , Alex Gaynor wrote: > Nick Coghlan writes: >> i = 88 >> >> def f(): >> nonlocal i from 17 >> print(i) >> i += 1 >> >> def outer(): >> i = 17 >> def f(): >> nonlocal i >> print(i) >> i += 1 >> return f >> >>>>> f = outer() > > You had me, you really did. Right up until you showed the current equivalent. > This strikes me as a few things. > > Most importantly, as you noted yourself, a pretty rare case, even in C static > variables are probably the rarest scope of variable. This strikes me as a) not > saving very much code, it's like crappy HFS instead of real sugar ;), and b) > not adding fundamental value, I think both blocks of code are equally readable. > Other examples of syntatic sugar, such as decorators, have code motion > properties that let you think about code in the places that makes sense, and I > don't think this has that. > An other thing which strikes me as weird is that the proposal is basically the creation of private instance attribute on functions. Could you not get the same by actually setting an attribute on the function (this can not be used in lambdas in any case)? def f(): print(f.i) f.i += 1 f.i = 17 and some of the verbosity (but mostly reverse-reading) could be sucrosed-away with a decorator: @setattribute(i=17) def f(): print(f.i) f.i += 1 because as far as I can tell, if this can make it there is little justification for keeping an explicit `self` (among other things). This proposal also does not help with the "reverse argument hack" in lambdas, since it's using a statement. From ncoghlan at gmail.com Mon Sep 26 15:47:28 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2011 09:47:28 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On Mon, Sep 26, 2011 at 7:45 AM, Paul Moore wrote: > On 26 September 2011 12:34, Nick Coghlan wrote: >> On Mon, Sep 26, 2011 at 2:56 AM, Greg Ewing wrote: >>> So, your proposed use of 'nonlocal' would actually be declaring >>> a name to be *local*. That strikes me as weird and perverse. >> >> Ah, but it *wouldn't* be local, that's the point - it would be stored >> on the function rather than on the frame, and hence be shared across >> invocations. > > Hmm, its lifetime is non-local, but the visibility is still local. My > instincts associate the word "(non-)local" with visibility rather than > lifetime. > > If you want a bikeshed to colour in, maybe "persistent" is a better > keyword for this: > > def counter(): > ? ?persistent n as 1 > ? ?print(n) > ? ?n += 1 Adding new keywords is a big, big step that demands a compelling justification. Now that I have come up with a way to make this syntactic sugar for existing usage of nonlocal rather than a new concept, I flatly oppose introduction of a new name for something which is, at a fundamental level, just a variation on existing functionality. I'd rather continue with the status quo indefinitely if people truly find this variant intolerable. It may be easier to turn things around and specifically look at it from the "syntactic sugar" point of view: # Current syntax def _outer(): # Boilerplate VAR = EXPR def FUNC(): # Real function name is hidden nonlocal VAR # VAR repeated # Do stuff with VAR, including rebinding it return f # Boilerplate FUNC = _outer() # Boilerplate and FUNC repeated Most of that code is noise: the interesting aspects are that: 1. There is a function called FUNC() available for use 2. VAR survives across invocations of FUNC() 3. At the first invocation of FUNC(), the initial value of VAR will be EXPR So, let's offer a syntax that just includes those 3 pieces of interesting information without the boilerplate: def FUNC(): # Function of interest is not hidden in a nested scope nonlocal VAR from EXPR # Shared variable and initial value # Do stuff with VAR Is anyone going to *guess* what that means without looking it up? Probably not. But are they going to *forget* what it means once they learn it? Also probably not. "I can guess what this means without reading the docs or having someone explain it to me" is setting the bar too high for what a single keyword can possibly hope to convey. The bar needs to be whether or not it serves as a useful mnemonic to recall the functionality once a user already know what means. For me, 'nonlocal' fits that bill, especially when the feature is described as syntactic sugar for a somewhat common use case for the existing lexical scoping functionality. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From arnodel at gmail.com Mon Sep 26 15:48:07 2011 From: arnodel at gmail.com (Arnaud Delobelle) Date: Mon, 26 Sep 2011 14:48:07 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On 26 September 2011 14:03, Masklinn wrote: > An other thing which strikes me as weird is that the proposal is basically the > creation of private instance attribute on functions. Could you not get the same > by actually setting an attribute on the function (this can not be used in > lambdas in any case)? > > ? ?def f(): > ? ? ? ?print(f.i) > ? ? ? ?f.i += 1 > ? ?f.i = 17 This has been talked about before (in this thread or a related one). It would be slower (requiring two dictionary lookups for each access, rather than a single LOAD_DEREF / STORE_DEREF) and would break if the name 'f' is bound to another object. -- Arnaud From ncoghlan at gmail.com Mon Sep 26 15:49:32 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2011 09:49:32 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On Mon, Sep 26, 2011 at 9:03 AM, Masklinn wrote: > An other thing which strikes me as weird is that the proposal is basically the > creation of private instance attribute on functions. Could you not get the same > by actually setting an attribute on the function (this can not be used in > lambdas in any case)? > > ? ?def f(): > ? ? ? ?print(f.i) > ? ? ? ?f.i += 1 > ? ?f.i = 17 No, because this fails if 'f' is rebound in the outer scope. > This proposal also does not help with the "reverse argument hack" in lambdas, since > it's using a statement. Correct, but the same can be said for 'nonlocal' itself. Besides, Guido has already nixed the lambda-friendly expression based suggestions. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ubershmekel at gmail.com Mon Sep 26 15:52:40 2011 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Mon, 26 Sep 2011 09:52:40 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: I liked the setattribute decorator. Also, static variables in C are just another way to do it (as opposed to TOOWTDI). The minor namespacing improvement isn't worth the unobvious "nonlocal .. as .." syntax in my mind. Pardon my android top-posting, --Yuval On Sep 26, 2011 9:04 AM, "Masklinn" wrote: > On 2011-09-26, at 14:26 , Alex Gaynor wrote: >> Nick Coghlan writes: >>> i = 88 >>> >>> def f(): >>> nonlocal i from 17 >>> print(i) >>> i += 1 >>> >>> def outer(): >>> i = 17 >>> def f(): >>> nonlocal i >>> print(i) >>> i += 1 >>> return f >>> >>>>>> f = outer() >> >> You had me, you really did. Right up until you showed the current equivalent. >> This strikes me as a few things. >> >> Most importantly, as you noted yourself, a pretty rare case, even in C static >> variables are probably the rarest scope of variable. This strikes me as a) not >> saving very much code, it's like crappy HFS instead of real sugar ;), and b) >> not adding fundamental value, I think both blocks of code are equally readable. >> Other examples of syntatic sugar, such as decorators, have code motion >> properties that let you think about code in the places that makes sense, and I >> don't think this has that. >> > An other thing which strikes me as weird is that the proposal is basically the > creation of private instance attribute on functions. Could you not get the same > by actually setting an attribute on the function (this can not be used in > lambdas in any case)? > > def f(): > print(f.i) > f.i += 1 > f.i = 17 > > and some of the verbosity (but mostly reverse-reading) could be sucrosed-away > with a decorator: > > @setattribute(i=17) > def f(): > print(f.i) > f.i += 1 > > because as far as I can tell, if this can make it there is little justification for > keeping an explicit `self` (among other things). > > This proposal also does not help with the "reverse argument hack" in lambdas, since > it's using a statement. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Sep 26 15:54:21 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2011 09:54:21 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On Mon, Sep 26, 2011 at 8:26 AM, Alex Gaynor wrote: > You had me, you really did. ?Right up until you showed the current equivalent. > This strikes me as a few things. > > Most importantly, as you noted yourself, a pretty rare case, even in C static > variables are probably the rarest scope of variable. ?This strikes me as a) not > saving very much code, it's like crappy HFS instead of real sugar ;), and b) > not adding fundamental value, I think both blocks of code are equally readable. > Other examples of syntatic sugar, such as decorators, have code motion > properties ?that let you think about code in the places that makes sense, and I > don't think this has that. To my mind, a 4:1 reduction in boilerplate lines, moving the function name out to the top level, making it clear that there's only one function (the inner one) that is kept around, avoiding repetition of the variable name and the function name all count as fairly substantial wins on the 'sugary goodness' front :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Mon Sep 26 16:08:56 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2011 10:08:56 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On Mon, Sep 26, 2011 at 9:54 AM, Nick Coghlan wrote: > On Mon, Sep 26, 2011 at 8:26 AM, Alex Gaynor wrote: >> You had me, you really did. ?Right up until you showed the current equivalent. >> This strikes me as a few things. >> >> Most importantly, as you noted yourself, a pretty rare case, even in C static >> variables are probably the rarest scope of variable. ?This strikes me as a) not >> saving very much code, it's like crappy HFS instead of real sugar ;), and b) >> not adding fundamental value, I think both blocks of code are equally readable. >> Other examples of syntatic sugar, such as decorators, have code motion >> properties ?that let you think about code in the places that makes sense, and I >> don't think this has that. > > To my mind, a 4:1 reduction in boilerplate lines, moving the function > name out to the top level, making it clear that there's only one > function (the inner one) that is kept around, avoiding repetition of > the variable name and the function name all count as fairly > substantial wins on the 'sugary goodness' front :) Oops, forget to mention that the toy examples I've been using for the sake of brevity fail to convey one of the benefits of the syntax (i.e. the elimination of all trailing boilerplate following the function definition), since the body of the inner function is so short. To be fair, a more accurate comparison would be to a '@closure' decorator along the lines of something I posted in the previous thread: def closure(f): # Not clear if this is actually the right thing to do # It depends on how you decide to handle annotations # and whether decorators are applied to the inner or the # outer function return functools.wraps(f)(f()) @closure def FUNC(): """Real function is the inner one""" VAR = EXPR def _inner(real, params, here): # Hidden signature! nonlocal VAR # Still have to repeat VAR # Arbitrarily long complex code here return _inner # Still have this trailing boilerplate Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From p.f.moore at gmail.com Mon Sep 26 16:24:49 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 26 Sep 2011 15:24:49 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On 26 September 2011 14:47, Nick Coghlan wrote: > On Mon, Sep 26, 2011 at 7:45 AM, Paul Moore wrote: >> On 26 September 2011 12:34, Nick Coghlan wrote: >>> On Mon, Sep 26, 2011 at 2:56 AM, Greg Ewing wrote: >>>> So, your proposed use of 'nonlocal' would actually be declaring >>>> a name to be *local*. That strikes me as weird and perverse. >>> >>> Ah, but it *wouldn't* be local, that's the point - it would be stored >>> on the function rather than on the frame, and hence be shared across >>> invocations. >> >> Hmm, its lifetime is non-local, but the visibility is still local. My >> instincts associate the word "(non-)local" with visibility rather than >> lifetime. >> >> If you want a bikeshed to colour in, maybe "persistent" is a better >> keyword for this: >> >> def counter(): >> ? ?persistent n as 1 >> ? ?print(n) >> ? ?n += 1 > > Adding new keywords is a big, big step that demands a compelling > justification. Now that I have come up with a way to make this > syntactic sugar for existing usage of nonlocal rather than a new > concept, I flatly oppose introduction of a new name for something > which is, at a fundamental level, just a variation on existing > functionality. I'd rather continue with the status quo indefinitely if > people truly find this variant intolerable. I agree entirely. My point here wasn't to suggest that this needs a new keyword, but rather that the proposal uses an unnatural keyword to avoid needing a new keyword. Your argument that this is a simple extension of the semantics of "nonlocal" is reasonable when viewing nonlocal in terms of lifetimes. My contention is that most people view nonlocal in terms of visibility (and in that view, the two uses of nonlocal are jarringly dissimilar). > It may be easier to turn things around and specifically look at it > from the "syntactic sugar" point of view: > > # Current syntax > > def _outer(): # Boilerplate > ? ?VAR = EXPR > ? ?def FUNC(): # Real function name is hidden > ? ? ? ?nonlocal VAR # VAR repeated > ? ? ? ?# Do stuff with VAR, including rebinding it > ? ?return f # Boilerplate > FUNC = _outer() # Boilerplate and FUNC repeated > > Most of that code is noise: the interesting aspects are that: > 1. There is a function called FUNC() available for use > 2. VAR survives across invocations of FUNC() > 3. At the first invocation of FUNC(), the initial value of VAR will be EXPR > > So, let's offer a syntax that just includes those 3 pieces of > interesting information without the boilerplate: > > def FUNC(): # Function of interest is not hidden in a nested scope > ? ?nonlocal VAR from EXPR ?# Shared variable and initial value > ? ?# Do stuff with VAR I have no problem with the contention that this would be useful. Minor, as you concede at the start of the thread, but certainly useful. I'm certainly bikeshedding here over the name of the keyword. But I think I'm arguing that green is the wrong colour for this stop sign, because people will misinterpret it, whereas you are arguing it's a great colour because we have this tin of green paint here, and the paint shop's closed. (End of overwrought analogy :-)) > Is anyone going to *guess* what that means without looking it up? > Probably not. But are they going to *forget* what it means once they > learn it? Also probably not. > > "I can guess what this means without reading the docs or having > someone explain it to me" is setting the bar too high for what a > single keyword can possibly hope to convey. The bar needs to be > whether or not it serves as a useful mnemonic to recall the > functionality once a user already know what means. For me, 'nonlocal' > fits that bill, especially when the feature is described as syntactic > sugar for a somewhat common use case for the existing lexical scoping > functionality. But readability matters, and I worry that this isn't "readable". Maybe the fact that neither of us is Dutch is relevant here, too :-) Paul. From ron3200 at gmail.com Mon Sep 26 16:31:47 2011 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 26 Sep 2011 09:31:47 -0500 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: <1317047507.13868.26.camel@Gutsy> On Mon, 2011-09-26 at 10:08 -0400, Nick Coghlan wrote: > On Mon, Sep 26, 2011 at 9:54 AM, Nick Coghlan wrote: > > On Mon, Sep 26, 2011 at 8:26 AM, Alex Gaynor wrote: > >> You had me, you really did. Right up until you showed the current equivalent. > >> This strikes me as a few things. > >> > >> Most importantly, as you noted yourself, a pretty rare case, even in C static > >> variables are probably the rarest scope of variable. This strikes me as a) not > >> saving very much code, it's like crappy HFS instead of real sugar ;), and b) > >> not adding fundamental value, I think both blocks of code are equally readable. > >> Other examples of syntatic sugar, such as decorators, have code motion > >> properties that let you think about code in the places that makes sense, and I > >> don't think this has that. > > > > To my mind, a 4:1 reduction in boilerplate lines, moving the function > > name out to the top level, making it clear that there's only one > > function (the inner one) that is kept around, avoiding repetition of > > the variable name and the function name all count as fairly > > substantial wins on the 'sugary goodness' front :) > > Oops, forget to mention that the toy examples I've been using for the > sake of brevity fail to convey one of the benefits of the syntax (i.e. > the elimination of all trailing boilerplate following the function > definition), since the body of the inner function is so short. > > To be fair, a more accurate comparison would be to a '@closure' > decorator along the lines of something I posted in the previous > thread: > > def closure(f): > # Not clear if this is actually the right thing to do > # It depends on how you decide to handle annotations > # and whether decorators are applied to the inner or the > # outer function > return functools.wraps(f)(f()) > > @closure > def FUNC(): > """Real function is the inner one""" > VAR = EXPR > def _inner(real, params, here): # Hidden signature! > nonlocal VAR # Still have to repeat VAR > # Arbitrarily long complex code here > return _inner # Still have this trailing boilerplate > Could this help in cleaning up the lru_cash decorator in functools? That seems like it may be a good test case as it has both defaults and wrappers in it. Cheers, Ron (Looking in Lib for things this will help with.) From ncoghlan at gmail.com Mon Sep 26 16:59:38 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2011 10:59:38 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <1317047507.13868.26.camel@Gutsy> References: <4E802202.4080009@canterbury.ac.nz> <1317047507.13868.26.camel@Gutsy> Message-ID: On Mon, Sep 26, 2011 at 10:31 AM, Ron Adam wrote: > Could this help in cleaning up the lru_cash decorator in functools? > > That seems like it may be a good test case as it has both defaults and > wrappers in it. Not really - lru_cache actually *needs* the outer function because it is a decorator factory. Once you have the outer function as a factory anyway, then the new shorthand syntax becomes somewhat less significant. lru_cache also shares the state amongst multiple functions (there are a couple of query operations that are provided as attributes on the function object). (It's written in a somewhat opaque way both to minimise runtime overhead and also to discourage people from becoming dependent on its internal implementation details) The only thing you could use it to clear up is the manual micro-optimisation in the decorating function's default arguments, and that could already be handled by doing the bindings in the outer function. That's one of the reasons this is such a niche feature regardless of how we spell it - it only works when you don't want to share the closure values across multiple inner functions. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From steve at pearwood.info Mon Sep 26 17:16:56 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 27 Sep 2011 01:16:56 +1000 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: <4E809768.4030600@pearwood.info> Masklinn wrote: > On 2011-09-26, at 14:26 , Alex Gaynor wrote: >> Nick Coghlan writes: >>> i = 88 >>> >>> def f(): >>> nonlocal i from 17 >>> print(i) >>> i += 1 >>> >>> def outer(): >>> i = 17 >>> def f(): >>> nonlocal i >>> print(i) >>> i += 1 >>> return f >>> >>>>>> f = outer() [...] > An other thing which strikes me as weird is that the proposal is basically the > creation of private instance attribute on functions. Could you not get the same > by actually setting an attribute on the function (this can not be used in > lambdas in any case)? > > def f(): > print(f.i) > f.i += 1 > f.i = 17 The advantages of the attribute solution are: - you can do it right now (functions have supported public writable attributes since v2.1); - the attribute is writable by the caller. If you want a public, writable attribute on a function, and I frequently do, this works fine. But: - the attribute is exposed to the caller even if it shouldn't be; - it's slow compared to local lookup; - lookup is by name, so if the function is renamed, it breaks; - initial value for the attribute is assigned *after* the function is created -- this is the same problem with decorators that the @ syntax was designed to fix. You can "fix" that last issue by moving the assignment inside the function, only this is even worse: def f(): try: f.i except AttributeError: f.i = 17 print(f.i) f.i += 1 or with a decorator, as you suggested. But still, it's pretty crappy to have slow public attribute access for something which should be fast and private. I have gradually warmed to Nick's suggestion. I'm not completely sold on the "nonlocal var from expr" syntax. Paul Moore's criticism makes a lot of sense to me. At the risk of dooming the proposal, "static" seems to me to be a more sensible keyword than "nonlocal". But for the sake of the argument, I'll stick to nonlocal for now. Some use-cases: (1) Early-binding: you use some constant value in a function, and nowhere else, so you don't want to make it a global, but it's expensive to calculate so you only want to do it once: # old way: _var = some_expensive_calculation() def spam(): do_stuff_with(_var) # or: def spam(_var=some_expensive_calculation()): do_stuff_with(_var) # proposed way: def spam(): nonlocal _var = some_expensive_calculation() do_stuff_with(_var) This puts the calculation inside the function where it belongs and is a win for encapsulation, without the ugly "looks like an argument, quacks like an argument, swims like an argument, but please don't try treating it as an argument" hack. (2) Persistent non-global storage: you have some value which needs to persist between calls to the function, but shouldn't be exposed as a global. A neat example comes from Guido's essay on graphs: def find_path(graph, start, end, path=[]): path = path + [start] if start == end: return path if not graph.has_key(start): return None for node in graph[start]: if node not in path: newpath = find_path(graph, node, end, path) if newpath: return newpath return None http://www.python.org/doc/essays/graphs.html I expect that could be re-written as: def find_path(graph, start, end): nonlocal path from [] path = path + [start] if start == end: return path if not graph.has_key(start): return None for node in graph[start]: if node not in path: newpath = find_path(graph, node, end) if newpath: return newpath return None The downside of this would be that the caller can now no longer seed the path argument with nodes. But for some applications, maybe that's a plus rather than a minus. (3) Micro-optimizations. An example from the random module: def randrange(self, start, stop=None, step=1, int=int): """Choose a random item from ... Do not supply the 'int' argument. """ If we're not supposed to supply the int argument, why is it an argument? Even uglier: def _randbelow(self, n, int=int, maxsize=1< 0: return recurse(x-1)+1 return 1 func = recurse del recurse Note: this use-case implies that the initial binding can't happen until *after* the function exists, otherwise recurse won't exist. -- Steven From ncoghlan at gmail.com Mon Sep 26 17:33:47 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2011 11:33:47 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On Mon, Sep 26, 2011 at 10:24 AM, Paul Moore wrote: > On 26 September 2011 14:47, Nick Coghlan wrote: >> Adding new keywords is a big, big step that demands a compelling >> justification. Now that I have come up with a way to make this >> syntactic sugar for existing usage of nonlocal rather than a new >> concept, I flatly oppose introduction of a new name for something >> which is, at a fundamental level, just a variation on existing >> functionality. I'd rather continue with the status quo indefinitely if >> people truly find this variant intolerable. > > I agree entirely. My point here wasn't to suggest that this needs a > new keyword, but rather that the proposal uses an unnatural keyword to > avoid needing a new keyword. > > Your argument that this is a simple extension of the semantics of > "nonlocal" is reasonable when viewing nonlocal in terms of lifetimes. > My contention is that most people view nonlocal in terms of visibility > (and in that view, the two uses of nonlocal are jarringly dissimilar). I agree it's certainly a *new* way of looking at the problem, but I don't agree that that necessarily makes it a *wrong* way to look at it. Should we invent a new keyword, or abandon the concept of (more) elegant syntax for shared internal function state, just to avoid teaching people that reference locality involves elements of both visibility *and* lifecycle? Instead, I contend that it is precisely this aspect of the language that makes mutable default arguments such a point of confusion. Since we *don't* separate out the explanations of visibility vs lifetime when discussing closure references, default arguments become a special snowflake that people need to learn about without being able to slot it into a broader framework. I believe we can actually make the concepts involved in both default argument lifecycles and the 3.x super() implementation more semantically coherent by *embracing* nonlocal as influencing both visibility *and* lifetime and clearly articulating that as a core language concept in relation to closures. We *already* have the following possibilities: local visibility, local lifetime: 'ordinary' local variable referenced solely from the executing frame local visibility, nonlocal lifetime: default argument expressions, local variable referenced from inner function that survives current invocation nonlocal visibility, nonlocal lifetime: 3.x __class__ reference, closure reference to outer lexically containing scope, global references My proposed syntax is just a way to explicitly create new entries in that middle category - variables with local visibility and nonlocal lifetime. So 'global VAR' and 'nonlocal VAR' would continue to declare that a NAME belongs in the third category and is initialised somewhere else, while 'nonlocal VAR from EXPR' would say "this has nonlocal lifetime, but local visibility, so here's the initial value since it won't be provided anywhere else". Regards, Nick. P.S. Really nice point about the visibility vs lifecycle distinction. I think it actually *strengthens* my argument about the current lack of a coherent semantic framework in this space, though. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From mwm at mired.org Mon Sep 26 17:45:38 2011 From: mwm at mired.org (Mike Meyer) Date: Mon, 26 Sep 2011 08:45:38 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: When this conversation started discussing namespaces, it occurred to me that we've had a number of suggestions for "statement-local" namespaces shot down. It seems that they'd solve this case as well as the intended case. I don't expect this to be acceptable, but since it solves part of this problem as well as dealing with the issues for which it was originally created, I thought I'd point it out. I'm talking about the requests that we add the let/where statement found in functional languages. Both add one new keyword. The syntax is: let: assignments in: statements or statement where: assignments which creates a namespace containing names bound in "assignments" for the duration of statement (or statements, as the case may be). I'm going to go with the let version, because it's not clear how what the syntax should be for where on a def statement. The original examples were things like: let: sumsq = sum(a * a for a in mylist) in: value = sumsq * 3 - sumsq So you can deal with the case of wanting to preserve values during a binding with something like: res = [] for i in range(20): let: i = i in: res.append(lambda x: x + i) Allowing a def in the statements means you can write the counter example with something like: let: counter = 0 in: def inc(change=1): nonlocal counter counter += change return counter We have to declare counter nonlocal in order to rebind it in this case. The odd thing here is that bindings in statements happen in the namespace that the let occurs in, but lookups of nonlocal variables include the namespace created by the let. From ubershmekel at gmail.com Mon Sep 26 17:46:59 2011 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Mon, 26 Sep 2011 11:46:59 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E809768.4030600@pearwood.info> References: <4E802202.4080009@canterbury.ac.nz> <4E809768.4030600@pearwood.info> Message-ID: I just want to chime in with how static variables can be bad for multi threaded code, yet they aren't considered as bad as global variables. E.g. the find_path guido example will explode if called by 2 threads in tandem. I believe this is more apparent and frowned upon when caused by global variables. Hasn't python given enough temptations to avoid multithreading already? Allowing these fast, local, persistent static variables is like allowing braces instead of whitespace - the good coders will manage either way but the bad will do bad things with it. --Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Sep 26 17:55:46 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 27 Sep 2011 01:55:46 +1000 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E809768.4030600@pearwood.info> Message-ID: <4E80A082.5070705@pearwood.info> Yuval Greenfield wrote: > I just want to chime in with how static variables can be bad for multi > threaded code, yet they aren't considered as bad as global variables. E.g. > the find_path guido example will explode if called by 2 threads in tandem. I > believe this is more apparent and frowned upon when caused by global > variables. Are you saying that Guido's find_path example is thread-safe when written with the default argument hack, but will fail with the new proposal? > Hasn't python given enough temptations to avoid multithreading already? One can never have enough reasons to avoid multi-threading! But seriously, unless this proposal makes things *worse*, I don't see why this is an objection. > Allowing these fast, local, persistent static variables is like allowing > braces instead of whitespace - the good coders will manage either way but > the bad will do bad things with it. That's an analogy that makes no sense to me. -- Steven From bruce at leapyear.org Mon Sep 26 18:27:47 2011 From: bruce at leapyear.org (Bruce Leban) Date: Mon, 26 Sep 2011 09:27:47 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On Mon, Sep 26, 2011 at 6:47 AM, Nick Coghlan wrote: > Adding new keywords is a big, big step that demands a compelling > justification. Agree. > Now that I have come up with a way to make this > syntactic sugar for existing usage of nonlocal rather than a new > concept, I flatly oppose introduction of a new name for something > which is, at a fundamental level, just a variation on existing > functionality. I'd rather continue with the status quo indefinitely if > people truly find this variant intolerable. You can always abuse an existing keyword to avoid adding a new one. For example, this could instead be pass x in from expression for x from expression with x from expression etc. These would surely be rejected because the keywords are used with an entirely different meaning and that makes the language harder to read and write. The keyword nonlocal means that this binding is not local to this scope but can be found up the call stack. In contrast, your usage means the binding is local to this function, created before the function is called the first time and shared with all calls to this function. Those are orthogonal scopes. --- Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Sep 26 19:01:17 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2011 13:01:17 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On Mon, Sep 26, 2011 at 11:45 AM, Mike Meyer wrote: > When this conversation started discussing namespaces, it occurred to me that > we've had a number of suggestions for "statement-local" namespaces shot > down. It seems that they'd solve this case as well as the intended case. I > don't expect this to be acceptable, but since it solves part of this problem > as well as dealing with the issues for which it was originally created, I > thought I'd point it out. See PEP 3150 - I've written fairly extensively on the topic of statement local namespaces, including their potential application to this use case :) There are some fairly significant problems with the idea, which are articulated in the PEP. If you'd like to discuss it further, please start a new thread so this one can stay focused on the more limited 'nonlocal' suggestion. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Mon Sep 26 19:05:13 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2011 13:05:13 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E80A082.5070705@pearwood.info> References: <4E802202.4080009@canterbury.ac.nz> <4E809768.4030600@pearwood.info> <4E80A082.5070705@pearwood.info> Message-ID: On Mon, Sep 26, 2011 at 11:55 AM, Steven D'Aprano wrote: > Yuval Greenfield wrote: >> Hasn't python given enough temptations to avoid multithreading already? > > One can never have enough reasons to avoid multi-threading! > > But seriously, unless this proposal makes things *worse*, I don't see why > this is an objection. I actually think the 'attractive nuisance' aspect wrt multi-threading is a potentially valid objection. OTOH, it also makes synchronising parts of process global algorithms *much* easier, since you can just do "nonlocal lock from threading.RLock()" to get a shared lock variable. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From jeanpierreda at gmail.com Mon Sep 26 19:16:08 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 26 Sep 2011 13:16:08 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: > The keyword nonlocal means that this binding is not local to this scope but > can be found up the call stack. Lexical stack. (The call stack would be dynamic scope. [I suspect you already know this; I am pedant > In contrast, your usage means the binding is > local to this function, created before the function is called the first time > and shared with all calls to this function. Those are orthogonal scopes. Not really. Created differently, yes. But after creation it works identically, by definition. However, this is not what "nonlocal" means to me, and from what others have said, it's not what it means to them either. Devin On Mon, Sep 26, 2011 at 12:27 PM, Bruce Leban wrote: > > On Mon, Sep 26, 2011 at 6:47 AM, Nick Coghlan wrote: >> >> Adding new keywords is a big, big step that demands a compelling >> justification. > > Agree. > >> >> Now that I have come up with a way to make this >> syntactic sugar for existing usage of nonlocal rather than a new >> concept, I flatly oppose introduction of a new name for something >> which is, at a fundamental level, just a variation on existing >> functionality. I'd rather continue with the status quo indefinitely if >> people truly find this variant intolerable. > > You can always abuse an existing keyword to avoid adding a new one. For > example, this could instead be > pass x in from expression > for x from?expression > with x from?expression > etc. These would surely be rejected because the keywords are used with an > entirely different meaning and that makes the language harder to read and > write. > The keyword nonlocal means that this binding is not local to this scope but > can be found up the call stack. In contrast, your usage means the binding is > local to this function, created before the function is called the first time > and shared with all calls to this function. Those are orthogonal scopes. > --- Bruce > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > From bruce at leapyear.org Mon Sep 26 19:32:40 2011 From: bruce at leapyear.org (Bruce Leban) Date: Mon, 26 Sep 2011 10:32:40 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On Mon, Sep 26, 2011 at 10:16 AM, Devin Jeanpierre wrote: > > The keyword nonlocal means that this binding is not local to this scope > but > > can be found up the call stack. > > Lexical stack. (The call stack would be dynamic scope. [I suspect you > already know this; I am pedant > Yeah, I know that but a perfect example of how easy it is to get confused. :-) > > > In contrast, your usage means the binding is > > local to this function, created before the function is called the first > time > > and shared with all calls to this function. Those are orthogonal scopes. > > Not really. Created differently, yes. But after creation it works > identically, by definition. > To a python developer, sure. To a python programmer, I don't think so. --- Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Sep 26 21:08:48 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 26 Sep 2011 12:08:48 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: <4E80CDC0.90909@stoneleaf.us> Bruce Leban wrote: > The keyword nonlocal means that this binding is not local to this scope > but can be found up the call stack. In contrast, your usage means the > binding is local to this function, created before the function is called > the first time and shared with all calls to this function. Those are > orthogonal scopes. Another way of looking at this is that the keyword nonlocal means the name is bound in a surrounding scope; which is still true with Nick's proposal. Nick's proposal has the added benefit of fast lookup. nonlocal also carries with it the implication of persistence, which Nick's proposal also has (indeed, it's one of the main points). ~Ethan~ From ethan at stoneleaf.us Mon Sep 26 21:09:39 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 26 Sep 2011 12:09:39 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: Message-ID: <4E80CDF3.8070104@stoneleaf.us> Nick Coghlan wrote: > nonlocal VAR from EXPR # Strongly indicates there's more than a > # simple assignment going on here +1 From tjreedy at udel.edu Mon Sep 26 21:45:48 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Sep 2011 15:45:48 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On 9/26/2011 7:34 AM, Nick Coghlan wrote: > Ah, but it *wouldn't* be local, that's the point - it would be stored > on the function rather than on the frame, and hence be shared across > invocations. Today's default arg values are readonly (because stored in an immutable structure), quasi-anonymous* closure values used to initialize local names. If people understood that better, there would be less puzzlement over their behavior. Some of this is tied up with the confusion over name and value versus 'variable'. * They are anonymous in that they are access by index rather than by name. On the other hand, the call mechanism deterministically associates them with local names. Nick, I think you are on to something, but I don't know how to get from here to there. One of the explicit reasons for introducing closures defined by nesting was to reduce default value use. Personally, though, I think writing an outer function just for that reason (when only one function object is every going to be created, is a cure as bad or worse than the disease. When multiple functions are to be created, the new closures are an improvement. -- Terry Jan Reedy From ncoghlan at gmail.com Mon Sep 26 21:57:02 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2011 15:57:02 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On Mon, Sep 26, 2011 at 12:27 PM, Bruce Leban wrote: > The keyword nonlocal means that this binding is not local to this scope but > can be found up the call stack. In contrast, your usage means the binding is > local to this function, created before the function is called the first time > and shared with all calls to this function. Those are orthogonal scopes. See my reply to Paul - while I agree that visibility and lifetime are different aspects of named references, it isn't like they're independent. A nonlocal reference from an inner function has a direct effect on the lifecycle of an otherwise local variable in the outer function. Consider the following two functions: def f(): x = 0 return x Here, 'x' is an ordinary local variable - it only exists while the frame is executing. def f(): x = 0 def inner(): nonlocal x x += 1 return x return inner Now, the use of 'nonlocal' in the inner function has *changed the lifecycle* of x. It is now a nonlocal variable - it survives beyond the execution of the function that defines it. Paul was quite right to point out that most developers only think about the visibility aspect of 'nonlocal', since they're looking at the impact from the point of view of the inner function, but that doesn't mean the lifecycle impact on the outer function that I want to highlight is arbitrary or irrelevant. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tjreedy at udel.edu Mon Sep 26 22:01:40 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Sep 2011 16:01:40 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On 9/26/2011 10:24 AM, Paul Moore wrote: > I agree entirely. My point here wasn't to suggest that this needs a > new keyword, but rather that the proposal uses an unnatural keyword to > avoid needing a new keyword. > > Your argument that this is a simple extension of the semantics of > "nonlocal" is reasonable when viewing nonlocal in terms of lifetimes. > My contention is that most people view nonlocal in terms of visibility > (and in that view, the two uses of nonlocal are jarringly dissimilar). In the template below, the visibility of VAR in _outer is both ephemeral (lasting only for one quick call) and non-essential (in that VAR is never *used* within _outer. Today's default value objects *are* visible non-locally: >>> def f(a=3): pass >>> f.__defaults__[0] 3 They optionally get bound to local names but they are *not* local values any more than cell values are. Lifetime and visibility are actually coupled. You cannot see something that does not exist; invisible objects tend to be garbage collected and cease to exist. >> It may be easier to turn things around and specifically look at it >> from the "syntactic sugar" point of view: >> >> # Current syntax >> >> def _outer(): # Boilerplate >> VAR = EXPR >> def FUNC(): # Real function name is hidden >> nonlocal VAR # VAR repeated >> # Do stuff with VAR, including rebinding it >> return f # Boilerplate >> FUNC = _outer() # Boilerplate and FUNC repeated >> >> Most of that code is noise: the interesting aspects are that: >> 1. There is a function called FUNC() available for use >> 2. VAR survives across invocations of FUNC() >> 3. At the first invocation of FUNC(), the initial value of VAR will be EXPR >> >> So, let's offer a syntax that just includes those 3 pieces of >> interesting information without the boilerplate: >> >> def FUNC(): # Function of interest is not hidden in a nested scope >> nonlocal VAR from EXPR # Shared variable and initial value >> # Do stuff with VAR -- Terry Jan Reedy From ncoghlan at gmail.com Mon Sep 26 22:17:30 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2011 16:17:30 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On Mon, Sep 26, 2011 at 4:01 PM, Terry Reedy wrote: > On 9/26/2011 10:24 AM, Paul Moore wrote: > >> I agree entirely. My point here wasn't to suggest that this needs a >> new keyword, but rather that the proposal uses an unnatural keyword to >> avoid needing a new keyword. >> >> Your argument that this is a simple extension of the semantics of >> "nonlocal" is reasonable when viewing nonlocal in terms of lifetimes. >> My contention is that most people view nonlocal in terms of visibility >> (and in that view, the two uses of nonlocal are jarringly dissimilar). > > In the template below, the visibility of VAR in _outer is both ephemeral > (lasting only for one quick call) and non-essential (in that VAR is never > *used* within _outer. > > Today's default value objects *are* visible non-locally: > >>>> def f(a=3): pass > >>>> f.__defaults__[0] > 3 Indeed, it could be said that the defining feature of a nonlocal reference is that it is accessible via the function object, whether that's through __defaults__, __kwdefaults__ or __closure__. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ericsnowcurrently at gmail.com Tue Sep 27 00:23:16 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 26 Sep 2011 16:23:16 -0600 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On Mon, Sep 26, 2011 at 2:17 PM, Nick Coghlan wrote: > Indeed, it could be said that the defining feature of a nonlocal > reference is that it is accessible via the function object, whether > that's through __defaults__, __kwdefaults__ or __closure__. First of all, I like the idea. I do agree that nonlocal is not the optimal name (neither is static). However, the concept matches well enough and it's not like nonlocal is used so much that everyone would have to rewire their brains. I do have some questions. Would the value from the nonlocal-from be stored in a cell in __closure__ or in a new special attribute? Would the name be stored in co_freevars or somewhere else? I realize that's an implementation detail and perhaps it's premature to worry about. Still, I'm curious about if you'll be able to distinguish this kind of value from existing ones when inspecting function and code objects. How does this impact the other Python implementations? I would think not much, since it's not that different from how closures are already handled (if I understood the idea correctly). As Steven already pointed out, this may allow the actual function object to be referenced in the function body, but not by default. Sounds good. However, this hinges on the point at which the cell is created relative to different results of the compilation process (like the decorator calls). From what I understand, this should not be hard to take care of. Would it be a big deal to make sure a function's self-referencing nonlocal-from would work? As Terry noted, default argument values are sort of read-only. A function's __defaults__ (and __kwdefaults__) is writable and bound to a tuple. The local name that maps to each value in __defaults__ is re-initialized for each call. The proposed nonlocal-from syntax would not reflect these characteristics. Instead it is more of a persistent/static mechanism, just like closures. So is the nonlocal-from an alternate closure syntax inspired by the default argument hack, or should it actually behave more like default arguments do? -eric > > Regards, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From greg.ewing at canterbury.ac.nz Tue Sep 27 01:35:09 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 27 Sep 2011 12:35:09 +1300 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: <4E810C2D.7040607@canterbury.ac.nz> Paul Moore wrote: > Your argument that this is a simple extension of the semantics of > "nonlocal" is reasonable when viewing nonlocal in terms of lifetimes. > My contention is that most people view nonlocal in terms of visibility > (and in that view, the two uses of nonlocal are jarringly dissimilar). Same here. What's more, when I think about what 'nonlocal' means at a conceptual level, I'm not thinking about cells attached to function objects. I'm thinking about reaching out to something pre-existing outside of the function. In contrast, the proposed feature would be creating something new, and putting it somewhere that the programmer can't even see. > But readability matters, and I worry that this isn't "readable". Maybe > the fact that neither of us is Dutch is relevant here, too :-) But Guido doesn't appear to like it either, so Dutchness does not seem to be a sufficient condition for appreciating this brand of logic. :-( -- Greg From tjreedy at udel.edu Tue Sep 27 01:57:03 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 26 Sep 2011 19:57:03 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: On 9/26/2011 6:23 PM, Eric Snow wrote: > As Terry noted, default argument values are sort of read-only. A > function's __defaults__ (and __kwdefaults__) is writable and bound to > a tuple. Oh, right, I keep forgetting that. The tuple of defaults in immutable but in current Python, at least, the tuple is replaceable. So a function can indirectly rewrite its defaults as long as it is still bound to its definition name. > The local name that maps to each value in __defaults__ is > re-initialized for each call. The proposed nonlocal-from syntax would > not reflect these characteristics. Instead it is more of a > persistent/static mechanism, just like closures. For a default that is intended to be permanent and never overshadowed, there is no need to rebind with every call. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Tue Sep 27 02:03:30 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 27 Sep 2011 13:03:30 +1300 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: <4E8112D2.809@canterbury.ac.nz> Nick Coghlan wrote: > My proposed syntax is just a way to explicitly create new entries in > that middle category - variables with local visibility and nonlocal > lifetime. I don't think that's the right way to characterise default argument values. While the *values* have nonlocal lifetime, the *name* being declared is a parameter, so both its scope and lifetime are local. Your proposal would be creating a new category of name that doesn't currently exist. I don't see how your proposal would do anything to clarify the distinction between visibility and lifetime. Currently, 'nonlocal' always refers to visibility. Your way, it would sometimes mean visibility and sometimes lifetime, with only a very obtuse clue as to the difference. -- Greg From guido at python.org Tue Sep 27 03:32:19 2011 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Sep 2011 18:32:19 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E8112D2.809@canterbury.ac.nz> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> Message-ID: Interesting discussion! Very enlightening. But after all that I'm still unconvinced that the new concept should be tied to the nonlocal keyword. Nick's right that the *mechanism* for nonlocals and __class__ is the same, and the mechanism for the new concept might well be the same. And possibly a hypothetical other Python implementation could use the same mechanism to keep track of default argument values. Or not. But at a different level, nonlocal and the new concept are quite different, and the difference has to do with the scope of the name, not its lifetime (this was a very important insight in the discussion!). I think it is actually very telling that the syntax for nonlocal, __class__ and default argument values is so different: to most users, the lifetime is much less of an issue than the scope. Now, let's pretend the new feature is called "own". We'd write: def counter(): own x = 0 x += 1 return x If we had another function (say, defined in the same scope): def count_by_twos(): own x = 0 x += 2 return x the x in counter() and count_by_twos() are unrelated. This is different from the way nonlocal works (I don't have to type the example I hope :-). Also, nonlocal is illegal in a global function -- syntactically, it must be in a function that is nested inside another function. For all these reasons, sorry Nick, but I really don't think any syntax based on nonlocal will do it, even "nonlocal from ". I see two options open at this point. 1) Bikeshed until we've come up with the right keyword. I think that apart from the keyword the optimal syntax is entirely fixed; " = " is the way to go. (I suppose we could argue about how to declare multiple variables this way.) I would be okay with using either "static" or "own" for the keyword. Personally, I don't really mind all the negative baggage that static has, but a Google code search for \bown\s= shows way fewer hits than for \bstatic\s= (both with lang:^python$ case:yes). 2) Give up, and keep using the default argument hack. You can protect your arguments from accidentally being overridden by using a "lone star" in the argument list (PEP 3102) and a __private naming convention. The lone star prevents accidental overriding when too many positional arguments are given. The __private provides sufficient protection against misguided attempts to provide a value for such "arguments", even if for functions outside a class it is purely psychological. (Note that someone who *really* wanted to mess with a cell's variable could also do it.) -- --Guido van Rossum (python.org/~guido) From zuo at chopin.edu.pl Tue Sep 27 03:40:12 2011 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Tue, 27 Sep 2011 03:40:12 +0200 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: Message-ID: <20110927014012.GG3218@chopin.edu.pl> Nick Coghlan dixit (2011-09-26, 11:01): > nonlocal VAR from EXPR # Strongly indicates there's more than a > nonlocal EXPR as VAR # Parser may struggle with this one The construct itself appeals to me, but I share reservations against using `nonlocal` for something used within the local lexical scope only. I'd like to use `deflocal` keyword -- it'd be suggest something local (in terms of scope), something constant (definition-time-related) and something complementary to `nonlocal`. deflocal VAR from EXPR -- would be perfect for me. But indeed -- that's a new keyword. Regards. *j From ncoghlan at gmail.com Tue Sep 27 04:23:27 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2011 22:23:27 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E8112D2.809@canterbury.ac.nz> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> Message-ID: On Mon, Sep 26, 2011 at 8:03 PM, Greg Ewing wrote: > Nick Coghlan wrote: > >> My proposed syntax is just a way to explicitly create new entries in >> that middle category - variables with local visibility and nonlocal >> lifetime. > > I don't think that's the right way to characterise default > argument values. While the *values* have nonlocal lifetime, > the *name* being declared is a parameter, so both its scope > and lifetime are local. Your proposal would be creating a > new category of name that doesn't currently exist. Yes, I did state that for default arguments it referred to the anonymous values. The only name currently in this category is the magic __class__ reference in the new super() implementation. Part of the idea here is to make it so that that name is less of a special case. > I don't see how your proposal would do anything to clarify > the distinction between visibility and lifetime. Currently, > 'nonlocal' always refers to visibility. Your way, it would > sometimes mean visibility and sometimes lifetime, with only > a very obtuse clue as to the difference. Actually, with my proposal it would always refer to lifetime, and only sometimes to visibility. The latter case would be implied by the fact that it isn't initialised inside the current function definition (currently you *can't* initialise it locally, so the term always refers to both lifetime *and* visibility). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Tue Sep 27 04:43:38 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Sep 2011 22:43:38 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> Message-ID: On Mon, Sep 26, 2011 at 9:32 PM, Guido van Rossum wrote: > Interesting discussion! Very enlightening. Indeed! It has certainly clarified many aspects for me. > But after all that I'm still unconvinced that the new concept should > be tied to the nonlocal keyword. > For all these reasons, sorry Nick, but I really don't think any syntax > based on nonlocal will do it, even "nonlocal from ". Well, the semantics of a 'nonlocal-from' initialised nonlocal variable would be different from a bare nonlocal declaration (i.e. the new form would create a new name binding only visible in local scope, so it wouldn't get shared across functions within the same outer function and is legal in top-level functions), just as they would be with a new keyword. So I'm not prepared to completely give up on this one yet - it took me years of poking at the issue to come up with the idea of reusing nonlocal as the keyword, so I hope people will take some time to consider the idea of using the term to refer to variations in both visibility and lifecycle rather than strictly limiting it to refer only to the visibility aspect. > I see two options open at this point. > > 1) Bikeshed until we've come up with the right keyword. I think that > apart from the keyword the optimal syntax is entirely fixed; > " = " is the way to go. (I suppose we > could argue about how to declare multiple variables this way.) I would > be okay with using either "static" or "own" for the keyword. > Personally, I don't really mind all the negative baggage that static > has, but a Google code search for \bown\s= shows way fewer hits than > for \bstatic\s= (both with lang:^python$ case:yes). There have certainly been plenty of proposals over the years. The ones I recall off the top of my head are: once static persistent shared closure own atdef deftime The baggage something like 'static' brings with it would be even more misleading than that associated with 'nonlocal', and the others are either seriously ugly or else aren't particularly suggestive as mnemonics. At least 'nonlocal' indicates something related to function closures is in play, even if it isn't the exact same thing as is currently meant by a bare declaration. Really, the mechanism suffers from the same problem that led to the choice of nonlocal for PEP 3104 in the first place: the important part of the declaration lies in making it clear to the compiler and the reader that the specified name is *not* a normal local variable. Exactly what it *is* instead will depend on the specifics of the code - it may be shared state, it may be something else, the only way to find out for sure is to look at the code and see how it is used. Choosing a keyword based on any given use case (such as 'shared') makes the construct seem odd when used for another use case (such as early binding). I agree the two concepts (bare nonlocal and initialised nonlocal) are not the same. However, I also believe they're close enough that a slight broadening of the definition of 'nonlocal' is preferable to trying to come up with a completely new term that is both readable, usefully mnemonic and not laden down with unrelated baggage either from Python itself or from other languages. > 2) Give up, and keep using the default argument hack. You can protect > your arguments from accidentally being overridden by using a "lone > star" in the argument list (PEP 3102) and a __private naming > convention. The lone star prevents accidental overriding when too many > positional arguments are given. The __private provides sufficient > protection against misguided attempts to provide a value for such > "arguments", even if for functions outside a class it is purely > psychological. (Note that someone who *really* wanted to mess with a > cell's variable could also do it.) Despite what I wrote earlier in the thread, I don't think we actually need to declare the idea *dead* if you don't currently like 'nonlocal' as the keyword. Perhaps the 'nonlocal' idea will grow on you (it certainly did on me, the longer I considered the idea), or perhaps you or someone else will come up with an appropriate keyword that everyone considers reasonable. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Tue Sep 27 04:48:14 2011 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Sep 2011 19:48:14 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> Message-ID: On Mon, Sep 26, 2011 at 7:43 PM, Nick Coghlan wrote: > Despite what I wrote earlier in the thread, I don't think we actually > need to declare the idea *dead* if you don't currently like 'nonlocal' > as the keyword. Perhaps the 'nonlocal' idea will grow on you (it > certainly did on me, the longer I considered the idea), or perhaps you > or someone else will come up with an appropriate keyword that everyone > considers reasonable. Sorry, from reading the responses, the objection that the other use of nonlocal refers to scope, not lifetime, seems pretty common, and I don't think it will go away. You might as well use "def var = expr" as the syntax and justify it by saying that it is clearly defining something and syntactically different from function definitions... OTOH I am happy to let you all bikeshed on a better name. -- --Guido van Rossum (python.org/~guido) From ghostwriter402 at gmail.com Tue Sep 27 05:53:26 2011 From: ghostwriter402 at gmail.com (Spectral One) Date: Mon, 26 Sep 2011 22:53:26 -0500 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> Message-ID: <4E8148B6.3060408@gmail.com> On 9/26/2011 9:48 PM, Guido van Rossum wrote: > On Mon, Sep 26, 2011 at 7:43 PM, Nick Coghlan wrote: >> Despite what I wrote earlier in the thread, I don't think we actually >> need to declare the idea *dead* if you don't currently like 'nonlocal' >> as the keyword. Perhaps the 'nonlocal' idea will grow on you (it >> certainly did on me, the longer I considered the idea), or perhaps you >> or someone else will come up with an appropriate keyword that everyone >> considers reasonable. > Sorry, from reading the responses, the objection that the other use of > nonlocal refers to scope, not lifetime, seems pretty common, and I > don't think it will go away. You might as well use "def var = expr" as > the syntax and justify it by saying that it is clearly defining > something and syntactically different from function definitions... > > OTOH I am happy to let you all bikeshed on a better name. I was actually considering asking whether using 'def.static' or 'def.initial' would be a possible way to avoid making a new keyword. I honestly prefer def to nonlocal for this. I like that "our" is short, but while it's less confusing, I don't see is as clear. Questions on options: Could there be an importable "function" that, when used, declares a variable to be one of these? Could the keyword be toggled in or out of namespaces based on, say, an imported module? If so, the keyword wouldn't be terribly burdensome. The following could also add to Nick's list of keyword options: init (or Initial or even initialize) retain hold held persist preserve keep kept lasting stash survive ark reshiyth Given the frequency of the use case, clarity probably trumps concerns about keyword length. To my eye, those all seem more congruous than "nonlocal." (Nick's list: nonlocal once static persistent shared closure own atdef deftime) -Nate From guido at python.org Tue Sep 27 06:12:07 2011 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Sep 2011 21:12:07 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E8148B6.3060408@gmail.com> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> Message-ID: On Mon, Sep 26, 2011 at 8:53 PM, Spectral One wrote: > On 9/26/2011 9:48 PM, Guido van Rossum wrote: >> >> On Mon, Sep 26, 2011 at 7:43 PM, Nick Coghlan ?wrote: >>> >>> Despite what I wrote earlier in the thread, I don't think we actually >>> need to declare the idea *dead* if you don't currently like 'nonlocal' >>> as the keyword. Perhaps the 'nonlocal' idea will grow on you (it >>> certainly did on me, the longer I considered the idea), or perhaps you >>> or someone else will come up with an appropriate keyword that everyone >>> considers reasonable. >> >> Sorry, from reading the responses, the objection that the other use of >> nonlocal refers to scope, not lifetime, seems pretty common, and I >> don't think it will go away. You might as well use "def var = expr" as >> the syntax and justify it by saying that it is clearly defining >> something and syntactically different from function definitions... >> >> OTOH I am happy to let you all bikeshed on a better name. > > I was actually considering asking whether using 'def.static' or > 'def.initial' would be a possible way to avoid making a new keyword. I > honestly prefer def to nonlocal for this. I like that "our" is short, but > while it's less confusing, I don't see is as clear. "our" is something in Perl, right? My proposal was "own", which is much older (Algol-60). Probably nobody else here remembers it. > Questions on options: > ? Could there be an importable "function" that, when used, declares a > variable to be one of these? The compiler would have to recognize that function as special, which is not really possible, since the Python compiler has no access to the runtime environment, so it can't know what is imported (__future__ is the exception, and that's why it's a __xxx__ word). > ? Could the keyword be toggled in or out of namespaces based on, say, an > imported module? If so, the keyword wouldn't be terribly burdensome. We do that regularly with "from __future__ import " and when the parser sees it, it modifies itself (slightly). Modifying the list of reserved words is easy. But it's not a long-term solution. Somehow you triggered a thought in my head: maybe se should stop focusing on the syntax for a while and focus on the code we want to be generated for it. If we can design the bytecode, perhaps that would help us come up with the right keyword. > The following could also add to Nick's list of keyword options: > ?init (or Initial or even initialize) retain hold held persist preserve keep > kept lasting stash survive ark reshiyth > > Given the frequency of the use case, clarity probably trumps concerns about > keyword length. > To my eye, those all seem more congruous than "nonlocal." > > (Nick's list: > nonlocal once static persistent shared closure own atdef deftime) -- --Guido van Rossum (python.org/~guido) From hetchkay at gmail.com Tue Sep 27 07:01:42 2011 From: hetchkay at gmail.com (H. Krishnan) Date: Mon, 26 Sep 2011 22:01:42 -0700 (PDT) Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: <3978419.810.1317099702327.JavaMail.geo-discussion-forums@prfc6> Since the expression will not be evaluated in the body of the function, would it make sense to define it outside the function body: def FUNC() given VAR1 [= EXPR1], VAR2 [= EXPR2]: # Do stuff with VAR1, VAR2 VAR1, VAR2 etc. will be evaluated in order so that EXPR2 can refer to VAR1. Values of VAR1, VAR2 etc. are preserved across function calls. This will be similar to scheme's let: (define func (let ((var1 expr1) (var2 expr2)) (lambda () ))) Regards, Krishnan -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Sep 27 07:05:47 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 27 Sep 2011 18:05:47 +1300 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: <4E8159AB.2030506@canterbury.ac.nz> Terry Reedy wrote: > One of the explicit reasons for introducing closures > defined by nesting was to reduce default value use. Indeed, and the question that needs to be asked is: Why does the default argument hack persist, when we now have a lexical scope system that's just as good as other languages such as Scheme and Haskell have, yet no such kludge seems to be needed in those languages? I don't believe it's because there's anything wrong with our functions or closures. Rather, it's a symptom of lingering deficiencies in *other* parts of the language. For example, when someone writes for i in things: def f(): dosomethingwith(i) squirrelaway(f) and get surprised by the result, he's effectively assuming that the for-loop creates a new binding for i each time around. He may not *realise* he's assuming that, but he is. Given that people seem to unconsciously make that assumption, perhaps we should consider making the for-loop actually behave that way. There's a precedent for this -- list comprehensions used to leak their loop variable into the surrounding scope, but that was eventually changed. Another point I'd like to make is that I don't think Nick's proposal, or any spelling variation of it, helps all that much in this case. Having to do *anything* special in or around the function to work around this problem is something of a kludge, especially if it involves writing anything as tautological-looking as "i = i". -- Greg From greg.ewing at canterbury.ac.nz Tue Sep 27 07:15:28 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 27 Sep 2011 18:15:28 +1300 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> Message-ID: <4E815BF0.9010807@canterbury.ac.nz> Nick Coghlan wrote: > Now, the use of 'nonlocal' in the inner function has *changed the > lifecycle* of x. It is now a nonlocal variable - it survives beyond > the execution of the function that defines it. Yes, but that's just a logical consequence of the fact that it's *visible* from the inner function, together with the fact that Python never throws anything away while it's still accessible. Visibility is still the primary concept attached to 'nonlocal'. -- Greg From cmjohnson.mailinglist at gmail.com Tue Sep 27 08:34:39 2011 From: cmjohnson.mailinglist at gmail.com (Carl Matthew Johnson) Date: Mon, 26 Sep 2011 20:34:39 -1000 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E8159AB.2030506@canterbury.ac.nz> References: <4E802202.4080009@canterbury.ac.nz> <4E8159AB.2030506@canterbury.ac.nz> Message-ID: <465D00FC-D144-4606-AF90-2F5014C4ED3A@gmail.com> On Sep 26, 2011, at 7:05 PM, Greg Ewing wrote: > For example, when someone writes > > for i in things: > def f(): > dosomethingwith(i) > squirrelaway(f) > > and get surprised by the result, he's effectively > assuming that the for-loop creates a new binding for > i each time around. He may not *realise* he's assuming > that, but he is. For the record, I remember being surprised when I learned that for doesn't create a new scope. This seems like a kind of Python 4000 feature though? At the very least you'd need to preserve the ability to use "old-style" for-loops by writing something like, i = None for i in things: ... if i is not None: print("Processed one or more things...") -- Carl Johnson From p.f.moore at gmail.com Tue Sep 27 09:09:56 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Sep 2011 08:09:56 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> Message-ID: On 27 September 2011 05:12, Guido van Rossum wrote: > "our" is something in Perl, right? My proposal was "own", which is > much older (Algol-60). Probably nobody else here remembers it. I do :-) Not sure that Algol-60 using it is a good enough reason for recommending it, though. The name didn't make sense to me even back then... (Actually, after this discussion, the logic is clearer - so congratulations, Nick, for clarifying a 40-year old puzzle for me!) Call by name, anyone? Paul. From ron3200 at gmail.com Tue Sep 27 09:36:27 2011 From: ron3200 at gmail.com (Ron Adam) Date: Tue, 27 Sep 2011 02:36:27 -0500 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> Message-ID: <1317108987.16933.90.camel@Gutsy> On Mon, 2011-09-26 at 22:43 -0400, Nick Coghlan wrote: > On Mon, Sep 26, 2011 at 9:32 PM, Guido van Rossum wrote: > > Interesting discussion! Very enlightening. > > Indeed! It has certainly clarified many aspects for me. > > > But after all that I'm still unconvinced that the new concept should > > be tied to the nonlocal keyword. > > > > > For all these reasons, sorry Nick, but I really don't think any syntax > > based on nonlocal will do it, even "nonlocal from ". I think I agree. It's close, but not close enough to what you want. > So I'm not prepared to completely give up on this one yet - it took me > years of poking at the issue to come up with the idea of reusing > nonlocal as the keyword, so I hope people will take some time to > consider the idea of using the term to refer to variations in both > visibility and lifecycle rather than strictly limiting it to refer > only to the visibility aspect. Don't give up just yet, I think you may just need an out of the box view point to see it from a different angle... Besides keywords, there is also syntax and or symbols. One of the things I like with python objects, is a symbol or operator, is really just a nicer way to call a method. Maybe this is option #3. ;-) Which reminds me... the '@' used for decorators doesn't follow that pattern. Too bad. If it did, it could be used for other things. The assignment nature of the '@' sort of fits this problem. def foo(x): @own {"y":12} #--> foo.__at__("own", **kwds): ... ... Ok, it's weird, I admit. @validate Anyway, this is just an idea example not to be taken too seriously. Maybe something could work in a similar way that doesn't require a keyword. Cheers, Ron From ncoghlan at gmail.com Tue Sep 27 11:35:05 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Sep 2011 05:35:05 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E815BF0.9010807@canterbury.ac.nz> References: <4E802202.4080009@canterbury.ac.nz> <4E815BF0.9010807@canterbury.ac.nz> Message-ID: On Tue, Sep 27, 2011 at 1:15 AM, Greg Ewing wrote: > Nick Coghlan wrote: > >> Now, the use of 'nonlocal' in the inner function has *changed the >> lifecycle* of x. It is now a nonlocal variable - it survives beyond >> the execution of the function that defines it. > > Yes, but that's just a logical consequence of the fact > that it's *visible* from the inner function, together > with the fact that Python never throws anything away > while it's still accessible. Visibility is still the > primary concept attached to 'nonlocal'. Hmm, I think you just summarised the key insight that led to me suggesting nonlocal in the first place (even though I wasn't thinking of it in exactly these terms at the time). A local name binding is visible only from the current *invocation* of a function, hence it only lasts until the call returns. On the other hand, an explicitly 'nonlocal' name binding is visible from: - the current invocation of the outer function where the name is defined - all invocations of any functions defined within that outer function As you note, the changes in lifecycle then follow from the change in visibility - as long as any of those inner functions survive, so does the name binding. Non-local name bindings can of course also happen implicitly if there's no local assignment to override. My proposal is merely that we add a *new* meaning to 'nonlocal' that supplements the existing meaning, to declare that a name is visible from all invocations of the function currently being defined rather than being local to each invocation. As with existing nonlocal usage, the change in visibility (all invocations vs current invocation) then leads directly to a change in lifecycle (linked to the lifetime of the function object rather than until an individual call returns). The 'nonlocal' terminology could also be seen as referring to the evaluation scope for the initialisation expression, since it would run in the surrounding scope, just like default argument values. Since the new semantics slots in neatly between actual locals and the current usage of 'nonlocal', and because 'nonlocal' itself is such a generic term, it just seems easier to me to teach people about the link between visibility and lifetime and two possible meanings for nonlocal than it would be to: 1. find a new term (we've been trying and failing at this for years now) 2. teach people what it means Step 2 doesn't get any easier just because we use a different name - if anything it gets harder, since people are already used to nonlocal as a modifier on name visibility, and the new term will need to indicate a closely related concept. Is giving 'nonlocal' a second related-but-not-identical meaning a perfect answer? No, I don't think so. However, when generators were added, the 'def' keyword went from unambiguously declaring a function to declaring either an ordinary function or a generator-iterator factory and people seemed to cope. 'yield' and 'yield from' don't mean exactly the same thing, but I expect to be able to cope with that as well. If someone can understand the concept of function scoped variables at all, then they'll be able to understand the difference between 'nonlocal VAR' and 'nonlocal VAR from EXPR'. Given past history, I seriously doubt our ability to come up with a keyword for function scoped variables that is any more intuitive than a variation on nonlocal. We would then have the following hierarchy of visibility: Local scope (VAR = EXPR): visible from current function invocation Function scope (nonlocal VAR from EXPR): visible from all invocations of this function Lexical scope (nonlocal VAR; VAR = EXPR): visible from outer function where VAR is a local and all invocations of this and any peer functions Module scope (global VAR; VAR = EXPR): visible from all code in the current module Process scope (import builtins; builtins.VAR = EXPR): visible from all code in the current process For name *lookup*, lexical scoping, module scoping and process scoping can all happen implicitly when a name doesn't refer to a local. Function scope lookup would only happen with an explicit declaration (regardless of how it was spelt). Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Tue Sep 27 12:08:53 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Sep 2011 06:08:53 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> Message-ID: On Tue, Sep 27, 2011 at 12:12 AM, Guido van Rossum wrote: > Somehow you triggered a thought in my head: maybe se should stop > focusing on the syntax for a while and focus on the code we want to be > generated for it. If we can design the bytecode, perhaps that would > help us come up with the right keyword. Heh, doing exactly that (i.e. thinking in terms of how it would *work* and how I would *explain* those semantics to someone in terms of existing language concepts) is where my nonlocal suggestion came from in the first place :) As I see it, there are 3 possible ways to approach this feature, and they vary mainly in terms of when the initialisation expressions get evaluated: 1. Prior to function definition (i.e. the same as default argument values) 2. After function definition but before decorators are applied 3. After decorator application I don't like number 3, since it means the closure references haven't been populated yet when the decorators are executed, so decorators that needed to call the function wouldn't work any more and I think number 2 would just be plain confusing. Hence, I prefer number 1, with a decorator based idiom if anyone wants to play recursion tricks. The semantics for a function with a function scoped variable declared would then be along the lines of: @example def global_accumulator(x): nonlocal tally from 0 # Insert preferred spelling here tally += x return tally meaning: def _outer(): tally = 0 @example def global_accumulator(x): tally += x return tally return global_accumulator global_accumulator = _outer() Of course, the compiler wouldn't have to generate the bytecode like that - it could easily do the expression evaluations, stash them on the stack, and then create the inner function. The MAKE_CLOSURE opcode could be adjusted to look at the code object and pop the initialisation values for the function scoped variables and create the additional cell references in the __closure__ attribute. However, one advantage of actually going the implicit outer scope route is that the function scoped variables would naturally be able to refer to each other, so you could easily cache an initial value and then some further derived values that depended on the first one. For the recursion use cases, I'd suggest actually offering a 'functools.rebind_closure' function that provided an officially supported way to modify a closure reference in a function. You could then write a function with an early binding reference to itself as follows: def recursive(f): f.rebind_closure('recurse', f) @recursive def factorial(x): nonlocal recurse from recursive # This value gets replaced by the decorator if x = 0: return 1 return x * recurse(x-1) Yes, it would be rather clumsy, but it is also incredibly uncommon for the simple 'recurse by name' approach to actually fail in practice. If we did decide to support recursive references directly, as in: def factorial(x): nonlocal recurse from factorial # Early binding - and nonlocal suggesting evaluation in outer scope ;) if x = 0: return 1 return x * recurse(x-1) Then the way to go would be the second option above where the function scope variables can see the undecorated version of the function: def _outer(): # First we define our function def global_accumulator(x): tally += x return tally # Then evaluate the function scope variables tally = 0 # Then we apply our decorators return example(global_accumulator) global_accumulator = _outer() Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Tue Sep 27 12:14:08 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Sep 2011 06:14:08 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <1317108987.16933.90.camel@Gutsy> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Tue, Sep 27, 2011 at 3:36 AM, Ron Adam wrote: > Don't give up just yet, ?I think you may just need an out of the box > view point to see it from a different angle... As far as I can see from the discussion so far, proposing 'nonlocal' for this use case *is* the out of the box viewpoint seeing it from a different angle ;) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From greg.ewing at canterbury.ac.nz Tue Sep 27 09:03:37 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 27 Sep 2011 20:03:37 +1300 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <465D00FC-D144-4606-AF90-2F5014C4ED3A@gmail.com> References: <4E802202.4080009@canterbury.ac.nz> <4E8159AB.2030506@canterbury.ac.nz> <465D00FC-D144-4606-AF90-2F5014C4ED3A@gmail.com> Message-ID: <4E817549.7080107@canterbury.ac.nz> Carl Matthew Johnson wrote: > This seems like a kind of Python 4000 feature though? > At the very least you'd need to preserve the ability to use > "old-style" for-loops by writing something like, > > i = None > for i in things: > ... > > if i is not None: print("Processed one or more things...") There's actually a way of implementing this so that the above code continues to work. If the loop variable is referenced by a nested function, it will be in a cell. All you need to do is create a *new* cell each time round the loop instead of replacing the contents of the existing cell. Code after the loop will then see whatever was last assigned to the loop variable, as it does currently. -- Greg From guido at python.org Tue Sep 27 16:57:33 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Sep 2011 07:57:33 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> Message-ID: On Tue, Sep 27, 2011 at 12:09 AM, Paul Moore wrote: > On 27 September 2011 05:12, Guido van Rossum wrote: >> "our" is something in Perl, right? My proposal was "own", which is >> much older (Algol-60). Probably nobody else here remembers it. > > I do :-) > > Not sure that Algol-60 using it is a good enough reason for > recommending it, though. The name didn't make sense to me even back > then... (Actually, after this discussion, the logic is clearer - so > congratulations, Nick, for clarifying a 40-year old puzzle for me!) In the days of Algol-60 they were pretty bad at explaining things. I was puzzled by "own" as well. (However, the biggest mystery for me, for a long time, were pointers in Pascal. It didn't clear up until I learned assembly.) > Call by name, anyone? ABC, Python's predecessor, had it, IIRC. -- --Guido van Rossum (python.org/~guido) From p.f.moore at gmail.com Tue Sep 27 18:03:54 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Sep 2011 17:03:54 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E815BF0.9010807@canterbury.ac.nz> Message-ID: On 27 September 2011 10:35, Nick Coghlan wrote: > If someone can understand the concept of function scoped variables at > all, then they'll be able to understand the difference between > 'nonlocal VAR' and 'nonlocal VAR from EXPR'. Given past history, I > seriously doubt our ability to come up with a keyword for function > scoped variables that is any more intuitive than a variation on > nonlocal. I think I'm starting to get o the point where I understand "nonlocal VAR from EXPR" when I see it. But that doesn't mean it's becoming clearer, rather it means that I'm learning to mentally translate it to "static VAR = EXPR" - or own, or whatever - the concept doesn't have a clear name to me, which is why choosing a keyword isn't easy, but whatever it is, it's not really internalised as "nonlocal". On the other hand, I do (sort of) understand "nonlocal VAR" as meaning "look outside for VAR" directly. Only somewhat, because I don't think "nonlocal" is a particularly good name even for that concept, but that ship's sailed. > We would then have the following hierarchy of visibility: > > Local scope (VAR = EXPR): visible from current function invocation > Function scope (nonlocal VAR from EXPR): visible from all invocations > of this function > Lexical scope (nonlocal VAR; VAR = EXPR): visible from outer function > where VAR is a local and all invocations of this and any peer > functions > Module scope (global VAR; VAR = EXPR): visible from all code in the > current module > Process scope (import builtins; builtins.VAR = EXPR): visible from all > code in the current process > > For name *lookup*, lexical scoping, module scoping and process scoping > can all happen implicitly when a name doesn't refer to a local. > Function scope lookup would only happen with an explicit declaration > (regardless of how it was spelt). Nice summary, but it does emphasise that function scope is (still) an odd one out, as it's not available for lookup without a declaration. So we're close, but things still aren't completely unified. And yes, I know that lookup semantics are so baked into the language that they aren't going to change - that's the point, think of it the other way round, that declaring variables as function-scope is odd, because it's the only one that doesn't have an implicit form for lookup semantics of code later in the function. Meh. I tried to give an example of what I mean, to make it clearer, but I can't quite get one to work right now, and I need to go. I'll let it stew for a while and try to clarify a bit more later. Paul. Paul. From ncoghlan at gmail.com Tue Sep 27 19:03:02 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Sep 2011 13:03:02 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E815BF0.9010807@canterbury.ac.nz> Message-ID: On Tue, Sep 27, 2011 at 12:03 PM, Paul Moore wrote: > Meh. I tried to give an example of what I mean, to make it clearer, > but I can't quite get one to work right now, and I need to go. I'll > let it stew for a while and try to clarify a bit more later. It's actually (kind of) the same in C. Names come from the current or surrounding scope, or are retrieved from another module by the linker, with static used as a modifier to alter lifetime or visibility. At the module level in C, "static VAR = EXPR;" doesn't change the lifecycle of VAR, but it does change the visibility by hiding it from the linker. Inside a function, it doesn't really change the visibility (you can still only access it from code inside the function), but it does alter the lifecycle by sharing the attribute across all invocations of the function. (Since C doesn't have nested functions, such variables effectively become process global). That's why I really don't like 'static' as the keyword - it has two different meanings in C, and C++ added a third by using it to denote class attributes. Python already adopted that third meaning when Guido created 'staticmethod', so I'm not a fan of also using it for 'static' in the "function state shared across invocations" sense. (Although, if we *did* go that way, I'd probably describe it as "a static method remains the same regardless of which instance you retrieve it from and a static reference remains the same across all invocations of the function") If we're going to settle for a term with multiple meanings, then I still think 'nonlocal' is the one that makes the most sense: - it's already a keyword - the scope to be described lies between 'local' and the existing meaning of 'nonlocal' on the visibility scale - it is suggestive of the fact that the initialisation expression is not evaluated in the local scope - the two usages would relate to different aspects of function closures and use the same underlying implementation at call time All that 'nonlocal' really says is 'not a local'. Since a function scoped variable *isn't* a local variable, I don't understand the preference for the even more vague 'static'. I know which one I'd prefer to try to explain to someone learning Python that didn't already know C :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ethan at stoneleaf.us Tue Sep 27 19:08:43 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 27 Sep 2011 10:08:43 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E815BF0.9010807@canterbury.ac.nz> Message-ID: <4E82031B.4050202@stoneleaf.us> Nick Coghlan wrote: > All that 'nonlocal' really says is 'not a local'. Since a function > scoped variable *isn't* a local variable, I don't understand the > preference for the even more vague 'static'. nonlocal makes sense to me: - initialization: local? no - lifetime: local? no ~Ethan~ From guido at python.org Tue Sep 27 19:10:35 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Sep 2011 10:10:35 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Tue, Sep 27, 2011 at 3:14 AM, Nick Coghlan wrote: > On Tue, Sep 27, 2011 at 3:36 AM, Ron Adam wrote: >> Don't give up just yet, ?I think you may just need an out of the box >> view point to see it from a different angle... > > As far as I can see from the discussion so far, proposing 'nonlocal' > for this use case *is* the out of the box viewpoint seeing it from a > different angle ;) Which reminds me of a Calvin and Hobbes cartoon. Calvin has a bad report card and says "You know how Einstein's grades were bad? Well mine are even *worse*!" IOW, out of the box thinking doesn't automatically lead to the right conclusion. :-) You can argue that nonlocal is about lifetime instead of visibility until you're blue in the face, but I still disagree. It is first and foremost about visibility. Lifetime is secondary. Note how a nonlocal variable may actually have a lifetime longer than that of the function using it that declares it as nonlocal -- that function object may be deleted from the containing scope, but the variable of course still exists in the containing scope. -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue Sep 27 19:12:43 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Sep 2011 10:12:43 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> Message-ID: On Tue, Sep 27, 2011 at 3:08 AM, Nick Coghlan wrote: > As I see it, there are 3 possible ways to approach this feature, and > they vary mainly in terms of when the initialisation expressions get > evaluated: > 1. Prior to function definition (i.e. the same as default argument values) > 2. After function definition but before decorators are applied > 3. After decorator application Undoubtedly #1 is the way to go. -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue Sep 27 19:15:43 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Sep 2011 10:15:43 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E817549.7080107@canterbury.ac.nz> References: <4E802202.4080009@canterbury.ac.nz> <4E8159AB.2030506@canterbury.ac.nz> <465D00FC-D144-4606-AF90-2F5014C4ED3A@gmail.com> <4E817549.7080107@canterbury.ac.nz> Message-ID: On Tue, Sep 27, 2011 at 12:03 AM, Greg Ewing wrote: > Carl Matthew Johnson wrote: > >> This seems like a kind of Python 4000 feature though? > >> At the very least you'd need to preserve the ability to use >> >> "old-style" for-loops by writing something like, >> >> i = None >> for i in things: >> ? ?... >> >> if i is not None: print("Processed one or more things...") > > There's actually a way of implementing this so that the > above code continues to work. If the loop variable is > referenced by a nested function, it will be in a cell. > All you need to do is create a *new* cell each time > round the loop instead of replacing the contents of the > existing cell. Code after the loop will then see > whatever was last assigned to the loop variable, as > it does currently. Greg, can you start a separate thread for this? It seems to be a much bigger change to the language, and it deserves its own thread, not mixed in with Nick's idea for nonlocal (even if the mechanism for both will end up using cells :-). (And yes, agreed that that can work.) -- --Guido van Rossum (python.org/~guido) From mwm at mired.org Tue Sep 27 19:35:07 2011 From: mwm at mired.org (Mike Meyer) Date: Tue, 27 Sep 2011 10:35:07 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: How about "eustace" (from Eric Frank Russell's "Next of Kin")? You're declaring the variable to be a eustace for this function. "bopamagilvie" would be better, but is a bit long. From p.f.moore at gmail.com Tue Sep 27 20:46:55 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Sep 2011 19:46:55 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E815BF0.9010807@canterbury.ac.nz> Message-ID: Before I start this, please note that I am not in favour of "static" as the keyword to use here. I agree it has the wrong implications. Although my C-based instincts make it feel right, which implies to me that there's a lot more to this whole debate than purely logical arguments over what is "right". People's tastes and instincts are crucial here, for better or worse. On 27 September 2011 18:03, Nick Coghlan wrote: > If we're going to settle for a term with multiple meanings, then I > still think 'nonlocal' is the one that makes the most sense: > - it's already a keyword To be honest, I think you need to stop raising this. You're right, and it's certainly a lot easier to accept reusing an existing keyword than adding a new one. But you need to convince people that "nonlocal" is right, not that it's easier. You're at risk of giving a distinct "I've got a hammer so it must be a nail" impression here... > - the scope to be described lies between 'local' and the existing > meaning of 'nonlocal' on the visibility scale Or, to put that more negatively, "it's different from the existing meaning of nonlocal". I see your point, but the argument's a bit weak. > - it is suggestive of the fact that the initialisation expression is > not evaluated in the local scope Not really (at least not to me). Nonlocal is associated (by its position) with the VAR, not with the EXPR. If anything, it says "assign this local EXPR to the nonlocal VAR" which implies it's equivalent to "nonlocal VAR; VAR = EXPR". > - the two usages would relate to different aspects of function > closures and use the same underlying implementation at call time I don't really follow this - it's too obscure to be compelling, and on an initial reading, the first thing I noticed was "different aspects". Anyway, I'd rather similarity of end user experience dictate similarity of keyword than similarity of implementation... > All that 'nonlocal' really says is 'not a local'. Since a function > scoped variable *isn't* a local variable, I don't understand the > preference for the even more vague 'static'. I know which one I'd > prefer to try to explain to someone learning Python that didn't > already know C :) Agreed. Personally, I'm not trying to argue that "static" is better than "nonlocal". To be precise over my position: 1. I like the functionality, it's a minor improvement, but it seems to fill a gap. (Although it is minor enough that I doubt I'd use it much in practice, which makes me wonder if I'm qualified to have an opinion anyway...) 2. I don't think it's important enough to warrant a new keyword. If there was an "obvious" existing keyword, I'd support it like a shot. But otherwise, unless Guido suddenly says he'd agree to a new keyword for this, I agree we're stuck trying to make existing keywords work if we want the functionality. 3. Intellectually, I can see your arguments as to why "nonlocal" is logical. But I never really liked the term in the first place, and my instinct is that it doesn't work for this new usage, regardless of logic. I keep trying to like it, though (see points 1 and 2 :-)) 4. I'm getting more and more able to simply read "nonlocal VAR from EXPR" as you intend it. But that's just familiarity - I still don't have a proper term to describe the concept (hence my use of the phrase "as you intend it" - I couldn't think of anything natural in the way that "as a global variable" feels). Oddly enough, now that Guido referred me back to the Algol-60 "own" variable concept, I can see why that name fits now (it's one of the function's "own" variables, not belonging to any other scope) and I'm starting to think of these as "own variables" in my mind. That's only appropriate for old fuddy-duddies like me (or experienced language designers like Guido!) though :-) Overall, I'm probably -0 on using "nonlocal". I'm +1 on the functionality, but if it needs a new keyword, that trumps my +1. Paul. From ron3200 at gmail.com Tue Sep 27 20:51:39 2011 From: ron3200 at gmail.com (ron adam) Date: Tue, 27 Sep 2011 13:51:39 -0500 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: <1317149499.18697.1.camel@Gutsy> On Tue, 2011-09-27 at 06:14 -0400, Nick Coghlan wrote: > On Tue, Sep 27, 2011 at 3:36 AM, Ron Adam wrote: > > Don't give up just yet, I think you may just need an out of the box > > view point to see it from a different angle... > > As far as I can see from the discussion so far, proposing 'nonlocal' > for this use case *is* the out of the box viewpoint seeing it from a > different angle ;) This box is starting to look like a tesseract. :-) From ncoghlan at gmail.com Tue Sep 27 21:18:24 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Sep 2011 15:18:24 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E815BF0.9010807@canterbury.ac.nz> Message-ID: On Tue, Sep 27, 2011 at 2:46 PM, Paul Moore wrote: >> - it is suggestive of the fact that the initialisation expression is >> not evaluated in the local scope > > Not really (at least not to me). Nonlocal is associated (by its > position) with the VAR, not with the EXPR. If anything, it says > "assign this local EXPR to the nonlocal VAR" which implies it's > equivalent to "nonlocal VAR; VAR = EXPR". That comment gets us back to the "nonlocal EXPR as VAR" syntax. That one looks really odd when used with constants though (what's a nonlocal zero?), and I'm not sure the parser could handle it. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From gisle at activestate.com Tue Sep 27 21:50:04 2011 From: gisle at activestate.com (Gisle Aas) Date: Tue, 27 Sep 2011 21:50:04 +0200 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> Message-ID: <054DF9F5-625D-4205-8148-5758E04E4708@activestate.com> On Sep 27, 2011, at 6:12 , Guido van Rossum wrote: > "our" is something in Perl, right? That's right, and "our" is similar to Python's "global". Perl went with "state" as the keyword the concept described in this thread. state $foo = 1; --Gisle From ncoghlan at gmail.com Tue Sep 27 21:53:39 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Sep 2011 15:53:39 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Tue, Sep 27, 2011 at 1:10 PM, Guido van Rossum wrote: > On Tue, Sep 27, 2011 at 3:14 AM, Nick Coghlan wrote: > You can argue that nonlocal is about lifetime instead of visibility > until you're blue in the face, but I still disagree. It is first and > foremost about visibility. Lifetime is secondary. Note how a nonlocal > variable may actually have a lifetime longer than that of the function > using it that declares it as nonlocal -- that function object may be > deleted from the containing scope, but the variable of course still > exists in the containing scope. I actually changed my mind on that front when replying to Greg - I now agree that visibility is the important aspect, but I also think there are two levels of visibility within a function: local scope, which is distinct for every call to the function, and "function scope", which is shared across all invocations to the function. This difference is reflected at the implementation level in where the references are stored (i.e. as locals on the frame in the first case, as references from the function object in the latter). Currently the only references which live in that second scope without also being visible outside the function's own lexical scope are default arguments, and those get bound to ordinary local references when it comes to actually accessing them from the body of the function. Accordingly, even after conceding the visibility vs lifetime point, I'm still in favour of extending nonlocal downwards to include "function scope" in addition to its current usage for lexical scoping. Finding a different word for "not as local as a local variable, but not as nonlocal as a lexically scoped variable" has proven to be more than a little challenging and the flat negation of "nonlocal" suggests that any addition would be at least as ill fitting as "global" is at the module end of the scale. Unfortunately, what I see as a minor tweak to the meaning of nonlocal, yourself, Paul and Greg seem to see as referring to a profoundly different concept. While that remains the case, I doubt I'm going to have much luck persuading any of you as to the reasonableness of the syntax. I do find it interesting that at least some of the possible keywords that have been suggested for the function scope use case were *also* put forward as possibilities in the context of PEP 3104 (which eventually gave us 'nonlocal') [1]. I'm not sure how much significance to ascribe to that fact, but I do think it provides at least some level of support for my thesis that the line between the two concepts of function scoping and lexical scoping is blurrier than it first appears. Regards, Nick. [1] http://www.python.org/dev/peps/pep-3104/#id14 -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Tue Sep 27 21:55:58 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Sep 2011 15:55:58 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <054DF9F5-625D-4205-8148-5758E04E4708@activestate.com> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <054DF9F5-625D-4205-8148-5758E04E4708@activestate.com> Message-ID: On Tue, Sep 27, 2011 at 3:50 PM, Gisle Aas wrote: > On Sep 27, 2011, at 6:12 , Guido van Rossum wrote: > >> "our" is something in Perl, right? > > That's right, and "our" is similar to Python's "global". ?Perl went with "state" as the keyword the concept described in this thread. > > ? state $foo = 1; I actually like that as a term for the concept (I've certainly used it in my own descriptions in this very thread), but I shudder to think how much code would have to change if we made 'state' a keyword :P Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From p.f.moore at gmail.com Tue Sep 27 22:15:28 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Sep 2011 21:15:28 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On 27 September 2011 20:53, Nick Coghlan wrote: > On Tue, Sep 27, 2011 at 1:10 PM, Guido van Rossum wrote: >> On Tue, Sep 27, 2011 at 3:14 AM, Nick Coghlan wrote: >> You can argue that nonlocal is about lifetime instead of visibility >> until you're blue in the face, but I still disagree. It is first and >> foremost about visibility. Lifetime is secondary. Note how a nonlocal >> variable may actually have a lifetime longer than that of the function >> using it that declares it as nonlocal -- that function object may be >> deleted from the containing scope, but the variable of course still >> exists in the containing scope. > > I actually changed my mind on that front when replying to Greg - I now > agree that visibility is the important aspect, but I also think there > are two levels of visibility within a function: local scope, which is > distinct for every call to the function, and "function scope", which > is shared across all invocations to the function. That description just makes my head hurt. To me, visibility is simply which lines of code, when using a name, refer to that particular variable. So a variable x at module level has "global scope" because it is visible from all lines (in the module, ignoring shadowing concerns). If x is defined within a function, it has local (to the function) scope, because only lines within the function can see that variable. In other words scope is essentially a lexical concept tied to lexical visibility. To that extent, these new closed-over values have local scope because they are visible from the same lines of code as a local variable. But their lifetime differs. The ideas here are more commonly seen in the history of Lisp, which originally had "dynamic" scope (which is more lifetime-oriented) but moved to lexical (visibility-based) scope. There was a time when Lisps were described as Lisp-1 or Lisp-2 depending on their position on scope. I need to go and do some research into old Lisp concepts to see how they might apply here. I may be some time :-) Paul. From greg.ewing at canterbury.ac.nz Tue Sep 27 22:38:50 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 28 Sep 2011 09:38:50 +1300 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> Message-ID: <4E82345A.3010105@canterbury.ac.nz> With regard to "own" or "static", my feeling is that this kind of feature (i.e. mutable shared state visible only to one function) is an anti-pattern in language design. We have much better ways nowadays of sharing state without making it completely global, such as modules and classes. There was a discussion a while back about a "const" statement, which was essentially what we are talking about here except that the name would be read-only. This seems like both a better name and better semantics to me. As I remember, the idea foundered because it was thought too confusing to have an expression that was written inside the function but evaluated outside of it. If we're seriously considering the current proposal, presumably we've gotten over that? Or is it still a valid objection? -- Greg From arnodel at gmail.com Tue Sep 27 23:04:26 2011 From: arnodel at gmail.com (Arnaud Delobelle) Date: Tue, 27 Sep 2011 22:04:26 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E82345A.3010105@canterbury.ac.nz> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: On 27 September 2011 21:38, Greg Ewing wrote: [...] > As I remember, the idea foundered because it was thought > too confusing to have an expression that was written > inside the function but evaluated outside of it. If > we're seriously considering the current proposal, > presumably we've gotten over that? Or is it still a > valid objection? I have been reading this thread from the start and FWIW, it's something that I have been feeling uncomfortable about. I value the guarantee that no code in the body of a def statement is executed when the def statement itself is executed. This proposal would break this guarantee. -- Arnaud From tjreedy at udel.edu Tue Sep 27 23:10:27 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 27 Sep 2011 17:10:27 -0400 Subject: [Python-ideas] For loop binding (was Re: Tweaking closures ...) In-Reply-To: <4E8159AB.2030506@canterbury.ac.nz> References: <4E802202.4080009@canterbury.ac.nz> <4E8159AB.2030506@canterbury.ac.nz> Message-ID: On 9/27/2011 1:05 AM, Greg Ewing wrote: > > For example, when someone writes > > for i in things: > def f(): > dosomethingwith(i) > squirrelaway(f) > > and get surprised by the result, People who post their 'surprise' on python-list have nearly always used lambda, not def, to define the function. This is so much the case that I have concluded that 'lambda' has a hypnotic effect that 'def' does not. > he's effectively assuming that the for-loop creates a new binding for > i each time around. No, for loops *do* rebind loop variables each time around. People are assuming that 'i' is immediately bound to its 'current' value, just like default args. (This is the opposite of people who mis-assume that default arg expressions are re-evaluated with each call.) This assumption is tied much more to 'lambda' than 'def'. > He may not *realise* he's assuming that, but he is. > > Given that people seem to unconsciously make that > assumption, *Some* people -- those who come from a language that does whatever it is that python does not do that you want it to do. > perhaps we should consider making the > for-loop actually behave that way. There's a precedent > for this -- list comprehensions used to leak their > loop variable into the surrounding scope, but that > was eventually changed. This has *nothing* to do with people assuming early binding of names in lambda expressions. The posted 'surprise code' typically uses list comps. So changing for loops to be like list comps would have no effect on that mis-assumptions and the resulting surprise. -- Terry Jan Reedy From guido at python.org Tue Sep 27 23:23:09 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Sep 2011 14:23:09 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: On Tue, Sep 27, 2011 at 2:04 PM, Arnaud Delobelle wrote: > On 27 September 2011 21:38, Greg Ewing wrote: > [...] >> As I remember, the idea foundered because it was thought >> too confusing to have an expression that was written >> inside the function but evaluated outside of it. If >> we're seriously considering the current proposal, >> presumably we've gotten over that? Or is it still a >> valid objection? > > I have been reading this thread from the start and FWIW, it's > something that I have been feeling uncomfortable about. ?I value the > guarantee that no code in the body of a def statement is executed when > the def statement itself is executed. ?This proposal would break this > guarantee. That seems a guarantee made up specifically to prevent this proposal. What benefit do you see from it? -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Wed Sep 28 00:23:56 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Sep 2011 18:23:56 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <054DF9F5-625D-4205-8148-5758E04E4708@activestate.com> Message-ID: Hmm... 'local', 'statelocal', 'nonlocal', 'global', 'builtin'? -- Nick Coghlan (via Gmail on Android, so likely to be more terse than usual) On Sep 27, 2011 3:55 PM, "Nick Coghlan" wrote: > On Tue, Sep 27, 2011 at 3:50 PM, Gisle Aas wrote: >> On Sep 27, 2011, at 6:12 , Guido van Rossum wrote: >> >>> "our" is something in Perl, right? >> >> That's right, and "our" is similar to Python's "global". Perl went with "state" as the keyword the concept described in this thread. >> >> state $foo = 1; > > I actually like that as a term for the concept (I've certainly used it > in my own descriptions in this very thread), but I shudder to think > how much code would have to change if we made 'state' a keyword :P > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Wed Sep 28 00:24:55 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 27 Sep 2011 16:24:55 -0600 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Tue, Sep 27, 2011 at 2:15 PM, Paul Moore wrote: > On 27 September 2011 20:53, Nick Coghlan wrote: >> I actually changed my mind on that front when replying to Greg - I now >> agree that visibility is the important aspect, but I also think there >> are two levels of visibility within a function: local scope, which is >> distinct for every call to the function, and "function scope", which >> is shared across all invocations to the function. > > That description just makes my head hurt. To me, visibility is simply > which lines of code, when using a name, refer to that particular > variable. So a variable x at module level has "global scope" because > it is visible from all lines (in the module, ignoring shadowing > concerns). If x is defined within a function, it has local (to the > function) scope, because only lines within the function can see that > variable. In other words scope is essentially a lexical concept tied > to lexical visibility. Perhaps that is why people sometimes miss that default argument values persist across calls. The names are locals, but the values live in an often unconsidered (and relatively artificial) "function definition scope". The variables in the proposed nonlocal-from statement are in that same function definition scope implicitly. However, unlike other names, they're bound to their scope by a statement in a different scope (the local one). Also, since they are variables they would be stored as cells (like __closure__ does). This is another contrast to default arguments, which are stored only as values (in __defaults__ and __kwdefaults__). The function definition scope isn't the only source of confusion here. In one context "nonlocal" would mean "not local (and not global)". In another context it would mean "in function definition scope". Here's a different angle to show what I mean (I'm not proposing this). Currently, if a nonlocal statement refers to a name that is not bound in an enclosing function scope, it gives a syntax error: >> def f(): ... def g(): ... nonlocal x ... return g ... SyntaxError: no binding for nonlocal 'x' found If that situation instead implied "function definition scope", you would get a similar effect to nonlocal-from: persistent state for select variables across calls. However, the behavior would be consistent across all uses of nonlocal. "nonlocal x=5" (if allowed) would be the same as "nonlocal x; x=5". You'd lose the ability to set a one-time initial value, but the point is that nonlocal would become the means to interact with variables in the function definition scope. Ultimately, it seems like we are looking for a way to do three things here: 1. bless a concrete function definition scope (persist across calls), 2. allow one-time initialization of those persistent variables, 3. (relatedly) have some expressions evaluated at definition time but available at execution time. Default arguments cover some of this. Closures cover some of it. The nonlocal-from statement covers a bunch. -eric > > To that extent, these new closed-over values have local scope because > they are visible from the same lines of code as a local variable. But > their lifetime differs. > > The ideas here are more commonly seen in the history of Lisp, which > originally had "dynamic" scope (which is more lifetime-oriented) but > moved to lexical (visibility-based) scope. There was a time when Lisps > were described as Lisp-1 or Lisp-2 depending on their position on > scope. > > I need to go and do some research into old Lisp concepts to see how > they might apply here. I may be some time :-) > > Paul. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From arnodel at gmail.com Wed Sep 28 00:24:58 2011 From: arnodel at gmail.com (Arnaud Delobelle) Date: Tue, 27 Sep 2011 23:24:58 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: On 27 September 2011 22:23, Guido van Rossum wrote: > On Tue, Sep 27, 2011 at 2:04 PM, Arnaud Delobelle wrote: >> On 27 September 2011 21:38, Greg Ewing wrote: >> [...] >>> As I remember, the idea foundered because it was thought >>> too confusing to have an expression that was written >>> inside the function but evaluated outside of it. If >>> we're seriously considering the current proposal, >>> presumably we've gotten over that? Or is it still a >>> valid objection? >> >> I have been reading this thread from the start and FWIW, it's >> something that I have been feeling uncomfortable about. ?I value the >> guarantee that no code in the body of a def statement is executed when >> the def statement itself is executed. ?This proposal would break this >> guarantee. > > That seems a guarantee made up specifically to prevent this proposal. I wouldn't do this :) I like new things. > What benefit do you see from it? It keeps things simple. To be specific, when I read code I can "fold" (mentally or with the help of a code editor) the body of a def statement, knowing that I won't miss anything until the function is called. So the following won't raise: [some code] x = 42 def foo(): [I don't need to read the body] assert x == 42 But now def statements can have non-obvious side-effects. For example: def spam_x(): global x x = "spam" x = 42 def foo(): # Some body nonlocal y from spam_x() # More body assert x == 42 I may be unduly worrying about this, but it feels to me that it lessens the overall tidiness of the language. I would much rather there was a way to initialise these "own" variables outside the body of the function. -- Arnaud From guido at python.org Wed Sep 28 00:32:43 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Sep 2011 15:32:43 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: On Tue, Sep 27, 2011 at 3:24 PM, Arnaud Delobelle wrote: > On 27 September 2011 22:23, Guido van Rossum wrote: >> On Tue, Sep 27, 2011 at 2:04 PM, Arnaud Delobelle wrote: >>> On 27 September 2011 21:38, Greg Ewing wrote: >>> [...] >>>> As I remember, the idea foundered because it was thought >>>> too confusing to have an expression that was written >>>> inside the function but evaluated outside of it. If >>>> we're seriously considering the current proposal, >>>> presumably we've gotten over that? Or is it still a >>>> valid objection? >>> >>> I have been reading this thread from the start and FWIW, it's >>> something that I have been feeling uncomfortable about. ?I value the >>> guarantee that no code in the body of a def statement is executed when >>> the def statement itself is executed. ?This proposal would break this >>> guarantee. >> >> That seems a guarantee made up specifically to prevent this proposal. > > I wouldn't do this :) ?I like new things. > >> What benefit do you see from it? > > It keeps things simple. ?To be specific, when I read code I can "fold" > (mentally or with the help of a code editor) the body of a def > statement, knowing that I won't miss anything until the function is > called. ?So the following won't raise: > > ? ?[some code] > > ? ?x = 42 > ? ?def foo(): > ? ? ? ?[I don't need to read the body] > ? ?assert x == 42 > > But now def statements can have non-obvious side-effects. ?For example: > > ? ?def spam_x(): > ? ? ? ?global x > ? ? ? ?x = "spam" > > ? ?x = 42 > ? ?def foo(): > ? ? ? ?# Some body > ? ? ? ?nonlocal y from spam_x() > ? ? ? ?# More body > ? ?assert x == 42 > > I may be unduly worrying about this, but it feels to me that it > lessens the overall tidiness of the language. ?I would much rather > there was a way to initialise these "own" variables outside the body > of the function. Yes, you have a point. Hiding side effects this way could be nasty, and this is actually the first point in favor of the default argument hack I've heard in a while. :-) C++ has special syntax to force the programmer to write certain things after the argument list but before the function body (I think only for constructors). I have seen too many constructors with a short argument list, a very short body, and a ton of code cramped in between, to like the idea very much: even though it is semantically something totally different, syntactically it would have the same awkwardness. -- --Guido van Rossum (python.org/~guido) From jeanpierreda at gmail.com Wed Sep 28 00:33:59 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 27 Sep 2011 18:33:59 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: I never liked the nontestability of nested functions, and I also don't like the nontestability of these non-global semiglobalnonlocal variables. Real globals have the benefit that a test suite can check what happens to that variable as functions are called fairly easily. Here you have to test against my_func.__closure__[i].cell_contents for some i whose value depends on the order in which you created the variables and the order in which they're added to existing nonlocals (provided that isn't an implementation detail). > It keeps things simple. To be specific, when I read code I can "fold" > (mentally or with the help of a code editor) the body of a def > statement, knowing that I won't miss anything until the function is > called. +1. Yes. > I may be unduly worrying about this, but it feels to me that it > lessens the overall tidiness of the language. I would much rather > there was a way to initialise these "own" variables outside the body > of the function. There is. Have this "nonlocal assignment" as a magical decorator, or as syntax as part of the def statement. I'm not sure to what degree Nick's proposal is about the nonlocal syntax versus the nonlocal semantics. But you can keep the semantics (fast access to variables that are shared as global state) while changing the syntax fairly radically. Devin On Tue, Sep 27, 2011 at 6:24 PM, Arnaud Delobelle wrote: > On 27 September 2011 22:23, Guido van Rossum wrote: >> On Tue, Sep 27, 2011 at 2:04 PM, Arnaud Delobelle wrote: >>> On 27 September 2011 21:38, Greg Ewing wrote: >>> [...] >>>> As I remember, the idea foundered because it was thought >>>> too confusing to have an expression that was written >>>> inside the function but evaluated outside of it. If >>>> we're seriously considering the current proposal, >>>> presumably we've gotten over that? Or is it still a >>>> valid objection? >>> >>> I have been reading this thread from the start and FWIW, it's >>> something that I have been feeling uncomfortable about. ?I value the >>> guarantee that no code in the body of a def statement is executed when >>> the def statement itself is executed. ?This proposal would break this >>> guarantee. >> >> That seems a guarantee made up specifically to prevent this proposal. > > I wouldn't do this :) ?I like new things. > >> What benefit do you see from it? > > It keeps things simple. ?To be specific, when I read code I can "fold" > (mentally or with the help of a code editor) the body of a def > statement, knowing that I won't miss anything until the function is > called. ?So the following won't raise: > > ? ?[some code] > > ? ?x = 42 > ? ?def foo(): > ? ? ? ?[I don't need to read the body] > ? ?assert x == 42 > > But now def statements can have non-obvious side-effects. ?For example: > > ? ?def spam_x(): > ? ? ? ?global x > ? ? ? ?x = "spam" > > ? ?x = 42 > ? ?def foo(): > ? ? ? ?# Some body > ? ? ? ?nonlocal y from spam_x() > ? ? ? ?# More body > ? ?assert x == 42 > > I may be unduly worrying about this, but it feels to me that it > lessens the overall tidiness of the language. ?I would much rather > there was a way to initialise these "own" variables outside the body > of the function. > > -- > Arnaud > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From ericsnowcurrently at gmail.com Wed Sep 28 00:42:19 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 27 Sep 2011 16:42:19 -0600 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Tue, Sep 27, 2011 at 4:24 PM, Eric Snow wrote: > Ultimately, it seems like we are looking for a way to do three things here: > > 1. bless a concrete function definition scope (persist across calls), > 2. allow one-time initialization of those persistent variables, > 3. (relatedly) have some expressions evaluated at definition time but > available at execution time. > > Default arguments cover some of this. ?Closures cover some of it. ?The > nonlocal-from statement covers a bunch. As do function attributes... However, the big motivator here is execution performance where it matters: in functions. > > -eric From zuo at chopin.edu.pl Wed Sep 28 01:08:09 2011 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Wed, 28 Sep 2011 01:08:09 +0200 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> Message-ID: <20110927230809.GA2882@chopin.edu.pl> Guido van Rossum dixit (2011-09-26, 19:48): > OTOH I am happy to let you all bikeshed on a better name. `deflocal`? (definition-time-bound local-scoped var...) `ownlocal`? (function's own local-scoped var...) `boundlocal`? -- that would be nice to have it somehow similar to the `nonlocal` keyword; suggesting *something different from* though *somehow similar or complementary* to the `nonlocal` mechanism. Cheers. *j From ethan at stoneleaf.us Wed Sep 28 01:11:25 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 27 Sep 2011 16:11:25 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <20110927230809.GA2882@chopin.edu.pl> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <20110927230809.GA2882@chopin.edu.pl> Message-ID: <4E82581D.9050907@stoneleaf.us> Jan Kaliszewski wrote: > Guido van Rossum dixit (2011-09-26, 19:48): > >> OTOH I am happy to let you all bikeshed on a better name. > > `deflocal`? (definition-time-bound local-scoped var...) > `ownlocal`? (function's own local-scoped var...) > `boundlocal`? I kinda like boundlocal. ~Ethan~ From bruce at leapyear.org Wed Sep 28 02:56:58 2011 From: bruce at leapyear.org (Bruce Leban) Date: Tue, 27 Sep 2011 17:56:58 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: I started to write this mail saying that we could use spellings like nonlocal.scoped, nonlocal.own, nonlocal.global and nonlocal.const but then read the message about declarations in the middle of a function having side effects at definition time. That seems like a big problem. And it made me realize it would be weird if (to use one possible bikeshed color): own x = y was not the same as own x x = y but I don't see how these can possibly be the same. You need the first case to do one time initialization and you must be able to change the value as in the second case or the feature is useless. I proposed a special decorator-like syntax ($inject) which would pull the declaration out of the function entirely (putting it at the scope where it evaluates). There wasn't much discussion other than that it wasn't feasible with current implementation. --- Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Sep 28 03:17:16 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 27 Sep 2011 21:17:16 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: On 9/27/2011 6:32 PM, Guido van Rossum wrote: > On Tue, Sep 27, 2011 at 3:24 PM, Arnaud Delobelle wrote: >> I may be unduly worrying about this, but it feels to me that it >> lessens the overall tidiness of the language. I would much rather >> there was a way to initialise these "own" variables outside the body >> of the function. > > Yes, you have a point. Hiding side effects this way could be nasty, > and this is actually the first point in favor of the default argument > hack I've heard in a while. :-) I have several times written on python-list that Python has a simple rule: header expressions (for default args) are evaluated one-time when the function is defined; body expressions are evaluated (perhaps) each time when the function is called. If people understand this and do not fight it, they are not surprised that mutating a once-defined default arg mutates it, nor that defining a function inside a loop magically causes define-time binding of names in the body. I would hate for Python to lose this simplicity. I consider it one of its positive features. -- Terry Jan Reedy From ethan at stoneleaf.us Wed Sep 28 03:26:53 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 27 Sep 2011 18:26:53 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: <4E8277DD.5060407@stoneleaf.us> Terry Reedy wrote: > On 9/27/2011 6:32 PM, Guido van Rossum wrote: >> On Tue, Sep 27, 2011 at 3:24 PM, Arnaud Delobelle >> wrote: > >>> I may be unduly worrying about this, but it feels to me that it >>> lessens the overall tidiness of the language. I would much rather >>> there was a way to initialise these "own" variables outside the body >>> of the function. >> >> Yes, you have a point. Hiding side effects this way could be nasty, >> and this is actually the first point in favor of the default argument >> hack I've heard in a while. :-) > > I have several times written on python-list that Python has a simple > rule: header expressions (for default args) are evaluated one-time when > the function is defined; body expressions are evaluated (perhaps) each > time when the function is called. If people understand this and do not > fight it, they are not surprised that mutating a once-defined default > arg mutates it, nor that defining a function inside a loop magically > causes define-time binding of names in the body. > > I would hate for Python to lose this simplicity. I consider it one of > its positive features. Simplicity is good. I don't remember who first suggested it, and I don't remember any discussion on it, but what if decorator functions grew a __prepare__ method similar to metaclasses? If the method exists it is called /before/ the function is evaluated, it returns a namespace that is then used as the basis for the function's (the interpreter can take everything it finds in there and add it as closures to the function), and folks can call it whatever they want. ;) If the method does not exist, nothing is different from what we have now. Seeing-if-we-can-have-more-exploding-heads'ly yours, ~Ethan~ From stephen at xemacs.org Wed Sep 28 04:18:59 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 28 Sep 2011 11:18:59 +0900 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E815BF0.9010807@canterbury.ac.nz> Message-ID: <87y5x9h0j0.fsf@uwakimon.sk.tsukuba.ac.jp> Paul Moore writes: > On 27 September 2011 18:03, Nick Coghlan wrote: > > If we're going to settle for a term with multiple meanings, then I > > still think 'nonlocal' is the one that makes the most sense: > > - it's already a keyword > > To be honest, I think you need to stop raising this. You're right, and > it's certainly a lot easier to accept reusing an existing keyword than > adding a new one. But you need to convince people that "nonlocal" is > right, not that it's easier. You're at risk of giving a distinct "I've > got a hammer so it must be a nail" impression here... As I read Nick, that is *precisely* the impression he *wants* to give: "if this ain't a nail in da fust place, it don't need no hammerin' nohow". From stephen at xemacs.org Wed Sep 28 04:26:17 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 28 Sep 2011 11:26:17 +0900 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E82581D.9050907@stoneleaf.us> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <20110927230809.GA2882@chopin.edu.pl> <4E82581D.9050907@stoneleaf.us> Message-ID: <87wrcth06u.fsf@uwakimon.sk.tsukuba.ac.jp> Ethan Furman writes: > I kinda like boundlocal. People coming from Lisp will find that both repetitive and redundant. From pyideas at rebertia.com Wed Sep 28 05:47:52 2011 From: pyideas at rebertia.com (Chris Rebert) Date: Tue, 27 Sep 2011 20:47:52 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <054DF9F5-625D-4205-8148-5758E04E4708@activestate.com> Message-ID: On Tue, Sep 27, 2011 at 3:23 PM, Nick Coghlan wrote: > On Sep 27, 2011 3:55 PM, "Nick Coghlan" wrote: >> On Tue, Sep 27, 2011 at 3:50 PM, Gisle Aas wrote: >>> On Sep 27, 2011, at 6:12 , Guido van Rossum wrote: >>> >>>> "our" is something in Perl, right? >>> >>> That's right, and "our" is similar to Python's "global". ?Perl went with >>> "state" as the keyword the concept described in this thread. >>> >>> ? state $foo = 1; >> >> I actually like that as a term for the concept (I've certainly used it >> in my own descriptions in this very thread), but I shudder to think >> how much code would have to change if we made 'state' a keyword :P > > Hmm...? 'local', 'statelocal', 'nonlocal', 'global', 'builtin'? I'll just toss the following variants out there. They tend to read nicely if one assumes the syntax " as ", though " = " would also work for some of them: * Emphasizing that the eval happens in the outer scope: take, grab, seize, capture * Emphasizing that the var/val persists between calls: hold, keep, stow, retain Cheers, Chris -- I <3 Thesauri. From tjreedy at udel.edu Wed Sep 28 06:12:26 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 28 Sep 2011 00:12:26 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E8277DD.5060407@stoneleaf.us> References: <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> < CAJ6cK1YKJx8VjrRepCfxJGTA5H23k243+iSb-mCcs1pforRFqg@mail.gmail.com> <4E8277DD.5060407@stoneleaf.us> Message-ID: On 9/27/2011 9:26 PM, Ethan Furman wrote: > Terry Reedy wrote: >> On 9/27/2011 6:32 PM, Guido van Rossum wrote: >>> On Tue, Sep 27, 2011 at 3:24 PM, Arnaud >>> Delobelle wrote: >> >>>> I may be unduly worrying about this, but it feels to me that it >>>> lessens the overall tidiness of the language. I would much rather >>>> there was a way to initialise these "own" variables outside the body >>>> of the function. >>> >>> Yes, you have a point. Hiding side effects this way could be nasty, >>> and this is actually the first point in favor of the default argument >>> hack I've heard in a while. :-) >> >> I have several times written on python-list that Python has a simple >> rule: header expressions (for default args) are evaluated one-time >> when the function is defined; body expressions are evaluated (perhaps) >> each time when the function is called. If people understand this and >> do not fight it, they are not surprised that mutating a once-defined >> default arg mutates it, nor that defining a function inside a loop >> magically causes define-time binding of names in the body. >> >> I would hate for Python to lose this simplicity. I consider it one of >> its positive features. > > Simplicity is good. > > I don't remember who first suggested it, and I don't remember any > discussion on it, but what if decorator functions grew a __prepare__ > method similar to metaclasses? At least three people, including Guido, have noted than a keyword-only _private parameter solves most of the problems with using default args for constants. Since the _underscore convention is already partly built into the language (import *, help(module), __mangling, __reserved__), we can have other tools like signature introspection also respect the convention. This solution *also* works for the semi-private accumulator parameter of most tail-recursive functions. Example: sum the values in a linked list (without iterators or explicit loops*: def llsum(ll, *, _total = 0): if ll: return llsum(ll[1], _total = _total + ll[0]) else: return _total The private parameter _total is constant for outside calls but gets a new value with each recursive call. It should not appear in the public signature but is used internally. The non-parameter constants proposed as a substitute for default args would not work for this case. * I am QUITE aware that this can be rewritten with while, and indeed I write the two if branches in the order I do to make it easy. def llsum(ll) total = 0): while ll: ll, total = ll[1], total + ll[0] else: return total But this is a different issue. -- Terry Jan Reedy From arnodel at gmail.com Wed Sep 28 07:23:42 2011 From: arnodel at gmail.com (Arnaud Delobelle) Date: Wed, 28 Sep 2011 06:23:42 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On 28 September 2011 01:56, Bruce Leban wrote: [...] > I proposed a special decorator-like syntax ($inject) which would pull the > declaration out of the function entirely (putting it at the scope where it > evaluates). There wasn't much discussion other than that it wasn't feasible > with current implementation. A hybrid approach would be possible, taking from both approaches (decorator and keyword): * a keyword to declare in the body of the function that a variable is of the "own" kind. * function objects would grow an __own__ attribute which is a dict-like object: - f.__own__["i"] would return the contents of the cell that the own variable "i" refers to - f.__own__["i"] = 42 would set the cell contents of the own variable "i" to 42 (or some other equivalent mechanism) * then one could create a kind of "inject" decorator. Here's an example: def setown(**kwargs): def decorator(f): for k, v in kwargs.items(): f.__own__[k] = v return decorator @setown(i=42) def foo(x): own i i += 1 return i -- Arnaud From ron3200 at gmail.com Wed Sep 28 07:48:31 2011 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 28 Sep 2011 00:48:31 -0500 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> Message-ID: <1317188911.20301.81.camel@Gutsy> On Tue, 2011-09-27 at 06:08 -0400, Nick Coghlan wrote: > On Tue, Sep 27, 2011 at 12:12 AM, Guido van Rossum > wrote: > > Somehow you triggered a thought in my head: maybe se should stop > > focusing on the syntax for a while and focus on the code we want to > be > > generated for it. If we can design the bytecode, perhaps that would > > help us come up with the right keyword. > > Heh, doing exactly that (i.e. thinking in terms of how it would *work* > and how I would *explain* those semantics to someone in terms of > existing language concepts) is where my nonlocal suggestion came from > in the first place :) > > As I see it, there are 3 possible ways to approach this feature, and > they vary mainly in terms of when the initialisation expressions get > evaluated: > 1. Prior to function definition (i.e. the same as default argument > values) > 2. After function definition but before decorators are applied > 3. After decorator application I keep going back to the point I was when I wanted to unpack **kwds into locals(). But that is only part of the problem. The other half is making the function use the decorator as it's closure, or to supply a new one. > For the recursion use cases, I'd suggest actually offering a > 'functools.rebind_closure' function that provided an officially > supported way to modify a closure reference in a function. You could > then write a function with an early binding reference to itself as > follows: +1, I think this is the right approach. But it needs to accept a dictionary because we can't unpack an argument list into locals, and we don't want to specialize (rewrite) the decorator for each individual case! We want to be able to write a *generalized* decorator. > @example > def global_accumulator(x): > nonlocal tally from 0 # Insert preferred spelling here > tally += x > return tally # A generalized decorator for adding closure values. def add_closure(f): """ A decorator to add explicit closure values. """ def _(**kwds): re_define(f, closure=kwds) #As if it was defined here! return f return _ @add_closure(tally=0): def global_accumulator(x): nonlocal tally #Nothing new here. ;-) tally += x return tally A re_define function could be extended to make other adjustments as well. For example a 'defaults=" keyword, that add to locals instead of closures. Those would be set back to their default value on each call. It may be better for a rebind or redefine to return a new function. If we add a way to update locals from kwds, Then the rebind or redefine function can be simpler. def add_closure(f): """ A decorator to add explicit closure values. """ def _(**kwds): replace_locals(kwds) return redefine(f) # as if it was defined here. return _ # Recursive example that works even if the original function is renamed. def set_this(f): """ A decorator that gives a function a visible name "this". """ f.rebind_closure(this=f) @set_this def factorial(x): if x = 0: return 1 return x * this(x-1) Nonlocal isn't needed as we don't write to "this". The rebind_closure function would be able to take '**kwds' as it's argument, so it could be used in the accumulator example as well. No changes to the nonlocal statement are needed if we can rebind, or re_define functions. These complement nonlocal and allow doing things that are not possible (or easy) at this time. Mainly a way to write nice *GENERALIZED* decorators to address the issues we've been discussing. Cheers, Ron From p.f.moore at gmail.com Wed Sep 28 09:10:04 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 28 Sep 2011 08:10:04 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On 28 September 2011 01:56, Bruce Leban wrote: > And it made > me realize it would be weird if (to use one possible bikeshed color): > ? ? own x = y > was not the same as > ? ? own x > ? ? x = y > but I don't see how these can possibly be the same. You need the first case > to do one time initialization and you must be able to change the value as in > the second case or the feature is useless. Equally to the point, if a function has own x = 1 x = 1 it is *not* valid to remove the second line, as it runs at a different time (runtime rather than define time). That is very weird. Between these points and Arnaud Delobelle's point that code inside a function should do nothing when the def itself is executed, I'm getting more convinced that objects with persistent local scope should be introduced *outside* the function body. That either validates the default-argument hack as a valid response to a specific requirement, or suggests some syntax added to the function definition line. Or of course, just go with a normal (nested function) closure. Paul. From jh at improva.dk Wed Sep 28 11:27:45 2011 From: jh at improva.dk (Jacob Holm) Date: Wed, 28 Sep 2011 11:27:45 +0200 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: <4E82E891.6030300@improva.dk> On 2011-09-28 07:23, Arnaud Delobelle wrote: > A hybrid approach would be possible, taking from both approaches > (decorator and keyword): > > * a keyword to declare in the body of the function that a variable is > of the "own" kind. > * function objects would grow an __own__ attribute which is a dict-like object: > - f.__own__["i"] would return the contents of the cell that the > own variable "i" refers to > - f.__own__["i"] = 42 would set the cell contents of the own > variable "i" to 42 > (or some other equivalent mechanism) > * then one could create a kind of "inject" decorator. > > Here's an example: > > def setown(**kwargs): > def decorator(f): > for k, v in kwargs.items(): > f.__own__[k] = v > return decorator > > @setown(i=42) > def foo(x): > own i > i += 1 > return i > I like this idea, but I think we should at least consider using "nonlocal" instead of "own" as the keyword (and __nonlocal__ instead of __own__ for the dict-like object). Pros: - No new keyword needed, just a change in how "nonlocal x" is treated when there is no 'x' variable in the defining scope. (Today this is a SyntaxError). Cons: - No way to define an "own" variable with the same name as a variable in the defining scope. - Ambiguity as to exactly what the lifetime of the nonlocal is. (If an outer function with no "x" variable in scope defines two inner functions that both use "nonlocal x", do they see the same "x"?) Ok, so I don't actually like repurposing nonlocal for this. +1 to your "own" suggestion. - Jacob From ncoghlan at gmail.com Wed Sep 28 11:59:17 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Sep 2011 05:59:17 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Wed, Sep 28, 2011 at 3:10 AM, Paul Moore wrote: > Between these points and Arnaud Delobelle's point that code inside a > function should do nothing when the def itself is executed, I'm > getting more convinced that objects with persistent local scope should > be introduced *outside* the function body. > > That either validates the default-argument hack as a valid response to > a specific requirement, or suggests some syntax added to the function > definition line. Or of course, just go with a normal (nested function) > closure. I'll point out that there's one thing the default argument hack *doesn't* do that is relevant to the discussion: because it doesn't actually move the name into a separate scope (only the value), it has no effect on rebinding. It's a lot like closures in the days before nonlocal declarations: def accumulator(): def incr(x, *, _tally=0): _tally += x return _tally return incr That doesn't actually work, since _tally is an ordinary local that gets reinitialised to zero on each invocation. The only syntax suggested so far that meets the criteria of being both in the function header and creating a separate namespace for the namebindings is the one I put forward in Ron's previous thread (I've excluded the 'after-**' syntax here, since this thread has convinced me it would be crazy to use something like that when the name binding semantics are so different from those of a default argument): def accumulator(): def incr(x) [tally=0]: tally += x return tally return incr As the rough equivalent of today's: def accumulator(): tally = 0 def incr(x): nonlocal tally tally += x return tally return incr The objections to that proposal that I recall were: 1. Can't look up [] in the help index (Agreed, but you can look up 'def' which is where these function state variables would be documented) 2. Noise in the header line (Agreed, but no worse than the default argument hack) 3. The behaviour is very different from a list (Agreed, but function parameters and arguments are very different from tuples and people can cope with that) 4. It should be in the function body (I think Arnaud and Paul have raised some very good points in this thread for why it *shouldn't* be in the function body) As far as the 'hidden state' problem goes, I'd prefer to address that by providing better utilities in functools for accessing closure state (and perhaps even the current internal state of suspended generators). As I said in the previous thread, the "algorithm with shared state" model represented by closures (and, with a slightly different spin on the concept, generators) and the "data with behaviour" model represented by object-oriented programming are very different ways of framing a problem and forcing people to switch models because of internal language issues unrelated to the real world problem they're trying to describe isn't a good thing. I agree closures have testability problems, but I don't think telling people "avoid closures because they have testability problems" is the right answer to that. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Sep 28 12:05:26 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Sep 2011 06:05:26 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Wed, Sep 28, 2011 at 5:59 AM, Nick Coghlan wrote: > As far as the 'hidden state' problem goes, I'd prefer to address that > by providing better utilities in functools for accessing closure state > (and perhaps even the current internal state of suspended generators). Oops, s/functools/inspect/. (The original version of that sentence suggested read/write access to the closure state, so functools made more sense, but inspect is a better choice for read-only access) Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From cmjohnson.mailinglist at gmail.com Wed Sep 28 12:07:55 2011 From: cmjohnson.mailinglist at gmail.com (Carl Matthew Johnson) Date: Wed, 28 Sep 2011 00:07:55 -1000 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <20110927230809.GA2882@chopin.edu.pl> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <20110927230809.GA2882@chopin.edu.pl> Message-ID: <4B0AD508-2513-4E13-9BC6-EE241368491F@gmail.com> On Sep 27, 2011, at 1:08 PM, Jan Kaliszewski wrote: > Guido van Rossum dixit (2011-09-26, 19:48): > >> OTOH I am happy to let you all bikeshed on a better name. > > `deflocal`? (definition-time-bound local-scoped var...) > `ownlocal`? (function's own local-scoped var...) > `boundlocal`? I think ``rebind expression as name`` reads fine. Not sure if "rebind" exactly gives the right impression though. From cmjohnson.mailinglist at gmail.com Wed Sep 28 12:14:13 2011 From: cmjohnson.mailinglist at gmail.com (Carl Matthew Johnson) Date: Wed, 28 Sep 2011 00:14:13 -1000 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: <22C523DC-361A-4106-B21F-222D98CBC66E@gmail.com> On Sep 27, 2011, at 11:59 PM, Nick Coghlan wrote: > > def accumulator(): > def incr(x) [tally=0]: > tally += x > return tally > return incr Brackets feel too much like line noise? A keyword? def incr(x) and rebind 0 as tally: ? def incr(x) with static tally = 0: ? Just tossing out ideas? From ncoghlan at gmail.com Wed Sep 28 12:15:23 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Sep 2011 06:15:23 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <87y5x9h0j0.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4E802202.4080009@canterbury.ac.nz> <4E815BF0.9010807@canterbury.ac.nz> <87y5x9h0j0.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Sep 27, 2011 at 10:18 PM, Stephen J. Turnbull wrote: > Paul Moore writes: > > ?> On 27 September 2011 18:03, Nick Coghlan wrote: > ?> > If we're going to settle for a term with multiple meanings, then I > ?> > still think 'nonlocal' is the one that makes the most sense: > ?> > - it's already a keyword > ?> > ?> To be honest, I think you need to stop raising this. You're right, and > ?> it's certainly a lot easier to accept reusing an existing keyword than > ?> adding a new one. But you need to convince people that "nonlocal" is > ?> right, not that it's easier. You're at risk of giving a distinct "I've > ?> got a hammer so it must be a nail" impression here... > > As I read Nick, that is *precisely* the impression he *wants* to give: > "if this ain't a nail in da fust place, it don't need no hammerin' > nohow". Indeed, although there was also a teensy bit of hyperbole involved :) Arnaud and Paul's examples have persuaded me that embedding this inside the function isn't a good idea, though: x = 42 def f(): # Folded by editor (or even just mentally) assert x == 42 # May fail! def f(): statelocal x = 1 x =1 We'd been considering the latter as arguably tolerable all the way through, but the combination with making the function body relevant when understanding the effect of def statements on the scope that contains them was enough to tip the balance for me. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From p.f.moore at gmail.com Wed Sep 28 12:26:00 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 28 Sep 2011 11:26:00 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On 28 September 2011 08:10, Paul Moore wrote: > That either validates the default-argument hack as a valid response to > a specific requirement, or suggests some syntax added to the function > definition line. Or of course, just go with a normal (nested function) > closure. Taking a step back from all this, using an explicit closure to implement the counter example being used throughout this thread looks something like this: >>> # I thought apply still existed in some module. Never mind, can't find it so I'll reimplement a quick version here... >>> def apply(f): return f() ... >>> @apply ... def counter(): ... n = 1 ... def inner(): ... nonlocal n ... print(n) ... n += 1 ... return inner ... >>> counter() 1 >>> counter() 2 >>> counter() 3 To be honest, that doesn't actually look that bad. Explicit is better than implicit and all that, and the boilerplate doesn't really bother me that much. The real annoying boilerplate here is the "def dummy_name()...return dummy_name" bit. But fixing that leads us directly back to the perennial discussion on anonymous multiline functions... Paul. From ncoghlan at gmail.com Wed Sep 28 12:27:43 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Sep 2011 06:27:43 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <22C523DC-361A-4106-B21F-222D98CBC66E@gmail.com> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> <22C523DC-361A-4106-B21F-222D98CBC66E@gmail.com> Message-ID: On Wed, Sep 28, 2011 at 6:14 AM, Carl Matthew Johnson wrote: > > On Sep 27, 2011, at 11:59 PM, Nick Coghlan wrote: >> >> ? ?def accumulator(): >> ? ? ? ?def incr(x) [tally=0]: >> ? ? ? ? ? ?tally += x >> ? ? ? ? ? ?return tally >> ? ? ? ?return incr > > > Brackets feel too much like line noise? A keyword? Hah, def statements are already line noise if you use the full syntax :) @what def theheck(do:all, these=1, *different, symbols, **mean) -> seriously: """And this is special, too!""" I think the combination of position with the different bracket style is enough to separate them out from the parameter list and their position in the function header eliminates much of the confusion associated with the various syntax proposals for state variables inside the function body. I actually think it also serves as a mild deterrent to abuse - code that overuses function state variables is likely reaching the point where it needs to be rewritten as a class, and this would be reflected in the unwieldiness of the function definition. Cheers, Nick. [1] http://docs.python.org/py3k/reference/compound_stmts.html#grammar-token-funcdef -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From cmjohnson.mailinglist at gmail.com Wed Sep 28 12:36:25 2011 From: cmjohnson.mailinglist at gmail.com (Carl Matthew Johnson) Date: Wed, 28 Sep 2011 00:36:25 -1000 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: <48C98E5B-E618-44FB-A60F-A0A2F9FA20B8@gmail.com> On Sep 28, 2011, at 12:26 AM, Paul Moore wrote: > The real annoying boilerplate here is the "def dummy_name()...return > dummy_name" bit. But fixing that leads us directly back to the > perennial discussion on anonymous multiline functions? Truly, they are the "Godwin's Law" of Python-ideas? From cmjohnson.mailinglist at gmail.com Wed Sep 28 12:39:31 2011 From: cmjohnson.mailinglist at gmail.com (Carl Matthew Johnson) Date: Wed, 28 Sep 2011 00:39:31 -1000 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> <22C523DC-361A-4106-B21F-222D98CBC66E@gmail.com> Message-ID: On Sep 28, 2011, at 12:27 AM, Nick Coghlan wrote: > Hah, def statements are already line noise if you use the full syntax :) > > @what > def theheck(do:all, these=1, *different, symbols, **mean) -> seriously: > """And this is special, too!""" Slightly OT, but does anyone else feel like function annotations are a bit of a failed experiment? Or have you guys seen them in a real project, not just a demonstration? I find the idea reasonable enough, but I've never thought to use them for anything serious. I wonder if we need to learn any lessons from that experience? The counterpoint here is that the defaults hack is something one often runs across, so a replacement is likely to have decent pickup. From steve at pearwood.info Wed Sep 28 12:42:33 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 28 Sep 2011 20:42:33 +1000 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: <4E82FA19.8080806@pearwood.info> Paul Moore wrote: > Equally to the point, if a function has > > own x = 1 > x = 1 > > it is *not* valid to remove the second line, as it runs at a different > time (runtime rather than define time). That is very weird. def f(x=1): x = 1 has the same behaviour. I don't think either case is weird. > Between these points and Arnaud Delobelle's point that code inside a > function should do nothing when the def itself is executed, I'm > getting more convinced that objects with persistent local scope should > be introduced *outside* the function body. I don't think having code inside a function execute is any worse than having code inside a class execute. class K: print("this is executed at definition time") def f(x=print("this is also executed at definition time")): own y=print("and so is this") To say nothing of decorators. -- Steven From ncoghlan at gmail.com Wed Sep 28 12:45:08 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Sep 2011 06:45:08 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Wed, Sep 28, 2011 at 6:26 AM, Paul Moore wrote: > On 28 September 2011 08:10, Paul Moore wrote: >> That either validates the default-argument hack as a valid response to >> a specific requirement, or suggests some syntax added to the function >> definition line. Or of course, just go with a normal (nested function) >> closure. > > Taking a step back from all this, using an explicit closure to > implement the counter example being used throughout this thread looks > something like this: > >>>> # I thought apply still existed in some module. Never mind, can't find it so I'll reimplement a quick version here... >>>> def apply(f): return f() > ... >>>> @apply > ... def counter(): > ... ? ? n = 1 > ... ? ? def inner(): > ... ? ? ? ? nonlocal n > ... ? ? ? ? print(n) > ... ? ? ? ? n += 1 > ... ? ? return inner > ... >>>> counter() > 1 >>>> counter() > 2 >>>> counter() > 3 > > To be honest, that doesn't actually look that bad. Explicit is better > than implicit and all that, and the boilerplate doesn't really bother > me that much. > > The real annoying boilerplate here is the "def dummy_name()...return > dummy_name" bit. But fixing that leads us directly back to the > perennial discussion on anonymous multiline functions... Actually, there are some additional aspects that annoy me: - the repetition of the variable name 'n' - the scope of the variable name is wrong (since we only need it in the inner function) - indentation matters (cf try/except/finally) - hidden signature for the actual function - hoops to jump through to get decent introspection values (e.g. __name__) I find the following 5 line toy example: from threading import Lock def global_counter() [n=1, lock=Lock()]: with lock: print(n) n += 1 Far more readable than the 10-line closure equivalent: from threading import Lock @apply # I sometimes wonder if we should bring this back in functools... def global_counter(): n = 1 local = Lock() def global_counter(): with lock: print(n) n += 1 return global_counter Or the 10-line 7-self class equivalent: from threading import Lock @apply class global_counter: def __init__(self): self.n = 1 self.lock = Lock() def __call__(self): with self.lock: print(self.n) self.n += 1 Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From greg.ewing at canterbury.ac.nz Wed Sep 28 12:54:24 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 28 Sep 2011 23:54:24 +1300 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E815BF0.9010807@canterbury.ac.nz> Message-ID: <4E82FCE0.7040702@canterbury.ac.nz> Paul Moore wrote: > Nonlocal is associated (by its > position) with the VAR, not with the EXPR. If anything, it says > "assign this local EXPR to the nonlocal VAR" which implies it's > equivalent to "nonlocal VAR; VAR = EXPR". Something that might be slightly more suggestive of Nick's meaning is i = nonlocal i but then there's no indication that i isn't just an ordinary local being assigned at call time. -- Greg From arnodel at gmail.com Wed Sep 28 13:19:44 2011 From: arnodel at gmail.com (Arnaud Delobelle) Date: Wed, 28 Sep 2011 12:19:44 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E82FA19.8080806@pearwood.info> References: <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> <4E82FA19.8080806@pearwood.info> Message-ID: On 28 September 2011 11:42, Steven D'Aprano wrote: > Paul Moore wrote: >> Between these points and Arnaud Delobelle's point that code inside a >> function should do nothing when the def itself is executed, I'm >> getting more convinced that objects with persistent local scope should >> be introduced *outside* the function body. > > I don't think having code inside a function execute is any worse than having > code inside a class execute. You can't compare the two. When a class statement is executed, the *whole* of its body is executed; when a def statement is executed, *none* of its body is. -- Arnaud From greg.ewing at canterbury.ac.nz Wed Sep 28 13:21:18 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 29 Sep 2011 00:21:18 +1300 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: <4E83032E.4020502@canterbury.ac.nz> Paul Moore wrote: > There was a time when Lisps > were described as Lisp-1 or Lisp-2 depending on their position on > scope. Actually that distinction is whether there is a separate namespace for functions or not, see http://en.wikipedia.org/wiki/Lisp-1_vs._Lisp-2#The_function_namespace -- Greg From p.f.moore at gmail.com Wed Sep 28 14:11:06 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 28 Sep 2011 13:11:06 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E83032E.4020502@canterbury.ac.nz> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> <4E83032E.4020502@canterbury.ac.nz> Message-ID: On 28 September 2011 12:21, Greg Ewing wrote: > Paul Moore wrote: >> >> There was a time when Lisps >> were described as Lisp-1 or Lisp-2 depending on their position on >> scope. > > Actually that distinction is whether there is a separate > namespace for functions or not, see > > http://en.wikipedia.org/wiki/Lisp-1_vs._Lisp-2#The_function_namespace Pah. Fading memory. Dynamic vs lexical scoping was in there somewhere, though, just not by the names I thought :-) Paul From p.f.moore at gmail.com Wed Sep 28 14:15:29 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 28 Sep 2011 13:15:29 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On 28 September 2011 11:45, Nick Coghlan wrote: > Actually, there are some additional aspects that annoy me: > - the repetition of the variable name 'n' > - the scope of the variable name is wrong (since we only need it in > the inner function) > - indentation matters (cf try/except/finally) > - hidden signature for the actual function > - hoops to jump through to get decent introspection values (e.g. __name__) Fair points, all (except the scope one, the fact that the scope extends to cover a couple of lines of boilerplate doesn't really bother me, but I concede that you're a purist in these matters :-)) > I find the following 5 line toy example: > > from threading import Lock > > def global_counter() [n=1, lock=Lock()]: > ? ?with lock: > ? ? ? ?print(n) > ? ? ? ?n += 1 > > Far more readable than the 10-line closure equivalent: [...] Agreed entirely. If we're back to talking about line-noise syntax rather than using "nonlocal" I'm with you all the way (including the original bit about it still being a large change for a relatively small benefit...) Paul. From steve at pearwood.info Wed Sep 28 15:48:40 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 28 Sep 2011 23:48:40 +1000 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4B0AD508-2513-4E13-9BC6-EE241368491F@gmail.com> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <20110927230809.GA2882@chopin.edu.pl> <4B0AD508-2513-4E13-9BC6-EE241368491F@gmail.com> Message-ID: <4E8325B8.4050806@pearwood.info> Carl Matthew Johnson wrote: > I think ``rebind expression as name`` reads fine. Not sure if "rebind" exactly gives the right impression though. Why "rebind"? The expression hasn't been bound once yet, so you can't REbind it. Not that I like ``bind expression as name`` any better. -- Steven From ethan at stoneleaf.us Wed Sep 28 16:25:01 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 28 Sep 2011 07:25:01 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4B0AD508-2513-4E13-9BC6-EE241368491F@gmail.com> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <20110927230809.GA2882@chopin.edu.pl> <4B0AD508-2513-4E13-9BC6-EE241368491F@gmail.com> Message-ID: <4E832E3D.5020908@stoneleaf.us> Carl Matthew Johnson wrote: > On Sep 27, 2011, at 1:08 PM, Jan Kaliszewski wrote: > >> Guido van Rossum dixit (2011-09-26, 19:48): >> >>> OTOH I am happy to let you all bikeshed on a better name. >> `deflocal`? (definition-time-bound local-scoped var...) >> `ownlocal`? (function's own local-scoped var...) >> `boundlocal`? > > I think ``rebind expression as name`` reads fine. Not sure if "rebind" exactly gives the right impression though. I have to retract my support for 'boundlocal', as well as phrase including bind or bound -- nearly every assignment is a binding, so why single out this particular one with the name? ~Ethan~ From guido at python.org Wed Sep 28 17:38:04 2011 From: guido at python.org (Guido van Rossum) Date: Wed, 28 Sep 2011 08:38:04 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Wed, Sep 28, 2011 at 3:45 AM, Nick Coghlan wrote: > Actually, there are some additional aspects that annoy me: > - the repetition of the variable name 'n' > - the scope of the variable name is wrong (since we only need it in > the inner function) > - indentation matters (cf try/except/finally) > - hidden signature for the actual function > - hoops to jump through to get decent introspection values (e.g. __name__) > > I find the following 5 line toy example: > > from threading import Lock > > def global_counter() [n=1, lock=Lock()]: > ? ?with lock: > ? ? ? ?print(n) > ? ? ? ?n += 1 Hm, this syntax is *too* concise. It's lack of keywords makes it open to misinterpretation. > Far more readable than the 10-line closure equivalent: > > from threading import Lock > > @apply # I sometimes wonder if we should bring this back in functools... > def global_counter(): > ? ?n = 1 > ? ?local = Lock() > ? ?def global_counter(): > ? ? ? ?with lock: > ? ? ? ? ? ?print(n) > ? ? ? ? ? ?n += 1 > ? ?return global_counter True, the outer function completely obscures what's going on. > Or the 10-line 7-self class equivalent: > > from threading import Lock > > @apply > class global_counter: > ? ?def __init__(self): > ? ? ? ?self.n = 1 > ? ? ? ?self.lock = Lock() > > ? ?def __call__(self): > ? ? ? ?with self.lock: > ? ? ? ? ? ?print(self.n) > ? ? ? ? ? ?self.n += 1 Yeah, anything involving __call__ is hard to figure out. So after reading some previous messages, I almost wish we could implement the "decorator + nonlocal statement" proposal: @init(n=1, lock=Lock()) def global_counter(): nonlocal n, lock with lock: print(n) n += 1 Alas, the problem is that you can't actually implement such a decorator, not even using an extension module, because the compiler, upon seeing the "nonlocal n, lock", ignoring the decorator (since it should be applied at run time, not at compile time), will complain that there isn't actually an intermediary outer scope defining either n or lock. Some ways out of this: - let the compiler special-case a certain built-in decorator name at compile-time (unorthodox, not impossible) - different syntax, e.g. @[n=1, lock=Lock()] or @in(n=1, lock=Lock()) or even in n=1, lock=Lock() (all followed by "def global_counter() etc.") Of course once there's different syntax, the nonlocal declaration in the function is redundant. And clearly I'm back-peddling. :-) -- --Guido van Rossum (python.org/~guido) From p.f.moore at gmail.com Wed Sep 28 17:58:24 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 28 Sep 2011 16:58:24 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On 28 September 2011 16:38, Guido van Rossum wrote: > Of course once there's different syntax, the nonlocal declaration in > the function is redundant. And clearly I'm back-peddling. :-) If we're back to syntax proposals on the def statement, how about def fn() with i=1, lock=Lock(): whatever ? This is basically another bikeshed to paint, though... Paul. From python at mrabarnett.plus.com Wed Sep 28 18:24:47 2011 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 28 Sep 2011 17:24:47 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: <4E834A4F.3030702@mrabarnett.plus.com> On 28/09/2011 16:58, Paul Moore wrote: > On 28 September 2011 16:38, Guido van Rossum wrote: >> Of course once there's different syntax, the nonlocal declaration in >> the function is redundant. And clearly I'm back-peddling. :-) > > If we're back to syntax proposals on the def statement, how about > > def fn() with i=1, lock=Lock(): > whatever > > ? This is basically another bikeshed to paint, though... > I'd already arrived at the same solution! :-) The complaint about putting "static" or "own" in the body of the function is that it's not clear that it's evaluated at definition time. The complaint about putting it outside the function is that it's not clear that the name is visible only inside the function (and it's also harder for the compiler to recognise it and optimise for it). The only place where the expression can be evaluated at definition time and bound to a name which is visible only inside the function is in the function's header. We don't want it in the parameter list, therefore it can go after the parameter list. From jeanpierreda at gmail.com Wed Sep 28 18:39:14 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 28 Sep 2011 12:39:14 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: > @init(n=1, lock=Lock()) > def global_counter(): > nonlocal n, lock > with lock: > print(n) > n += 1 This use-case is no good. Consider: _n = 1; _lock = Lock() def global_counter(): global _n with _lock: print(n) n += 1 Devin On Wed, Sep 28, 2011 at 11:38 AM, Guido van Rossum wrote: > On Wed, Sep 28, 2011 at 3:45 AM, Nick Coghlan wrote: >> Actually, there are some additional aspects that annoy me: >> - the repetition of the variable name 'n' >> - the scope of the variable name is wrong (since we only need it in >> the inner function) >> - indentation matters (cf try/except/finally) >> - hidden signature for the actual function >> - hoops to jump through to get decent introspection values (e.g. __name__) >> >> I find the following 5 line toy example: >> >> from threading import Lock >> >> def global_counter() [n=1, lock=Lock()]: >> ? ?with lock: >> ? ? ? ?print(n) >> ? ? ? ?n += 1 > > Hm, this syntax is *too* concise. It's lack of keywords makes it open > to misinterpretation. > >> Far more readable than the 10-line closure equivalent: >> >> from threading import Lock >> >> @apply # I sometimes wonder if we should bring this back in functools... >> def global_counter(): >> ? ?n = 1 >> ? ?local = Lock() >> ? ?def global_counter(): >> ? ? ? ?with lock: >> ? ? ? ? ? ?print(n) >> ? ? ? ? ? ?n += 1 >> ? ?return global_counter > > True, the outer function completely obscures what's going on. > >> Or the 10-line 7-self class equivalent: >> >> from threading import Lock >> >> @apply >> class global_counter: >> ? ?def __init__(self): >> ? ? ? ?self.n = 1 >> ? ? ? ?self.lock = Lock() >> >> ? ?def __call__(self): >> ? ? ? ?with self.lock: >> ? ? ? ? ? ?print(self.n) >> ? ? ? ? ? ?self.n += 1 > > Yeah, anything involving __call__ is hard to figure out. > > So after reading some previous messages, I almost wish we could > implement the "decorator + nonlocal statement" proposal: > > @init(n=1, lock=Lock()) > def global_counter(): > ?nonlocal n, lock > ?with lock: > ? ?print(n) > ? ?n += 1 > > Alas, the problem is that you can't actually implement such a > decorator, not even using an extension module, because the compiler, > upon seeing the "nonlocal n, lock", ignoring the decorator (since it > should be applied at run time, not at compile time), will complain > that there isn't actually an intermediary outer scope defining either > n or lock. > > Some ways out of this: > > - let the compiler special-case a certain built-in decorator name at > compile-time (unorthodox, not impossible) > > - different syntax, e.g. > ?@[n=1, lock=Lock()] > or > ?@in(n=1, lock=Lock()) > or even > ?in n=1, lock=Lock() > > (all followed by "def global_counter() etc.") > > Of course once there's different syntax, the nonlocal declaration in > the function is redundant. And clearly I'm back-peddling. :-) > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From ubershmekel at gmail.com Wed Sep 28 18:49:24 2011 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Wed, 28 Sep 2011 12:49:24 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Sep 28, 2011 8:58 AM, "Paul Moore" wrote: > > On 28 September 2011 16:38, Guido van Rossum wrote: > > Of course once there's different syntax, the nonlocal declaration in > > the function is redundant. And clearly I'm back-peddling. :-) > > If we're back to syntax proposals on the def statement, how about > > def fn() with i=1, lock=Lock(): > whatever > > ? This is basically another bikeshed to paint, though... > > Paul. > I also had the same idea with: def fn() with i=1, lock=Lock(): Whatever So I guess it's not unobvious. Though "with" is used for something different unless we disallow def fn() with NAME = EXPR: and instead use def fn() with CONTEXT as NAME: And the existing enter/exit mechanism for initialization. Though I agree with the protesters that this construct can be an attractive nuisance or an "anti pattern" as Greg Ewing said. Though I haven't yet been able to articulate in which cases. --Yuval On Sep 28, 2011 8:58 AM, "Paul Moore" wrote: > On 28 September 2011 16:38, Guido van Rossum wrote: >> Of course once there's different syntax, the nonlocal declaration in >> the function is redundant. And clearly I'm back-peddling. :-) > > If we're back to syntax proposals on the def statement, how about > > def fn() with i=1, lock=Lock(): > whatever > > ? This is basically another bikeshed to paint, though... > > Paul. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Wed Sep 28 20:08:28 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 28 Sep 2011 12:08:28 -0600 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Wed, Sep 28, 2011 at 9:38 AM, Guido van Rossum wrote: > On Wed, Sep 28, 2011 at 3:45 AM, Nick Coghlan wrote: >> Actually, there are some additional aspects that annoy me: >> - the repetition of the variable name 'n' >> - the scope of the variable name is wrong (since we only need it in >> the inner function) >> - indentation matters (cf try/except/finally) >> - hidden signature for the actual function >> - hoops to jump through to get decent introspection values (e.g. __name__) >> >> I find the following 5 line toy example: >> >> from threading import Lock >> >> def global_counter() [n=1, lock=Lock()]: >> ? ?with lock: >> ? ? ? ?print(n) >> ? ? ? ?n += 1 > > Hm, this syntax is *too* concise. It's lack of keywords makes it open > to misinterpretation. > >> Far more readable than the 10-line closure equivalent: >> >> from threading import Lock >> >> @apply # I sometimes wonder if we should bring this back in functools... >> def global_counter(): >> ? ?n = 1 >> ? ?local = Lock() >> ? ?def global_counter(): >> ? ? ? ?with lock: >> ? ? ? ? ? ?print(n) >> ? ? ? ? ? ?n += 1 >> ? ?return global_counter > > True, the outer function completely obscures what's going on. > >> Or the 10-line 7-self class equivalent: >> >> from threading import Lock >> >> @apply >> class global_counter: >> ? ?def __init__(self): >> ? ? ? ?self.n = 1 >> ? ? ? ?self.lock = Lock() >> >> ? ?def __call__(self): >> ? ? ? ?with self.lock: >> ? ? ? ? ? ?print(self.n) >> ? ? ? ? ? ?self.n += 1 > > Yeah, anything involving __call__ is hard to figure out. > > So after reading some previous messages, I almost wish we could > implement the "decorator + nonlocal statement" proposal: > > @init(n=1, lock=Lock()) > def global_counter(): > ?nonlocal n, lock > ?with lock: > ? ?print(n) > ? ?n += 1 > > Alas, the problem is that you can't actually implement such a > decorator, not even using an extension module, because the compiler, > upon seeing the "nonlocal n, lock", ignoring the decorator (since it > should be applied at run time, not at compile time), will complain > that there isn't actually an intermediary outer scope defining either > n or lock. > > Some ways out of this: > > - let the compiler special-case a certain built-in decorator name at > compile-time (unorthodox, not impossible) > > - different syntax, e.g. > ?@[n=1, lock=Lock()] > or > ?@in(n=1, lock=Lock()) > or even > ?in n=1, lock=Lock() > > (all followed by "def global_counter() etc.") > > Of course once there's different syntax, the nonlocal declaration in > the function is redundant. And clearly I'm back-peddling. :-) I like the idea of a decorator approach too, though just normal decorators. Here's another alternative. It could require exposing function internals a little (impacting the language definition?). There are two distinct outcomes we could pursue: def-time values (like default arguments), and def-time variables (static-like). The Def-time Value Approach ------------------------------------ For def-time values, the values would have to be stored in a function attribute similar to __defaults__ and __kwdefaults__. Let's call it __spam__. A keyword (not "nonlocal" ) in the function body would indicate that the name should be initialized with each call to the appropriate value in __spam__. Then we would have a (builtin?) decorator factory that would update values in __spam__: # def init_deftime(**kwargs): def decorator(f): for name in kwargs: index = figure_out_index(f, name) f.__spam__[index] = kwargs[name] #functools.update_spam(f, name, kwargs[name]) return f return decorator @init_deftime(x=5): def f(): some_keyword x print(x) # prints 5 # Currently using default arguments: def f(x=5): print(x) # As suggested already, the keyword simply indicates to the compiler that x should be a local and it should be initialized from __spam__. Also, if the decorator is not used to initialize a name marked by some_keyword, then it would either just use some default value or raise some exception (just after all decorators have been applied or at call time). The Def-time Variable Approach --------------------------------------- Perhaps we want static-like variables in functions instead. Unless I misread the more recent posts, such variables are a no-go. Just in case, I'll summarize a decorator solution. Basically, all we do is indicate through a keyword that a name will be found in the "function definition scope". Then closure happens like normal. The trick is to reuse nonlocal and add the function definition scope as the implicit default scope. Currently you get a syntax error when nonlocal refers to a name that is not found by the compiler in any enclosing function scope. This would change that. The decorator would be the mechanism for initializing the variable: # def init_deftime(**kwargs): def decorator(f): for name in kwargs: functools.update_closure(f.__closure__, name, kwargs[name]) return f return decorator @init_deftime(x=5): def f(): nonlocal x print(x) # prints 5 # Again, if the nonlocal name is never initialized (and not bound-to in the body), either a default would have to be used or an exception raised. Wrap-up ---------- So there's a decorator approach. For me it's a toss-up between decorators and the bracket syntax in the signature that Nick suggested. I find the deftime-value-in-the-body proposals not nearly as clear. I haven't had much use for static-like variables so I prefer def-time value approaches. Of course, the simplest solution is to do like Terry (et al.) suggested and just encourage the use of "private" default args. -eric > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From arnodel at gmail.com Wed Sep 28 21:38:27 2011 From: arnodel at gmail.com (Arnaud Delobelle) Date: Wed, 28 Sep 2011 20:38:27 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On 28 September 2011 19:08, Eric Snow wrote: [...] > The Def-time Value Approach > ------------------------------------ > > For def-time values, the values would have to be stored in a function > attribute similar to __defaults__ and __kwdefaults__. ?Let's call it > __spam__. ?A keyword (not "nonlocal" ) in the function body > would indicate that the name should be initialized with each call to > the appropriate value in __spam__. ?Then we would have a (builtin?) > decorator factory that would update values in __spam__: > > # > > def init_deftime(**kwargs): > ? ?def decorator(f): > ? ? ? ?for name in kwargs: > ? ? ? ? ? ?index = figure_out_index(f, name) > ? ? ? ? ? ?f.__spam__[index] = kwargs[name] > ? ? ? ? ? ?#functools.update_spam(f, name, kwargs[name]) > ? ? ? ?return f > ? ?return decorator > > @init_deftime(x=5): > def f(): > ? ?some_keyword x > ? ?print(x) Mmh. This looks exactly like one I posted earlier on: http://mail.python.org/pipermail/python-ideas/2011-September/011892.html -- Arnaud From ericsnowcurrently at gmail.com Wed Sep 28 22:30:19 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 28 Sep 2011 14:30:19 -0600 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Wed, Sep 28, 2011 at 1:38 PM, Arnaud Delobelle wrote: > On 28 September 2011 19:08, Eric Snow wrote: > [...] >> The Def-time Value Approach >> ------------------------------------ >> >> For def-time values, the values would have to be stored in a function >> attribute similar to __defaults__ and __kwdefaults__. ?Let's call it >> __spam__. ?A keyword (not "nonlocal" ) in the function body >> would indicate that the name should be initialized with each call to >> the appropriate value in __spam__. ?Then we would have a (builtin?) >> decorator factory that would update values in __spam__: >> >> # >> >> def init_deftime(**kwargs): >> ? ?def decorator(f): >> ? ? ? ?for name in kwargs: >> ? ? ? ? ? ?index = figure_out_index(f, name) >> ? ? ? ? ? ?f.__spam__[index] = kwargs[name] >> ? ? ? ? ? ?#functools.update_spam(f, name, kwargs[name]) >> ? ? ? ?return f >> ? ?return decorator >> >> @init_deftime(x=5): >> def f(): >> ? ?some_keyword x >> ? ?print(x) > > Mmh. ?This looks exactly like one I posted earlier on: > http://mail.python.org/pipermail/python-ideas/2011-September/011892.html Yeah, very similar. Somehow I missed your post earlier. It seems like a decent approach. It is essentially a variation on how default arguments work right now, using the decorator instead of the function signature. -eric > > -- > Arnaud > From zuo at chopin.edu.pl Wed Sep 28 23:57:47 2011 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Wed, 28 Sep 2011 23:57:47 +0200 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: <20110928215747.GA2263@chopin.edu.pl> Terry Reedy dixit (2011-09-27, 21:17): > defining a function > inside a loop magically causes define-time binding of names in the > body. No, it does not cause such a binding. That is one of the cases the proposition of this or that early-binding syntax comes back repeatedly: def strangish_factorission(): chairs = [] for spam in (1,2,3,4,5): def comfy_chair(fmt): # we naively try to make use of early binding here # but there is no such binding here result = fmt % spam return result chairs.append(comfy_chair) return chairs for comfy_chair in strangish_factorission(): print comfy_chair('%d'), -- will print "5 5 5 5 5", not "1 2 3 4 5". To obtain the latter you need to use eigher default argument hack (which is ugly and unsafe in some situations) or a closure (which for now needs another nested scope, which is even worse in terms of readability). Cheers, *j From zuo at chopin.edu.pl Thu Sep 29 00:02:25 2011 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Thu, 29 Sep 2011 00:02:25 +0200 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E8325B8.4050806@pearwood.info> References: <4E8112D2.809@canterbury.ac.nz> <20110927230809.GA2882@chopin.edu.pl> <4B0AD508-2513-4E13-9BC6-EE241368491F@gmail.com> <4E8325B8.4050806@pearwood.info> Message-ID: <20110928220225.GB2263@chopin.edu.pl> Steven D'Aprano dixit (2011-09-28, 23:48): > >I think ``rebind expression as name`` reads fine. Not sure if "rebind" exactly gives the right impression though. > > Why "rebind"? The expression hasn't been bound once yet, so you can't > REbind it. Maybe: freeze EXPR as VAR or frozen VAR=EXPR Cheers. *j From zuo at chopin.edu.pl Thu Sep 29 00:32:35 2011 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Thu, 29 Sep 2011 00:32:35 +0200 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <1317108987.16933.90.camel@Gutsy> Message-ID: <20110928223235.GC2263@chopin.edu.pl> Nick Coghlan dixit (2011-09-28, 05:59): > def accumulator(): > def incr(x) [tally=0]: > tally += x > return tally > return incr > > As the rough equivalent of today's: > > def accumulator(): > tally = 0 > def incr(x): > nonlocal tally > tally += x > return tally > return incr IMHO such a syntax (or a special or non-special decorator) should simply allow to add free variable(s) to a function. Then, if we want to modify such a variable in the function body we should use nonlocal, as with today's closures: def accumulator(): def incr(x) [tally=0]: nonlocal tally tally += x return tally return incr But we would not need to use nonlocal for read-only access, as with today's closures: def my_func() [lock=Lock()]: with lock: "foo" Cheers. *j From zuo at chopin.edu.pl Thu Sep 29 00:36:14 2011 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Thu, 29 Sep 2011 00:36:14 +0200 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <1317108987.16933.90.camel@Gutsy> Message-ID: <20110928223614.GD2263@chopin.edu.pl> Guido van Rossum dixit (2011-09-28, 08:38): > @init(n=1, lock=Lock()) > def global_counter(): > nonlocal n, lock > with lock: > print(n) > n += 1 > > Alas, the problem is that you can't actually implement such a > decorator, not even using an extension module, because the compiler, > upon seeing the "nonlocal n, lock", ignoring the decorator (since it > should be applied at run time, not at compile time), will complain > that there isn't actually an intermediary outer scope defining either > n or lock. > > Some ways out of this: > > - let the compiler special-case a certain built-in decorator name at > compile-time (unorthodox, not impossible) > > - different syntax, e.g. > @[n=1, lock=Lock()] > or > @in(n=1, lock=Lock()) > or even > in n=1, lock=Lock() > > (all followed by "def global_counter() etc.") > > Of course once there's different syntax, the nonlocal declaration in > the function is redundant. And clearly I'm back-peddling. :-) And what about: @nonlocal n=1, lock=Lock() def global_counter(): "we don't need to repeat nonlocal in the body" or @nonlocal(n=1, lock=Lock()) def global_counter(): "we don't need to repeat nonlocal in the body" ? Cheers. *j From tjreedy at udel.edu Thu Sep 29 00:46:15 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 28 Sep 2011 18:46:15 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <20110928215747.GA2263@chopin.edu.pl> References: <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <20110928215747.GA2263@chopin.edu.pl> Message-ID: On 9/28/2011 5:57 PM, Jan Kaliszewski wrote: > Terry Reedy dixit (2011-09-27, 21:17): > >> defining a function inside a loop Insert 'does not', which somehow got omitted or deleted. >> magically causes define-time binding of names in the body. > > No, it does not cause such a binding. Of course not, as I have said many times over the last decade plus, most recently just 4 hours earlier (at 17:10), when I said "People are assuming [wrongly, when using a local name that matches an outer enclosing loop name] that 'i' is immediately bound to its 'current' value, just like default args." Sorry for the confusing omission. My intention was to list this as a delusion, not as a fact. -- Terry Jan Reedy From anacrolix at gmail.com Thu Sep 29 01:35:58 2011 From: anacrolix at gmail.com (Matt Joiner) Date: Thu, 29 Sep 2011 09:35:58 +1000 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> <22C523DC-361A-4106-B21F-222D98CBC66E@gmail.com> Message-ID: Other than making my pocket vibrate every 5 mins at work, where is this discussion headed? is there a PEP I can get up to speed with? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimjjewett at gmail.com Thu Sep 29 02:41:41 2011 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 28 Sep 2011 20:41:41 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <4E8277DD.5060407@stoneleaf.us> Message-ID: On Wed, Sep 28, 2011 at 12:12 AM, Terry Reedy wrote: > ... Python has a simple rule: header expressions (for default args) are > evaluated one-time when the function is defined; body expressions are > evaluated (perhaps) each time when the function is called. Note that this also matches indentation, since decorators are indented with the header, rather than with the body. > If people understand this and do not fight it, they are not surprised that > mutating a once-defined default arg mutates it, nor that defining a function > inside a loop magically causes define-time binding of names in the body. Though they may still be surprised in the opposite direction; that you have to write an explicit and redundant i=i to capture the current value when binding. > I would hate for Python to lose this simplicity. Agreed. > At least three people, including Guido, have noted than a keyword-only > _private parameter solves most of the problems with using default args for > constants. Since the _underscore convention is already partly built into the > language (import *, help(module), __mangling, __reserved__), we can have > other tools like signature introspection also respect the convention. > This solution *also* works for the semi-private accumulator parameter of > most tail-recursive functions. Additional advantages: (a) Because it is still a parameter, it *can* be altered for test code; it is just obvious that you're doing something unsupported. (b) The only patch required is to documentation. Instead of saying "Don't do that", the documentation should say "If you just want to save state between calls, make it a keyword-only parameter and indicate that it is private by prefixing the name with an underscore." That is pretty hard to beat from a backwards-compatibility standpoint. -jJ From jimjjewett at gmail.com Thu Sep 29 02:46:25 2011 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 28 Sep 2011 20:46:25 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Wed, Sep 28, 2011 at 1:23 AM, Arnaud Delobelle wrote: > A hybrid approach would be possible, taking from both approaches > (decorator and keyword): > > * a keyword to declare in the body of the function that a variable is > of the "own" kind. > * function objects would grow an __own__ attribute which is a dict-like object: > ? ?- f.__own__["i"] would return the contents of the cell that the > own variable "i" refers to > ? ?- f.__own__["i"] = 42 would set the cell contents of the own > variable "i" to 42 > (or some other equivalent mechanism) > * then one could create a kind of "inject" decorator. Any reason not to just use the function itself, as opposed to a designated dictionary? Right now, we don't have a good way to refer to a function either before it defined (decorator time) or while it is being defined, but that seems like a subset of what you would need for the above proposal. (Of course, I did like the functionality of PEP 3130.) -jJ From guido at python.org Thu Sep 29 04:54:26 2011 From: guido at python.org (Guido van Rossum) Date: Wed, 28 Sep 2011 19:54:26 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <4E8277DD.5060407@stoneleaf.us> Message-ID: On Wed, Sep 28, 2011 at 5:41 PM, Jim Jewett wrote: > On Wed, Sep 28, 2011 at 12:12 AM, Terry Reedy wrote: > >> ... Python has a simple rule: header expressions (for default args) are >> evaluated one-time when the function is defined; body expressions are >> evaluated (perhaps) each time when the function is called. > > Note that this also matches indentation, since decorators are indented > with the header, rather than with the body. > >> If people understand this and do not fight it, they are not surprised that >> mutating a once-defined default arg mutates it, nor that defining a function >> inside a loop magically causes define-time binding of names in the body. > > Though they may still be surprised in the opposite direction; that you > have to write an explicit and redundant i=i to capture the current > value when binding. > >> I would hate for Python to lose this simplicity. > > Agreed. > >> At least three people, including Guido, have noted than a keyword-only >> _private parameter solves most of the problems with using default args for >> constants. ?Since the _underscore convention is already partly built into the >> language (import *, help(module), __mangling, __reserved__), we can have >> other tools like signature introspection also respect the convention. > >> This solution *also* works for the semi-private accumulator parameter of >> most tail-recursive functions. > > Additional advantages: > > (a) ?Because it is still a parameter, it *can* be altered for test > code; it is just obvious that you're doing something unsupported. > > (b) ?The only patch required is to documentation. ?Instead of saying > "Don't do that", the documentation should say "If you just want to > save state between calls, make it a keyword-only parameter and > indicate that it is private by prefixing the name with an underscore." > ?That is pretty hard to beat from a backwards-compatibility > standpoint. Um, but you can't save state between calls in a default argument value, except by the hack of making it a list (or some other mutable object) and mutating that. -- --Guido van Rossum (python.org/~guido) From matthew at exanimo.com Thu Sep 29 08:27:39 2011 From: matthew at exanimo.com (Matthew J Tretter) Date: Thu, 29 Sep 2011 02:27:39 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <4E8277DD.5060407@stoneleaf.us> Message-ID: So has Jan's post-** argument list proposal from June ( http://mail.python.org/pipermail/python-ideas/2011-June/010479.html) been definitively nixed? It seemed the perfect answer given that its similarity to the default argument hack makes it intuitive when the evaluation is taking place. On Wed, Sep 28, 2011 at 10:54 PM, Guido van Rossum wrote: > On Wed, Sep 28, 2011 at 5:41 PM, Jim Jewett wrote: > > On Wed, Sep 28, 2011 at 12:12 AM, Terry Reedy wrote: > > > >> ... Python has a simple rule: header expressions (for default args) are > >> evaluated one-time when the function is defined; body expressions are > >> evaluated (perhaps) each time when the function is called. > > > > Note that this also matches indentation, since decorators are indented > > with the header, rather than with the body. > > > >> If people understand this and do not fight it, they are not surprised > that > >> mutating a once-defined default arg mutates it, nor that defining a > function > >> inside a loop magically causes define-time binding of names in the body. > > > > Though they may still be surprised in the opposite direction; that you > > have to write an explicit and redundant i=i to capture the current > > value when binding. > > > >> I would hate for Python to lose this simplicity. > > > > Agreed. > > > >> At least three people, including Guido, have noted than a keyword-only > >> _private parameter solves most of the problems with using default args > for > >> constants. Since the _underscore convention is already partly built > into the > >> language (import *, help(module), __mangling, __reserved__), we can have > >> other tools like signature introspection also respect the convention. > > > >> This solution *also* works for the semi-private accumulator parameter of > >> most tail-recursive functions. > > > > Additional advantages: > > > > (a) Because it is still a parameter, it *can* be altered for test > > code; it is just obvious that you're doing something unsupported. > > > > (b) The only patch required is to documentation. Instead of saying > > "Don't do that", the documentation should say "If you just want to > > save state between calls, make it a keyword-only parameter and > > indicate that it is private by prefixing the name with an underscore." > > That is pretty hard to beat from a backwards-compatibility > > standpoint. > > Um, but you can't save state between calls in a default argument > value, except by the hack of making it a list (or some other mutable > object) and mutating that. > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Sep 29 12:52:43 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 29 Sep 2011 23:52:43 +1300 Subject: [Python-ideas] For loop binding (was Re: Tweaking closures ...) In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8159AB.2030506@canterbury.ac.nz> Message-ID: <4E844DFB.7070909@canterbury.ac.nz> Terry Reedy wrote: > No, for loops *do* rebind loop variables each time around. I probably should have avoided the word "bind", because it can be ambiguous. What I mean is that it assigns a different value to the same variable instead of creating a new variable. This is different from what equivalent constructs in some other languages do. > People are > assuming that 'i' is immediately bound to its 'current' value, Yes, that's another possible misinterpretation they may be making -- if they're thinking that deeply about the issue at all. However, changing the language to make *that* true would break a lot of other things, such as mutual recursion, and isn't really an option. > The posted 'surprise code' typically uses list > comps. So changing for loops to be like list comps would have no effect > on that mis-assumptions and the resulting surprise. Agreed, because list comps in Python have the same problem -- even if the name is local to the list comp, it's still just one variable being reassigned. I only mentioned list comps as a precedent for making some kind of change to the way a loop variable is treated. I didn't mean to suggest that the way they currently work would solve the closure problem. -- Greg > From masklinn at masklinn.net Thu Sep 29 13:16:09 2011 From: masklinn at masklinn.net (Masklinn) Date: Thu, 29 Sep 2011 13:16:09 +0200 Subject: [Python-ideas] For loop binding (was Re: Tweaking closures ...) In-Reply-To: <4E844DFB.7070909@canterbury.ac.nz> References: <4E802202.4080009@canterbury.ac.nz> <4E8159AB.2030506@canterbury.ac.nz> <4E844DFB.7070909@canterbury.ac.nz> Message-ID: <68FAAEA0-6566-48A7-AA64-6287C7681CDE@masklinn.net> On 2011-09-29, at 12:52 , Greg Ewing wrote: > Terry Reedy wrote: >> No, for loops *do* rebind loop variables each time around. > > I probably should have avoided the word "bind", because it > can be ambiguous. What I mean is that it assigns a different > value to the same variable instead of creating a new > variable. This is different from what equivalent constructs > in some other languages do. That's because most (not all, Javascript is a significant breaker of those rules) C-type languages create a new scope every time instead of being solely function-scoped no? So the construct closes over a new name in a different lexical scope instead of (as in Python, or Javascript) the same name in the same lexical. It's not about rebinding so much as about the breadth of lexical scopes. From ncoghlan at gmail.com Thu Sep 29 14:00:14 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Sep 2011 08:00:14 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <4E8277DD.5060407@stoneleaf.us> Message-ID: On Thu, Sep 29, 2011 at 2:27 AM, Matthew J Tretter wrote: > So has Jan's post-** argument list proposal from June > (http://mail.python.org/pipermail/python-ideas/2011-June/010479.html) been > definitively nixed? It seemed the perfect answer given that its similarity > to the default argument hack makes it intuitive when the evaluation is > taking place. It has in my mind. As Guido noted, default argument *names* refer to ordinary locals, and hence exhibit the same rebinding behaviour as pre-nonlocal closure references (i.e. reassignments don't persist across calls, only mutations do). Since we expressly *don't* want that behaviour, anything directly in the argument list (even after **) would be inappropriate. I'm still mulling over the idea of '@state' either as a keyword in its own right or else as a 'known to the compiler' builtin like super so that the following would work (as several people have suggested): @state(n=0, lock=threading.Lock()) def global_counter(): n += 1 return n The special casing of super() is something that annoys me though, and making 'state' a keyword would be extraordinarily disruptive, so I'm also intrigued by the suggestion of reusing 'nonlocal' for the same purpose: @nonlocal(n=0, lock=threading.Lock()) def global_counter(): n += 1 return n With that spelling, the nonlocal statement would declare references to pre-existing closure variables in an outer scope, while the nonlocal decorator would declare new function state variables, implicitly creating the outer scope without all the boilerplate. I must admit, the 'some kind of decorator' syntax is growing on me, especially the nonlocal variant. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Thu Sep 29 14:03:28 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Sep 2011 08:03:28 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Wed, Sep 28, 2011 at 12:39 PM, Devin Jeanpierre wrote: >> @init(n=1, lock=Lock()) >> def global_counter(): >> ?nonlocal n, lock >> ?with lock: >> ? ?print(n) >> ? ?n += 1 > > This use-case is no good. Consider: > > _n = 1; _lock = Lock() > def global_counter(): > ? ?global _n > ? ?with _lock: > ? ? ? ?print(n) > ? ? ? ?n += 1 And consider what happens as soon as you have two module level functions that you want to assign a lock to - you end up have to use clumsy name hacks to avoid collisions in the module namespace. I'll stick with this as my toy example (although I forgot to actually acquire the lock in my last post) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Thu Sep 29 14:05:12 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Sep 2011 08:05:12 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Wed, Sep 28, 2011 at 11:58 AM, Paul Moore wrote: > On 28 September 2011 16:38, Guido van Rossum wrote: >> Of course once there's different syntax, the nonlocal declaration in >> the function is redundant. And clearly I'm back-peddling. :-) > > If we're back to syntax proposals on the def statement, how about > > def fn() with i=1, lock=Lock(): > ? whatever > > ? This is basically another bikeshed to paint, though... The PEP 3150 discussion on keywords is relevant to that kind of keyword-based proposal (there's a reason the statement local namespaces proposal suggests 'given' rather than 'with' or 'where') Cheers, Nick -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Thu Sep 29 15:30:35 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Sep 2011 09:30:35 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E82345A.3010105@canterbury.ac.nz> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: On Tue, Sep 27, 2011 at 4:38 PM, Greg Ewing wrote: > With regard to "own" or "static", my feeling is that this > kind of feature (i.e. mutable shared state visible only to > one function) is an anti-pattern in language design. We > have much better ways nowadays of sharing state without > making it completely global, such as modules and classes. In explaining to someone why I think this issue of function state variables is worth exploring, I described it as being important for the same reason as closures are important. That's also why I've been describing it as syntactic sugar for a particular *kind* of closure usage. The reason I think closures themselves are important is that gets back into the 'algorithm with state' vs 'data with behaviour' contrasting views of a problem. Sometimes one view is more productive, sometimes the other, and there's blurry ground in the middle where either can work. As programmers, our best bet is to learn both modelling tools and then judge which is most appropriate for the problem at hand. One thing I definitely got out of this thread is that I think we need better tools for introspecting closure and generator internal state, so I'd be interested in seeing proposals for additions to the inspect module on that front. However, getting back to the original topic, the @nonlocal suggestion is definitely growing on me, since it seems to combine all the elements of the current equivalent that I'd like to shorten into one concise syntax: def _outer(): n = 0 lock = threading.Lock() def global_counter(): nonlocal n with lock: n += 1 return n return global_counter global_counter = _outer() would become: @nonlocal(n=0, lock=threading.Lock()) def global_counter(): with lock: n += 1 return n It's presence in the decorator list hints that this is something happening outside the functions ordinary local scope, as well as indicating when the initialisation happens. The 'nonlocal' then ties in with how they behave from the point of view of the code in the function body. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From p.f.moore at gmail.com Thu Sep 29 16:05:58 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 29 Sep 2011 15:05:58 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: On 29 September 2011 14:30, Nick Coghlan wrote: > > ? ?@nonlocal(n=0, lock=threading.Lock()) > ? ?def global_counter(): > ? ? ? ?with lock: > ? ? ? ? ? ?n += 1 > ? ? ? ? ? ?return n > > It's presence in the decorator list hints that this is something > happening outside the functions ordinary local scope, as well as > indicating when the initialisation happens. The 'nonlocal' then ties > in with how they behave from the point of view of the code in the > function body. I agree, this is a nice option. My biggest discomfort with it is the idea of a magic decorator name handled specially by the compiler (and worse still, it's a keyword so it's syntactically very weird indeed). I had similar discomfort over the new super, but I could get over that by simply assuming that super was a normal function that was just more magical than I understood. "@keyword" decorators don't even fit into my model of valid syntax :-( That said, I like it otherwise. The above says to me "the following values are nonlocal to this function", which I can read exactly the way it actually works. Whether that's a result of a week's immersion in Nick's propaganda, I can't say, though :-) Paul. From ethan at stoneleaf.us Thu Sep 29 17:00:38 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 29 Sep 2011 08:00:38 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: <4E848816.4060205@stoneleaf.us> Nick Coghlan wrote: > @nonlocal(n=0, lock=threading.Lock()) > def global_counter(): > with lock: > n += 1 > return n > > It's presence in the decorator list hints that this is something > happening outside the functions ordinary local scope, as well as > indicating when the initialisation happens. The 'nonlocal' then ties > in with how they behave from the point of view of the code in the > function body. I think this fits in nicely with the @staticmethod and @classmethod decorator usage as well. ~Ethan~ From p.f.moore at gmail.com Thu Sep 29 17:15:14 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 29 Sep 2011 16:15:14 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: On 29 September 2011 15:05, Paul Moore wrote: > On 29 September 2011 14:30, Nick Coghlan wrote: >> >> ? ?@nonlocal(n=0, lock=threading.Lock()) >> ? ?def global_counter(): >> ? ? ? ?with lock: >> ? ? ? ? ? ?n += 1 >> ? ? ? ? ? ?return n >> >> It's presence in the decorator list hints that this is something >> happening outside the functions ordinary local scope, as well as >> indicating when the initialisation happens. The 'nonlocal' then ties >> in with how they behave from the point of view of the code in the >> function body. > > I agree, this is a nice option. My biggest discomfort with it is the > idea of a magic decorator name handled specially by the compiler (and > worse still, it's a keyword so it's syntactically very weird indeed). > I had similar discomfort over the new super, but I could get over that > by simply assuming that super was a normal function that was just more > magical than I understood. "@keyword" decorators don't even fit into > my model of valid syntax :-( > > That said, I like it otherwise. The above says to me "the following > values are nonlocal to this function", which I can read exactly the > way it actually works. Whether that's a result of a week's immersion > in Nick's propaganda, I can't say, though :-) Actually, while we're using keywords as decorator names, @with(n=0, lock=threading.Lock()) def global_counter(): ? ?with lock: ? ? ? ?n += 1 ? ? ? ?return n reads well to me, and avoids the technical/jargon word "nonlocal". Let the @keyword floodgates open :-) Paul. From masklinn at masklinn.net Thu Sep 29 17:20:58 2011 From: masklinn at masklinn.net (Masklinn) Date: Thu, 29 Sep 2011 17:20:58 +0200 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: On 2011-09-29, at 17:15 , Paul Moore wrote: > > Actually, while we're using keywords as decorator names, > > @with(n=0, lock=threading.Lock()) > def global_counter(): > with lock: > n += 1 > return n > > reads well to me, and avoids the technical/jargon word "nonlocal". Let > the @keyword floodgates open :-) > The problem with that one is that it's going to be read in light of the corresponding keyword, so it'd make people think the purpose is to __enter__/__exit__ all objects provided around the function call somehow (pretty nice for the lock, nonsensical for the integer, of course). From ethan at stoneleaf.us Thu Sep 29 17:21:07 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 29 Sep 2011 08:21:07 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: <4E848CE3.1030009@stoneleaf.us> Paul Moore wrote: > Actually, while we're using keywords as decorator names, > > @with(n=0, lock=threading.Lock()) > def global_counter(): > with lock: > n += 1 > return n > > reads well to me, and avoids the technical/jargon word "nonlocal". I think 'nonlocal' is the better choice as it mirrors what nonlocal does inside the function. Because of its current usage 'with' will generate questions about __enter__ and __exit__. ~Ethan~ From him at online.de Thu Sep 29 17:31:16 2011 From: him at online.de (=?ISO-8859-1?Q?Joachim_K=F6nig?=) Date: Thu, 29 Sep 2011 17:31:16 +0200 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E848CE3.1030009@stoneleaf.us> References: <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <4E848CE3.1030009@stoneleaf.us> Message-ID: <4E848F44.6090401@online.de> On 29/09/2011 17:21, Ethan Furman wrote: > Paul Moore wrote: >> Actually, while we're using keywords as decorator names, >> >> @with(n=0, lock=threading.Lock()) >> def global_counter(): >> with lock: >> n += 1 >> return n >> >> reads well to me, and avoids the technical/jargon word "nonlocal". > > I think 'nonlocal' is the better choice as it mirrors what nonlocal > does inside the function. Because of its current usage 'with' will > generate questions about __enter__ and __exit__. > But what if the decorator is used inside an outer function? Wouldn't that be confusing? What about @func_attrs(...) ? - it would give a hint where the objects are actually located. From guido at python.org Thu Sep 29 18:51:46 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 29 Sep 2011 09:51:46 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E848CE3.1030009@stoneleaf.us> References: <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <4E848CE3.1030009@stoneleaf.us> Message-ID: On Thu, Sep 29, 2011 at 8:21 AM, Ethan Furman wrote: > Paul Moore wrote: >> >> Actually, while we're using keywords as decorator names, >> >> @with(n=0, lock=threading.Lock()) >> def global_counter(): >> ? ?with lock: >> ? ? ? ?n += 1 >> ? ? ? ?return n >> >> reads well to me, and avoids the technical/jargon word "nonlocal". > > I think 'nonlocal' is the better choice as it mirrors what nonlocal does > inside the function. ?Because of its current usage 'with' will generate > questions about __enter__ and __exit__. +1 -- --Guido van Rossum (python.org/~guido) From jeanpierreda at gmail.com Thu Sep 29 19:10:39 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 29 Sep 2011 13:10:39 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: > And consider what happens as soon as you have two module level > functions that you want to assign a lock to - you end up have to use > clumsy name hacks to avoid collisions in the module namespace. I'll > stick with this as my toy example (although I forgot to actually > acquire the lock in my last post) Surely the solution is a new namespace (that's still publicly accessible)? It's absolutely pannoying that you have to prefix the globals like in C, but deftime-assigned nonlocals are less convenient in other respects too. e.g. try sharing a lock among multiple functions, or try testing the usage of a lock, etc. Devin On Thu, Sep 29, 2011 at 8:03 AM, Nick Coghlan wrote: > On Wed, Sep 28, 2011 at 12:39 PM, Devin Jeanpierre > wrote: >>> @init(n=1, lock=Lock()) >>> def global_counter(): >>> ?nonlocal n, lock >>> ?with lock: >>> ? ?print(n) >>> ? ?n += 1 >> >> This use-case is no good. Consider: >> >> _n = 1; _lock = Lock() >> def global_counter(): >> ? ?global _n >> ? ?with _lock: >> ? ? ? ?print(n) >> ? ? ? ?n += 1 > > And consider what happens as soon as you have two module level > functions that you want to assign a lock to - you end up have to use > clumsy name hacks to avoid collisions in the module namespace. I'll > stick with this as my toy example (although I forgot to actually > acquire the lock in my last post) > > Cheers, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > From ncoghlan at gmail.com Thu Sep 29 19:26:01 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Sep 2011 13:26:01 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Thu, Sep 29, 2011 at 1:10 PM, Devin Jeanpierre wrote: > It's absolutely pannoying that you have to prefix the globals like in > C, but deftime-assigned nonlocals are less convenient in other > respects too. e.g. try sharing a lock among multiple functions, or try > testing the usage of a lock, etc. Sharing state amongst multiple functions is a solved problem - you use either a closure or a class instance to create a shared namespace, depending on which approach makes more sense for the problem at hand. However, both of those solutions feel very heavy when all you want to do is share state across multiple invocations of the *same* function. Hence the current discussion (and the numerous ones that have preceded it over the years). It's a tricky problem precisely because it doesn't take much to tip any given use case over the threshold of complexity to where it's a better idea to use a full-fledged closure or class. And, as I have said several times, I agree closures currently impose testability problems, but I think the answer there lies in providing better introspection tools to lessen those problems rather than advising people not to use closures specifically for those reasons. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From jeanpierreda at gmail.com Thu Sep 29 19:41:48 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 29 Sep 2011 13:41:48 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: > And, as I have said several times, I agree closures currently impose > testability problems, but I think the answer there lies in providing > better introspection tools to lessen those problems rather than > advising people not to use closures specifically for those reasons. OK, that's fair. In order to support testing you'd need to be able to assign to closure variables (for e.g. mocking) and read them (to test alterations to global state). At that point it's a short leap to defining a decorator that creates these nonlocals. Is there a compelling reason to use a new syntax? Devin On Thu, Sep 29, 2011 at 1:26 PM, Nick Coghlan wrote: > On Thu, Sep 29, 2011 at 1:10 PM, Devin Jeanpierre > wrote: >> It's absolutely pannoying that you have to prefix the globals like in >> C, but deftime-assigned nonlocals are less convenient in other >> respects too. e.g. try sharing a lock among multiple functions, or try >> testing the usage of a lock, etc. > > Sharing state amongst multiple functions is a solved problem - you use > either a closure or a class instance to create a shared namespace, > depending on which approach makes more sense for the problem at hand. > However, both of those solutions feel very heavy when all you want to > do is share state across multiple invocations of the *same* function. > Hence the current discussion (and the numerous ones that have preceded > it over the years). It's a tricky problem precisely because it doesn't > take much to tip any given use case over the threshold of complexity > to where it's a better idea to use a full-fledged closure or class. > > And, as I have said several times, I agree closures currently impose > testability problems, but I think the answer there lies in providing > better introspection tools to lessen those problems rather than > advising people not to use closures specifically for those reasons. > > Regards, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > From ncoghlan at gmail.com Thu Sep 29 19:44:57 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Sep 2011 13:44:57 -0400 Subject: [Python-ideas] Proposed additions to inspect module for function and generator state introspection Message-ID: Based on the testability comments in the closure threads, I created http://bugs.python.org/issue13062 to propose two new introspection functions: inspect.getclosure(func) Returns a dictionary mapping closure references from the supplied function to their current values. inspect.getgeneratorlocals(generator) Returns the same result as would be reported by calling locals() in the generator's frame of execution The former would just involve syncing up the names on the code object with the cell references on the function object, while the latter would be equivalent to doing generator.gi_frame.f_locals with some nice error checking for when the generator's frame is already gone (or the supplied object isn't a generator iterator). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Thu Sep 29 20:03:31 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Sep 2011 14:03:31 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Thu, Sep 29, 2011 at 1:41 PM, Devin Jeanpierre wrote: >> And, as I have said several times, I agree closures currently impose >> testability problems, but I think the answer there lies in providing >> better introspection tools to lessen those problems rather than >> advising people not to use closures specifically for those reasons. > > OK, that's fair. > > In order to support testing you'd need to be able to assign to closure > variables (for e.g. mocking) and read them (to test alterations to > global state). At that point it's a short leap to defining a decorator > that creates these nonlocals. Is there a compelling reason to use a > new syntax? I don't think supporting monkey-patching is necessary - the code under test will presumably provide some mechanism to control how the hidden state is initialised or modified. The missing part from a testing point of view is the ability to get that information back out in a clean, portable way so you can make assertions about it for whitebox testing. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ericsnowcurrently at gmail.com Thu Sep 29 20:20:43 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 29 Sep 2011 12:20:43 -0600 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Thu, Sep 29, 2011 at 11:26 AM, Nick Coghlan wrote: > And, as I have said several times, I agree closures currently impose > testability problems, but I think the answer there lies in providing > better introspection tools to lessen those problems rather than > advising people not to use closures specifically for those reasons. +1 > > Regards, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From jeanpierreda at gmail.com Thu Sep 29 20:25:37 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 29 Sep 2011 14:25:37 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: > I don't think supporting monkey-patching is necessary Nothing is necessary. I maintain it is extremely useful for testing. There are entire libraries built around monkeypatching code for the purpose of testing it. For example, recently I saw Exocet, which allows you to create a new module object but replacing the imports with whatever you want.(The analogue here would be copying a function, but with new closure variables). Without this you can't really _unit_ test a closure, at least as I understand it. It becomes inextricably linked with its scope, which is not what you want to test. Devin On Thu, Sep 29, 2011 at 2:03 PM, Nick Coghlan wrote: > On Thu, Sep 29, 2011 at 1:41 PM, Devin Jeanpierre > wrote: >>> And, as I have said several times, I agree closures currently impose >>> testability problems, but I think the answer there lies in providing >>> better introspection tools to lessen those problems rather than >>> advising people not to use closures specifically for those reasons. >> >> OK, that's fair. >> >> In order to support testing you'd need to be able to assign to closure >> variables (for e.g. mocking) and read them (to test alterations to >> global state). At that point it's a short leap to defining a decorator >> that creates these nonlocals. Is there a compelling reason to use a >> new syntax? > > I don't think supporting monkey-patching is necessary - the code under > test will presumably provide some mechanism to control how the hidden > state is initialised or modified. The missing part from a testing > point of view is the ability to get that information back out in a > clean, portable way so you can make assertions about it for whitebox > testing. > > Cheers, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > From ericsnowcurrently at gmail.com Thu Sep 29 21:03:55 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 29 Sep 2011 13:03:55 -0600 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: On Thu, Sep 29, 2011 at 8:05 AM, Paul Moore wrote: > On 29 September 2011 14:30, Nick Coghlan wrote: >> >> ? ?@nonlocal(n=0, lock=threading.Lock()) >> ? ?def global_counter(): >> ? ? ? ?with lock: >> ? ? ? ? ? ?n += 1 >> ? ? ? ? ? ?return n >> >> It's presence in the decorator list hints that this is something >> happening outside the functions ordinary local scope, as well as >> indicating when the initialisation happens. The 'nonlocal' then ties >> in with how they behave from the point of view of the code in the >> function body. > > I agree, this is a nice option. My biggest discomfort with it is the > idea of a magic decorator name handled specially by the compiler (and > worse still, it's a keyword so it's syntactically very weird indeed). > I had similar discomfort over the new super, but I could get over that > by simply assuming that super was a normal function that was just more > magical than I understood. "@keyword" decorators don't even fit into > my model of valid syntax :-( I feel the same way. The alternative is to leave nonlocal as just a simple statement, but change its behavior when the name is not found inside a containing function scope. Currently that is a syntax error. Instead it could imply that the name should fall back to the definition-time "scope" of the current function. In other words, the compiler would treat the name like a closed variable anyway, but with an empty cell (or some default initial value, like None)[1]. A decorator would still be used to explicitly initialize the value: @init_nonlocal(n=0, lock=threading.Lock()) def global_counter(): nonlocal n with lock: n += 1 return n This approach has some advantages: * doesn't require the compiler to do any special handling of decorators, * fewer changes to the compiler (and less substantial), * keeps the advantages of using a decorator for initialization, * the nonlocal nature of the variable is clear in the function body (where it is used), * allows for "uninitialized" def-time names (see [1]), * more intuitively consistent use of the keyword (says my gut), * decorator could be applied to any function at any time (not just def-time); ...and some disadvantages: * the statement may be redundant for def-time nonlocals (if it will always be accompanied by the decorator), * implicit initialization (if used) is...implicit (EIBTI), * implicit default def-time scope is as well, * (maybe) makes it too easy to modify closure cells, if that's a bad thing, * decorator could be applied to any function at any time (not just def-time). The init_nonlocal decorator [factory] would change the value of the appropriate cell in __closure__. Either the decorator would be the function that exposed the ability to modify cells or it would leverage another function that did so. I would think the latter, so it could look like this: def init_nonlocal(**kwargs): def decorator(f): freevars = f.__code__.co_freevars indices = {} for name in kwargs: try: index = freevars.index(name) except ValueError: raise NameError(...) indices[name] = index for name, index in indices.items(): #f.__closure__[index].cell_contents = kwargs[name] functools.update_cell(f, index, kwargs[name] return f return decorator or def init_nonlocal(**kwargs): def decorator(f): functools.update_freevars(**kwargs) return f return decorator The cell-modifying capabilty would have to apply to any cell in __closure__ and not just the ones associated with def-time nonlocals. If the nonlocal statement gets associated with a name in an enclosing function scope, then init_nonlocal would be used to explicitly force the name to be a def-time nonlocal. Maybe that's an argument for syntax to explicitly identify a def-time nonlocal rather than relying on any implicit behavior of the nonlocal keyword. -eric [1] explicitly uninitialized variables in some languages (like C) get implicitly initialized to some default value. I'm not sure what the use case would be for an uninitialized nonlocal (or even if it makes sense for a cell to be uninitialized). In that situation I would expect a NameError when the name is accessed. > > That said, I like it otherwise. The above says to me "the following > values are nonlocal to this function", which I can read exactly the > way it actually works. Whether that's a result of a week's immersion > in Nick's propaganda, I can't say, though :-) > > Paul. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From ericsnowcurrently at gmail.com Thu Sep 29 21:08:19 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 29 Sep 2011 13:08:19 -0600 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: On Thu, Sep 29, 2011 at 1:03 PM, Eric Snow wrote: > On Thu, Sep 29, 2011 at 8:05 AM, Paul Moore wrote: >> On 29 September 2011 14:30, Nick Coghlan wrote: >>> >>> ? ?@nonlocal(n=0, lock=threading.Lock()) >>> ? ?def global_counter(): >>> ? ? ? ?with lock: >>> ? ? ? ? ? ?n += 1 >>> ? ? ? ? ? ?return n >>> >>> It's presence in the decorator list hints that this is something >>> happening outside the functions ordinary local scope, as well as >>> indicating when the initialisation happens. The 'nonlocal' then ties >>> in with how they behave from the point of view of the code in the >>> function body. >> >> I agree, this is a nice option. My biggest discomfort with it is the >> idea of a magic decorator name handled specially by the compiler (and >> worse still, it's a keyword so it's syntactically very weird indeed). >> I had similar discomfort over the new super, but I could get over that >> by simply assuming that super was a normal function that was just more >> magical than I understood. "@keyword" decorators don't even fit into >> my model of valid syntax :-( > > I feel the same way. > > The alternative is to leave nonlocal as just a simple statement, but > change its behavior when the name is not found inside a containing > function scope. ?Currently that is a syntax error. > > Instead it could imply that the name should fall back to the > definition-time "scope" of the current function. ?In other words, the > compiler would treat the name like a closed variable anyway, but with > an empty cell (or some default initial value, like None)[1]. ?A > decorator would still be used to explicitly initialize the value: s/would/could/ > > ? @init_nonlocal(n=0, lock=threading.Lock()) > ? def global_counter(): > ? ? ? nonlocal n > ? ? ? with lock: > ? ? ? ? ? n += 1 > ? ? ? ? ? return n > > This approach has some advantages: > > * doesn't require the compiler to do any special handling of decorators, > * fewer changes to the compiler (and less substantial), > * keeps the advantages of using a decorator for initialization, > * the nonlocal nature of the variable is clear in the function body > (where it is used), > * allows for "uninitialized" def-time names (see [1]), > * more intuitively consistent use of the keyword (says my gut), > * decorator could be applied to any function at any time (not just def-time); > > ...and some disadvantages: > > * the statement may be redundant for def-time nonlocals (if it will > always be accompanied by the decorator), > * implicit initialization (if used) is...implicit (EIBTI), > * implicit default def-time scope is as well, > * (maybe) makes it too easy to modify closure cells, if that's a bad thing, > * decorator could be applied to any function at any time (not just def-time). > > The init_nonlocal decorator [factory] would change the value of the > appropriate cell in __closure__. ?Either the decorator would be the > function that exposed the ability to modify cells or it would leverage > another function that did so. ?I would think the latter, so it could > look like this: > > ?def init_nonlocal(**kwargs): > ? ? ?def decorator(f): > ? ? ? ? ?freevars = f.__code__.co_freevars > ? ? ? ? ?indices = {} > ? ? ? ? ?for name in kwargs: > ? ? ? ? ? ? ?try: > ? ? ? ? ? ? ? ? ?index = freevars.index(name) > ? ? ? ? ? ? ?except ValueError: > ? ? ? ? ? ? ? ? ?raise NameError(...) > ? ? ? ? ? ? ?indices[name] = index > ? ? ? ? ?for name, index in indices.items(): > ? ? ? ? ? ? ?#f.__closure__[index].cell_contents = kwargs[name] > ? ? ? ? ? ? ?functools.update_cell(f, index, kwargs[name] > ? ? ? ? ?return f > ? ? ?return decorator > > or > > ?def init_nonlocal(**kwargs): > ? ? ?def decorator(f): > ? ? ? ? ?functools.update_freevars(**kwargs) > ? ? ? ? ?return f > ? ? ?return decorator > > The cell-modifying capabilty would have to apply to any cell in > __closure__ and not just the ones associated with def-time nonlocals. > If the nonlocal statement gets associated with a name in an enclosing > function scope, then init_nonlocal would be used to explicitly force > the name to be a def-time nonlocal. ?Maybe that's an argument for > syntax to explicitly identify a def-time nonlocal rather than relying > on any implicit behavior of the nonlocal keyword. > > -eric > > > [1] explicitly uninitialized variables in some languages (like C) get > implicitly initialized to some default value. ?I'm not sure what the > use case would be for an uninitialized nonlocal (or even if it makes > sense for a cell to be uninitialized). ?In that situation I would > expect a NameError when the name is accessed. > >> >> That said, I like it otherwise. The above says to me "the following >> values are nonlocal to this function", which I can read exactly the >> way it actually works. Whether that's a result of a week's immersion >> in Nick's propaganda, I can't say, though :-) >> >> Paul. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas >> > From ncoghlan at gmail.com Thu Sep 29 21:24:45 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Sep 2011 15:24:45 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <1317108987.16933.90.camel@Gutsy> Message-ID: On Thu, Sep 29, 2011 at 2:25 PM, Devin Jeanpierre wrote: >> I don't think supporting monkey-patching is necessary > > Nothing is necessary. I maintain it is extremely useful for testing. > There are entire libraries built around monkeypatching code for the > purpose of testing it. For example, recently I saw Exocet, which > allows you to create a new module object but replacing the imports > with whatever you want.(The analogue here would be copying a function, > but with new closure variables). > > Without this you can't really _unit_ test a closure, at least as I > understand it. It becomes inextricably linked with its scope, which is > not what you want to test. Improving introspection of closures doesn't magically remove the need to design for testability. Some ways of using closures are fairly easy to test (e.g. you just call the outer function again with monkey-patched globals and get a monkey-patched closure). Others are not (you define excessively complex state in the closure without provide a way to replace it for testing purposes). If the amount of state in a closure is causing serious testing problems, it's a hint that the code is attempting to handle a "data with behaviour" situation more suited to an OOP style rather than an "algorithm with state" that is particularly amenable to modelling with closures. Providing a nicer API for features the language already supports to simplify whitebox testing is one thing, providing a completely new feature (i.e. modifying closure references from outside the function) is something else entirely. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tjreedy at udel.edu Thu Sep 29 21:47:08 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 29 Sep 2011 15:47:08 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <4E8277DD.5060407@stoneleaf.us> Message-ID: On 9/29/2011 8:00 AM, Nick Coghlan wrote: > It has in my mind. As Guido noted, default argument *names* refer to > ordinary locals, and hence exhibit the same rebinding behaviour as > pre-nonlocal closure references (i.e. reassignments don't persist > across calls, only mutations do). Since we expressly *don't* want that > behaviour, anything directly in the argument list (even after **) > would be inappropriate. /'we'/'I and some of the discussants'/ It has occurred to me that one reason this threads is not getting anywhere very fast is that there are two quite different proposals being intermixed. People interested in and discussing the two different proposals have been talking past each other. One is to reproduce the behavior of defaulted parameters without having the name appear in the signature to produce *read-only* private locals, which amount to private define-time constants. This includes the int=int type speed hack. I think '*, _int=int', with adjustment of introspection, is sufficient. This make '_int' pragmatically constant, while still externally over-rideable for exceptional purposes such as testing or tracing. The other is to introduce something new: private *read-write* closure (persistent) name-value pairs. I agree that such a new thing should not be in the param list between '(' and ')'. -- Terry Jan Reedy From ncoghlan at gmail.com Thu Sep 29 21:48:55 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Sep 2011 15:48:55 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: On Thu, Sep 29, 2011 at 3:03 PM, Eric Snow wrote: > Instead it could imply that the name should fall back to the > definition-time "scope" of the current function. ?In other words, the > compiler would treat the name like a closed variable anyway, but with > an empty cell (or some default initial value, like None). Yeah, I'd thought of that possibility and have a few problems with it: 1. Replacing the current SyntaxError when a nonlocal statement refers to a variable that doesn't exist in an outer scope with a call time "UnboundNonLocalError" or an implicit None reference doesn't thrill me. Either choice would make UnboundLocalError look positively intuitive. 2. Adding a general ability to edit closure references from pure Python code outside the function body is a bridge I'm highly reluctant to cross. That kind of thing is what classes are for. 3. The syntax is repetitive and noisy enough that I think people could be legitimately excused for continuing to use the default argument hack instead. It's important to keep in mind that for folks that are familiar with how it works, the default argument hack is actually pretty easy to read and write. It's quite clearly exploiting a side effect of a feature that was added for other reasons, though, which is why blessing and documenting it as the preferred mechanism for function state variables (if you choose to use them) also doesn't feel right. All that said, the @nonlocal idea certainly has its own problems. In particular, it looks superficially like an ordinary call to a decorator factory, but is actually quite restricted syntactically (keyword arguments only, no * or ** expansion), has significant effects on the resolution of names in the function's namespace and can't be used for post-decoration via an ordinary function call with the just defined function as an argument. It would end up in the same category of "special case that doesn't actually look all that special" as the 3.x version of super(). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Thu Sep 29 21:56:20 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Sep 2011 15:56:20 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <4E8277DD.5060407@stoneleaf.us> Message-ID: On Thu, Sep 29, 2011 at 3:47 PM, Terry Reedy wrote: > One is to reproduce the behavior of defaulted parameters without having the > name appear in the signature to produce *read-only* private locals, which > amount to private define-time constants. This includes the int=int type > speed hack. I think '*, _int=int', with adjustment of introspection, is > sufficient. This make '_int' pragmatically constant, while still externally > over-rideable for exceptional purposes such as testing or tracing. Well, the references would be constant. The state itself could still be mutable, depending on the kinds of objects referred to. I agree this use case could be addressed by blessing the status quo and a couple of conventions. > The other is to introduce something new: private *read-write* closure > (persistent) name-value pairs. I agree that such a new thing should not be > in the param list between '(' and ')'. It's not actually new as far as overall language capabilities go - since PEP 3104, you can get the functionality through appropriate use of an ordinary closure. It's really just about providing a slightly shorter syntax for a particular way of using closures. Well noted that not all proposals are addressing the same functionality, though. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Thu Sep 29 22:01:12 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 29 Sep 2011 13:01:12 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: On Thu, Sep 29, 2011 at 12:03 PM, Eric Snow wrote: > The alternative is to leave nonlocal as just a simple statement, but > change its behavior when the name is not found inside a containing > function scope. ?Currently that is a syntax error. For a reason. It would be too easy for a typo to produce the wrong interpretation. -- --Guido van Rossum (python.org/~guido) From jimjjewett at gmail.com Thu Sep 29 22:24:05 2011 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 29 Sep 2011 16:24:05 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: On Thu, Sep 29, 2011 at 3:03 PM, Eric Snow wrote: > The alternative is to leave nonlocal as just a simple statement, but > change its behavior when the name is not found inside a containing > function scope. ?Currently that is a syntax error. Until this discussion started, I had not realized that nonlocal was limited to names from containing *function* scopes; I had thought that it would end up using the module-level globals if need be. Having it indicate a scope *closer* than all the scopes where it already looked but failed to find the name seems wrong. That said, my intuition may be suspect; outside examples, I have never used nonlocal, and work fairly hard to avoid global. I don't *like* boxing up my variables, but it still seems less offensive than a global statement and the resulting side effects. -jJ From jeanpierreda at gmail.com Thu Sep 29 22:26:44 2011 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 29 Sep 2011 16:26:44 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: > I don't *like* > boxing up my variables, but it still seems less offensive than a > global statement and the resulting side effects. How is that? Devin On Thu, Sep 29, 2011 at 4:24 PM, Jim Jewett wrote: > On Thu, Sep 29, 2011 at 3:03 PM, Eric Snow wrote: > >> The alternative is to leave nonlocal as just a simple statement, but >> change its behavior when the name is not found inside a containing >> function scope. ?Currently that is a syntax error. > > Until this discussion started, I had not realized that nonlocal was > limited to names from containing *function* scopes; I had thought that > it would end up using the module-level globals if need be. > > Having it indicate a scope *closer* than all the scopes where it > already looked but failed to find the name seems wrong. > > That said, my intuition may be suspect; outside examples, I have never > used nonlocal, and work fairly hard to avoid global. ?I don't *like* > boxing up my variables, but it still seems less offensive than a > global statement and the resulting side effects. > > -jJ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From greg.ewing at canterbury.ac.nz Thu Sep 29 23:24:15 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 30 Sep 2011 10:24:15 +1300 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <4E8277DD.5060407@stoneleaf.us> Message-ID: <4E84E1FF.9070808@canterbury.ac.nz> Nick Coghlan wrote: > As Guido noted, default argument *names* refer to > ordinary locals, and hence exhibit the same rebinding behaviour as > pre-nonlocal closure references (i.e. reassignments don't persist > across calls, only mutations do). Since we expressly *don't* want that > behaviour, Are you sure we don't? The ability to persistently re-assign the name is not needed to address the main use cases under consideration, as I understand them to be. In fact, disallowing assignment to the name at all would be fine by me. Hence I would be +0 on using 'const' as the keyword. -- Greg From greg.ewing at canterbury.ac.nz Thu Sep 29 23:31:51 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 30 Sep 2011 10:31:51 +1300 Subject: [Python-ideas] Changing semantics of for-loop variable (was: Tweaking closures ...) In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8159AB.2030506@canterbury.ac.nz> <465D00FC-D144-4606-AF90-2F5014C4ED3A@gmail.com> <4E817549.7080107@canterbury.ac.nz> Message-ID: <4E84E3C7.2030102@canterbury.ac.nz> Guido has asked me to start a new thread for discussing this idea. To recap, instead of trying to come up with some new sugar to make the default-argument hack taste slightly less bitter, I suggest making a small change to the semantics of for-loops: If the loop variable is referenced from an inner scope, instead of replacing the contents of its cell, create a *new* cell on each iteration. Code following the loop would then continue to see the last value bound to the loop variable, as now, but inner functions would capture different versions of it. -- Greg From ncoghlan at gmail.com Thu Sep 29 23:33:16 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Sep 2011 17:33:16 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E84E1FF.9070808@canterbury.ac.nz> References: <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <4E8277DD.5060407@stoneleaf.us> <4E84E1FF.9070808@canterbury.ac.nz> Message-ID: On Thu, Sep 29, 2011 at 5:24 PM, Greg Ewing wrote: > Are you sure we don't? The ability to persistently re-assign > the name is not needed to address the main use cases under > consideration, as I understand them to be. In fact, disallowing > assignment to the name at all would be fine by me. Hence I > would be +0 on using 'const' as the keyword. Fixed references to mutable objects makes 'const' a rather problematic choice. Besides, if we set the bar at that level of functionality then documenting some conventions for the status quo (i.e. 'private' keyword only arguments) is an adequate response. As Terry noted, there are actually two different features being discussed in the thread, and they differ according to whether or not rebinding is supported. By casting the proposal as syntactic sugar for a particular usage of closures, I'm firmly in the camp that if we change anything here the updated syntax should support rebinding of the shared names. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From greg.ewing at canterbury.ac.nz Thu Sep 29 23:36:13 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 30 Sep 2011 10:36:13 +1300 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: <4E84E4CD.90603@canterbury.ac.nz> Paul Moore wrote: > On 29 September 2011 14:30, Nick Coghlan wrote: > >> @nonlocal(n=0, lock=threading.Lock()) >> def global_counter(): >> with lock: >> n += 1 >> return n >> > The above says to me "the following > values are nonlocal to this function", It still has the problem that the *visibility* is *local*, directly contradicting what the keyword says. -- Greg From greg.ewing at canterbury.ac.nz Thu Sep 29 23:37:23 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 30 Sep 2011 10:37:23 +1300 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E848816.4060205@stoneleaf.us> References: <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <4E848816.4060205@stoneleaf.us> Message-ID: <4E84E513.8000202@canterbury.ac.nz> Ethan Furman wrote: > Nick Coghlan wrote: > >> @nonlocal(n=0, lock=threading.Lock()) > > I think this fits in nicely with the @staticmethod and @classmethod > decorator usage as well. Maybe @iwastoolazytocreateaclassforthis would be a better name? :-) -- Greg From p.f.moore at gmail.com Thu Sep 29 23:48:58 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 29 Sep 2011 22:48:58 +0100 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E84E4CD.90603@canterbury.ac.nz> References: <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <4E84E4CD.90603@canterbury.ac.nz> Message-ID: On 29 September 2011 22:36, Greg Ewing wrote: > Paul Moore wrote: >> >> On 29 September 2011 14:30, Nick Coghlan wrote: >> >>> ?@nonlocal(n=0, lock=threading.Lock()) >>> ?def global_counter(): >>> ? ? ?with lock: >>> ? ? ? ? ?n += 1 >>> ? ? ? ? ?return n >>> >> The above says to me "the following >> values are nonlocal to this function", > > It still has the problem that the *visibility* is *local*, > directly contradicting what the keyword says. I see what you mean. But I think with this version, the fact that the variable is visible in the "decorator" call (i.e., just slightly outside the function body) is enough to make me feel less uncomfortable than the purely-within-the-function "nonlocal VAR from EXPR" did... Paul. From ethan at stoneleaf.us Thu Sep 29 23:53:10 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 29 Sep 2011 14:53:10 -0700 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E84E513.8000202@canterbury.ac.nz> References: <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <4E848816.4060205@stoneleaf.us> <4E84E513.8000202@canterbury.ac.nz> Message-ID: <4E84E8C6.7080307@stoneleaf.us> Greg Ewing wrote: > Ethan Furman wrote: >> Nick Coghlan wrote: >> >>> @nonlocal(n=0, lock=threading.Lock()) >> >> I think this fits in nicely with the @staticmethod and @classmethod >> decorator usage as well. > > Maybe @iwastoolazytocreateaclassforthis would be a > better name? :-) Nah, it should be @thisisbetterasanalgorythmwithstatethanastinkinoopclass ! ~Ethan~ From ncoghlan at gmail.com Fri Sep 30 00:04:47 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Sep 2011 18:04:47 -0400 Subject: [Python-ideas] Changing semantics of for-loop variable (was: Tweaking closures ...) In-Reply-To: <4E84E3C7.2030102@canterbury.ac.nz> References: <4E802202.4080009@canterbury.ac.nz> <4E8159AB.2030506@canterbury.ac.nz> <465D00FC-D144-4606-AF90-2F5014C4ED3A@gmail.com> <4E817549.7080107@canterbury.ac.nz> <4E84E3C7.2030102@canterbury.ac.nz> Message-ID: On Thu, Sep 29, 2011 at 5:31 PM, Greg Ewing wrote: > Guido has asked me to start a new thread for discussing > this idea. > > To recap, instead of trying to come up with some new > sugar to make the default-argument hack taste slightly > less bitter, I suggest making a small change to the > semantics of for-loops: > > If the loop variable is referenced from an inner scope, > instead of replacing the contents of its cell, create > a *new* cell on each iteration. > > Code following the loop would then continue to see the > last value bound to the loop variable, as now, but > inner functions would capture different versions of > it. That's potentially really nasty from an eval loop point of view. However, I think there's a way to make it work fairly naturally. Currently, the compiler takes all of the following names and effectively mashes them into a flat list: ordinary locals, cell vars (i.e. locals referenced from inner scopes) and free vars (i.e. names from outer lexical scopes). While technically 'free vars' in the normal usage of the term, global and builtin references are handled separately. The bytecode for the function then includes the following kinds of operations: LOAD/STORE_FAST: work on ordinary locals via a numeric index LOAD/STORE_DEREF: work on cells (i.e. cell vars and free vars) via a numeric index LOAD/STORE_GLOBAL: dynamic lookup in the module globals and then builtins by name The key point is that the compiler has enough information to figure all this out at compile time, so any change to loop semantics would need to work in with that. To make the loop rebinding work, it would probably be enough to change for loops and comprehensions to emit a new REPLACE_CELL opcode such that instead of replacing the contents of the existing cell they created a *new* cell and replaced the entire cell. Nested scopes from previous iterations would still have a reference to the old cell but all future references would see the new cell. This wouldn't have any impact on ordinary loops, since the new STORE_CLOSURE would only be used where STORE_DEREF is used currently. If there aren't any nested scopes involved, then STORE_FAST (or STORE_NAME at module or class level) would still get used. However, I'm not sure how we could handle the following pathological case: def outer(): i = 0 def loop(): nonlocal i for i in range(10): def inner(): return i yield inner def shared(): return i return loop, shared >>> loop, shared = outer() >>> [x() for x in [x for x in loop()]] [9, 9, 9, 9, 9, 9, 9, 9, 9, 9] >>> shared() 9 If that inner loop was modified to replace the cells in the STORE_DEREF case then that final call would return 0 rather than 9. If we did this, I think we'd have to make reusing a nonlocal reference as a loop variable a SyntaxError since the two would flatly contradict each other (one says "share via this existing cell" the other says "create a new cell on each iteration"). Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ron3200 at gmail.com Fri Sep 30 03:07:48 2011 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 29 Sep 2011 20:07:48 -0500 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: <1317344868.2369.32.camel@Gutsy> On Thu, 2011-09-29 at 13:01 -0700, Guido van Rossum wrote: > On Thu, Sep 29, 2011 at 12:03 PM, Eric Snow wrote: > > The alternative is to leave nonlocal as just a simple statement, but > > change its behavior when the name is not found inside a containing > > function scope. Currently that is a syntax error. > > For a reason. It would be too easy for a typo to produce the wrong > interpretation. But this is exactly how it works now! The behavior of nonlocal doesn't need to be changed. It already behaves that way with closures. :-) Cheers, Ron (PS... Pleas repost if this doesn't show up on the python-ideas. My emails aren't getting there. My apologies for duplicates.) Python 3.2rc2+ (py3k:88358, Feb 6 2011, 08:56:00) [GCC 4.4.5] on linux2 Type "help", "copyright", "credits" or "license" for more information. Closure as defaults. >>> def a(x): ... def b(y): ... return x+y ... return b ... >>> c = a(5) >>> c(4) 9 >>> c(5) 10 Closures with nonlocal. >>> def a(x): ... def b(y): ... nonlocal x ... x = x + y ... return x ... return b ... >>> c = a(3) >>> c(1) 4 >>> c(2) 6 >>> c(3) 9 Closure won't allow changes to cells without nonlocal. >>> def a(x): ... def b(y): ... x = x + y ... return x ... return b ... >>> c = a(3) >>> c(1) Traceback (most recent call last): File "", line 1, in File "", line 3, in b UnboundLocalError: local variable 'x' referenced before assignment >>> From jxo6948 at rit.edu Fri Sep 30 03:23:26 2011 From: jxo6948 at rit.edu (John O'Connor) Date: Thu, 29 Sep 2011 21:23:26 -0400 Subject: [Python-ideas] Add peekline(), peeklines(n) and optional maxlines argument to readlines() Message-ID: It seems there could be a cleaner way of reading the first n lines of a file and additionally not seeking past those lines (ie peek). This is obviously a trivial task for 1 line ie... f.readline() f.seek(0) but one that I think would make sense to add to the IO implementation, given that we already have readline, readlines, and peek I think peekline() or peeklines(n) is only a natural addition. The argument for doing so (in 3.3 of course), is primarily readability but also that the maintenance burden *seems* like it would be low. This addition would also be helpful and more concise where n > 1. I think readlines() should also take an optional argument for a max number of lines to read. It seems more common/helpful to me than 'hint' for max bytes. In n>1 case one could do... f.readlines(maxlines=10) or for the 'peek' case f.peeklines(10) I also didn't find any of the answers from http://stackoverflow.com/questions/1767513/read-first-n-lines-of-a-file-in-python to be very compelling. I am more than willing to propose a patch if the idea(s) are supported. - John From ncoghlan at gmail.com Fri Sep 30 03:32:12 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 29 Sep 2011 21:32:12 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <1317344868.2369.32.camel@Gutsy> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <1317344868.2369.32.camel@Gutsy> Message-ID: On Thu, Sep 29, 2011 at 9:07 PM, Ron Adam wrote: > On Thu, 2011-09-29 at 13:01 -0700, Guido van Rossum wrote: >> On Thu, Sep 29, 2011 at 12:03 PM, Eric Snow wrote: >> > The alternative is to leave nonlocal as just a simple statement, but >> > change its behavior when the name is not found inside a containing >> > function scope. ?Currently that is a syntax error. >> >> For a reason. It would be too easy for a typo to produce the wrong >> interpretation. > > But this is exactly how it works now! ?The behavior of nonlocal doesn't > need to be changed. ?It already behaves that way with closures. :-) That isn't what Guido and Eric are talking about here. They're talking about this syntax error: >>> def f(): ... nonlocal x ... SyntaxError: no binding for nonlocal 'x' found We had the opportunity in PEP 3104 to make 'nonlocal x' a synonym for 'global x' and chose not to do so. After deliberately passing up that more obvious interpretation, we aren't likely to now decide to use it to denote function state variables. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From wuwei23 at gmail.com Fri Sep 30 04:36:51 2011 From: wuwei23 at gmail.com (alex23) Date: Thu, 29 Sep 2011 19:36:51 -0700 (PDT) Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E84E513.8000202@canterbury.ac.nz> References: <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <4E848816.4060205@stoneleaf.us> <4E84E513.8000202@canterbury.ac.nz> Message-ID: On Sep 30, 7:37?am, Greg Ewing wrote: > Maybe @iwastoolazytocreateaclassforthis would be a > better name? :-) +1000 From ron3200 at gmail.com Fri Sep 30 05:35:26 2011 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 29 Sep 2011 22:35:26 -0500 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <1317344868.2369.32.camel@Gutsy> Message-ID: <1317353726.2369.146.camel@Gutsy> On Thu, 2011-09-29 at 21:32 -0400, Nick Coghlan wrote: > On Thu, Sep 29, 2011 at 9:07 PM, Ron Adam wrote: > > On Thu, 2011-09-29 at 13:01 -0700, Guido van Rossum wrote: > >> On Thu, Sep 29, 2011 at 12:03 PM, Eric Snow wrote: > >> > The alternative is to leave nonlocal as just a simple statement, but > >> > change its behavior when the name is not found inside a containing > >> > function scope. Currently that is a syntax error. > >> > >> For a reason. It would be too easy for a typo to produce the wrong > >> interpretation. > > > > But this is exactly how it works now! The behavior of nonlocal doesn't > > need to be changed. It already behaves that way with closures. :-) > > That isn't what Guido and Eric are talking about here. They're talking > about this syntax error: > > >>> def f(): > ... nonlocal x > ... > SyntaxError: no binding for nonlocal 'x' found > > We had the opportunity in PEP 3104 to make 'nonlocal x' a synonym for > 'global x' and chose not to do so. After deliberately passing up that > more obvious interpretation, we aren't likely to now decide to use it > to denote function state variables. Ah, ok, There are so many suggestions it's hard to keep up. I think, all that is needed is a way to rebind a function inside a decorator. If we can do that, then everything will pretty much just work. And nonlocal should do the right thing as it is. Could a method be added to the Function class, that will copy a function and capture the current scope and any additional cells? class function(object) | function(code, globals[, name[, argdefs[, closure]]]) | | Create a function object from a code object and a dictionary. | The optional name string overrides the name from the code object. | The optional argdefs tuple specifies the default argument values. | The optional closure tuple supplies the bindings for free variables. Looking at this, it may already be able to do that. But it could be easier. How about if we add the ability for it to copy a function, so that we can do... func = type(func)(func) # get a copy func = type(func)(func, closure={'x':0}) # with closures Then we can do... def set_closure(**kwds): def wrapper(f): f = type(f)(f, closure=kwds) return f return wrapper @set_closure(x=0) def adder(y): nonlocal x x += y return x That looks fine to me. def set_this(f): kwds = {'__this__':f} f = type(f)(f, closure={'__this__':f) return f This decorator won't work as the function '__this__' points to is the old function, the new one hasn't been created yet. Hmmm. Cheers, Ron From ron3200 at gmail.com Fri Sep 30 05:40:34 2011 From: ron3200 at gmail.com (ron adam) Date: Thu, 29 Sep 2011 22:40:34 -0500 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <1317353726.2369.146.camel@Gutsy> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <1317344868.2369.32.camel@Gutsy> <1317353726.2369.146.camel@Gutsy> Message-ID: <1317354034.2369.149.camel@Gutsy> On Thu, 2011-09-29 at 22:35 -0500, Ron Adam wrote: > That looks fine to me. > > > > def set_this(f): > kwds = {'__this__':f} > f = type(f)(f, closure={'__this__':f) > return f > > This decorator won't work as the function '__this__' points to is the > old function, the new one hasn't been created yet. Hmmm. Correct that too... def set_this(f): f = type(f)(f, closure={'__this__':f}) return f It still won't work, for the reasons stated. > Cheers, > Ron From cmjohnson.mailinglist at gmail.com Fri Sep 30 05:45:32 2011 From: cmjohnson.mailinglist at gmail.com (Carl Matthew Johnson) Date: Thu, 29 Sep 2011 17:45:32 -1000 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <1317353726.2369.146.camel@Gutsy> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <1317344868.2369.32.camel@Gutsy> <1317353726.2369.146.camel@Gutsy> Message-ID: On Sep 29, 2011, at 5:35 PM, Ron Adam wrote: > @set_closure(x=0) > def adder(y): > nonlocal x > x += y > return x Won't work as written because the nonlocal will cause an error before the decorator is applied. The other day, I suggested we add the ability for decorators to have a __prepare__ method that creates the namespace the function being created will use, but it was shot down as impractical. It would however do what you want to do here. From tjreedy at udel.edu Fri Sep 30 07:06:31 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 30 Sep 2011 01:06:31 -0400 Subject: [Python-ideas] Changing semantics of for-loop variable In-Reply-To: <4E84E3C7.2030102@canterbury.ac.nz> References: <4E802202.4080009@canterbury.ac.nz> <4E8159AB.2030506@canterbury.ac.nz> <465D00FC-D144-4606-AF90-2F5014C4ED3A@gmail.com> <4E817549.7080107@canterbury.ac.nz> <4E84E3C7.2030102@canterbury.ac.nz> Message-ID: On 9/29/2011 5:31 PM, Greg Ewing wrote: > Guido has asked me to start a new thread for discussing > this idea. > > To recap, instead of trying to come up with some new > sugar to make the default-argument hack taste slightly > less bitter, I suggest making a small change to the > semantics of for-loops: > > If the loop variable is referenced from an inner scope, > instead of replacing the contents of its cell, create > a *new* cell on each iteration. Since loop variables do not normally have cells, I really do not understand from this what you are proposing. What I do understand is that you would have the content of the body of a loop change the behavior of the loop. This is a radical change with what to me is little practical justification and not a 'small change'. I have used about 20 languages and in all of them, a 'variable' refers to a specific memory block or or name. You must have a very different background to think that doing something else is normal. > Code following the loop would then continue to see the > last value bound to the loop variable, as now, but > inner functions would capture different versions of > it. > -- Terry Jan Reedy From ron3200 at gmail.com Fri Sep 30 07:25:04 2011 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 30 Sep 2011 00:25:04 -0500 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <1317344868.2369.32.camel@Gutsy> <1317353726.2369.146.camel@Gutsy> Message-ID: <1317360304.4082.22.camel@Gutsy> On Thu, 2011-09-29 at 17:45 -1000, Carl Matthew Johnson wrote: > On Sep 29, 2011, at 5:35 PM, Ron Adam wrote: > > > @set_closure(x=0) > > def adder(y): > > nonlocal x > > x += y > > return x > > Won't work as written because the nonlocal will cause an error before the decorator is applied. You are right! I didn't see that. But, first lets understand what nonlocal actually does. Doing a lot of comparisons, it tells the compiler to generate STORE_DEREF byte codes instead of STORE_FAST byte codes. In otherwords the nonlocal feature can be moved into the Function class constructor. It doesn't need to actually be in the function it self. So the Function class could take a list of "nonlocal" names, along with the a dict of closure names and values. Then, the function the decorator is on, will work as if it had nonlocal statement in it on those names. def set_closure(**kwds): def wrapper(f): f = type(f)(f, closure=kwds, nonlocals=kwds.keys()) return f return wrapper @set_closure(x=0) def adder(y): x += y return x That would get around the nonlocal error. > The other day, I suggested we add the ability for decorators to have a __prepare__ method > that creates the namespace the function being created will use, > but it was shot down as impractical. It would however do what you want to do here. I think it would have the same problems, because the namespace isn't associated to the function at the time it's created. If the above is out, then were back to syntax. How about anonymous wrappers? @@(x=0) def adder(y): nonlocal x x += y return x which would be the same as... def _(x=0): def adder(y): nonlocal x x += y return x return adder adder = _() And of course decorators could be stacked above that. Cheers, Ron From g.rodola at gmail.com Fri Sep 30 11:42:31 2011 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Fri, 30 Sep 2011 11:42:31 +0200 Subject: [Python-ideas] Add peekline(), peeklines(n) and optional maxlines argument to readlines() In-Reply-To: References: Message-ID: +0.5 about f.readlines(maxlines=10); it might be a good addition since the sizehint parameter does not allow an actual prediction of how many lines will be returned. On the other side the two arguments are mutually exclusive, therefore I'm not sure what expect in case both are specified (maybe ValueError, but such kind of APIs always leave me a little skeptical). +0 about f.peeklines(10) as it only saves one line of code: f.peeklines(10) f.seek(0) ...or 2 in case you're not at the beginning of the file. before = f.tell() f.peeklines(10) f.seek(before) Not a great advantage vs. the fact of introducing (and remembering) a new function, in my opinion. Regards, --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ 2011/9/30 John O'Connor > It seems there could be a cleaner way of reading the first n lines of > a file and additionally not seeking past those lines (ie peek). This > is obviously a trivial task for 1 line ie... > f.readline() > f.seek(0) > but one that I think would make sense to add to the IO implementation, > given that we already have readline, readlines, and peek I think > peekline() or peeklines(n) is only a natural addition. The argument > for doing so (in 3.3 of course), is primarily readability but also > that the maintenance burden *seems* like it would be low. This > addition would also be helpful and more concise where n > 1. > > I think readlines() should also take an optional argument for a max > number of lines to read. It seems more common/helpful to me than > 'hint' for max bytes. In n>1 case one could do... > > f.readlines(maxlines=10) > > or for the 'peek' case > > f.peeklines(10) > > > I also didn't find any of the answers from > > http://stackoverflow.com/questions/1767513/read-first-n-lines-of-a-file-in-python > to be very compelling. > > I am more than willing to propose a patch if the idea(s) are supported. > > - John > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Fri Sep 30 11:49:23 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 30 Sep 2011 22:49:23 +1300 Subject: [Python-ideas] PEP 335 Revision 2 (Overloadable Boolean Operators) Message-ID: <4E8590A3.4040004@canterbury.ac.nz> Here's a draft of an update to PEP 335. It includes a couple of fully worked and tested examples, plus discussion of some potential simplifications and ways to optimise the generated bytecode. ------------------------------------------------------------- PEP: 335 Title: Overloadable Boolean Operators Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 29-Aug-2004 Python-Version: Unspecified Post-History: 05-Sep-2004 Abstract ======== This PEP proposes an extension to permit objects to define their own meanings for the boolean operators 'and', 'or' and 'not', and suggests an efficient strategy for implementation. A prototype of this implementation is available for download. Background ========== Python does not currently provide any '__xxx__' special methods corresponding to the 'and', 'or' and 'not' boolean operators. In the case of 'and' and 'or', the most likely reason is that these operators have short-circuiting semantics, i.e. the second operand is not evaluated if the result can be determined from the first operand. The usual technique of providing special methods for these operators therefore would not work. There is no such difficulty in the case of 'not', however, and it would be straightforward to provide a special method for this operator. The rest of this proposal will therefore concentrate mainly on providing a way to overload 'and' and 'or'. Motivation ========== There are many applications in which it is natural to provide custom meanings for Python operators, and in some of these, having boolean operators excluded from those able to be customised can be inconvenient. Examples include: 1. NumPy, in which almost all the operators are defined on arrays so as to perform the appropriate operation between corresponding elements, and return an array of the results. For consistency, one would expect a boolean operation between two arrays to return an array of booleans, but this is not currently possible. There is a precedent for an extension of this kind: comparison operators were originally restricted to returning boolean results, and rich comparisons were added so that comparisons of NumPy arrays could return arrays of booleans. 2. A symbolic algebra system, in which a Python expression is evaluated in an environment which results in it constructing a tree of objects corresponding to the structure of the expression. 3. A relational database interface, in which a Python expression is used to construct an SQL query. A workaround often suggested is to use the bitwise operators '&', '|' and '~' in place of 'and', 'or' and 'not', but this has some drawbacks. The precedence of these is different in relation to the other operators, and they may already be in use for other purposes (as in example 1). There is also the aesthetic consideration of forcing users to use something other than the most obvious syntax for what they are trying to express. This would be particularly acute in the case of example 3, considering that boolean operations are a staple of SQL queries. Rationale ========= The requirements for a successful solution to the problem of allowing boolean operators to be customised are: 1. In the default case (where there is no customisation), the existing short-circuiting semantics must be preserved. 2. There must not be any appreciable loss of speed in the default case. 3. Ideally, the customisation mechanism should allow the object to provide either short-circuiting or non-short-circuiting semantics, at its discretion. One obvious strategy, that has been previously suggested, is to pass into the special method the first argument and a function for evaluating the second argument. This would satisfy requirements 1 and 3, but not requirement 2, since it would incur the overhead of constructing a function object and possibly a Python function call on every boolean operation. Therefore, it will not be considered further here. The following section proposes a strategy that addresses all three requirements. A `prototype implementation`_ of this strategy is available for download. .. _prototype implementation: http://www.cosc.canterbury.ac.nz/~greg/python/obo//Python_OBO.tar.gz Specification ============= Special Methods --------------- At the Python level, objects may define the following special methods. =============== ================= ======================== Unary Binary, phase 1 Binary, phase 2 =============== ================= ======================== * __not__(self) * __and1__(self) * __and2__(self, other) * __or1__(self) * __or2__(self, other) * __rand2__(self, other) * __ror2__(self, other) =============== ================= ======================== The __not__ method, if defined, implements the 'not' operator. If it is not defined, or it returns NotImplemented, existing semantics are used. To permit short-circuiting, processing of the 'and' and 'or' operators is split into two phases. Phase 1 occurs after evaluation of the first operand but before the second. If the first operand defines the relevant phase 1 method, it is called with the first operand as argument. If that method can determine the result without needing the second operand, it returns the result, and further processing is skipped. If the phase 1 method determines that the second operand is needed, it returns the special value NeedOtherOperand. This triggers the evaluation of the second operand, and the calling of a relevant phase 2 method. During phase 2, the __and2__/__rand2__ and __or2__/__ror2__ method pairs work as for other binary operators. Processing falls back to existing semantics if at any stage a relevant special method is not found or returns NotImplemented. As a special case, if the first operand defines a phase 2 method but no corresponding phase 1 method, the second operand is always evaluated and the phase 2 method called. This allows an object which does not want short-circuiting semantics to simply implement the phase 2 methods and ignore phase 1. Bytecodes --------- The patch adds four new bytecodes, LOGICAL_AND_1, LOGICAL_AND_2, LOGICAL_OR_1 and LOGICAL_OR_2. As an example of their use, the bytecode generated for an 'and' expression looks like this:: . . . evaluate first operand LOGICAL_AND_1 L evaluate second operand LOGICAL_AND_2 L: . . . The LOGICAL_AND_1 bytecode performs phase 1 processing. If it determines that the second operand is needed, it leaves the first operand on the stack and continues with the following code. Otherwise it pops the first operand, pushes the result and branches to L. The LOGICAL_AND_2 bytecode performs phase 2 processing, popping both operands and pushing the result. Type Slots ---------- At the C level, the new special methods are manifested as five new slots in the type object. In the patch, they are added to the tp_as_number substructure, since this allows making use of some existing code for dealing with unary and binary operators. Their existence is signalled by a new type flag, Py_TPFLAGS_HAVE_BOOLEAN_OVERLOAD. The new type slots are:: unaryfunc nb_logical_not; unaryfunc nb_logical_and_1; unaryfunc nb_logical_or_1; binaryfunc nb_logical_and_2; binaryfunc nb_logical_or_2; Python/C API Functions ---------------------- There are also five new Python/C API functions corresponding to the new operations:: PyObject *PyObject_LogicalNot(PyObject *); PyObject *PyObject_LogicalAnd1(PyObject *); PyObject *PyObject_LogicalOr1(PyObject *); PyObject *PyObject_LogicalAnd2(PyObject *, PyObject *); PyObject *PyObject_LogicalOr2(PyObject *, PyObject *); Alternatives and Optimisations ============================== This section discusses some possible variations on the proposal, and ways in which the bytecode sequences generated for boolean expressions could be optimised. Reduced special method set -------------------------- For completeness, the full version of this proposal includes a mechanism for types to define their own customised short-circuiting behaviour. However, the full mechanism is not needed to address the main use cases put forward here, and it would be possible to define a simplified version that only includes the phase 2 methods. There would then only be 5 new special methods (__and2__, __rand2__, __or2__, __ror2__, __not__) with 3 associated type slots and 3 API functions. This simplified version could be expanded to the full version later if desired. Additional bytecodes -------------------- As defined here, the bytecode sequence for code that branches on the result of a boolean expression would be slightly longer than it currently is. For example, in Python 2.7, :: if a and b: statement1 else: statement2 generates LOAD_GLOBAL a POP_JUMP_IF_FALSE false_branch LOAD_GLOBAL b POP_JUMP_IF_FALSE false_branch JUMP_FORWARD end_branch false_branch: end_branch: Under this proposal as described so far, it would become something like :: LOAD_GLOBAL a LOGICAL_AND_1 test LOAD_GLOBAL b LOGICAL_AND_2 test: POP_JUMP_IF_FALSE false_branch JUMP_FORWARD end_branch false_branch: end_branch: This involves executing one extra bytecode in the short-circuiting case and two extra bytecodes in the non-short-circuiting case. However, by introducing extra bytecodes that combine the logical operations with testing and branching on the result, it can be reduced to the same number of bytecodes as the original: :: LOAD_GLOBAL a AND1_JUMP true_branch, false_branch LOAD_GLOBAL b AND2_JUMP_IF_FALSE false_branch true_branch: JUMP_FORWARD end_branch false_branch: end_branch: Here, AND1_JUMP performs phase 1 processing as above, and then examines the result. If there is a result, it is popped from the stack, its truth value is tested and a branch taken to one of two locations. Otherwise, the first operand is left on the stack and execution continues to the next bytecode. The AND2_JUMP_IF_FALSE bytecode performs phase 2 processing, pops the result and branches if it tests false For the 'or' operator, there would be corresponding OR1_JUMP and OR2_JUMP_IF_TRUE bytecodes. If the simplified version without phase 1 methods is used, then early exiting can only occur if the first operand is false for 'and' and true for 'or'. Consequently, the two-target AND1_JUMP and OR1_JUMP bytecodes can be replaced with AND1_JUMP_IF_FALSE and OR1_JUMP_IF_TRUE, these being ordinary branch instructions with only one target. Optimisation of 'not' --------------------- Recent versions of Python implement a simple optimisation in which branching on a negated boolean expression is implemented by reversing the sense of the branch, saving a UNARY_NOT opcode. Taking a strict view, this optimisation should no longer be performed, because the 'not' operator may be overridden to produce quite different results from usual. However, in typical use cases, it is not envisaged that expressions involving customised boolean operations will be used for branching -- it is much more likely that the result will be used in some other way. Therefore, it would probably do little harm to specify that the compiler is allowed to use the laws of boolean algebra to simplify any expression that appears directly in a boolean context. If this is inconvenient, the result can always be assigned to a temporary name first. This would allow the existing 'not' optimisation to remain, and would permit future extensions of it such as using De Morgan's laws to extend it deeper into the expression. Usage Examples ============== Example 1: NumPy Arrays ----------------------- :: #----------------------------------------------------------------- # # This example creates a subclass of numpy array to which # 'and', 'or' and 'not' can be applied, producing an array # of booleans. # #----------------------------------------------------------------- from numpy import array, ndarray class BArray(ndarray): def __str__(self): return "barray(%s)" % ndarray.__str__(self) def __and2__(self, other): return (self & other) def __or2__(self, other): return (self & other) def __not__(self): return (self == 0) def barray(*args, **kwds): return array(*args, **kwds).view(type = BArray) a0 = barray([0, 1, 2, 4]) a1 = barray([1, 2, 3, 4]) a2 = barray([5, 6, 3, 4]) a3 = barray([5, 1, 2, 4]) print "a0:", a0 print "a1:", a1 print "a2:", a2 print "a3:", a3 print "not a0:", not a0 print "a0 == a1 and a2 == a3:", a0 == a1 and a2 == a3 print "a0 == a1 or a2 == a3:", a0 == a1 or a2 == a3 Example 1 Output --------------- :: a0: barray([0 1 2 4]) a1: barray([1 2 3 4]) a2: barray([5 6 3 4]) a3: barray([5 1 2 4]) not a0: barray([ True False False False]) a0 == a1 and a2 == a3: barray([False False False True]) a0 == a1 or a2 == a3: barray([False False False True]) Example 2: Database Queries --------------------------- :: #----------------------------------------------------------------- # # This example demonstrates the creation of a DSL for database # queries allowing 'and' and 'or' operators to be used to # formulate the query. # #----------------------------------------------------------------- class SQLNode(object): def __and2__(self, other): return SQLBinop("and", self, other) def __rand2__(self, other): return SQLBinop("and", other, self) def __eq__(self, other): return SQLBinop("=", self, other) class Table(SQLNode): def __init__(self, name): self.__tablename__ = name def __getattr__(self, name): return SQLAttr(self, name) def __sql__(self): return self.__tablename__ class SQLBinop(SQLNode): def __init__(self, op, opnd1, opnd2): self.op = op.upper() self.opnd1 = opnd1 self.opnd2 = opnd2 def __sql__(self): return "(%s %s %s)" % (sql(self.opnd1), self.op, sql(self.opnd2)) class SQLAttr(SQLNode): def __init__(self, table, name): self.table = table self.name = name def __sql__(self): return "%s.%s" % (sql(self.table), self.name) class SQLSelect(SQLNode): def __init__(self, targets): self.targets = targets self.where_clause = None def where(self, expr): self.where_clause = expr return self def __sql__(self): result = "SELECT %s" % ", ".join([sql(target) for target in self.targets]) if self.where_clause: result = "%s WHERE %s" % (result, sql(self.where_clause)) return result def sql(expr): if isinstance(expr, SQLNode): return expr.__sql__() elif isinstance(expr, str): return "'%s'" % expr.replace("'", "''") else: return str(expr) def select(*targets): return SQLSelect(targets) #-------------------------------------------------------------------------------- dishes = Table("dishes") customers = Table("customers") orders = Table("orders") query = select(customers.name, dishes.price, orders.amount).where( customers.cust_id == orders.cust_id and orders.dish_id == dishes.dish_id and dishes.name == "Spam, Eggs, Sausages and Spam") print repr(query) print sql(query) Example 2 Output ---------------- :: <__main__.SQLSelect object at 0x1cc830> SELECT customers.name, dishes.price, orders.amount WHERE (((customers.cust_id = orders.cust_id) AND (orders.dish_id = dishes.dish_id)) AND (dishes.name = 'Spam, Eggs, Sausages and Spam')) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: From greg.ewing at canterbury.ac.nz Fri Sep 30 12:03:55 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 30 Sep 2011 23:03:55 +1300 Subject: [Python-ideas] Changing semantics of for-loop variable In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8159AB.2030506@canterbury.ac.nz> <465D00FC-D144-4606-AF90-2F5014C4ED3A@gmail.com> <4E817549.7080107@canterbury.ac.nz> <4E84E3C7.2030102@canterbury.ac.nz> Message-ID: <4E85940B.70305@canterbury.ac.nz> Nick Coghlan wrote: > I'm not sure how we could handle the following pathological case: > > def outer(): > i = 0 > def loop(): > nonlocal i > for i in range(10): > def inner(): > return i > yield inner > def shared(): > return i > return loop, shared Yeow. That's nasty indeed. Really handling it properly would require cells containing cells, which could get quite tricky. I'm inclined to punt and say that you deserve whatever you get if you do that. In any of the intended use cases, the loop variable will be local. > If we did this, I think we'd have to make reusing a nonlocal reference > as a loop variable a SyntaxError That might be a bit harsh, since it would make currently valid code illegal. Another possibility would be to say that the new semantics only apply when the loop variable is local; if it's declared nonlocal or global, the old semantics apply. -- Greg From ncoghlan at gmail.com Fri Sep 30 13:07:38 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Sep 2011 07:07:38 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <1317360304.4082.22.camel@Gutsy> References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <1317344868.2369.32.camel@Gutsy> <1317353726.2369.146.camel@Gutsy> <1317360304.4082.22.camel@Gutsy> Message-ID: On Fri, Sep 30, 2011 at 1:25 AM, Ron Adam wrote: > In otherwords the nonlocal feature can be moved into the Function class > constructor. ?It doesn't need to actually be in the function it self. It's *really* important to remember the difference between compilation time, definition time and call time in this discussion, since name binding affects all of them. Compilation time (usually at module import): Compiler decides between STORE_FAST (function local), STORE_DEREF (function closure/nonlocal), STORE_GLOBAL (global declaration) and STORE_NAME (class/module scope). Definition time (i.e. when the def statement itself is executed to create the function object): Compiler creates any decorators, any default arguments, loads references to any outer cells on to the stack, loads the code object for the function body, creates the function (passing in the default arguments, cell references and code object), and only then applies the decorators. Call time (i.e. when the function is called): Initialises the execution frame with space for the local variables and references to the cells for variables that may persist beyond this call, binds the arguments to the appropriate locals (potentially filling some in from the default arguments) and then passes the code object to the main eval loop to be executed. The reason nonlocal and global declarations need to exist in the first place is precisely to override the compiler's default assumption that all name bindings in function scope are references to local variables. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From g.rodola at gmail.com Fri Sep 30 13:08:35 2011 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Fri, 30 Sep 2011 13:08:35 +0200 Subject: [Python-ideas] Add peekline(), peeklines(n) and optional maxlines argument to readlines() In-Reply-To: References: Message-ID: Whops! In my peekline examples obviously f.peeklines must be replaced with f.readlines. =) Regards, --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ 2011/9/30 Giampaolo Rodol? > +0.5 about f.readlines(maxlines=10); it might be a good addition since the > sizehint parameter does not allow an actual prediction of how many lines > will be returned. > On the other side the two arguments are mutually exclusive, therefore I'm > not sure what expect in case both are specified (maybe ValueError, but such > kind of APIs always leave me a little skeptical). > +0 about f.peeklines(10) as it only saves one line of code: > > f.peeklines(10) > f.seek(0) > > ...or 2 in case you're not at the beginning of the file. > > before = f.tell() > f.peeklines(10) > f.seek(before) > > Not a great advantage vs. the fact of introducing (and remembering) a new > function, in my opinion. > > Regards, > > --- Giampaolo > http://code.google.com/p/pyftpdlib/ > http://code.google.com/p/psutil/ > > 2011/9/30 John O'Connor > >> It seems there could be a cleaner way of reading the first n lines of >> a file and additionally not seeking past those lines (ie peek). This >> is obviously a trivial task for 1 line ie... >> f.readline() >> f.seek(0) >> but one that I think would make sense to add to the IO implementation, >> given that we already have readline, readlines, and peek I think >> peekline() or peeklines(n) is only a natural addition. The argument >> for doing so (in 3.3 of course), is primarily readability but also >> that the maintenance burden *seems* like it would be low. This >> addition would also be helpful and more concise where n > 1. >> >> I think readlines() should also take an optional argument for a max >> number of lines to read. It seems more common/helpful to me than >> 'hint' for max bytes. In n>1 case one could do... >> >> f.readlines(maxlines=10) >> >> or for the 'peek' case >> >> f.peeklines(10) >> >> >> I also didn't find any of the answers from >> >> http://stackoverflow.com/questions/1767513/read-first-n-lines-of-a-file-in-python >> to be very compelling. >> >> I am more than willing to propose a patch if the idea(s) are supported. >> >> - John >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Sep 30 13:28:19 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Sep 2011 07:28:19 -0400 Subject: [Python-ideas] Add peekline(), peeklines(n) and optional maxlines argument to readlines() In-Reply-To: References: Message-ID: On Fri, Sep 30, 2011 at 5:42 AM, Giampaolo Rodol? wrote: > ...or 2 in case you're not at the beginning of the file. > before = f.tell() > f.peeklines(10) > f.seek(before) A context manager to handle the tell()/seek() may be an interesting and more general purpose idea: # In the io module class _TellSeek: def __init__(self, f): self._f = f def __enter__(self): self._position = self._f.tell() def __exit__(self, *args): self._f.seek(self._position) def restore_position(f): return _TellSeek(f) # Usage with io.restore_position(f): for i, line in enumerate(f, 1): # Do stuff if i == 10: break else: # Oops, didn't get as many lines as we wanted Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From jimjjewett at gmail.com Fri Sep 30 16:17:22 2011 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 30 Sep 2011 10:17:22 -0400 Subject: [Python-ideas] Changing semantics of for-loop variable In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8159AB.2030506@canterbury.ac.nz> <465D00FC-D144-4606-AF90-2F5014C4ED3A@gmail.com> <4E817549.7080107@canterbury.ac.nz> <4E84E3C7.2030102@canterbury.ac.nz> Message-ID: On Fri, Sep 30, 2011 at 1:06 AM, Terry Reedy wrote: > On 9/29/2011 5:31 PM, Greg Ewing wrote: >> If the loop variable is referenced from an inner scope, >> instead of replacing the contents of its cell, create >> a *new* cell on each iteration. > Since loop variables do not normally have cells, I really do not understand > from this what you are proposing. What I do understand is that you would > have the content of the body of a loop change the behavior of the loop. Not really. If the loop (or other) variable is not used in creating closures, then it doesn't really matter whether the variable is stored as a local or a cell -- it happens to be stored as a local for speed reasons. If a variable (including the loop variable) is used inside a closure, then it already creates a cell. This means that loop content already affects the way the loop is compiled, though again, it affects only efficiency, not semantics. The difference is that now the loop will create N separate cells instead of just one, so that each closure will see its own private variable, instead of all sharing the same one. That is a semantic difference, but if the closed-over variable is also a loop variable, it will normally be a bugfix. > I have used about 20 languages and in all of them, a 'variable' refers to a > specific memory block or or name. There are languages (Erlang is the best known) that make "variables" write-once; in those cases, loops will always create N separate "variables". (Expensive on memory; good for concurrency.) This proposal says that python should do so when (and perhaps only when) those separate variable instances are used in different closures. -jJ From jimjjewett at gmail.com Fri Sep 30 16:27:36 2011 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 30 Sep 2011 10:27:36 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> Message-ID: me: >> I don't *like* >> boxing up my variables, but it still seems less offensive than a >> global statement and the resulting side effects. On Thu, Sep 29, 2011 at 4:26 PM, Devin Jeanpierre asked: > How is that? Normally, things at the module level can only be modified by other lines at the module level, or by monkey-patching. The global statement changes that. >>> x=5 # Safe ... no other assignments on the far left ... >>> class T: x=4 # OK, so there is a class level variable def __init__(self): global x # which, alas, isn't what gets used x=3 >>> x 5 >>> T() <__main__.T object at 0x0137A270> >>> x 3 Wait -- how did x get changed? *I* didn't change it, did I? -jJ From jimjjewett at gmail.com Fri Sep 30 17:02:36 2011 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 30 Sep 2011 11:02:36 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <4E84E1FF.9070808@canterbury.ac.nz> References: <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <4E8277DD.5060407@stoneleaf.us> <4E84E1FF.9070808@canterbury.ac.nz> Message-ID: On Thu, Sep 29, 2011 at 5:24 PM, Greg Ewing wrote: > Nick Coghlan wrote: >> As Guido noted, default argument *names* refer to >> ordinary locals, and hence exhibit the same rebinding behaviour as >> pre-nonlocal closure references (i.e. reassignments don't persist >> across calls, only mutations do). Since we expressly *don't* want that >> behaviour, > Are you sure we don't? The ability to persistently re-assign > the name is not needed to address the main use cases under > consideration, as I understand them to be. In fact, disallowing > assignment to the name at all would be fine by me. Hence I > would be +0 on using 'const' as the keyword. What do you think the main use cases are? If you can't rebind, it obviously doesn't solve the Counter case. As best I can tell, const only works for speed hacks. And there are other ways to solve that, if you're willing to use a different VM, like pypy does. -jJ From ncoghlan at gmail.com Fri Sep 30 17:07:38 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Sep 2011 11:07:38 -0400 Subject: [Python-ideas] Changing semantics of for-loop variable In-Reply-To: <4E85940B.70305@canterbury.ac.nz> References: <4E802202.4080009@canterbury.ac.nz> <4E8159AB.2030506@canterbury.ac.nz> <465D00FC-D144-4606-AF90-2F5014C4ED3A@gmail.com> <4E817549.7080107@canterbury.ac.nz> <4E84E3C7.2030102@canterbury.ac.nz> <4E85940B.70305@canterbury.ac.nz> Message-ID: On Fri, Sep 30, 2011 at 6:03 AM, Greg Ewing wrote: > Nick Coghlan wrote: >> If we did this, I think we'd have to make reusing a nonlocal reference >> as a loop variable a SyntaxError > > That might be a bit harsh, since it would make currently valid > code illegal. > > Another possibility would be to say that the new semantics only > apply when the loop variable is local; if it's declared nonlocal > or global, the old semantics apply. Yeah, I think if we decide how to handle the global case, then the same answer can be applied in the nonlocal case. The local case would be straightforward: def func_gen(): for i in range(3): def inner(): return i yield i, inner >>> [i, f() for i, f in list(func_gen())] # Today reports [(0, 2), (1, 2), (2, 2)] [(0, 0), (1, 1), (2, 2)] To maintain semantic equivalence between for loops and while loops, the new semantics would need to affect *all* name binding inside loops for names referenced from inner scopes, not just the iteration variable in for loops (FWIW, this would likely make implementation easier rather than harder): def func_gen(): i = 0 while i < 3: def inner(): return i yield i, inner i += 1 >>> [i, f() for i, f in list(func_gen())] # Today reports [(0, 2), (1, 2), (2, 2)] [(0, 0), (1, 1), (2, 2)] Explicit nonlocal and global declarations would then override the new semantics completely, just as they override ordinary local semantics today. While locals would gain early binding semantics, declared globals and nonlocals would retain late binding semantics: def func_gen(): global i for i in range(3): def inner(): return i yield i, inner >>> [i, f() for i, f in list(func_gen())] # Remains unchanged under new regime [(0, 2), (1, 2), (2, 2)] >>> i 2 Code that used the default argument hack would continue to work under the new regime. Code that deliberately exploited the current late binding semantics would break. I think I'm still overall -1 on the idea, since it creates some rather subtle weirdness with the hidden switch to early binding semantics for locals, but the retention of late binding semantics for nonlocals and globals. While late binding for locals has its problems, it at least has the virtues of consistency and familiarity. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ziade.tarek at gmail.com Fri Sep 30 17:30:18 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 30 Sep 2011 17:30:18 +0200 Subject: [Python-ideas] startsin ? Message-ID: Hey, not sure how people do this, or if I missed something obvious in the stdlib, but I often have this pattern: starts = ('a', 'b', 'c') somestring = 'acapulco' for start in starts: if somestring.startswith(start): print "yeah" So what about a startsin() method, that would iterate over a sequence: if somestring.startsin('a', 'b', 'c'): print "yeah" Implementing it in C should be faster as well same deal with .endswith I guess Cheers Tarek -- Tarek Ziad? | http://ziade.org From dstanek at dstanek.com Fri Sep 30 17:46:38 2011 From: dstanek at dstanek.com (David Stanek) Date: Fri, 30 Sep 2011 11:46:38 -0400 Subject: [Python-ideas] startsin ? In-Reply-To: References: Message-ID: On Fri, Sep 30, 2011 at 11:30 AM, Tarek Ziad? wrote: > Hey, > > not sure how people do this, or if I missed something obvious in the > stdlib, but I often have this pattern: > > starts = ('a', 'b', 'c') > somestring = 'acapulco' > > for start in starts: > if somestring.startswith(start): > print "yeah" > > > So what about a startsin() method, that would iterate over a sequence: > > if somestring.startsin('a', 'b', 'c'): > print "yeah" > > Implementing it in C should be faster as well > > same deal with .endswith I guess > I tend to do something like this a lot; any(somestring.startswith(x) for x in starts) Probably enough that having a method would be nice. -- David blog: http://www.traceback.org twitter: http://twitter.com/dstanek www: http://dstanek.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ziade.tarek at gmail.com Fri Sep 30 17:47:56 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 30 Sep 2011 17:47:56 +0200 Subject: [Python-ideas] startsin ? In-Reply-To: References: Message-ID: On Fri, Sep 30, 2011 at 5:46 PM, David Stanek wrote: ... > I tend to do something like this a lot; > ? any(somestring.startswith(x) for x in starts) > Probably enough that having a method would be nice. Yeah we can have all kind of compressed loops, but I feel like a method would be cool :) Cheers Tarek -- Tarek Ziad? | http://ziade.org From ubershmekel at gmail.com Fri Sep 30 17:49:41 2011 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Fri, 30 Sep 2011 11:49:41 -0400 Subject: [Python-ideas] startsin ? In-Reply-To: References: Message-ID: If accepted, think this could be: 1. Added as a tool in a std lib module. 2. Allow startswith/endswith to accept a tuple. 3. Added as another method to str. I'm +1 for the first option. Either way, there's an O(len_str + len_max_needle) algorithm for this: http://en.m.wikipedia.org/wiki/Aho%E2%80%93Corasick_string_matching_algorithm On Sep 30, 2011 8:30 AM, "Tarek Ziad?" wrote: > Hey, > > not sure how people do this, or if I missed something obvious in the > stdlib, but I often have this pattern: > > starts = ('a', 'b', 'c') > somestring = 'acapulco' > > for start in starts: > if somestring.startswith(start): > print "yeah" > > > So what about a startsin() method, that would iterate over a sequence: > > if somestring.startsin('a', 'b', 'c'): > print "yeah" > > Implementing it in C should be faster as well > > same deal with .endswith I guess > > Cheers > Tarek > > -- > Tarek Ziad? | http://ziade.org > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Sep 30 17:51:39 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 30 Sep 2011 16:51:39 +0100 Subject: [Python-ideas] startsin ? In-Reply-To: References: Message-ID: On 30 September 2011 16:30, Tarek Ziad? wrote: > Hey, > > not sure how people do this, or if I missed something obvious in the > stdlib, but I often have this pattern: > > starts = ('a', 'b', 'c') > somestring = 'acapulco' > > for start in starts: > ? ?if somestring.startswith(start): > ? ? ? ?print "yeah" > > > So what about a startsin() method, that would iterate over a sequence: > > if somestring.startsin('a', 'b', 'c'): > ? ?print "yeah" if any(somestring.startswith(s) for s in starts) print "yeah" > Implementing it in C should be faster as well Probably slightly, but how critical is speed here? > same deal with .endswith I guess A benefit of the any() recipe is that it generalises, so you don't need to have endsin as well, (or whatever else people might think of...) But it is a common pattern, and there's also the case where you want to know *which" pattern matched, which any can't do, so there may be some mileage in this. But it might be simpler just to accept that you've reached the point where a simple regex is the best answer. Yes, now you have two problems, I know :-) Paul. From ncoghlan at gmail.com Fri Sep 30 18:02:47 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Sep 2011 12:02:47 -0400 Subject: [Python-ideas] startsin ? In-Reply-To: References: Message-ID: On Fri, Sep 30, 2011 at 11:30 AM, Tarek Ziad? wrote: > Hey, > > not sure how people do this, or if I missed something obvious in the > stdlib, but I often have this pattern: > > starts = ('a', 'b', 'c') > somestring = 'acapulco' > > for start in starts: > ? ?if somestring.startswith(start): > ? ? ? ?print "yeah" You missed something. startswith() and endswith() already accept tuples of strings to check and have done so since 2.5 [1]: >>> "example".startswith(('c', 'd', 'e')) True [1] http://docs.python.org/library/stdtypes.html#str.startswith Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Fri Sep 30 18:00:19 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 30 Sep 2011 18:00:19 +0200 Subject: [Python-ideas] Add peekline(), peeklines(n) and optional maxlines argument to readlines() References: Message-ID: <20110930180019.466ebbdb@pitrou.net> On Fri, 30 Sep 2011 07:28:19 -0400 Nick Coghlan wrote: > On Fri, Sep 30, 2011 at 5:42 AM, Giampaolo Rodol? wrote: > > ...or 2 in case you're not at the beginning of the file. > > before = f.tell() > > f.peeklines(10) > > f.seek(before) > > A context manager to handle the tell()/seek() may be an interesting > and more general purpose idea: But it still only works on seekable streams. Some time ago I proposed a general prefetch() method that would allow easy protocol-specific buffering on top of non-seekable streams, but there didn't seem to be a lot of enthusiasm at the time: http://mail.python.org/pipermail/python-ideas/2010-September/008179.html Regards Antoine. From ziade.tarek at gmail.com Fri Sep 30 18:03:53 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 30 Sep 2011 18:03:53 +0200 Subject: [Python-ideas] startsin ? In-Reply-To: References: Message-ID: On Fri, Sep 30, 2011 at 6:02 PM, Nick Coghlan wrote: > On Fri, Sep 30, 2011 at 11:30 AM, Tarek Ziad? wrote: >> Hey, >> >> not sure how people do this, or if I missed something obvious in the >> stdlib, but I often have this pattern: >> >> starts = ('a', 'b', 'c') >> somestring = 'acapulco' >> >> for start in starts: >> ? ?if somestring.startswith(start): >> ? ? ? ?print "yeah" > > You missed something. startswith() and endswith() already accept > tuples of strings to check and have done so since 2.5 [1]: > >>>> "example".startswith(('c', 'd', 'e')) > True > > [1] http://docs.python.org/library/stdtypes.html#str.startswith > > Cheers, > Nick. Arrggggg... thanks ! :) > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > -- Tarek Ziad? | http://ziade.org From matt at whoosh.ca Fri Sep 30 18:09:16 2011 From: matt at whoosh.ca (Matt Chaput) Date: Fri, 30 Sep 2011 12:09:16 -0400 Subject: [Python-ideas] startsin ? In-Reply-To: References: Message-ID: <4E85E9AC.8040401@whoosh.ca> On 30/09/2011 11:30 AM, Tarek Ziad? wrote: > not sure how people do this, or if I missed something obvious in the > stdlib, but I often have this pattern: str's interface is a bit cluttered with some questionable methods ("captialize"? "center"? "swapcase"? "title"?) that probably should have been functions in a text module instead of methods. On the other hand, I do think this is a fairly common operation. One thing is that the equivalent of .startsin() for .endswith() would be .endsin(). In English, "ends in" is a variation of "ends with", e.g. "What words end in 'a'?" I think it would be better for startswith/endswith to accept a tuple/list argument. Matt From matt at whoosh.ca Fri Sep 30 18:10:25 2011 From: matt at whoosh.ca (Matt Chaput) Date: Fri, 30 Sep 2011 12:10:25 -0400 Subject: [Python-ideas] startsin ? In-Reply-To: References: Message-ID: <4E85E9F1.6050308@whoosh.ca> On 30/09/2011 12:02 PM, Nick Coghlan wrote: > You missed something. startswith() and endswith() already accept > tuples of strings to check and have done so since 2.5 [1]: Ha ha, should have checked for emails that arrived while I was composing my stupid response before I sent it. Matt From jxo6948 at rit.edu Fri Sep 30 18:13:33 2011 From: jxo6948 at rit.edu (John O'Connor) Date: Fri, 30 Sep 2011 12:13:33 -0400 Subject: [Python-ideas] Add peekline(), peeklines(n) and optional maxlines argument to readlines() In-Reply-To: <20110930180019.466ebbdb@pitrou.net> References: <20110930180019.466ebbdb@pitrou.net> Message-ID: > Some time ago I proposed a general prefetch() method that would allow > easy protocol-specific buffering on top of non-seekable streams There is also the issue for it here http://bugs.python.org/issue12053. - John From songofacandy at gmail.com Fri Sep 30 18:18:20 2011 From: songofacandy at gmail.com (INADA Naoki) Date: Sat, 1 Oct 2011 01:18:20 +0900 Subject: [Python-ideas] strtr? (was startsin ? Message-ID: I think `strtr`_ in php is also very useful when escaping something. _ strtr: http://jp.php.net/manual/en/function.strtr.php For example: .. code-block:: php php> = strtr("foo\\\"bar\\'baz\\\\", array("\\\\"=>"\\", '\\"'=>'"', "\\'"=>"'")); "foo\"bar'baz\\" .. code-block:: python In [1]: "foo\\\"bar\\'baz\\\\".replace('\\"', '"').replace("\\'", "'").replace('\\\\', '\\') Out[1]: 'foo"bar\'baz\\' In Python, lookup of 'replace' method occurs many times and temporary strings is created many times too. It makes Python slower than php. And replacing order may cause very common mistake. .. code-block:: python In [4]: "foo\\\"bar\\'baz\\\\'".replace('\\\\', '\\').replace('\\"', '"').replace("\\'", "'") Out[4]: 'foo"bar\'baz\'' When I wrote HandlerSocket_ client in pure Python. I use dirty hack for speed. http://bazaar.launchpad.net/~songofacandy/+junk/pyhandlersocket/view/head:/handlersocket/client.py I believe Pythonic means simple and efficient. My code is not Pythonic at all. .. _HandlerSocket: https://github.com/ahiguti/HandlerSocket-Plugin-for-MySQL On Sat, Oct 1, 2011 at 12:30 AM, Tarek Ziad? wrote: > Hey, > > not sure how people do this, or if I missed something obvious in the > stdlib, but I often have this pattern: > > starts = ('a', 'b', 'c') > somestring = 'acapulco' > > for start in starts: > ? ?if somestring.startswith(start): > ? ? ? ?print "yeah" > > > So what about a startsin() method, that would iterate over a sequence: > > if somestring.startsin('a', 'b', 'c'): > ? ?print "yeah" > > Implementing it in C should be faster as well > > same deal with .endswith I guess > > Cheers > Tarek > > -- > Tarek Ziad? | http://ziade.org > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- INADA Naoki? From p.f.moore at gmail.com Fri Sep 30 18:21:32 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 30 Sep 2011 17:21:32 +0100 Subject: [Python-ideas] startsin ? In-Reply-To: References: Message-ID: On 30 September 2011 17:02, Nick Coghlan wrote: > On Fri, Sep 30, 2011 at 11:30 AM, Tarek Ziad? wrote: >> Hey, >> >> not sure how people do this, or if I missed something obvious in the >> stdlib, but I often have this pattern: >> >> starts = ('a', 'b', 'c') >> somestring = 'acapulco' >> >> for start in starts: >> ? ?if somestring.startswith(start): >> ? ? ? ?print "yeah" > > You missed something. startswith() and endswith() already accept > tuples of strings to check and have done so since 2.5 [1]: > >>>> "example".startswith(('c', 'd', 'e')) > True > > [1] http://docs.python.org/library/stdtypes.html#str.startswith The time machine strikes again :-) Paul. From zuo at chopin.edu.pl Fri Sep 30 18:45:35 2011 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Fri, 30 Sep 2011 18:45:35 +0200 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <1317360304.4082.22.camel@Gutsy> References: <4E82345A.3010105@canterbury.ac.nz> <1317344868.2369.32.camel@Gutsy> <1317353726.2369.146.camel@Gutsy> <1317360304.4082.22.camel@Gutsy> Message-ID: <20110930164535.GA2286@chopin.edu.pl> Ron Adam dixit (2011-09-30, 00:25): > How about anonymous wrappers? > > @@(x=0) > def adder(y): > nonlocal x > x += y > return x > > which would be the same as... > > def _(x=0): > def adder(y): > nonlocal x > x += y > return x > return adder > adder = _() +1, though I'd rather prefer simply: @(x=0) def adder(y): nonlocal x x += y return x And, of course, if you don't need to rebind the variable, `nonlocal` would not be needed: @(lock=threading.RLock()) def my_foo(): with lock: "do foo" IMHO it is better than @nonlocal because it uses already settled @-decorator idiom and at the same time it does not pretend to be a normal fuction-based decorator. I like it. :) I think it would be not only clear and useful but also beautiful. Cheers. *j From ericsnowcurrently at gmail.com Fri Sep 30 19:39:10 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 30 Sep 2011 11:39:10 -0600 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: <20110930164535.GA2286@chopin.edu.pl> References: <4E82345A.3010105@canterbury.ac.nz> <1317344868.2369.32.camel@Gutsy> <1317353726.2369.146.camel@Gutsy> <1317360304.4082.22.camel@Gutsy> <20110930164535.GA2286@chopin.edu.pl> Message-ID: On Fri, Sep 30, 2011 at 10:45 AM, Jan Kaliszewski wrote: > Ron Adam dixit (2011-09-30, 00:25): > >> How about anonymous wrappers? >> >> ? @@(x=0) >> ? def adder(y): >> ? ? ? nonlocal x >> ? ? ? x += y >> ? ? ? return x >> >> which would be the same as... >> >> ? def _(x=0): >> ? ? ? def adder(y): >> ? ? ? ? ? nonlocal x >> ? ? ? ? ? x += y >> ? ? ? ? ? return x >> ? ? ? return adder >> ? adder = _() > > +1, though I'd rather prefer simply: > > ? ?@(x=0) > ? ?def adder(y): > ? ? ? ?nonlocal x > ? ? ? ?x += y > ? ? ? ?return x > > And, of course, if you don't need to rebind the variable, `nonlocal` > would not be needed: > > ? ?@(lock=threading.RLock()) > ? ?def my_foo(): > ? ? ? ?with lock: > ? ? ? ? ? ?"do foo" > > IMHO it is better than @nonlocal because it uses already settled > @-decorator idiom and at the same time it does not pretend to be a > normal fuction-based decorator. > > I like it. :) I think it would be not only clear and useful but also > beautiful. On its own it actually looks very appealing. And it may still be fine. However, when mixed with decorators, it may be too ambiguous: @(lock=threading.RLock()) @does_something_else def my_foo(): with lock: "do foo" or @does_something @(lock=threading.RLock()) @does_something_else def my_foo(): with lock: "do foo" What does "@(lock=threading.RLock())" do here? It would be easy to miss that it affects "my_foo" (at compile-time) and not the result of "@does_something_else" (at def-time). This is the problem with mixing the syntax for a compile-time directive with that of a def-time directive, particularly when they can show up right next to each other. This is a possible solution: a syntax requirement that no real decorators come after this new syntax. I'm still cautious about the idea of sharing syntax between compile-time and def-time directives. However, default arguments sort of do this already. -eric > > Cheers. > *j > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From mikegraham at gmail.com Fri Sep 30 19:57:14 2011 From: mikegraham at gmail.com (Mike Graham) Date: Fri, 30 Sep 2011 13:57:14 -0400 Subject: [Python-ideas] startsin ? In-Reply-To: References: Message-ID: On Fri, Sep 30, 2011 at 11:46 AM, David Stanek wrote: > I tend to do something like this a lot; > > any(somestring.startswith(x) for x in starts) > > Probably enough that having a method would be nice. > > I wonder if it might be worthwhile to give any and all two-parameter API for predicate functions, so that any(f, xs) is the same as any(f(x) for x in xs) This eliminates some really common boilerplate, but it adds complication and has an ugly API. Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Fri Sep 30 20:11:58 2011 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 30 Sep 2011 13:11:58 -0500 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E82345A.3010105@canterbury.ac.nz> <1317344868.2369.32.camel@Gutsy> <1317353726.2369.146.camel@Gutsy> <1317360304.4082.22.camel@Gutsy> <20110930164535.GA2286@chopin.edu.pl> Message-ID: <1317406318.7015.5.camel@Gutsy> On Fri, 2011-09-30 at 11:39 -0600, Eric Snow wrote: > On Fri, Sep 30, 2011 at 10:45 AM, Jan Kaliszewski wrote: > > Ron Adam dixit (2011-09-30, 00:25): > > > >> How about anonymous wrappers? > >> > >> @@(x=0) > >> def adder(y): > >> nonlocal x > >> x += y > >> return x > >> > >> which would be the same as... > >> > >> def _(x=0): > >> def adder(y): > >> nonlocal x > >> x += y > >> return x > >> return adder > >> adder = _() > > > > +1, though I'd rather prefer simply: > > > > @(x=0) > > def adder(y): > > nonlocal x > > x += y > > return x > > > > And, of course, if you don't need to rebind the variable, `nonlocal` > > would not be needed: > > > > @(lock=threading.RLock()) > > def my_foo(): > > with lock: > > "do foo" > > > > IMHO it is better than @nonlocal because it uses already settled > > @-decorator idiom and at the same time it does not pretend to be a > > normal fuction-based decorator. > > > > I like it. :) I think it would be not only clear and useful but also > > beautiful. > > On its own it actually looks very appealing. And it may still be > fine. However, when mixed with decorators, it may be too ambiguous: > > @(lock=threading.RLock()) > @does_something_else > def my_foo(): > with lock: > "do foo" > > or > > @does_something > @(lock=threading.RLock()) > @does_something_else > def my_foo(): > with lock: > "do foo" > > What does "@(lock=threading.RLock())" do here? It would be easy to > miss that it affects "my_foo" (at compile-time) and not the result of > "@does_something_else" (at def-time). This is the problem with mixing > the syntax for a compile-time directive with that of a def-time > directive, particularly when they can show up right next to each > other. I was thinking it would only be allowed just before def statement, (Thats why I used '@@' instead of just '@') If it's out of place then it would be treated just like any other decorator. >>> @_() ... @deco ... def foo():pass ... Traceback (most recent call last): File "", line 1, in NameError: name '_' is not defined And of course cause an error. ;-) (Just maybe not that exact one.) Cheers, Ron > This is a possible solution: a syntax requirement that no real > decorators come after this new syntax. I'm still cautious about the > idea of sharing syntax between compile-time and def-time directives. > However, default arguments sort of do this already. > > -eric From ncoghlan at gmail.com Fri Sep 30 20:14:35 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Sep 2011 14:14:35 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E82345A.3010105@canterbury.ac.nz> <1317344868.2369.32.camel@Gutsy> <1317353726.2369.146.camel@Gutsy> <1317360304.4082.22.camel@Gutsy> <20110930164535.GA2286@chopin.edu.pl> Message-ID: On Fri, Sep 30, 2011 at 1:39 PM, Eric Snow wrote: > This is a possible solution: a syntax requirement that no real > decorators come after this new syntax. ?I'm still cautious about the > idea of sharing syntax between compile-time and def-time directives. > However, default arguments sort of do this already. All name binding operations have both compile time and runtime semantics, so that's OK. 'nonlocal' and 'global' are actually the odd ones out since they *only* affect compile time and don't actually correspond directly to anything in the generated bytecode (instead influencing the way the names they specify get treated in other parts of the code). If a "function state decorator" approach is used, then yeah, I agree it should come immediately before the main part of the function header. However, I'm not seeing a lot to recommend that kind of syntax over the post-arguments '[]' approach. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ericsnowcurrently at gmail.com Fri Sep 30 21:01:44 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 30 Sep 2011 13:01:44 -0600 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <1317344868.2369.32.camel@Gutsy> <1317353726.2369.146.camel@Gutsy> <1317360304.4082.22.camel@Gutsy> Message-ID: On Fri, Sep 30, 2011 at 5:07 AM, Nick Coghlan wrote: > On Fri, Sep 30, 2011 at 1:25 AM, Ron Adam wrote: >> In otherwords the nonlocal feature can be moved into the Function class >> constructor. ?It doesn't need to actually be in the function it self. > > It's *really* important to remember the difference between compilation > time, definition time and call time in this discussion, since name > binding affects all of them. +1 > > Compilation time (usually at module import): > ? ?Compiler decides between STORE_FAST (function local), STORE_DEREF > (function closure/nonlocal), STORE_GLOBAL (global declaration) and > STORE_NAME (class/module scope). Good point. And it's important to reiterate that compilation only results in code objects. When a module is compiled, the bodies of all function defs (and class defs and comprehensions) therein are compiled separately (but relative to the module's code). The resulting code objects are put into the co_consts of the module's code object. You'll find this if you look at a pyc file[1]. Likewise, if you have a nested function, the inner function's code object will be stored in the co_consts of the outer function's code object. All this was a mystery to me until recently. > > Definition time (i.e. when the def statement itself is executed to > create the function object): > ? ?Compiler creates any decorators, any default arguments, loads > references to any outer cells on to the stack, loads the code object > for the function body, creates the function (passing in the default > arguments, cell references and code object), and only then applies the > decorators. > > Call time (i.e. when the function is called): > ? ?Initialises the execution frame with space for the local variables > and references to the cells for variables that may persist beyond this > call, binds the arguments to the appropriate locals (potentially > filling some in from the default arguments) and then passes the code > object to the main eval loop to be executed. > > The reason nonlocal and global declarations need to exist in the first > place is precisely to override the compiler's default assumption that > all name bindings in function scope are references to local variables. Exactly! That's a critical point to the discussion that isn't so obvious. Compilation pre-computes a bunch of things that are used at call-time and these are stored on the code object. Two examples are the "space for the local variables" and the size of the frame's stack, which are partially dependent on the local/nonlocal nature of the various names used in the function body. Also, all the names in the signature or used in the body were mapped to indices at compile-time (by virtue of being stored in various tuples). The actual compiled code (in co_code) refers to these indices, which the eval loop uses when pulling values from the appropriate appropriate tuples. Those tuples were pulled from function attributes like __closure__ and __defaults__. So, any solution here must reflect that the compiler needs a way to calculate these things. On the flip side, any solution that relies on def-time hacks of the function object must take into account the subsequent changes that will be necessary on the function's code object. Particularly important is the relationship between __closure__ and co_freevars[2]. It's part of why turning a local into a closed variable is not a trivial thing. I'll admit that I may have gotten some of this wrong, not having years of experience to back me up here. So I definitely defer to knowledge of Nick, et al. I have been pretty immersed in this stuff for several months and the above is my resultant perspective, that of someone who has dived in without a lot of background in it. :) -eric [1] See http://code.activestate.com/recipes/577880/ [2] http://hg.python.org/cpython/file/default/Include/funcobject.h#l34 http://hg.python.org/cpython/file/default/Objects/funcobject.c#l454 > > Cheers, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From ncoghlan at gmail.com Fri Sep 30 21:21:46 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Sep 2011 15:21:46 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <1317344868.2369.32.camel@Gutsy> <1317353726.2369.146.camel@Gutsy> <1317360304.4082.22.camel@Gutsy> Message-ID: On Fri, Sep 30, 2011 at 3:01 PM, Eric Snow wrote: > On Fri, Sep 30, 2011 at 5:07 AM, Nick Coghlan wrote: > > I'll admit that I may have gotten some of this wrong, not having years > of experience to back me up here. ?So I definitely defer to knowledge > of Nick, et al. ?I have been pretty immersed in this stuff for several > months and the above is my resultant perspective, that of someone who > has dived in without a lot of background in it. :) > Heh, it's not like all this is something that comes up every day - the compiler and eval loop mostly sit in the corner chugging away quietly without bothering anyone. The main advantage of experience is knowing what all the moving parts are and where in the code base to look to refresh my recollection of any details I've forgotten :) The only comment I'd make about your explanation is that a lot of those details aren't actually part of the language spec - they're implementation details of CPython that other implementations may happen to follow because it's a reasonable way to do things. They can still be a useful intuition pump in helping to figure out what is and isn't feasible in the language design, though. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From jimjjewett at gmail.com Fri Sep 30 21:36:32 2011 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 30 Sep 2011 15:36:32 -0400 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <4E802202.4080009@canterbury.ac.nz> <4E8112D2.809@canterbury.ac.nz> <4E8148B6.3060408@gmail.com> <4E82345A.3010105@canterbury.ac.nz> <1317344868.2369.32.camel@Gutsy> <1317353726.2369.146.camel@Gutsy> <1317360304.4082.22.camel@Gutsy> Message-ID: On Fri, Sep 30, 2011 at 5:07 AM, Nick Coghlan wrote: > The reason nonlocal and global declarations need to exist in the first > place is precisely to override the compiler's default assumption that > all name bindings in function scope are references to local variables. And something that I missed the first time -- even if you know a name shows up in the closure, that doesn't prevent it from being shadowed locally inside the function. So the function responsible for adjusting another function's bytecode (or other function representation) really does have to know both the contents of the closure AND which names are being removed from the locals. -jJ From zuo at chopin.edu.pl Fri Sep 30 23:21:57 2011 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Fri, 30 Sep 2011 23:21:57 +0200 Subject: [Python-ideas] Proposed additions to inspect module for function and generator state introspection In-Reply-To: References: Message-ID: <20110930212156.GA3996@chopin.edu.pl> Nick Coghlan dixit (2011-09-29, 13:44): > Based on the testability comments in the closure threads, I created > http://bugs.python.org/issue13062 to propose two new introspection > functions: > > inspect.getclosure(func) > Returns a dictionary mapping closure references from the supplied > function to their current values. > > inspect.getgeneratorlocals(generator) > Returns the same result as would be reported by calling locals() > in the generator's frame of execution The only thing I can type in here is +1 Cheers. *j From zuo at chopin.edu.pl Fri Sep 30 23:32:31 2011 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Fri, 30 Sep 2011 23:32:31 +0200 Subject: [Python-ideas] Tweaking closures and lexical scoping to include the function being defined In-Reply-To: References: <1317344868.2369.32.camel@Gutsy> <1317353726.2369.146.camel@Gutsy> <1317360304.4082.22.camel@Gutsy> <20110930164535.GA2286@chopin.edu.pl> Message-ID: <20110930213231.GB3996@chopin.edu.pl> Nick Coghlan dixit (2011-09-30, 14:14): > If a "function state decorator" approach is used, then yeah, I agree > it should come immediately before the main part of the function > header. Yes, it seems to be a necessary requirement. > However, I'm not seeing a lot to recommend that kind of syntax > over the post-arguments '[]' approach. IMHO @(...) has two advantages over '[]' approach: 1. It keeps all that 'additional scope'-stuff in a separate line, making the code probably more clear visualy, and not making the crowd of elements in the '...):' line even more dense (especially if we did use annotations, e.g.: '...) -> "my annotation":'). 2. It is more consistent with the existing syntax (not introducing '='-based syntax within []; [] are already used for two different things: list literals and item lookup -- both somehow leading your thoughts to sequence/container-related stuff). Cheers. *j