From jcarlson at uci.edu Mon Jan 1 23:03:56 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 01 Jan 2007 14:03:56 -0800 Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations: overloadable -> In-Reply-To: References: <4599791F.9020505@acm.org> Message-ID: <20070101133749.82F5.JCARLSON@uci.edu> To Tony and Kay, my short answer is: use __returns__ . Moving to python-ideas as per Guido's request: "Guido van Rossum" wrote: > This is sufficiently controversial that I believe it ought to go to > python-ideas first. If it comes to a PEP it should be a separate one > from PEP 3107. > On 1/1/07, Talin wrote: > > Tony Lownds wrote: > > >> From: Tony Lownds > > > What do people here think? > > 1) Normally, we don't name operators based on their shape - we don't > > call '/' the __slash__ operator, for example, nor do we call '|' the > > "__vbar__" operator. Certainly, but those two operators, and basically every other operator used in Python have long-standing semantics in basically every language that Python is even remotely related to. > > 2) I think that all operators should have a "suggested semantic". When > > someone overloads the '+' operator, its a good bet that the meaning of > > the overload has something to do with addition or accumulation in a > > general sense. This won't *always* be true, but it will be true often > > enough. I don't buy your "suggested semantic" argument. And even if I did, operator overloading allows people to choose the semantics of operations for themselves; suggesting a semantic for an operation would be a set of documentation that would never be read, and if it was read, ignored. > > But an arbitrary operator with no guidelines as to what it means is > > anyone's guess; It means that when we see a '->' operator embedded in > > the code, we have no idea what is being said. Ahh, but in this case there is precisely one place where the '->' operator is planned on being found for Py3k: def () -> : In that sense, we don't need a fcn_obj.__becomes__() method, and that wasn't what the discussion was about, it was "what should the attribute be called for this already agreed upon *function annotation*?". Now, because this *particular* annotation was created to allow for the annotation of "returns", I agree with Kay's last suggestion, the attribute should be called __returns__, as that is *exactly* what the annotation was meant to convey. > > From an HCI perspective, punctuation symbols improve code readability > > only if their meanings are familiar to the reader; An operator whose > > meaning is constantly changing is a hindrance to readability rather than > > a help. Claiming "we want a particular operation to always refer to the same method/attribute" is only applicable if a particular operation has a chance of meaning more than one thing. Currently it means *exactly* one thing in the context of Python, 'this function returns X', so in my opinion, your argument isn't applicable. If you can manage to convince more people in python-ideas of arbitrary operations (as per your previous message(s) on the subject), and/or you can convince Guido to say "I would like more operations in Python", then your argument is applicable. However, I don't believe that you will be able to convince Guido that the large set of operations that you have previously posted about would be a good idea, and I certainly don't believe it would happen with sufficient time to make it into the first Py3k release. - Josiah From talin at acm.org Mon Jan 1 23:25:22 2007 From: talin at acm.org (Talin) Date: Mon, 01 Jan 2007 14:25:22 -0800 Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations: overloadable -> In-Reply-To: <20070101133749.82F5.JCARLSON@uci.edu> References: <4599791F.9020505@acm.org> <20070101133749.82F5.JCARLSON@uci.edu> Message-ID: <45998A52.9060801@acm.org> Josiah Carlson wrote: > To Tony and Kay, my short answer is: use __returns__ . > > Moving to python-ideas as per Guido's request: > > "Guido van Rossum" wrote: >> This is sufficiently controversial that I believe it ought to go to >> python-ideas first. If it comes to a PEP it should be a separate one >> from PEP 3107. > >> On 1/1/07, Talin wrote: >>> Tony Lownds wrote: >>>>> From: Tony Lownds >>>> What do people here think? >>> 1) Normally, we don't name operators based on their shape - we don't >>> call '/' the __slash__ operator, for example, nor do we call '|' the >>> "__vbar__" operator. > > Certainly, but those two operators, and basically every other operator > used in Python have long-standing semantics in basically every language > that Python is even remotely related to. > > >>> 2) I think that all operators should have a "suggested semantic". When >>> someone overloads the '+' operator, its a good bet that the meaning of >>> the overload has something to do with addition or accumulation in a >>> general sense. This won't *always* be true, but it will be true often >>> enough. > > I don't buy your "suggested semantic" argument. And even if I did, > operator overloading allows people to choose the semantics of operations > for themselves; suggesting a semantic for an operation would be a set of > documentation that would never be read, and if it was read, ignored. > > >>> But an arbitrary operator with no guidelines as to what it means is >>> anyone's guess; It means that when we see a '->' operator embedded in >>> the code, we have no idea what is being said. > > Ahh, but in this case there is precisely one place where the '->' > operator is planned on being found for Py3k: > > def () -> : > > > In that sense, we don't need a fcn_obj.__becomes__() method, > and that wasn't what the discussion was about, it was "what should the > attribute be called for this already agreed upon *function annotation*?". > > Now, because this *particular* annotation was created to allow for the > annotation of "returns", I agree with Kay's last suggestion, the > attribute should be called __returns__, as that is *exactly* what the > annotation was meant to convey. > > >>> From an HCI perspective, punctuation symbols improve code readability >>> only if their meanings are familiar to the reader; An operator whose >>> meaning is constantly changing is a hindrance to readability rather than >>> a help. > > Claiming "we want a particular operation to always refer to the same > method/attribute" is only applicable if a particular operation has a > chance of meaning more than one thing. Currently it means *exactly* one > thing in the context of Python, 'this function returns X', so in my > opinion, your argument isn't applicable. > > If you can manage to convince more people in python-ideas of arbitrary > operations (as per your previous message(s) on the subject), and/or you > can convince Guido to say "I would like more operations in Python", then > your argument is applicable. > > However, I don't believe that you will be able to convince Guido that > the large set of operations that you have previously posted about would > be a good idea, and I certainly don't believe it would happen with > sufficient time to make it into the first Py3k release. > > > - Josiah Sorry, I'm a habitual generalizer :) Anyway, I'm not going to push on this one any further, neither here nor in python-ideas. There are more important things to work on. -- Talin From tony at PageDNA.com Mon Jan 1 23:28:11 2007 From: tony at PageDNA.com (Tony Lownds) Date: Mon, 1 Jan 2007 14:28:11 -0800 Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations: overloadable -> In-Reply-To: <20070101133749.82F5.JCARLSON@uci.edu> References: <4599791F.9020505@acm.org> <20070101133749.82F5.JCARLSON@uci.edu> Message-ID: <571A7E3A-297F-40DA-86C0-E5E07F234B13@PageDNA.com> On Jan 1, 2007, at 2:03 PM, Josiah Carlson wrote: > > To Tony and Kay, my short answer is: use __returns__ . > Thanks Talin's arguments for a consistent semantic make me think that __implies__ would be a bad idea (among other things). Is anyone against "->" overloadable as __returns__ /__rreturns__ with no semantics for existing types? -Tony From tony at PageDNA.com Tue Jan 2 00:17:36 2007 From: tony at PageDNA.com (Tony Lownds) Date: Mon, 1 Jan 2007 15:17:36 -0800 Subject: [Python-ideas] PEP 3107 Function Annotations: interoperability (again) Message-ID: In discussing the Function Annotations PEP on python-list, interoperability between schemes came up again: http://mail.python.org/pipermail/python-list/2006-December/420645.html John Roth wrote: > Third, it's half of a proposal. Type checking isn't the only use > for metadata about functions/methods, classes, properties > and other objects, and the notion that there are only going to > be a small number of non-intersecting libraries out there is > an abdication of responsibility to think this thing through. This issue came up before in http://mail.python.org/pipermail/python-3000/2006-August/002796.html and a rather long thread followed. Here is the paragraph in the PEP that needs updating, at the least: There is no worry that these libraries will assign semantics at random, or that a variety of libraries will appear, each with varying semantics and interpretations of what, say, a tuple of strings means. The difficulty inherent in writing annotation interpreting libraries will keep their number low and their authorship in the hands of people who, frankly, know what they're doing. The notion that libraries don't intersect should be stripped from the PEP. The question in my mind is whether this PEP needs to take responsibility for interoperability. I contend that people who design an annotation-consuming library that ends up intersecting several others will likely be capable of finding a solution even if not ideal without a central mechanism, and will be far better situated to define a solution for a central mechanism. Any thoughts? Thanks -Tony From brett at python.org Tue Jan 2 01:17:27 2007 From: brett at python.org (Brett Cannon) Date: Mon, 1 Jan 2007 16:17:27 -0800 Subject: [Python-ideas] PEP 3107 Function Annotations: interoperability (again) In-Reply-To: References: Message-ID: On 1/1/07, Tony Lownds wrote: > > In discussing the Function Annotations PEP on python-list, > interoperability > between schemes came up again: > > http://mail.python.org/pipermail/python-list/2006-December/420645.html > > John Roth wrote: > > Third, it's half of a proposal. Type checking isn't the only use > > for metadata about functions/methods, classes, properties > > and other objects, and the notion that there are only going to > > be a small number of non-intersecting libraries out there is > > an abdication of responsibility to think this thing through. > > This issue came up before in > http://mail.python.org/pipermail/python-3000/2006-August/002796.html > and a rather long thread followed. Here is the paragraph in the PEP that > needs updating, at the least: > > There is no worry that these libraries will assign semantics at > random, or that a variety of libraries will appear, each with > varying semantics and interpretations of what, say, a tuple of > strings means. The difficulty inherent in writing annotation > interpreting libraries will keep their number low and their > authorship in the hands of people who, frankly, know what they're > doing. > > The notion that libraries don't intersect should be stripped from the > PEP. The question in my mind is whether this PEP needs to take > responsibility for interoperability. > > I contend that people who design an annotation-consuming library > that ends up intersecting several others will likely be capable of > finding > a solution even if not ideal without a central mechanism, and will be > far > better situated to define a solution for a central mechanism. > > Any thoughts? Until extensive usage happens with annotations we shouldn't try to shoehorn consumers of the annotations into a specific solution. As you said, something will probably organically grow and that can be supported in another PEP later on (probably with stdlib support). People tend to want to design up front but with something like this that is meant to be highly customizable by the people wanting to use it you just don't get that option without ample feedback from users. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jan 2 01:20:40 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 1 Jan 2007 16:20:40 -0800 Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations: overloadable -> In-Reply-To: <571A7E3A-297F-40DA-86C0-E5E07F234B13@PageDNA.com> References: <4599791F.9020505@acm.org> <20070101133749.82F5.JCARLSON@uci.edu> <571A7E3A-297F-40DA-86C0-E5E07F234B13@PageDNA.com> Message-ID: On 1/1/07, Tony Lownds wrote: > Is anyone against "->" overloadable as __returns__ /__rreturns__ > with no semantics for existing types? I suggest that you question *really* *hard* whether it's worth it. We'd be introducing two very different syntactic uses for '->'. I chose this operator to signify function return annotation specifically because it *does't* have another meaning in Python. (My first choice would've been ':', which is mostly a delimiter that derives its meaning from context, like ',', but that would've introduced too much syntactic ambiguity. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Jan 2 01:40:33 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 1 Jan 2007 16:40:33 -0800 Subject: [Python-ideas] PEP 3107 Function Annotations: interoperability (again) In-Reply-To: References: Message-ID: Well, it's your PEP, isn't it? Just submit a change to the PEP editor. If it were me, any suggestion of how annotations could/should be used should be stricken from the PEP, including the example def foo(a: int, b: dict, c: int = 5): The only place where an example like this could be mentioned would be a motivational section which could suffice by referring to Collin's existing type annotation library and explaining how it could be made more elegant by attaching the types directly to the arguments. The interop issue needn't be mentioned -- I imagine it's not easy to ensure interop between something like Collin's current library and some other library that makes extensive use of decorators, either. A later PEP, based on actual experience, could propose an interop convention to be used by frameworks that expect to be needing it. To me the current fuss about interop sounds like over-analyzing the situation based on zero data points. Remember YAGNI. --Guido On 1/1/07, Tony Lownds wrote: > In discussing the Function Annotations PEP on python-list, > interoperability > between schemes came up again: > > http://mail.python.org/pipermail/python-list/2006-December/420645.html > > John Roth wrote: > > Third, it's half of a proposal. Type checking isn't the only use > > for metadata about functions/methods, classes, properties > > and other objects, and the notion that there are only going to > > be a small number of non-intersecting libraries out there is > > an abdication of responsibility to think this thing through. > > This issue came up before in > http://mail.python.org/pipermail/python-3000/2006-August/002796.html > and a rather long thread followed. Here is the paragraph in the PEP that > needs updating, at the least: > > There is no worry that these libraries will assign semantics at > random, or that a variety of libraries will appear, each with > varying semantics and interpretations of what, say, a tuple of > strings means. The difficulty inherent in writing annotation > interpreting libraries will keep their number low and their > authorship in the hands of people who, frankly, know what they're > doing. > > The notion that libraries don't intersect should be stripped from the > PEP. The question in my mind is whether this PEP needs to take > responsibility for interoperability. > > I contend that people who design an annotation-consuming library > that ends up intersecting several others will likely be capable of > finding > a solution even if not ideal without a central mechanism, and will be > far > better situated to define a solution for a central mechanism. > > Any thoughts? > > Thanks > -Tony > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Tue Jan 2 02:39:17 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 01 Jan 2007 17:39:17 -0800 Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations: overloadable -> In-Reply-To: References: <571A7E3A-297F-40DA-86C0-E5E07F234B13@PageDNA.com> Message-ID: <20070101172715.82F8.JCARLSON@uci.edu> "Guido van Rossum" wrote: > > On 1/1/07, Tony Lownds wrote: > > Is anyone against "->" overloadable as __returns__ /__rreturns__ > > with no semantics for existing types? > > I suggest that you question *really* *hard* whether it's worth it. > We'd be introducing two very different syntactic uses for '->'. I > chose this operator to signify function return annotation specifically > because it *does't* have another meaning in Python. (My first choice > would've been ':', which is mostly a delimiter that derives its > meaning from context, like ',', but that would've introduced too much > syntactic ambiguity. Just to clarify, the only place '->' is allowable is in the case of function annotations... def () -> : Given a reading of version 53169 of PEP 3107, I see no reason to even offer a __returns__ attribute, automatic method call, etc. PEP 3107 already defines the annotation to be specified as a dictionary of named argument names, with a key of 'return' for the annotation of the return annotation. Is there any reason why this isn't sufficient? Even in the context of the signature PEP 362, there is no need for __returns__, as an annotation-consuming library would check the signature object for argument annotations, and the func_annotations dictionary for the 'return' key for any return annotations. - Josiah From tony at PageDNA.com Tue Jan 2 03:53:48 2007 From: tony at PageDNA.com (Tony Lownds) Date: Mon, 1 Jan 2007 18:53:48 -0800 Subject: [Python-ideas] Attribute Docstrings and Annotations Message-ID: I'd like to propose annotations and docstrings on attributes for Python 3000 with the following Grammar and properties: expr_stmt: test (':' test ['=' (yield_expr|testlist)] | augassign (yield_expr|testlist) | [',' testlist] ('=' (yield_expr|testlist))* ) * annotations can appear for attributes that are not defined yet * code to generate and populate __annotations__ and __attrdoc__ would appear for all modules and class bodies, not for functions. * attribute annotations allow a name as target only * attribute annotations without assignment are illegal for functions * attribute annotations with assignment should probably be illegal for functions * docstring annotations only apply to the first target, and only if it is a name * docstring annotations do not apply to augmented assignments * docstring and annotations on functions do not get populated in __annotations__ * the class docstring is not reused as a function docstring The basic rationale for annotations on attributes is completeness with PEP3107. I'm proposing attribute docstrings as well because I think it's preferable to have a spot for documentation that isn't tied to annotations, like functions' __doc__. Also attribute docstrings as specified look similar to function docstrings; both have statements consisting of only a string that is taken as documentation. Here is an interactive session and the code that might be generated. What do you think? Thanks -Tony >>> class X: ... "class docstring" ... foo: 1 = 1 ... bar: 2 ... "attribute docstring" ... attr = None ... "another attribute docstring" ... fields = __slots__ = ['fields', 'attr'] ... "docstring ignored" ... x, y = 1, 2 ... >>> X.__attrdoc__ {'fields': 'another attribute docstring', 'attr': 'attribute docstring'} >>> X.__annotations__ {'foo': 1, 'bar': 2} >>> X.foo 1 >>> X.bar Traceback (most recent call last): File "", line 1, in AttributeError: type object 'X' has no attribute 'bar' >>> def f(): ... x: 1 ... File "", line 2 SyntaxError: annotation without assignment not allowed in function 2 0 LOAD_NAME 0 (__name__) 3 STORE_NAME 1 (__module__) 6 BUILD_MAP 0 9 STORE_NAME 2 (__annotations__) 12 BUILD_MAP 0 15 STORE_NAME 3 (__attrdoc__) 3 18 LOAD_CONST 0 ('class docstring') 21 STORE_NAME 4 (__doc__) 5 24 LOAD_CONST 1 ('attribute docstring') 27 LOAD_NAME 3 (__attrdoc__) 30 LOAD_CONST 2 ('attr') 33 STORE_SUBSCR 34 LOAD_NAME 5 (None) 37 STORE_NAME 6 (attr) 6 40 LOAD_CONST 3 (1) 48 STORE_NAME 7 (foo) 44 LOAD_CONST 3 (1) 51 LOAD_NAME 2 (__annotations__) 54 LOAD_CONST 4 ('foo') 57 STORE_SUBSCR 7 58 LOAD_CONST 5 (2) 61 LOAD_NAME 2 (__annotations__) 64 LOAD_CONST 6 ('bar') 67 STORE_SUBSCR 9 68 LOAD_CONST 7 ('another attribute docstring') 71 LOAD_NAME 3 (__attrdoc__) 74 LOAD_CONST 8 ('fields') 77 STORE_SUBSCR 78 LOAD_CONST 8 ('fields') 81 LOAD_CONST 2 ('attr') 84 BUILD_LIST 2 87 DUP_TOP 88 STORE_NAME 8 (fields) 91 STORE_NAME 9 (__slots__) 94 LOAD_LOCALS 95 RETURN_VALUE From tony at pagedna.com Tue Jan 2 04:28:07 2007 From: tony at pagedna.com (Tony Lownds) Date: Mon, 1 Jan 2007 19:28:07 -0800 Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations: overloadable -> In-Reply-To: <20070101172715.82F8.JCARLSON@uci.edu> References: <571A7E3A-297F-40DA-86C0-E5E07F234B13@PageDNA.com> <20070101172715.82F8.JCARLSON@uci.edu> Message-ID: <78364BAD-1C96-4D15-927E-983DB909FBF5@pagedna.com> On Jan 1, 2007, at 5:39 PM, Josiah Carlson wrote: > Given a reading of version 53169 of PEP 3107, I see no reason to even > offer a __returns__ attribute, automatic method call, etc. PEP 3107 > already defines the annotation to be specified as a dictionary of > named > argument names, with a key of 'return' for the annotation of the > return > annotation. Is there any reason why this isn't sufficient? > > Even in the context of the signature PEP 362, there is no need for > __returns__, as an annotation-consuming library would check the > signature object for argument annotations, and the func_annotations > dictionary for the 'return' key for any return annotations. Something got lost in the movement between lists... __returns__ wasn't going to supplant or alter func_annotations or PEP 362 in any way. __returns__ was proposed as the special method attached to a new operator (->). That operator could be used to write expressions that look like a function signature. Here's Kay's original email. http://mail.python.org/pipermail/python-list/2007-January/420774.html -Tony From jcarlson at uci.edu Tue Jan 2 07:21:21 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 01 Jan 2007 22:21:21 -0800 Subject: [Python-ideas] Attribute Docstrings and Annotations In-Reply-To: References: Message-ID: <20070101221716.8306.JCARLSON@uci.edu> Tony Lownds wrote: > > I'd like to propose annotations and docstrings on attributes for > Python 3000 > with the following Grammar and properties: > > expr_stmt: test (':' test ['=' (yield_expr|testlist)] | > augassign (yield_expr|testlist) | > [',' testlist] ('=' (yield_expr|testlist))* > ) [snip] > >>> class X: > ... "class docstring" > ... foo: 1 = 1 > ... bar: 2 > ... "attribute docstring" > ... attr = None > ... "another attribute docstring" > ... fields = __slots__ = ['fields', 'attr'] > ... "docstring ignored" > ... x, y = 1, 2 > ... I have never needed attribute annotations, and I've never heard any core Python developer talk about it being useful to ahve them. -1 for the feature in any form. The syntax as described is ugly. -100 for the feature if it has the syntax provided. - Josiah From jcarlson at uci.edu Tue Jan 2 07:34:41 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 01 Jan 2007 22:34:41 -0800 Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations: overloadable -> In-Reply-To: <78364BAD-1C96-4D15-927E-983DB909FBF5@pagedna.com> References: <20070101172715.82F8.JCARLSON@uci.edu> <78364BAD-1C96-4D15-927E-983DB909FBF5@pagedna.com> Message-ID: <20070101222204.8309.JCARLSON@uci.edu> Tony Lownds wrote: > > > On Jan 1, 2007, at 5:39 PM, Josiah Carlson wrote: > > > Given a reading of version 53169 of PEP 3107, I see no reason to even > > offer a __returns__ attribute, automatic method call, etc. PEP 3107 > > already defines the annotation to be specified as a dictionary of > > named > > argument names, with a key of 'return' for the annotation of the > > return > > annotation. Is there any reason why this isn't sufficient? > > > > Even in the context of the signature PEP 362, there is no need for > > __returns__, as an annotation-consuming library would check the > > signature object for argument annotations, and the func_annotations > > dictionary for the 'return' key for any return annotations. > > Something got lost in the movement between lists... __returns__ wasn't > going to supplant or alter func_annotations or PEP 362 in any way. > > __returns__ was proposed as the special method attached to a new > operator (->). That operator could be used to write expressions that > look > like a function signature. > > Here's Kay's original email. > > http://mail.python.org/pipermail/python-list/2007-January/420774.html -1 on the -> operator as specified in the email you link. There is no reason to add aribtrary operators for call-site annotations. PEP 3107 has already defined the syntax for function definition site annotations as follows... def name(argument=default:annotation) -> annotation: body While PEP 3107 has not been formally accepted (according to the web page), the ideas contained received general approval from more or less everyone involved in the py3k discussion, or at least more or less everyone who has written and/or consumed annotations in the past. The mail you link, talks about an arbitrary -> operator for call-site annotations. Without seeing actual use-cases where a -> operator would be useful in the real-world, I can't help but be -1 on the -> operator and -1 on the __returns__ method. Use-case(s) before syntax and semantics, - Josiah From kay.schluehr at gmx.net Tue Jan 2 13:44:34 2007 From: kay.schluehr at gmx.net (Kay Schluehr) Date: Tue, 02 Jan 2007 13:44:34 +0100 Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations: overloadable -> In-Reply-To: <20070101222204.8309.JCARLSON@uci.edu> References: <20070101172715.82F8.JCARLSON@uci.edu> <78364BAD-1C96-4D15-927E-983DB909FBF5@pagedna.com> <20070101222204.8309.JCARLSON@uci.edu> Message-ID: <459A53B2.2030107@gmx.net> Josiah Carlson schrieb: >Tony Lownds wrote: > > >>On Jan 1, 2007, at 5:39 PM, Josiah Carlson wrote: >> >> >> >>>Given a reading of version 53169 of PEP 3107, I see no reason to even >>>offer a __returns__ attribute, automatic method call, etc. PEP 3107 >>>already defines the annotation to be specified as a dictionary of >>>named >>>argument names, with a key of 'return' for the annotation of the >>>return >>>annotation. Is there any reason why this isn't sufficient? >>> >>>Even in the context of the signature PEP 362, there is no need for >>>__returns__, as an annotation-consuming library would check the >>>signature object for argument annotations, and the func_annotations >>>dictionary for the 'return' key for any return annotations. >>> >>> >>Something got lost in the movement between lists... __returns__ wasn't >>going to supplant or alter func_annotations or PEP 362 in any way. >> >>__returns__ was proposed as the special method attached to a new >>operator (->). That operator could be used to write expressions that >>look >>like a function signature. >> >>Here's Kay's original email. >> >>http://mail.python.org/pipermail/python-list/2007-January/420774.html >> >> > >-1 on the -> operator as specified in the email you link. There is no >reason to add aribtrary operators for call-site annotations. PEP 3107 >has already defined the syntax for function definition site annotations >as follows... > > def name(argument=default:annotation) -> annotation: > body > > If a function consumes or returns another function, it is quite natural to check against the function type ( at least for certain usefull annotation handlers ). Some experts call those "higher order functions". Programmers have found those in Pythons __builtins__ module. Others have guessed that decorators could be HOF. I hope this is convenient enough for motivation. With current syntax one ends up defining function type annotations like this Function(int,str).returns(str) Function(int,str)*str (Function(int,str), str) "(int,str)->str" ... Maybe one can update PEP 8 for a recommendation, if there is just too much syntax angst for choosing the form Function(int, str)->str where wild hordes of Pythonistas might think about this as an implication - and are perfectly correct about it of course, at least according to Curry-Howard and any person, reading at least an introductory paper about type theory. The latter is also the reason why Tonys __implies__ proposal makes perfect sense to me. Regards and a happy new year, Kay From tony at pagedna.com Tue Jan 2 17:33:00 2007 From: tony at pagedna.com (Tony Lownds) Date: Tue, 2 Jan 2007 08:33:00 -0800 Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations: overloadable -> In-Reply-To: <20070101222204.8309.JCARLSON@uci.edu> References: <20070101172715.82F8.JCARLSON@uci.edu> <78364BAD-1C96-4D15-927E-983DB909FBF5@pagedna.com> <20070101222204.8309.JCARLSON@uci.edu> Message-ID: <338E39B3-446B-4322-816B-9783ED882393@pagedna.com> On Jan 1, 2007, at 10:34 PM, Josiah Carlson wrote: > -1 on the -> operator as specified in the email you link. There is no > reason to add aribtrary operators for call-site annotations. What was the arbitrary operator? -> is not arbitrary, and there was a reason given to add it. > PEP 3107 > has already defined the syntax for function definition site > annotations > as follows... > > def name(argument=default:annotation) -> annotation: > body > You mean def name(argument:annotation=default) > The mail you link, talks about an arbitrary -> operator for call-site > annotations. Without seeing actual use-cases where a -> operator > would > be useful in the real-world, I can't help but be -1 on the -> operator > and -1 on the __returns__ method. That email had a use case... The gain is being able to write def wrap(text: str, split: Function(str)-> list): instead of def wrap(text: str, split: Function(str) == list): or def wrap(text: str, split: Function(str, returns=list)): or one of the other methods that Kay came up with. -Tony From tony at pagedna.com Tue Jan 2 17:46:29 2007 From: tony at pagedna.com (Tony Lownds) Date: Tue, 2 Jan 2007 08:46:29 -0800 Subject: [Python-ideas] Attribute Docstrings and Annotations In-Reply-To: <20070101221716.8306.JCARLSON@uci.edu> References: <20070101221716.8306.JCARLSON@uci.edu> Message-ID: <77616156-108A-40EC-9CC3-1F59082CCD4C@pagedna.com> On Jan 1, 2007, at 10:21 PM, Josiah Carlson wrote: > > Tony Lownds wrote: >> >> I'd like to propose annotations and docstrings on attributes for >> Python 3000 >> with the following Grammar and properties: >> >> expr_stmt: test (':' test ['=' (yield_expr|testlist)] | >> augassign (yield_expr|testlist) | >> [',' testlist] ('=' (yield_expr|testlist))* >> ) > [snip] >>>>> class X: >> ... "class docstring" >> ... foo: 1 = 1 >> ... bar: 2 >> ... "attribute docstring" >> ... attr = None >> ... "another attribute docstring" >> ... fields = __slots__ = ['fields', 'attr'] >> ... "docstring ignored" >> ... x, y = 1, 2 >> ... > > I have never needed attribute annotations, and I've never heard any > core > Python developer talk about it being useful to ahve them. -1 for the > feature in any form. > > The syntax as described is ugly. -100 for the feature if it has the > syntax provided. > It's the same syntax as function annotations... def f(name: annotation = value): ^^^^^^^^^^^^^^^^^^^^^^^^ class F: name: annotation = value ^^^^^^^^^^^^^^^^^^^^^^^^ The syntax was presented on Guido's blog, too. http://www.artima.com/weblogs/viewpost.jsp?thread=87182 -Tony From jcarlson at uci.edu Tue Jan 2 18:21:31 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 02 Jan 2007 09:21:31 -0800 Subject: [Python-ideas] Attribute Docstrings and Annotations In-Reply-To: <77616156-108A-40EC-9CC3-1F59082CCD4C@pagedna.com> References: <20070101221716.8306.JCARLSON@uci.edu> <77616156-108A-40EC-9CC3-1F59082CCD4C@pagedna.com> Message-ID: <20070102090907.831E.JCARLSON@uci.edu> Tony Lownds wrote: > On Jan 1, 2007, at 10:21 PM, Josiah Carlson wrote: > > I have never needed attribute annotations, and I've never heard any > > core > > Python developer talk about it being useful to have them. -1 for the > > feature in any form. > > The syntax as described is ugly. -100 for the feature if it has the > > syntax provided. > > > > It's the same syntax as function annotations... > > def f(name: annotation = value): > ^^^^^^^^^^^^^^^^^^^^^^^^ I don't particularly like the look of function annotations either, but I don't have a better syntax (aside from swapping the annotation and value). In this case, I really don't like attribute annotations because it looks to me like a bunch of line noise without meaning, or some C-like conditional expression gone wrong. > class F: > name: annotation = value > ^^^^^^^^^^^^^^^^^^^^^^^^ > > The syntax was presented on Guido's blog, too. > > http://www.artima.com/weblogs/viewpost.jsp?thread=87182 I didn't like it when Guido posted it on his blog either. I would also point out that Guido lists attribute annotations as a "maybe". Perhaps he has become 100% on them, I don't know, but I'm still -1. In any case, you still haven't provided any use-cases, or an example where developers have been asking for the feature and could show that *not* having the feature was constraining them in some significant way. - Josiah From tony at pagedna.com Tue Jan 2 18:40:44 2007 From: tony at pagedna.com (Tony Lownds) Date: Tue, 2 Jan 2007 09:40:44 -0800 Subject: [Python-ideas] Attribute Docstrings and Annotations In-Reply-To: <20070102090907.831E.JCARLSON@uci.edu> References: <20070101221716.8306.JCARLSON@uci.edu> <77616156-108A-40EC-9CC3-1F59082CCD4C@pagedna.com> <20070102090907.831E.JCARLSON@uci.edu> Message-ID: On Jan 2, 2007, at 9:21 AM, Josiah Carlson wrote: > > Tony Lownds wrote: >> On Jan 1, 2007, at 10:21 PM, Josiah Carlson wrote: >>> I have never needed attribute annotations, and I've never heard any >>> core >>> Python developer talk about it being useful to have them. -1 for >>> the >>> feature in any form. >>> The syntax as described is ugly. -100 for the feature if it has the >>> syntax provided. >>> >> >> It's the same syntax as function annotations... >> >> def f(name: annotation = value): >> ^^^^^^^^^^^^^^^^^^^^^^^^ > > I don't particularly like the look of function annotations either, > but I > don't have a better syntax (aside from swapping the annotation and > value). > In this case, I really don't like attribute annotations because it > looks to me like a bunch of line noise without meaning, or some C-like > conditional expression gone wrong. > > >> class F: >> name: annotation = value >> ^^^^^^^^^^^^^^^^^^^^^^^^ >> >> The syntax was presented on Guido's blog, too. >> >> http://www.artima.com/weblogs/viewpost.jsp?thread=87182 > > I didn't like it when Guido posted it on his blog either. I would > also > point out that Guido lists attribute annotations as a "maybe". > Perhaps > he has become 100% on them, I don't know, but I'm still -1. > > In any case, you still haven't provided any use-cases, or an example > where developers have been asking for the feature and could show that > *not* having the feature was constraining them in some significant > way. As long as it's clear that the syntax as proposed has SOME existing thought behind it, I'm happy to leave the proposal alone. -Tony From jcarlson at uci.edu Tue Jan 2 19:29:06 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 02 Jan 2007 10:29:06 -0800 Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations: overloadable -> In-Reply-To: <338E39B3-446B-4322-816B-9783ED882393@pagedna.com> References: <20070101222204.8309.JCARLSON@uci.edu> <338E39B3-446B-4322-816B-9783ED882393@pagedna.com> Message-ID: <20070102090653.831B.JCARLSON@uci.edu> Tony Lownds wrote: > On Jan 1, 2007, at 10:34 PM, Josiah Carlson wrote: > > -1 on the -> operator as specified in the email you link. There is no > > reason to add aribtrary operators for call-site annotations. > > What was the arbitrary operator? -> is not arbitrary, and there was a > reason given to add it. I misunderstood portions of the post. > > PEP 3107 > > has already defined the syntax for function definition site > > annotations > > as follows... > > > > def name(argument=default:annotation) -> annotation: > > body > > You mean > > def name(argument:annotation=default) Thank you for the correction. Maybe I'm the only one, but I would prefer the default value to come before the annotation. > > The mail you link, talks about an arbitrary -> operator for call-site > > annotations. Without seeing actual use-cases where a -> operator > > would > > be useful in the real-world, I can't help but be -1 on the -> operator > > and -1 on the __returns__ method. > > That email had a use case... The gain is being able to write > > def wrap(text: str, split: Function(str)-> list): > > instead of > > def wrap(text: str, split: Function(str) == list): > > or > > def wrap(text: str, split: Function(str, returns=list)): > > or one of the other methods that Kay came up with. Kay's mail specified: maptype = Function( Function(object)->object, list) -> list That looks to me like the -> operator being used anywhere the user desires. With your examples above, I see the use-case, but I'm still -1 (for the same reasons). For other reasons to be -1 on ->, we can look at how it would be abused to violate the one obvious way to do things... Stack() -> 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 t = Tree() t -> (t -> (1,2), t -> (3,4)) Or for a concrete example... class mylist(list): def __returns__(self, arg): self.append(arg) return self mylist() -> 1 -> 2 -> 3 -> 4 -> 5 Right now those things aren't done because the punctuation for doing it isn't as natural as -> would be. - Josiah From guido at python.org Tue Jan 2 19:41:46 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 2 Jan 2007 10:41:46 -0800 Subject: [Python-ideas] Attribute Docstrings and Annotations In-Reply-To: References: <20070101221716.8306.JCARLSON@uci.edu> <77616156-108A-40EC-9CC3-1F59082CCD4C@pagedna.com> <20070102090907.831E.JCARLSON@uci.edu> Message-ID: Yes, I blogged about it, but in the discussion that followed on python-3000 it became clear that the "typed attribute" notation is not a favorite of many folks, and I'm no longer in favor of it myself. The use cases are a lot weaker than for signature annotations. So let's drop it. On 1/2/07, Tony Lownds wrote: > > On Jan 2, 2007, at 9:21 AM, Josiah Carlson wrote: > > > > > Tony Lownds wrote: > >> On Jan 1, 2007, at 10:21 PM, Josiah Carlson wrote: > >>> I have never needed attribute annotations, and I've never heard any > >>> core > >>> Python developer talk about it being useful to have them. -1 for > >>> the > >>> feature in any form. > >>> The syntax as described is ugly. -100 for the feature if it has the > >>> syntax provided. > >>> > >> > >> It's the same syntax as function annotations... > >> > >> def f(name: annotation = value): > >> ^^^^^^^^^^^^^^^^^^^^^^^^ > > > > I don't particularly like the look of function annotations either, > > but I > > don't have a better syntax (aside from swapping the annotation and > > value). > > In this case, I really don't like attribute annotations because it > > looks to me like a bunch of line noise without meaning, or some C-like > > conditional expression gone wrong. > > > > > >> class F: > >> name: annotation = value > >> ^^^^^^^^^^^^^^^^^^^^^^^^ > >> > >> The syntax was presented on Guido's blog, too. > >> > >> http://www.artima.com/weblogs/viewpost.jsp?thread=87182 > > > > I didn't like it when Guido posted it on his blog either. I would > > also > > point out that Guido lists attribute annotations as a "maybe". > > Perhaps > > he has become 100% on them, I don't know, but I'm still -1. > > > > In any case, you still haven't provided any use-cases, or an example > > where developers have been asking for the feature and could show that > > *not* having the feature was constraining them in some significant > > way. > > As long as it's clear that the syntax as proposed has SOME existing > thought > behind it, I'm happy to leave the proposal alone. > > -Tony > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Jan 2 19:48:50 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 2 Jan 2007 10:48:50 -0800 Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations: overloadable -> In-Reply-To: <20070102090653.831B.JCARLSON@uci.edu> References: <20070101222204.8309.JCARLSON@uci.edu> <338E39B3-446B-4322-816B-9783ED882393@pagedna.com> <20070102090653.831B.JCARLSON@uci.edu> Message-ID: On 1/2/07, Josiah Carlson wrote: > > > PEP 3107 > > > has already defined the syntax for function definition site > > > annotations > > > as follows... > > > > > > def name(argument=default:annotation) -> annotation: > > > body > > > > You mean > > > > def name(argument:annotation=default) > > Thank you for the correction. Maybe I'm the only one, but I would > prefer the default value to come before the annotation. You're about a year and a half too late with this objection. And anyway, I strongly favor putting the annotation first; that's how it is in other languages from which Python has borrowed before, e.g. Modula-3. Regarding the status of the PEP: I am mostly waiting for the wording to improve, e.g. not use examples that suggest this will have type checking semantics by default. I am +1 on the syntax used in the PEP, since it is what I have been proposing for a long time. I am against more additions like an '->' operator or attribute annotations or anything else at this point, at least until there has been actual experience with using the implementation as-is. (And yes, I have seen the use case, and remain unconvinced. Not every use case deserves new syntax.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jimjjewett at gmail.com Tue Jan 2 23:58:20 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 2 Jan 2007 17:58:20 -0500 Subject: [Python-ideas] Attribute Docstrings and Annotations In-Reply-To: <20070101221716.8306.JCARLSON@uci.edu> References: <20070101221716.8306.JCARLSON@uci.edu> Message-ID: On 1/2/07, Josiah Carlson wrote: > I have never needed attribute annotations For some classes, I document the intended use of certain attributes with a comment. Making this introspectable would be good. I've snipped the rest because I agree it wouldn't be worth the wordiness it would cause with any syntax I've seen so far. -jJ From cvrebert at gmail.com Wed Jan 3 01:07:31 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Tue, 02 Jan 2007 16:07:31 -0800 Subject: [Python-ideas] new operators via backquoting Message-ID: <459AF3C3.9020502@gmail.com> In Haskell, foo `baz` bar means (baz foo bar), which translates to baz(foo, bar) in Python. This allows Haskell programmers to use functions as infix operators. If I recall correctly, in Py3k, enclosing something in backticks will no longer cause it to be repr()-ed, leaving the backtick without a meaning in Python. Thus, I propose one of the following as the new use for the backtick (`): [Note: In both, the characters between the backticks must be a valid Python identifier.] (A) `baz` is treated as an operator, named "baz", just as / is "div". foo `baz` bar thus causes python to try to call foo.__baz__(bar), and failing that, bar.__rbaz__(foo), and if both those fail, raise TypeError. This is, if I understand correctly, how the builtin operators work. (B) `baz` is a special way to call a callable. foo `baz` bar is translated to baz(foo, bar) with the standard lookup rules for resolving "baz" Example use cases, stolen from Haskell: The Craft of Functional Programming: 2 `max` 5 => 5 7 `cons` tail => ConsCell(val=7, next=tail) matrix1 `crossproduct` matrix2 => cross-product of the matrices [1, 2, 3] `zip` ['a', 'b', 'c'] => [[1, 'a'], [2, 'c'], [3, 'c']] I believe that this would improve the readability of code, such as Numeric, without going off the deep end and offering programmable syntax. - Chris Rebert From rrr at ronadam.com Wed Jan 3 01:14:38 2007 From: rrr at ronadam.com (Ron Adam) Date: Tue, 02 Jan 2007 18:14:38 -0600 Subject: [Python-ideas] Attribute Docstrings and Annotations In-Reply-To: References: <20070101221716.8306.JCARLSON@uci.edu> Message-ID: <459AF56E.50500@ronadam.com> Jim Jewett wrote: > On 1/2/07, Josiah Carlson wrote: > >> I have never needed attribute annotations > > For some classes, I document the intended use of certain attributes > with a comment. Making this introspectable would be good. > > I've snipped the rest because I agree it wouldn't be worth the > wordiness it would cause with any syntax I've seen so far. > > -jJ I feel the correct way to do this is to just put the attribute comment in the doc string of the object that contains the attribute. Simple attributes are usually just that, simple values related to the object. Yes, it makes the doc string longer, but it also tends to makes for better documentation as well. Why make things more complex than they need to be? Ron From brett at python.org Wed Jan 3 01:33:58 2007 From: brett at python.org (Brett Cannon) Date: Tue, 2 Jan 2007 16:33:58 -0800 Subject: [Python-ideas] new operators via backquoting In-Reply-To: <459AF3C3.9020502@gmail.com> References: <459AF3C3.9020502@gmail.com> Message-ID: On 1/2/07, Chris Rebert wrote: > > In Haskell, foo `baz` bar means (baz foo bar), which translates to > baz(foo, bar) in Python. This allows Haskell programmers to use > functions as infix operators. > If I recall correctly, in Py3k, enclosing something in backticks will no > longer cause it to be repr()-ed, leaving the backtick without a meaning > in Python. Right. I removed that back in August at the Google sprint. Thus, I propose one of the following as the new use for the backtick (`): > [Note: In both, the characters between the backticks must be a valid > Python identifier.] > > (A) `baz` is treated as an operator, named "baz", just as / is "div". > foo `baz` bar thus causes python to try to call foo.__baz__(bar), and > failing that, bar.__rbaz__(foo), and if both those fail, raise > TypeError. This is, if I understand correctly, how the builtin operators > work. (B) `baz` is a special way to call a callable. foo `baz` bar is > translated to baz(foo, bar) with the standard lookup rules for > resolving "baz" > > > Example use cases, stolen from Haskell: The Craft of Functional > Programming: > 2 `max` 5 => 5 > 7 `cons` tail => ConsCell(val=7, next=tail) > matrix1 `crossproduct` matrix2 => cross-product of the matrices > [1, 2, 3] `zip` ['a', 'b', 'c'] => [[1, 'a'], [2, 'c'], [3, 'c']] > > I believe that this would improve the readability of code, such as > Numeric, without going off the deep end and offering programmable syntax. Big -1 from me. I hate this feature from Haskell. It is a step towards programmable syntax and I think that's just a messy. And having it become a magic method that is called instead of just some function that takes two arguments really sends us down that road. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcarlson at uci.edu Wed Jan 3 01:52:06 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 02 Jan 2007 16:52:06 -0800 Subject: [Python-ideas] new operators via backquoting In-Reply-To: <459AF3C3.9020502@gmail.com> References: <459AF3C3.9020502@gmail.com> Message-ID: <20070102164348.8CC1.JCARLSON@uci.edu> Chris Rebert wrote: > > In Haskell, foo `baz` bar means (baz foo bar), which translates to > baz(foo, bar) in Python. This allows Haskell programmers to use > functions as infix operators. > If I recall correctly, in Py3k, enclosing something in backticks will no > longer cause it to be repr()-ed, leaving the backtick without a meaning > in Python. [snip] > I believe that this would improve the readability of code, such as > Numeric, without going off the deep end and offering programmable syntax. Let us break down what you are more or less proposing... foo `baz` bar ... is translated into ... foo.__baz__(bar) bar.__rbaz__(foo) How is that any easier to read or understand than... baz(foo, bar) I don't believe it is easier to understand, and would claim that it is more difficult to understand, especially to users who have seen backticks being used for repr. The only people it could be more undestandable to is those who have used Haskell and have seen such operator use in the past. I dont know how many of those users there are, but I don't believe that the confusion would be worth it. If you want to call a method on an object, call the method. If you want to call a function that does automatic method invocation, that is fine too. But what you are offering is more or less arbitrary infix operators, which I can't see as being anything more than confusing to new and seasoned Python users alike. - Josiah From guido at python.org Wed Jan 3 02:33:36 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 2 Jan 2007 17:33:36 -0800 Subject: [Python-ideas] new operators via backquoting In-Reply-To: <459AF3C3.9020502@gmail.com> References: <459AF3C3.9020502@gmail.com> Message-ID: On 1/2/07, Chris Rebert wrote: > Thus, I propose one of the following as the new use for the backtick (`): You're missing one of the main reasons for removing the backtick syntax in the first place: the character itself causes trouble by looking too much like a regular quote (depending on your font), is routinely mangled by typesetting software (as every Python book author can testify), and requires a four-finger chord on Swiss keyboards. No new uses for it will be accepted in Python 3000 no matter how good the idea. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jimjjewett at gmail.com Wed Jan 3 16:27:11 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 3 Jan 2007 10:27:11 -0500 Subject: [Python-ideas] new operators via backquoting In-Reply-To: References: <459AF3C3.9020502@gmail.com> Message-ID: Patch 1627052, assigned to Georg, adds backtick-reuse to PEP 3099 (rejected ideas). On 1/2/07, Guido van Rossum wrote: > On 1/2/07, Chris Rebert wrote: > > Thus, I propose one of the following as the new use for the backtick (`): > > You're missing one of the main reasons for removing the backtick > syntax in the first place: the character itself causes trouble by > looking too much like a regular quote (depending on your font), is > routinely mangled by typesetting software (as every Python book author > can testify), and requires a four-finger chord on Swiss keyboards. No > new uses for it will be accepted in Python 3000 no matter how good the > idea. > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From george.sakkis at gmail.com Wed Jan 3 18:29:55 2007 From: george.sakkis at gmail.com (George Sakkis) Date: Wed, 3 Jan 2007 12:29:55 -0500 Subject: [Python-ideas] new operators via backquoting Message-ID: <91ad5bf80701030929i72d74cfeia4bfdcb66559a92f@mail.gmail.com> Chris Rebert wrote: > In Haskell, foo `baz` bar means (baz foo bar), which translates to > baz(foo, bar) in Python. This allows Haskell programmers to use > functions as infix operators. > If I recall correctly, in Py3k, enclosing something in backticks will no > longer cause it to be repr()-ed, leaving the backtick without a meaning > in Python. Perhaps you're not aware of the "best hack of 2005", which works in current Python and goes a step further by allowing you to even "override" the operator's delimiter: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/384122 George From cvrebert at gmail.com Wed Jan 3 18:39:33 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Wed, 03 Jan 2007 09:39:33 -0800 Subject: [Python-ideas] new operators via backquoting In-Reply-To: <91ad5bf80701030929i72d74cfeia4bfdcb66559a92f@mail.gmail.com> References: <91ad5bf80701030929i72d74cfeia4bfdcb66559a92f@mail.gmail.com> Message-ID: <459BEA55.1070908@gmail.com> George Sakkis wrote: > Perhaps you're not aware of the "best hack of 2005", which works in > current Python and goes a step further by allowing you to even > "override" the operator's delimiter: > > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/384122 > > George > _______________________________________________ In light of this and Guido's pronouncement that the backtick will not have any meaning in Py3k, I withdraw my proposal. - Chris Rebert From collinw at gmail.com Wed Jan 3 20:24:47 2007 From: collinw at gmail.com (Collin Winter) Date: Wed, 3 Jan 2007 13:24:47 -0600 Subject: [Python-ideas] PEP 3107 Function Annotations: interoperability (again) In-Reply-To: References: Message-ID: <43aa6ff70701031124h7137b946x77022e5e4ec34793@mail.gmail.com> On 1/1/07, Brett Cannon wrote: > Until extensive usage happens with annotations we shouldn't try to shoehorn > consumers of the annotations into a specific solution. As you said, > something will probably organically grow and that can be supported in > another PEP later on (probably with stdlib support). I agree that a convention will develop organically, but I would add that the emergence of that convention will be governed by two things: who has the bigger market share, and who's first to market. When the first big project like Zope or Twisted announces that "we will do annotation interop like so...", everyone else will be pressured to line up behind them. Until that happens, smaller projects like mine will either a) not support annotation interop (because there's no good, obvious solution), or b) pick an interop scheme at random (again, because there's no good, obvious solution). Thanks, Collin Winter From guido at python.org Wed Jan 3 21:58:22 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 3 Jan 2007 12:58:22 -0800 Subject: [Python-ideas] PEP 3107 Function Annotations: interoperability (again) In-Reply-To: <43aa6ff70701031124h7137b946x77022e5e4ec34793@mail.gmail.com> References: <43aa6ff70701031124h7137b946x77022e5e4ec34793@mail.gmail.com> Message-ID: On 1/3/07, Collin Winter wrote: > On 1/1/07, Brett Cannon wrote: > > Until extensive usage happens with annotations we shouldn't try to shoehorn > > consumers of the annotations into a specific solution. As you said, > > something will probably organically grow and that can be supported in > > another PEP later on (probably with stdlib support). > > I agree that a convention will develop organically, but I would add > that the emergence of that convention will be governed by two things: > who has the bigger market share, and who's first to market. When the > first big project like Zope or Twisted announces that "we will do > annotation interop like so...", everyone else will be pressured to > line up behind them. Until that happens, smaller projects like mine > will either a) not support annotation interop (because there's no > good, obvious solution), or b) pick an interop scheme at random > (again, because there's no good, obvious solution). Actually, I think that's a pretty good way to arrive at a good solution. I don't this is something you can analyze on paper ahead of time; there aren't really any precedents I suspect (do you know of any other language that has semantics-free signature annotations?). You must've "built two and thrown one away" before you really know what the constraints are on the solution. I realize this is frustrating for you, but I really don't think it's appropriate to pre-empty this with a standardization attempt before-the-fact (remember ISO networking?). You could be first to market yourself -- after all you already have a type checking package! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From collinw at gmail.com Wed Jan 3 22:55:25 2007 From: collinw at gmail.com (Collin Winter) Date: Wed, 3 Jan 2007 15:55:25 -0600 Subject: [Python-ideas] PEP 3107 Function Annotations: interoperability (again) In-Reply-To: References: <43aa6ff70701031124h7137b946x77022e5e4ec34793@mail.gmail.com> Message-ID: <43aa6ff70701031355g6535f02cnde7b13f95305bb36@mail.gmail.com> On 1/3/07, Guido van Rossum wrote: > On 1/3/07, Collin Winter wrote: > > I agree that a convention will develop organically, but I would add > > that the emergence of that convention will be governed by two things: > > who has the bigger market share, and who's first to market. When the > > first big project like Zope or Twisted announces that "we will do > > annotation interop like so...", everyone else will be pressured to > > line up behind them. Until that happens, smaller projects like mine > > will either a) not support annotation interop (because there's no > > good, obvious solution), or b) pick an interop scheme at random > > (again, because there's no good, obvious solution). [snip] > I realize this is frustrating for you, but I really don't think it's > appropriate to pre-empty this with a standardization attempt > before-the-fact (remember ISO networking?). You could be first to > market yourself -- after all you already have a type checking package! Unless someone comes up with a good interop solution before the Py3k release, I doubt my typechecking package will support annotation interop; I don't want to pick a mechanism, only to have to roll another release when Zope or Twisted or some other big project pushes out their own solution. I will support annotations, though; that way, I can at least gather real-world usage data when users complain about not being able to use more than one annotation consumer : ) Thanks, Collin Winter From guido at python.org Wed Jan 3 23:06:29 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 3 Jan 2007 14:06:29 -0800 Subject: [Python-ideas] PEP 3107 Function Annotations: interoperability (again) In-Reply-To: <43aa6ff70701031355g6535f02cnde7b13f95305bb36@mail.gmail.com> References: <43aa6ff70701031124h7137b946x77022e5e4ec34793@mail.gmail.com> <43aa6ff70701031355g6535f02cnde7b13f95305bb36@mail.gmail.com> Message-ID: On 1/3/07, Collin Winter wrote: > Unless someone comes up with a good interop solution before the Py3k > release, I doubt my typechecking package will support annotation > interop; I don't want to pick a mechanism, only to have to roll > another release when Zope or Twisted or some other big project pushes > out their own solution. > I will support annotations, though; that way, I can at least gather > real-world usage data when users complain about not being able to use > more than one annotation consumer : ) Those projects, due to sheer size, will be years behind you in their adoption of Python 3000, so you still have an advantage if you change your mind. Good luck! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From matt at draisey.ca Thu Jan 4 01:20:43 2007 From: matt at draisey.ca (Matt Draisey) Date: Wed, 03 Jan 2007 19:20:43 -0500 Subject: [Python-ideas] Module local namespaces Message-ID: <1167870043.1873.34.camel@della.draisey.ca> A while ago, I developed a small PyGtk programme that could dynamically reload all the working callbacks and logic while the GUI was still running. I could get away with this because of the flexible way Python loads modules at runtime, but it ended up being a waste of time as implementing it took more time that actually using it. For sanity's sake it quickly becomes clear you almost never want to rely on being able to refer to a half initialized module. And wouldn't it be nice if Python enforced this. My suggestion is that module importing occur in a temporary local namespace that exists only until the end of the module code is executed, then a small function could copy everything from the temporary namespace into the module object. The usual closure semantics would guarantee that top-level functions could still call each other, but they would effectively become immutable after the namespace wraps up. The 'global' keyword could be used at the top level in a module to force it to be defined in the module immediately, and to ensure internal references to the object go through the module object. This would be a big change in module import semantics, but should have remarkably few consequences, as it really is an enforcement mechanism for good style. The copying from the temporary namespace into the module object would be a good place to insert a hook function to filter what objects are actually published to the module. You could by default not copy any object indentified by a leading underscore. From brett at python.org Thu Jan 4 02:01:13 2007 From: brett at python.org (Brett Cannon) Date: Wed, 3 Jan 2007 17:01:13 -0800 Subject: [Python-ideas] Module local namespaces In-Reply-To: <1167870043.1873.34.camel@della.draisey.ca> References: <1167870043.1873.34.camel@della.draisey.ca> Message-ID: On 1/3/07, Matt Draisey wrote: > > A while ago, I developed a small PyGtk programme that could dynamically > reload all the working callbacks and logic while the GUI was still > running. I could get away with this because of the flexible way Python > loads modules at runtime, but it ended up being a waste of time as > implementing it took more time that actually using it. For sanity's > sake it quickly becomes clear you almost never want to rely on being > able to refer to a half initialized module. And wouldn't it be nice if > Python enforced this. How are you having an issue with a partially initialized module? The only way I can see that happening is that you have a circular import dependency where both modules want to execute code that the other one has. And if that happens the answer for that is "don't do it". My suggestion is that module importing occur in a temporary local > namespace that exists only until the end of the module code is executed, > then a small function could copy everything from the temporary namespace > into the module object. The usual closure semantics would guarantee > that top-level functions could still call each other, but they would > effectively become immutable after the namespace wraps up. But why would you want it to be immutable? Being able to change the function in a module and have all uses of it also change can be handy. The 'global' > keyword could be used at the top level in a module to force it to be > defined in the module immediately, and to ensure internal references to > the object go through the module object. > > This would be a big change in module import semantics, but should have > remarkably few consequences, as it really is an enforcement mechanism > for good style. The copying from the temporary namespace into the > module object would be a good place to insert a hook function to filter > what objects are actually published to the module. You could by default > not copy any object indentified by a leading underscore. Private namespaces are not exactly a popular thing in Python. =) -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcarlson at uci.edu Thu Jan 4 02:17:16 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 03 Jan 2007 17:17:16 -0800 Subject: [Python-ideas] Module local namespaces In-Reply-To: <1167870043.1873.34.camel@della.draisey.ca> References: <1167870043.1873.34.camel@della.draisey.ca> Message-ID: <20070103170636.8CDE.JCARLSON@uci.edu> Matt Draisey wrote: [snip] -1 . The only thing that possibly should be fixed is that modules should only be inserted into sys.modules after they have been imported completely and correctly. > This would be a big change in module import semantics, but should have > remarkably few consequences, as it really is an enforcement mechanism > for good style. How is it an enforcement for good style? If I can do exactly what I'm doing now, then it isn't an enforcement mechanism in any way. Regardless, the way to handle not allowing the rest of your program to muck with partially initialized modules is to not use the standard import machinery. There are various ways of emulating module loading, many of which can allow you to insert the module into sys.modules *after* the module was loaded completely correctly. I actually use one of these methods to handle the dynamic loading and reloading of Python macros in a source code editor, and none of the macros are available in sys.modules . > The copying from the temporary namespace into the > module object would be a good place to insert a hook function to filter > what objects are actually published to the module. You could by default > not copy any object indentified by a leading underscore. You can already do this with the following code: __gl = globals() for name in __gl.keys(): if name[:1] == '_' and len(name) > 1 and name.count('_') == 1: del __gl[name] del __gl - Josiah From guido at python.org Thu Jan 4 02:18:09 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 3 Jan 2007 17:18:09 -0800 Subject: [Python-ideas] Module local namespaces In-Reply-To: <20070103170636.8CDE.JCARLSON@uci.edu> References: <1167870043.1873.34.camel@della.draisey.ca> <20070103170636.8CDE.JCARLSON@uci.edu> Message-ID: On 1/3/07, Josiah Carlson wrote: > -1 . The only thing that possibly should be fixed is that modules > should only be inserted into sys.modules after they have been imported > completely and correctly. I agree, but there's the wrinkle that during recursive imports you want partially-imported modules to be importable. I believe we once agreed on a solution but I don't recall what it was. Does anybody remember? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Thu Jan 4 02:51:28 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 03 Jan 2007 17:51:28 -0800 Subject: [Python-ideas] Module local namespaces In-Reply-To: References: <20070103170636.8CDE.JCARLSON@uci.edu> Message-ID: <20070103174954.8CE3.JCARLSON@uci.edu> "Guido van Rossum" wrote: > On 1/3/07, Josiah Carlson wrote: > > -1 . The only thing that possibly should be fixed is that modules > > should only be inserted into sys.modules after they have been imported > > completely and correctly. > > I agree, but there's the wrinkle that during recursive imports you > want partially-imported modules to be importable. I believe we once > agreed on a solution but I don't recall what it was. Does anybody > remember? I don't remember what was agreed upon, but what about something like... try: newmodule = ... sys.modules[name] = newmodule handle_the_import(newmodule, ...) except: del sys.modules[name] raise - Josiah From matt at draisey.ca Thu Jan 4 03:25:16 2007 From: matt at draisey.ca (Matt Draisey) Date: Wed, 03 Jan 2007 21:25:16 -0500 Subject: [Python-ideas] Module local namespaces In-Reply-To: <20070103170636.8CDE.JCARLSON@uci.edu> References: <1167870043.1873.34.camel@della.draisey.ca> <20070103170636.8CDE.JCARLSON@uci.edu> Message-ID: <1167877516.1873.74.camel@della.draisey.ca> On Wed, 2007-01-03 at 17:17 -0800, Josiah Carlson wrote: > You can already do this with the following code: > > __gl = globals() > for name in __gl.keys(): > if name[:1] == '_' and len(name) > 1 and name.count('_') == 1: > del __gl[name] > del __gl > > - Josiah No, that is not what I meant. I wasn't talking about deleting temporaries but not publishing private objects. Brett Cannon understood when he said, "Private namespaces are not exactly a popular thing in Python". But functional programming style and generators are all the rage and they often drag in private state via a closure. From brett at python.org Thu Jan 4 03:29:05 2007 From: brett at python.org (Brett Cannon) Date: Wed, 3 Jan 2007 18:29:05 -0800 Subject: [Python-ideas] Module local namespaces In-Reply-To: <20070103174954.8CE3.JCARLSON@uci.edu> References: <20070103170636.8CDE.JCARLSON@uci.edu> <20070103174954.8CE3.JCARLSON@uci.edu> Message-ID: On 1/3/07, Josiah Carlson wrote: > > > "Guido van Rossum" wrote: > > On 1/3/07, Josiah Carlson wrote: > > > -1 . The only thing that possibly should be fixed is that modules > > > should only be inserted into sys.modules after they have been imported > > > completely and correctly. > > > > I agree, but there's the wrinkle that during recursive imports you > > want partially-imported modules to be importable. I believe we once > > agreed on a solution but I don't recall what it was. Does anybody > > remember? > > I don't remember what was agreed upon, but what about something like... > > try: > newmodule = ... > sys.modules[name] = newmodule > handle_the_import(newmodule, ...) > except: > del sys.modules[name] > raise My re-implementation does exactly that (I just sometimes postpone the module creation to the top of the handle_the_import function). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Jan 4 03:31:15 2007 From: brett at python.org (Brett Cannon) Date: Wed, 3 Jan 2007 18:31:15 -0800 Subject: [Python-ideas] Module local namespaces In-Reply-To: <1167877516.1873.74.camel@della.draisey.ca> References: <1167870043.1873.34.camel@della.draisey.ca> <20070103170636.8CDE.JCARLSON@uci.edu> <1167877516.1873.74.camel@della.draisey.ca> Message-ID: On 1/3/07, Matt Draisey wrote: > > On Wed, 2007-01-03 at 17:17 -0800, Josiah Carlson wrote: > > > You can already do this with the following code: > > > > __gl = globals() > > for name in __gl.keys(): > > if name[:1] == '_' and len(name) > 1 and name.count('_') == 1: > > del __gl[name] > > del __gl > > > > - Josiah > > No, that is not what I meant. I wasn't talking about deleting > temporaries but not publishing private objects. Brett Cannon understood > when he said, "Private namespaces are not exactly a popular thing in > Python". But functional programming style and generators are all the > rage and they often drag in private state via a closure. True, but you can still get at everything in either (generators, for instance, expose the paused execution frame and gives you access to everything). But I consider these more of an exception instead of the rule. The only way I can see myself approving something like this is if it helped with security (which it might, but I don't know if it would be the best solution). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From eopadoan at altavix.com Sat Jan 6 18:16:19 2007 From: eopadoan at altavix.com (Eduardo "EdCrypt" O. Padoan) Date: Sat, 6 Jan 2007 15:16:19 -0200 Subject: [Python-ideas] [Python-Dev] features i'd like [Python 3000] ... #3: fix super() In-Reply-To: References: <4573909F.5050405@666.com> <457402E1.6060705@gmail.com> Message-ID: > I was bitten by the urge to play with this today, and modified my > previous "self" hack to handle "super" also, so that the following > code works: > > class D (C): > @method > def sum(n): > return super.sum(n * 2) - self.base > > Posted as "evil2.py" here: > > http://www.lag.net/robey/code/surf/ > > Because hacking "super" requires having the class object handy, this > one needs a metaclass to do its magic, which is a shame. I guess if > it was implemented inside the cpython compiler, it would be less of a > problem. BTW, a "super-only" version of this decortor (that I think could be called "implement") has some more chances in Python. But think belongs more to Python-Ideas list, ok? -- EduardoOPadoan (eopadoan->altavix::com) Bookmarks: http://del.icio.us/edcrypt Blog: http://edcrypt.blogspot.com Jabber: edcrypt at jabber dot org ICQ: 161480283 GTalk: eduardo dot padoan at gmail dot com MSN: eopadoan at altavix dot com From eopadoan at altavix.com Sat Jan 6 18:17:29 2007 From: eopadoan at altavix.com (Eduardo "EdCrypt" O. Padoan) Date: Sat, 6 Jan 2007 15:17:29 -0200 Subject: [Python-ideas] [Python-Dev] features i'd like [Python 3000] ... #3: fix super() In-Reply-To: References: <4573909F.5050405@666.com> <457402E1.6060705@gmail.com> Message-ID: On 1/6/07, Eduardo EdCrypt O. Padoan wrote: > > I was bitten by the urge to play with this today, and modified my > > previous "self" hack to handle "super" also, so that the following > > code works: > > > > class D (C): > > @method > > def sum(n): > > return super.sum(n * 2) - self.base > > > > Posted as "evil2.py" here: > > > > http://www.lag.net/robey/code/surf/ > > > > Because hacking "super" requires having the class object handy, this > > one needs a metaclass to do its magic, which is a shame. I guess if > > it was implemented inside the cpython compiler, it would be less of a > > problem. > > BTW, a "super-only" version of this decortor (that I think could be > called "implement") has some more chances in Python. But think > belongs more to Python-Ideas list, ok? Ops: But *I* think *it* belongs more to Python-Ideas list, ok? -- EduardoOPadoan (eopadoan->altavix::com) Bookmarks: http://del.icio.us/edcrypt Blog: http://edcrypt.blogspot.com Jabber: edcrypt at jabber dot org ICQ: 161480283 GTalk: eduardo dot padoan at gmail dot com MSN: eopadoan at altavix dot com From tomerfiliba at gmail.com Sun Jan 14 22:19:01 2007 From: tomerfiliba at gmail.com (tomer filiba) Date: Sun, 14 Jan 2007 23:19:01 +0200 Subject: [Python-ideas] Fwd: multi-dispatch again In-Reply-To: <1d85506f0701140428q2ac7cf7eyd22c7b1ef72ed7e8@mail.gmail.com> References: <1d85506f0701140428q2ac7cf7eyd22c7b1ef72ed7e8@mail.gmail.com> Message-ID: <1d85506f0701141319o67630ad3v5d0bb2a7d7357f10@mail.gmail.com> Guido asked me to move it here. anyway, i might as well state my case better. in the formal definitions of object oriented programming, objects are said to encapsulate state and behavior. behavior is largely dependent on the type of the object, but the state is important nonetheless. for example, file objects are stateful: they can be opened for reading-only, writing-only, both, or be closed altogether. still, they are all instances of the file type. since generic functions/multi-dispatch come to help the programmer by reducing boilerplate early type-checking code and providing a more capable dispatch mechanism -- we can't overlook the state of the object. this early type-checking goes against the spirit of pure duck typing (which i'm fond of), but is crucial when the code has side effects. in this case, you can't just start executing the code and "hope it works", as the resources (i.e., files) involved are modified. in this kind of code, you want to check everything is alright *instead* of suddenly having an AttributeError/TypeError somewhere. here's an example: @dispatch def copy(src: file, dst: file): while True: buf = src.read(1000) if not buf: break dst.write(buf) suppose now that dst is mistakenly opened for reading. this means src would have already been modified, while dst.write is bound to fail. if src is a socket, for instance, this would be destructive. so if we already go as far as having multiple dispatch, which imposes constraints on the arguments a function accepts, we might as well base it on the type and state of the object, rather than only on it's type. we could say, for example: @dispatch def copy(src: file_for_reading, dst: file_for_writing): file_for_reading is not a type -- it's a checker. it may be defined as def file_for_reading(obj: file): return file.mode == "r" and not file.closed types, by default, would check using isinstance(), but costume checkers could check for stateful requirements too. and a note about performance: this check is required whether it's done explicitly or by the dispatch mechanism, and since most functions don't require so many overloads, i don't think it's an issue. besides, we can have a different decorator for dispatching by type or by checkers, i.e., @dispatch vs @stateful_dispatch, or something. the simple @dispatch would use a dictionary, while the stateful version would use a loop. -tomer ---------- Forwarded message ---------- From: tomer filiba Date: Jan 14, 2007 2:28 PM Subject: multi-dispatch again To: Python-3000 at python.org i just thought of a so-to-speak counter-example for ABCs... it's not really a counter-example, but i believe it shows a deficiency in the concept. theoretically speaking, objects are a made of type and state. ABCs, isinstance() and interfaces at general only check the type part. for example: @dispatch def log_to_file(text: str, device: file): file.write(text) this will constrain the *type* of the device, but not its *state*. practically speaking, i can pass a closed file, or a file open for reading-only, and it would pass silently. basing multi-dispatch on types is of course a leap forward, but if we already plan to take this leap, why not make it general enough to support more complex use-cases? this way we could rewrite the snippet above as @dispatch def log_to_file(text: str, device: open_file): file.write(text) where open_file isn't a type, but rather a "checker" that may also examine the state. by default, type objects would check for inheritance (via a special method), but checkers could extend this behavior. for efficiency purposes, we can have two decorators: @type_dispatch - dispatches based on type only @full_dispatch - dispatches based on type and state bottom line -- we can't just look at the type of the object for dispatching, overlooking its state. the state is meaningful, and we'd want the function not to be called at all if the state of the object is wrong. -tomer -------------- next part -------------- An HTML attachment was scrubbed... URL: From cvrebert at gmail.com Mon Jan 15 02:19:09 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Sun, 14 Jan 2007 17:19:09 -0800 Subject: [Python-ideas] fixing mutable default argument values Message-ID: <45AAD68D.5010502@gmail.com> If A.M. Kuchling's list of Python Warts is any indication, Python has removed many of the warts it once had. However, the behavior of mutable default argument values is still a frequent stumbling-block for newbies. It is also present on at least 3 different lists of Python's deficiencies ([0][1][2]). Example of current, unintuitive behavior (snipped from [0]): >>> def popo(x=[]): ... x.append(666) ... print x ... >>> popo() [666] >>> popo() [666, 666] >>> popo() [666, 666, 666] Whereas a newbie with experience with immutable default argument values would, by analogy, expect: >>> popo() [666] >>> popo() [666] >>> popo() [666] In scanning [0], [1], [2], and other similar lists, I have only found one mediocre use-case for this behavior: Using the default argument value to retain state between calls. However, as [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables. Other uses are alluded to be equally esoteric and unpythonic. To work around this behavior, the following idiom is used: def popo(x=None): if x is None: x = [] x.append(666) print x However, why should the programmer have to write this extra boilerplate code when the current, unusual behavior is only relied on by 1% of Python code? Therefore, I propose that default arguments be handled as follows in Py3K: 1. The initial default value is evaluated at definition-time (as in the current behavior). 2. That in a function call where the caller has not specified a value for an optional argument, Python calls copy.deepcopy(initial_default_value), and fills in the optional argument with the resulting value. This is fully backwards-compatible with the aforementioned workaround, and removes the need for the it, allowing one to write the first, simpler definition of popo(). Comments? - Chris Rebert [0] 10 Python pitfalls (http://zephyrfalcon.org/labs/python_pitfalls.html) [1] Python Gotchas (http://www.ferg.org/projects/python_gotchas.html#contents_item_6) [2] When Pythons Attack (http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2) From ironfroggy at gmail.com Mon Jan 15 03:58:00 2007 From: ironfroggy at gmail.com (Calvin Spealman) Date: Sun, 14 Jan 2007 21:58:00 -0500 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: <45AAD68D.5010502@gmail.com> References: <45AAD68D.5010502@gmail.com> Message-ID: <76fd5acf0701141858w7e87a67h3eb477f099e022bf@mail.gmail.com> There are a few problems here. Deep copy of the originally created default argument can be expensive and would not work in any useful way with non-literals as defaults, such as function calls or subscript lookups or even simple attributes. If any solution is possible, it would require a way to differentiate between mutable and immutable objects, and evaluate the immutables for every call. This brings on more problems as to when you do the initial evaluation, if you then might do further evaluations depending on the results of the first. Any solution would be an unjust addition of expense to defaults. On 1/14/07, Chris Rebert wrote: > If A.M. Kuchling's list of Python Warts is any indication, Python has > removed many of the warts it once had. However, the behavior of mutable > default argument values is still a frequent stumbling-block for newbies. > It is also present on at least 3 different lists of Python's > deficiencies ([0][1][2]). > > Example of current, unintuitive behavior (snipped from [0]): > >>> def popo(x=[]): > ... x.append(666) > ... print x > ... > >>> popo() > [666] > >>> popo() > [666, 666] > >>> popo() > [666, 666, 666] > > Whereas a newbie with experience with immutable default argument values > would, by analogy, expect: > >>> popo() > [666] > >>> popo() > [666] > >>> popo() > [666] > > In scanning [0], [1], [2], and other similar lists, I have only found > one mediocre use-case for this behavior: Using the default argument > value to retain state between calls. However, as [2] comments, this > purpose is much better served by decorators, classes, or (though less > preferred) global variables. Other uses are alluded to be equally > esoteric and unpythonic. > > To work around this behavior, the following idiom is used: > def popo(x=None): > if x is None: > x = [] > x.append(666) > print x > > However, why should the programmer have to write this extra boilerplate > code when the current, unusual behavior is only relied on by 1% of > Python code? > > Therefore, I propose that default arguments be handled as follows in Py3K: > 1. The initial default value is evaluated at definition-time (as in the > current behavior). > 2. That in a function call where the caller has not specified a value > for an optional argument, Python calls > copy.deepcopy(initial_default_value), and fills in the optional argument > with the resulting value. > > This is fully backwards-compatible with the aforementioned workaround, > and removes the need for the it, allowing one to write the first, > simpler definition of popo(). > > Comments? > > - Chris Rebert > > > [0] 10 Python pitfalls (http://zephyrfalcon.org/labs/python_pitfalls.html) > [1] Python Gotchas > (http://www.ferg.org/projects/python_gotchas.html#contents_item_6) > [2] When Pythons Attack > (http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/ From cvrebert at gmail.com Wed Jan 17 03:49:14 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Tue, 16 Jan 2007 18:49:14 -0800 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: <76fd5acf0701141858w7e87a67h3eb477f099e022bf@mail.gmail.com> References: <45AAD68D.5010502@gmail.com> <76fd5acf0701141858w7e87a67h3eb477f099e022bf@mail.gmail.com> Message-ID: <45AD8EAA.6040902@gmail.com> Calvin Spealman wrote: > There are a few problems here. Deep copy of the originally created > default argument can be expensive For immutables, yes, it would make an unnecessary copy, though it seems less likely to have complex immutable objects than complex mutable objects as defaults. > and would not work in any useful way > with non-literals as defaults, such as function calls or subscript > lookups or even simple attributes. Good point, I hadn't considered that. > If any solution is possible, it would require a way to differentiate > between mutable and immutable objects, and evaluate the immutables for > every call. That doesn't seem to consider the other cases you just mentioned, though I see merit in your idea. How about there be some way to specify that a default argument value be re-evaluated at every call it's required for, while all other arguments have existing semantics? Hypothetical syntax: def foo(a, b=4, c=): #b's default value is evaluated exactly once, at definition-time #c's default value is evaluated every time foo() is called and no value for c is given Where the <>s indicate these special semantics. > Any solution would be an unjust addition of expense to defaults. That sounds a bit like premature optimization to me. - Chris Rebert From jjb5 at cornell.edu Wed Jan 17 15:19:19 2007 From: jjb5 at cornell.edu (Joel Bender) Date: Wed, 17 Jan 2007 09:19:19 -0500 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: <45AD8EAA.6040902@gmail.com> References: <45AAD68D.5010502@gmail.com> <76fd5acf0701141858w7e87a67h3eb477f099e022bf@mail.gmail.com> <45AD8EAA.6040902@gmail.com> Message-ID: <45AE3067.2080605@cornell.edu> > Where the <>s indicate these special semantics. The syntax bothers me, but launched an idea: How about separating parameter default value issues from instance specific definition time objects? def foo(a, b=4, c=None): local d = Bar([2,3,4]) ... The function code can decide if it wants to use c, provided by the caller, or d when no value for c is given. The developer may decide that None is not a good value for 'no value for c', but that's a design decision. You can do this now with: def foo(a, b=4, c=None): ... foo.func_dict['d'] = Bar([2,3,4]) But I would rather see this inside foo(), jamming variables into the func_dict bothers me too :-). The new keyword would work for classes, but be a functional noop: class Snorf: local eggs = 3 spam = 4 Joel From ironfroggy at gmail.com Thu Jan 18 05:10:30 2007 From: ironfroggy at gmail.com (Calvin Spealman) Date: Wed, 17 Jan 2007 23:10:30 -0500 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: <45AE3067.2080605@cornell.edu> References: <45AAD68D.5010502@gmail.com> <76fd5acf0701141858w7e87a67h3eb477f099e022bf@mail.gmail.com> <45AD8EAA.6040902@gmail.com> <45AE3067.2080605@cornell.edu> Message-ID: <76fd5acf0701172010j554fa420q38f09151d46883af@mail.gmail.com> On 1/17/07, Joel Bender wrote: > > Where the <>s indicate these special semantics. > > The syntax bothers me, but launched an idea: How about separating > parameter default value issues from instance specific definition time > objects? > > def foo(a, b=4, c=None): > local d = Bar([2,3,4]) > ... > > The function code can decide if it wants to use c, provided by the > caller, or d when no value for c is given. The developer may decide > that None is not a good value for 'no value for c', but that's a design > decision. I dont understand how that would be different than doing c = c if c is not None else Bar([2,3,4]) > You can do this now with: > > def foo(a, b=4, c=None): > ... > > foo.func_dict['d'] = Bar([2,3,4]) This would not really work in practice. See this: >>> def f(): ... print a ... >>> f.func_dict['a'] = 10 >>> f() Traceback (most recent call last): File "", line 1, in File "", line 2, in f NameError: global name 'a' is not defined > But I would rather see this inside foo(), jamming variables into the > func_dict bothers me too :-). > > The new keyword would work for classes, but be a functional noop: > > class Snorf: > local eggs = 3 > spam = 4 > > > Joel > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/ From jjb5 at cornell.edu Thu Jan 18 14:44:12 2007 From: jjb5 at cornell.edu (Joel Bender) Date: Thu, 18 Jan 2007 08:44:12 -0500 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: <76fd5acf0701172010j554fa420q38f09151d46883af@mail.gmail.com> References: <45AAD68D.5010502@gmail.com> <76fd5acf0701141858w7e87a67h3eb477f099e022bf@mail.gmail.com> <45AD8EAA.6040902@gmail.com> <45AE3067.2080605@cornell.edu> <76fd5acf0701172010j554fa420q38f09151d46883af@mail.gmail.com> Message-ID: <45AF79AC.9090509@cornell.edu> Calvin Spealman wrote: > I dont understand how that would be different than doing > > c = c if c is not None else Bar([2,3,4]) > Because that would be calling Bar(), perhaps creating a new Bar object, every time foo() is called with None for c, which is not what a default argument values are about. I'm proposing a way to create function local singleton objects, removing them from the parameter list. def foo(x): local history = [] history.append(x) Rather than: def foo(x, history=[]): history.append(x) and then hoping that nobody calls foo() with a history parameter. Joel From jcarlson at uci.edu Thu Jan 18 17:42:00 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 18 Jan 2007 08:42:00 -0800 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: <45AF79AC.9090509@cornell.edu> References: <76fd5acf0701172010j554fa420q38f09151d46883af@mail.gmail.com> <45AF79AC.9090509@cornell.edu> Message-ID: <20070118084035.5974.JCARLSON@uci.edu> Joel Bender wrote: > > Calvin Spealman wrote: > > > I dont understand how that would be different than doing > > > > c = c if c is not None else Bar([2,3,4]) > > > > Because that would be calling Bar(), perhaps creating a new Bar object, > every time foo() is called with None for c, which is not what a default > argument values are about. I'm proposing a way to create function local > singleton objects, removing them from the parameter list. [snip] > and then hoping that nobody calls foo() with a history parameter. With your proposal, you are seeking to attempt to fix a "problem" that no one has complained about. -1 . - Josiah From jcarlson at uci.edu Thu Jan 18 17:47:14 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 18 Jan 2007 08:47:14 -0800 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: <45AAD68D.5010502@gmail.com> References: <45AAD68D.5010502@gmail.com> Message-ID: <20070118084211.5977.JCARLSON@uci.edu> Chris Rebert wrote: > If A.M. Kuchling's list of Python Warts is any indication, Python has > removed many of the warts it once had. However, the behavior of mutable > default argument values is still a frequent stumbling-block for newbies. > It is also present on at least 3 different lists of Python's > deficiencies ([0][1][2]). > > Example of current, unintuitive behavior (snipped from [0]): > >>> def popo(x=[]): > ... x.append(666) > ... print x > ... > >>> popo() > [666] > >>> popo() > [666, 666] > >>> popo() > [666, 666, 666] [snip] > Comments? As provided by Calvin Spealman, the above can be fixed with: def popo(x=None): x = x if x is not None else [] x.append(666) print x I would also mention that forcing users to learn about mutable arguments and procedural programming is not a bad thing. Learning the "gotcha" of mutable default arguments is a very useful lesson, and to remove that lesson, I believe, wouldn't necessarily help new users to Python, or new programmers in general. - Josiah From jimjjewett at gmail.com Thu Jan 18 20:00:12 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 18 Jan 2007 14:00:12 -0500 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: <20070118084035.5974.JCARLSON@uci.edu> References: <76fd5acf0701172010j554fa420q38f09151d46883af@mail.gmail.com> <45AF79AC.9090509@cornell.edu> <20070118084035.5974.JCARLSON@uci.edu> Message-ID: On 1/18/07, Josiah Carlson wrote: > Joel Bender wrote: > > Calvin Spealman wrote: > > > I dont understand how that would be different than doing > > > c = c if c is not None else Bar([2,3,4]) But he actually ones a variable that *does* keep state between calls (like a mutable default arg), but can't be overridden. > With your proposal, you are seeking to attempt to fix a "problem" that > no one has complained about. -1 . Sure they have, and they've solved it (under different names) in plenty of other languages. In python, the only current solution seems to be turning the function into a class (with self) or at least a closure. People have griped about this. For What Its Worth, my personal opinion is that having to create an object instead of a function is annoying, but not so bad (or so frequent) that it is worth special syntax. -jJ From gsakkis at rutgers.edu Thu Jan 18 21:39:01 2007 From: gsakkis at rutgers.edu (George Sakkis) Date: Thu, 18 Jan 2007 15:39:01 -0500 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: References: <76fd5acf0701172010j554fa420q38f09151d46883af@mail.gmail.com> <45AF79AC.9090509@cornell.edu> <20070118084035.5974.JCARLSON@uci.edu> Message-ID: <91ad5bf80701181239p1aa710afv4401861257d2eb4f@mail.gmail.com> On 1/18/07, Jim Jewett wrote: > On 1/18/07, Josiah Carlson wrote: > > > Joel Bender wrote: > > > Calvin Spealman wrote: > > > > I dont understand how that would be different than doing > > > > > c = c if c is not None else Bar([2,3,4]) > > But he actually ones a variable that *does* keep state between calls > (like a mutable default arg), but can't be overridden. > > > With your proposal, you are seeking to attempt to fix a "problem" that > > no one has complained about. -1 . > > Sure they have, and they've solved it (under different names) in > plenty of other languages. In python, the only current solution seems > to be turning the function into a class (with self) or at least a > closure. People have griped about this. User-defined function attributes is another handy solution. > For What Its Worth, my personal opinion is that having to create an > object instead of a function is annoying, but not so bad (or so > frequent) that it is worth special syntax. Function attributes fit the bill really good if writing a class is too much overhead. George From jimjjewett at gmail.com Thu Jan 18 23:12:52 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 18 Jan 2007 17:12:52 -0500 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: <91ad5bf80701181239p1aa710afv4401861257d2eb4f@mail.gmail.com> References: <76fd5acf0701172010j554fa420q38f09151d46883af@mail.gmail.com> <45AF79AC.9090509@cornell.edu> <20070118084035.5974.JCARLSON@uci.edu> <91ad5bf80701181239p1aa710afv4401861257d2eb4f@mail.gmail.com> Message-ID: On 1/18/07, George Sakkis wrote: > On 1/18/07, Jim Jewett wrote: > > But he actually ones a variable that *does* keep state between calls > > (like a mutable default arg), but can't be overridden. > > For What Its Worth, my personal opinion is that having to create an > > object instead of a function is annoying, but not so bad (or so > > frequent) that it is worth special syntax. > Function attributes fit the bill really good if writing a class is too > much overhead. Not really, because Python doesn't have the equivalent of "this". The only way for a function to access its own attributes is to hardcode a name and to assume the name will always refer to that same function object. In practice, it mostly works, but so does just using a global variable. -jJ From stephenemslie at gmail.com Fri Jan 19 18:08:17 2007 From: stephenemslie at gmail.com (stephen emslie) Date: Fri, 19 Jan 2007 17:08:17 +0000 Subject: [Python-ideas] tab completion in pdb Message-ID: <51f97e530701190908l7c0edff0r866ce45fac914166@mail.gmail.com> I make frequent use of the python's built-in debugger, which I think is brilliant in its simplicity. However an important feature seems to be missing: bash-like tab completion similar to that provided by the rlcompleter module. By default, Pdb and other instances of Cmd complete names for commands only. However in the context of pdb, I think it is more useful to complete identifiers and keywords in its current scope than to complete names of commands (most of which have single letter abbreviations). I believe this makes pdb a far more usable introspection tool. Implementation: I've attached a patch to pdb.py (on Python 2.4.4c1). The only real difference to rlcompleter's default complete method is that because pdb changes scope as you step through a program, rlcompleter's namespace is updated to reflect the current local and global namespace. This is my first attempt at a python patch. Any suggestions or improvements are welcome. Stephen Emslie -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pdb-python2.4.4-tabcomplete.patch Type: text/x-patch Size: 798 bytes Desc: not available URL: From aahz at pythoncraft.com Fri Jan 19 18:38:04 2007 From: aahz at pythoncraft.com (Aahz) Date: Fri, 19 Jan 2007 09:38:04 -0800 Subject: [Python-ideas] tab completion in pdb In-Reply-To: <51f97e530701190908l7c0edff0r866ce45fac914166@mail.gmail.com> References: <51f97e530701190908l7c0edff0r866ce45fac914166@mail.gmail.com> Message-ID: <20070119173804.GA8679@panix.com> On Fri, Jan 19, 2007, stephen emslie wrote: > > I've attached a patch to pdb.py (on Python 2.4.4c1). The only real > difference to rlcompleter's default complete method is that because pdb > changes scope as you step through a program, rlcompleter's namespace is > updated to reflect the current local and global namespace. > > This is my first attempt at a python patch. Any suggestions or improvements > are welcome. Please go ahead and upload this patch to SourceForge to make sure it doesn't get lost. Thanks! -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ Help a hearing-impaired person: http://rule6.info/hearing.html From cvrebert at gmail.com Sun Jan 21 07:09:46 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Sat, 20 Jan 2007 22:09:46 -0800 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: <20070118084211.5977.JCARLSON@uci.edu> References: <45AAD68D.5010502@gmail.com> <20070118084211.5977.JCARLSON@uci.edu> Message-ID: <45B303AA.1010707@gmail.com> Josiah Carlson wrote: > As provided by Calvin Spealman, the above can be fixed with: > > def popo(x=None): > x = x if x is not None else [] > x.append(666) > print x > > I would also mention that forcing users to learn about mutable arguments > and procedural programming is not a bad thing. Learning the "gotcha" > of mutable default arguments is a very useful lesson, and to remove that > lesson, I believe, wouldn't necessarily help new users to Python, or new > programmers in general. > > - Josiah > First, your 'fix' misses the point: though the proposed feature isn't necessary, and code can be written without using it, it allows mutable default argument values to be expressed more clearly and succinctly than the idiom your 'fix' uses. Second, Python isn't (inherently) about teaching new programmers about programming, and what is good for newbies isn't necessarily good for experienced programmers. And at any rate, the lesson would still exist in the form of having to use the new feature I proposed (in my strawman syntax, ), and also in doing multiplication on 2D lists (e.g. x = [mutable]*42; x[7].mutate(); x[0].mutated == True). Comments (from anyone) on my revised proposal (from the 2nd email)? I would particularly appreciate alternate syntax suggestions. - Chris Rebert From jcarlson at uci.edu Mon Jan 22 01:49:51 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun, 21 Jan 2007 16:49:51 -0800 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: <45B303AA.1010707@gmail.com> References: <20070118084211.5977.JCARLSON@uci.edu> <45B303AA.1010707@gmail.com> Message-ID: <20070121162511.59B9.JCARLSON@uci.edu> Chris Rebert wrote: > Josiah Carlson wrote: > > As provided by Calvin Spealman, the above can be fixed with: > > > > def popo(x=None): > > x = x if x is not None else [] > > x.append(666) > > print x > > > > I would also mention that forcing users to learn about mutable arguments > > and procedural programming is not a bad thing. Learning the "gotcha" > > of mutable default arguments is a very useful lesson, and to remove that > > lesson, I believe, wouldn't necessarily help new users to Python, or new > > programmers in general. > > > > - Josiah Maybe you are taking me a bit too seriously, but hopefully this will add some levity; I'm a poo-poo head. Moving on... > First, your 'fix' misses the point: though the proposed feature isn't > necessary, and code can be written without using it, it allows mutable > default argument values to be expressed more clearly and succinctly than > the idiom your 'fix' uses. As I stated, it wasn't my fix. And using previously existing syntax that adds 1 line to a function to support a particular desired result, I think, is perfectly reasonable. Had the conditional syntax been available for the last decade, those "gotchas" pages would have said "mutable default arguments are tricky, always use the following, and it will probably be the right thing" and moved on. > Second, Python isn't (inherently) about > teaching new programmers about programming, and what is good for newbies > isn't necessarily good for experienced programmers. Indeed, and what *may* be good for *certain* experienced programmers, may not be good for other experienced programmers, or for the language in general. And personally, I am not sure that I could be convinced that a syntax to support what can be emulated by a single line is even worthy of addition. In the case of decorators, or even the py3k support for argument annotation, there are certain operations that can be made *significantly* easier. In this case, I'm not convinced that the extra syntax baggage is worthwhile. Nevermind that it would be one more incompatible syntax that would make it difficult to write for 2.5/2.6 and 3.x . - Josiah From stephenemslie at gmail.com Mon Jan 22 12:57:38 2007 From: stephenemslie at gmail.com (stephen emslie) Date: Mon, 22 Jan 2007 11:57:38 +0000 Subject: [Python-ideas] tab completion in pdb In-Reply-To: <20070119173804.GA8679@panix.com> References: <51f97e530701190908l7c0edff0r866ce45fac914166@mail.gmail.com> <20070119173804.GA8679@panix.com> Message-ID: <51f97e530701220357sdc7c1d8nab5dfebca2aae605@mail.gmail.com> Thanks for taking a look. I've created a patch relative to pdb.py in svn and submitted it to sourceforge here: http://sourceforge.net/tracker/index.php?func=detail&aid=1641544&group_id=5470&atid=305470 On 1/19/07, Aahz wrote: > > On Fri, Jan 19, 2007, stephen emslie wrote: > > > > I've attached a patch to pdb.py (on Python 2.4.4c1). The only real > > difference to rlcompleter's default complete method is that because pdb > > changes scope as you step through a program, rlcompleter's namespace is > > updated to reflect the current local and global namespace. > > > > This is my first attempt at a python patch. Any suggestions or > improvements > > are welcome. > > Please go ahead and upload this patch to SourceForge to make sure it > doesn't get lost. Thanks! > -- > Aahz (aahz at pythoncraft.com) <*> > http://www.pythoncraft.com/ > > Help a hearing-impaired person: http://rule6.info/hearing.html > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ironfroggy at gmail.com Tue Jan 23 07:10:21 2007 From: ironfroggy at gmail.com (Calvin Spealman) Date: Tue, 23 Jan 2007 01:10:21 -0500 Subject: [Python-ideas] oneof() and multi split and replace for stirngs Message-ID: <76fd5acf0701222210p41f68c3asd79b03f0e508d6c7@mail.gmail.com> I don't know if this would make sense to try to push to 2.6 or 3.0. I was talking with some people about how the ability to split or replace on multiple substrings would be added to python, without adding new methods or having ugly tuple passing requirents like s.split(('foo', 'bar'), 4). This idea came to mind, so I wanted to toss it out there for scrutination. It would be a builtin, but can be implemented in python like this, basically: class oneof(list): def __init__(self, *args): list.__init__(self) self.extend(args) def __eq__(self, o): return o in self assert 'bar' == oneof('bar', 'baz') In addition to the new type, .replace, .split, and other appropriate functions would be updated to take this as the substring argument to locate and would match any one of the substrings it contains. I've asked a few people and gotten good responses on the general idea so far, but what do you all think? 1) Would the multi-substring operations be welcomed? 2) Could this be a good way to add those to the API without breaking things? 3) What version would it target? 4) What all functions and methods should support this or generally might gain value from some similar solution? -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/ From jcarlson at uci.edu Tue Jan 23 09:29:47 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 23 Jan 2007 00:29:47 -0800 Subject: [Python-ideas] oneof() and multi split and replace for stirngs In-Reply-To: <76fd5acf0701222210p41f68c3asd79b03f0e508d6c7@mail.gmail.com> References: <76fd5acf0701222210p41f68c3asd79b03f0e508d6c7@mail.gmail.com> Message-ID: <20070123000433.59CC.JCARLSON@uci.edu> "Calvin Spealman" wrote: > I don't know if this would make sense to try to push to 2.6 or 3.0. I > was talking with some people about how the ability to split or replace > on multiple substrings would be added to python, without adding new > methods or having ugly tuple passing requirents like s.split(('foo', > 'bar'), 4). This idea came to mind, so I wanted to toss it out there > for scrutination. It would be a builtin, but can be implemented in > python like this, basically: Whether it is a tuple being passed, or a "magic" container, I don't think it matters; though I would lean towards a tuple because it is 5 less characters to type out, and one fewer data types to worry about. > class oneof(list): > def __init__(self, *args): > list.__init__(self) > self.extend(args) > def __eq__(self, o): > return o in self > > assert 'bar' == oneof('bar', 'baz') If all you wanted to test was containment, it would be faster (in terms of string searching) to use a set...and perhaps just use containment. class oneof(set): def __init__(self, *args): set.__init__(self) set.update(args) assert 'bar' in oneof('bar', 'baz') The underlying string search implementation probably shouldn't concern itself with specific types. Though there are some ... gotchas if one were to provide it with an iterator that isn't restartable. > 1) Would the multi-substring operations be welcomed? This has been discussed before in python-dev, I believe the general consensus was that it would be convenient at times, but I also believe the general consensus was "use re"; def replace(options, becomes, source, count=0): pattern = '(%s)'%('|'.join(re.escape(i) for i in options)) return re.sub(pattern, becomes, source, count) > 2) Could this be a good way to add those to the API without breaking things? No. Adding a data structure that is so simple just to offer multi-substring search, replace, split, etc., isn't worthwhile. Use a tuple. > 3) What version would it target? If it were to be added, I would say 3.x. The fewer API alterations in 2.5 -> 2.6, the better. > 4) What all functions and methods should support this or generally > might gain value from some similar solution? If any, split. Beyond that, it smells to me like a regular expression job (though split still smells like a regular expression job to me). - Josiah From ironfroggy at gmail.com Tue Jan 23 09:44:59 2007 From: ironfroggy at gmail.com (Calvin Spealman) Date: Tue, 23 Jan 2007 03:44:59 -0500 Subject: [Python-ideas] oneof() and multi split and replace for stirngs In-Reply-To: <76fd5acf0701230044t6fd28878m822bf482c156839@mail.gmail.com> References: <76fd5acf0701222210p41f68c3asd79b03f0e508d6c7@mail.gmail.com> <20070123000433.59CC.JCARLSON@uci.edu> <76fd5acf0701230044t6fd28878m822bf482c156839@mail.gmail.com> Message-ID: <76fd5acf0701230044j6381e1e3n97ab0df3e99be018@mail.gmail.com> On 1/23/07, Josiah Carlson wrote: > Whether it is a tuple being passed, or a "magic" container, I don't > think it matters; though I would lean towards a tuple because it is 5 > less characters to type out, and one fewer data types to worry about. I had talked to others and the concensus was against the tuples for ugliness. s.split((a,b), c) wasn't a popular choice, but s.split(oneof(a, b), c) reads better. > This has been discussed before in python-dev, I believe the general > consensus was that it would be convenient at times, but I also believe > the general consensus was "use re"; It seems like we have a history of useful string operations being moved away from "use re" to "dont use re", such that the slightly recent startswith and endswith methods, and even split and replace themselves. I would like to see less reasons for people to worry with regular expressions until they actually need them. If we can provide a better way to get the job done, that seems like a great idea. ALSO The oneof type isn't just a single use thing. Usecode may often make use of it, and other types could benefit such as doing a lookup with d[oneof(1,2,3)] (where order would matter for priority). I think this semantic collection type would be very useful in a number of contexts where we would currently just loop or duplicate code. -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/ From jcarlson at uci.edu Tue Jan 23 18:32:02 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 23 Jan 2007 09:32:02 -0800 Subject: [Python-ideas] oneof() and multi split and replace for stirngs In-Reply-To: <76fd5acf0701230044j6381e1e3n97ab0df3e99be018@mail.gmail.com> References: <76fd5acf0701230044t6fd28878m822bf482c156839@mail.gmail.com> <76fd5acf0701230044j6381e1e3n97ab0df3e99be018@mail.gmail.com> Message-ID: <20070123090336.59CF.JCARLSON@uci.edu> "Calvin Spealman" wrote: > > On 1/23/07, Josiah Carlson wrote: > > Whether it is a tuple being passed, or a "magic" container, I don't > > think it matters; though I would lean towards a tuple because it is 5 > > less characters to type out, and one fewer data types to worry about. > > I had talked to others and the concensus was against the tuples for > ugliness. s.split((a,b), c) wasn't a popular choice, but > s.split(oneof(a, b), c) reads better. oneof = tuple Now it reads better. Stick with tuple. > > This has been discussed before in python-dev, I believe the general > > consensus was that it would be convenient at times, but I also believe > > the general consensus was "use re"; > > It seems like we have a history of useful string operations being > moved away from "use re" to "dont use re", such that the slightly > recent startswith and endswith methods, and even split and replace > themselves. I would like to see less reasons for people to worry with > regular expressions until they actually need them. If we can provide a > better way to get the job done, that seems like a great idea. Well, startswith and endswith is still computable without re or the methods, and turns out to be faster (for certain startswith and endswith operations) to do... if st[:X] == sub: or if st[-X:] == sub: (more specifically, when sub is a constant and X is previously-known) > The oneof type isn't just a single use thing. Usecode may often make > use of it, and other types could benefit such as doing a lookup with > d[oneof(1,2,3)] (where order would matter for priority). I think this > semantic collection type would be very useful in a number of contexts > where we would currently just loop or duplicate code. I took the time to look for a thread discussing something like this feature before. And I found it. Turns out I was wrong: http://mail.python.org/pipermail/python-dev/2005-September/056119.html That thread discusses an enhancement to str.[r|l|]strip(), where one could specify a list of strings to be trimmed from the end(s), not only just a group of characters (does the equivalent of .startswith and .endswith). What was the outcome? Use the X-line function that can be defined as Y. I'll stick with "use re". - Josiah From rrr at ronadam.com Tue Jan 23 21:23:02 2007 From: rrr at ronadam.com (Ron Adam) Date: Tue, 23 Jan 2007 14:23:02 -0600 Subject: [Python-ideas] oneof() and multi split and replace for stirngs In-Reply-To: <76fd5acf0701222210p41f68c3asd79b03f0e508d6c7@mail.gmail.com> References: <76fd5acf0701222210p41f68c3asd79b03f0e508d6c7@mail.gmail.com> Message-ID: <45B66EA6.5080108@ronadam.com> Calvin Spealman wrote: > I don't know if this would make sense to try to push to 2.6 or 3.0. I > was talking with some people about how the ability to split or replace > on multiple substrings would be added to python, without adding new > methods or having ugly tuple passing requirents like s.split(('foo', > 'bar'), 4). This idea came to mind, so I wanted to toss it out there > for scrutination. It would be a builtin, but can be implemented in > python like this, basically: > > class oneof(list): > def __init__(self, *args): > list.__init__(self) > self.extend(args) > def __eq__(self, o): > return o in self > > assert 'bar' == oneof('bar', 'baz') 'bar' in 'bar baz' ? > In addition to the new type, .replace, .split, and other appropriate > functions would be updated to take this as the substring argument to > locate and would match any one of the substrings it contains. I've > asked a few people and gotten good responses on the general idea so > far, but what do you all think? > 1) Would the multi-substring operations be welcomed? > 2) Could this be a good way to add those to the API without breaking things? > 3) What version would it target? > 4) What all functions and methods should support this or generally > might gain value from some similar solution? It doesn't feel right to me to have this as a built in. This falls in the category of mid level functionality, but I'm never sure where to draw the line between having simple objects to do more complex things on, vs more complex objects. I tend to prefer the first case. Having more functions to do more complex, but common, things to strings and lists of strings would be nice to have in the library. Examples of this are fnmatch.py, and textwrap.py. These both use re to do the work, but present an easier to use interface. The functions in textwrap could be extended to accept 'lists of lines'. And functions to do justifying, right and full, might also be useful. Having a simpler word matching alternative to do web style multi term searches on lists of strings, would be nice. (Something I could use to handle search requests right now.) It could be designed to be presentable to users like fnmatch. The simpler pattern matching would work in many programming situations as well. Like fnmatch, and textwrap, it would use re to do the actual work. Functions like split_pattern(), replace_pattern(), and partition_pattern() could also be available in the same module to handle the cases you suggested. Cheers, Ron From jan.kanis at phil.uu.nl Thu Jan 25 02:03:38 2007 From: jan.kanis at phil.uu.nl (Jan Kanis) Date: Thu, 25 Jan 2007 02:03:38 +0100 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: <20070121162511.59B9.JCARLSON@uci.edu> References: <20070118084211.5977.JCARLSON@uci.edu> <45B303AA.1010707@gmail.com> <20070121162511.59B9.JCARLSON@uci.edu> Message-ID: I don't like new syntax for something like this, but I think the default argument values can be fixed with semantic changes (which should not break the most common current uses): What I think should happen is compile a function like this def popo(x=[]): x.append(666) print x as if it had read def popo(x=__default_argument_marker__): if x == __default_argument_marker__: x = [] x.append(666) print x This way, every execution of popo gets its own list. Of course, __default_argument_marker__ is just a way to tell the python runtime that no argument was provided, it should not be exposed to the language. If a variable is used in the default argument, it becomes a closure variable: d = createMyListOfLists() n = getDefaultIndex() def foo(x=d[n]): x.append(666) print x this is compiled as if it had read d = createMyListOfLists() n = getDefaultIndex() def foo(x=__default_argument_marker__): if x == __default_argument_marker__: x = d[n] # d and n are closure variables x.append(666) print x d and n are looked up in foo's parent scope, which in this example is the global scope. Of course the bytecode compiler should make sure d and n don't name-clash with any variables used in the body of foo. When you use variables as default value instead of literals, I think most of the time you intend to have the function do something to the same object the variable is bound to, instead of the function creating it's own copy every time it's called. This behaviour still works with these semantics: >>> a = [] >>>def foo(x=[[],a]): >>> x[0].append(123) >>> x[1].append(123) >>> print x >>>foo() [[123], [123]] >>> foo() [[123], [123, 123]] >>> foo() [[123], [123, 123, 123]] foo is compiled as if it had read: def foo(x=__default_argument_marker__): if x == __default_argument_marker__: x = [[],a] # a is a closure variable x[0].append(123) x[1].append(123) print x An other difference between this proposal and the current situation is that it would be possible to change the value of a default argument after the function is defined. However I don't think that would really be a problem, and this behaviour is just the same as that of other closure variables. Besides, this (what I perceive as a) problem with closure variables is fixable on its own. Jan On Mon, 22 Jan 2007 01:49:51 +0100, Josiah Carlson wrote: > > Chris Rebert wrote: >> Josiah Carlson wrote: >> > As provided by Calvin Spealman, the above can be fixed with: >> > >> > def popo(x=None): >> > x = x if x is not None else [] >> > x.append(666) >> > print x >> > >> > I would also mention that forcing users to learn about mutable >> arguments >> > and procedural programming is not a bad thing. Learning the "gotcha" >> > of mutable default arguments is a very useful lesson, and to remove >> that >> > lesson, I believe, wouldn't necessarily help new users to Python, or >> new >> > programmers in general. >> > >> > - Josiah > > Maybe you are taking me a bit too seriously, but hopefully this will add > some levity; I'm a poo-poo head. Moving on... > > >> First, your 'fix' misses the point: though the proposed feature isn't >> necessary, and code can be written without using it, it allows mutable >> default argument values to be expressed more clearly and succinctly than >> the idiom your 'fix' uses. > > As I stated, it wasn't my fix. And using previously existing syntax > that adds 1 line to a function to support a particular desired result, I > think, is perfectly reasonable. Had the conditional syntax been > available for the last decade, those "gotchas" pages would have said > "mutable default arguments are tricky, always use the following, and it > will probably be the right thing" and moved on. > >> Second, Python isn't (inherently) about >> teaching new programmers about programming, and what is good for newbies >> isn't necessarily good for experienced programmers. > > Indeed, and what *may* be good for *certain* experienced programmers, > may not be good for other experienced programmers, or for the language > in general. And personally, I am not sure that I could be convinced > that a syntax to support what can be emulated by a single line is even > worthy of addition. In the case of decorators, or even the py3k support > for argument annotation, there are certain operations that can be made > *significantly* easier. In this case, I'm not convinced that the extra > syntax baggage is worthwhile. > > Nevermind that it would be one more incompatible syntax that would make > it difficult to write for 2.5/2.6 and 3.x . > > > - Josiah > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From jimjjewett at gmail.com Thu Jan 25 15:41:54 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 25 Jan 2007 09:41:54 -0500 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: References: <20070118084211.5977.JCARLSON@uci.edu> <45B303AA.1010707@gmail.com> <20070121162511.59B9.JCARLSON@uci.edu> Message-ID: On 1/24/07, Jan Kanis wrote: > I don't like new syntax for something like this, but I think the default > argument values can be fixed with semantic changes (which should not break > the most common current uses): > > What I think should happen is compile a function like this > > def popo(x=[]): > x.append(666) > print x > > as if it had read > > def popo(x=__default_argument_marker__): > if x == __default_argument_marker__: > x = [] > x.append(666) > print x How is this different from the x=None idiom of today? def f(inlist=None): if inlist is None: inlist=[] The if (either 2 lines or against PEP8) is a bit ugly, but Calvin pointed out that you can now write it as def f(inlist=None): inlist = inlist if (inlist is not None) else [] I see below that you give it slightly different semantics, but I'm not entirely sure how to tell when those different semantics should apply (always? when the variable name is marked with __*__? When a specific non-None singleton appears?), or why you would ever want them. > When you use variables as default value instead of literals, I think most > of the time you intend to have the function do something to the same > object the variable is bound to, instead of the function creating it's own > copy every time it's called. This behaviour still works with these > semantics: > >>> a = [] > >>>def foo(x=[[],a]): > >>> x[0].append(123) > >>> x[1].append(123) > >>> print x > >>>foo() > [[123], [123]] > >>> foo() > [[123], [123, 123]] > >>> foo() > [[123], [123, 123, 123]] So you're saying that x[1] should be persistent because it (also) has a name (as 'a'), but x[0] should be recreated fresh on each call because it doesn't? -jJ From tomerfiliba at gmail.com Thu Jan 25 21:48:00 2007 From: tomerfiliba at gmail.com (tomer filiba) Date: Thu, 25 Jan 2007 22:48:00 +0200 Subject: [Python-ideas] new pickle semantics/API In-Reply-To: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com> References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com> Message-ID: <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com> ---------- Forwarded message ---------- From: tomer filiba Date: Jan 24, 2007 10:45 PM Subject: new pickle semantics/API To: Python-3000 at python.org i'm having great trouble in RPyC with pickling object proxies. several users have asked for this feature, but no matter how hard i try to "bend the truth", pickle always complains. it uses type(obj) for the dispatching, which "uncovers" the object is actually a proxy, rather than a real object. recap: RPyC uses local proxies that refer to objects of a remote interpreter (another process/machine). if you'd noticed, every RPC framework has its own serializer. for example banna/jelly in twisted and bunch of other XML serializers, and what not. for RPyC i wrote yet another serializer, but for different purposes, so it's not relevant for the issue at hand. what i want is a standard serialization *API*. the idea is that any framework could make use of this API, and that it would be generic enough to eliminate copy_reg and other misfortunes. this also means the built in types should be familiarized with this API. - - - - - - - - for example, currently the builtin types don't support __reduce__, and require pickle to use it's own internal registry. moreover, __reduce__ is very pickle-specific (i.e., it takes the protocol number). what i'm after is an API for "simplifying" complex objects into simpler parts. here's the API i'm suggesting: def __getstate__(self): # return a tuple of (type(self), obj), where obj is a simplified # version of self @classmethod def __setstate__(cls, state): # return an instance of cls, with the given state well, you may already know these two, although their semantics are different. but wait, there's more! the idea is of having the following simple building blocks: * integers (int/long) * strings (str) * arrays (tuples) all picklable objects should be able to express themselves as a collection of these building blocks. of course this will be recursive, i.e., object X could simplify itself as object Y, where object Y might go further simplification, until we are left with building blocks only. for example: * int - return self * float - string in the format "[+-]X.YYYe[+-]EEE" * complex - two floats * tuple - tuple of its simplified elements * list - tuple of its simplified elements * dict - a tuple of (key, value) tuples * set - a tuple of its items * file - raises TypeError("can't be simplified") all in all, i choose to call that *simplification* rather than *serialization*, as serialization is more about converting the simplified objects into a sequence of bytes. my suggestion leaves that out for the implementers of specific serializers. so this is how a typical serializer (e.g., pickle) would be implemented: * define its version of a "recursive simplifier" * optionally use a "memo" to remember objects that were already visited (so they would be serialized by reference rather than by value) * define its flavor of converting ints, strings, and arrays to bytes (binary, textual, etc. etc.) - - - - - - - - the default implementation of __getstate__, in object.__getstate__, will simply return self.__dict__ and any self.__slots__ this removes the need for __reduce__, __reduce_ex__, and copy_reg, and simplifies pickle greatly. it does require, however, adding support for simplification for all builtin types... but this doesn't call for much code: def PyList_GetState(self): state = tuple(PyObject_GetState(item) for item in self) return PyListType, state also note that it makes the copy module much simpler: def copy(obj): state = obj.__getstate__() return type(obj).__setstate__(state) - - - - - - - - executive summary: simplifying object serialization and copying by revising __getstate__ and __setstate__, so that they return a "simplified" version of the object. this new mechanism should become an official API to getting or setting the "contents" of objects (either builtin or user-defined). having this unified mechanism, pickling proxy objects would work as expected. if there's interest, i'll write a pep-like document to explain all the semantics. -tomer -------------- next part -------------- An HTML attachment was scrubbed... URL: From tomerfiliba at gmail.com Thu Jan 25 22:15:04 2007 From: tomerfiliba at gmail.com (tomer filiba) Date: Thu, 25 Jan 2007 23:15:04 +0200 Subject: [Python-ideas] new pickle semantics/API In-Reply-To: <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com> References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com> <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com> Message-ID: <1d85506f0701251315p2a447195g401e272861a75bdc@mail.gmail.com> there's a bug in the copy function that i wanted to fix: def copy(obj): cls, state = obj.__getstate__() return cls.__setstate__(state) also, but there's a reason why __getstate__ returns "(cls, state)" rather than just "state", and that's to keep things agile. i don't want to be necessarily tightly-coupled to a certain type. the cls will be used to reconstruct the object (cls.__setstate__), much like the function returned by __reduce__, so normally, __getstate__ would just return self.__class__ for cls. but there are times, especially when object proxies are involved, that we may want to "lie" about the actual type, i.e., use a different type to reconstruct the object with. here's an example that shows why: class ListProxy: ... def __getstate__(self): return list, self._value instances of ListProxy, when stored in a file (i.e., shelf), want to be pickled by value. moreover, when they are loaded from a file, they want to loaded as actual lists, not proxies, as the proxied object is long lost. so returning a tuple of (cls, state) gives more freedom to frameworks and other utilities. of course, again, most code would just return (self.__class__, state) -tomer From collinw at gmail.com Thu Jan 25 22:21:54 2007 From: collinw at gmail.com (Collin Winter) Date: Thu, 25 Jan 2007 15:21:54 -0600 Subject: [Python-ideas] new pickle semantics/API In-Reply-To: <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com> References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com> <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com> Message-ID: <43aa6ff70701251321y8885892j28916f83b9a4d8c0@mail.gmail.com> On 1/25/07, tomer filiba wrote: > for example: > * int - return self > * float - string in the format "[+-]X.YYYe[+-]EEE" > * complex - two floats > * tuple - tuple of its simplified elements > * list - tuple of its simplified elements > * dict - a tuple of (key, value) tuples > * set - a tuple of its items > * file - raises TypeError("can't be simplified") [snip] > the default implementation of __getstate__, in object.__getstate__, > will simply return self.__dict__ and any self.__slots__ How will e.g. classes be simplified? Can I simplify a dictionary with function objects for values? Collin Winter From jcarlson at uci.edu Thu Jan 25 22:43:54 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 25 Jan 2007 13:43:54 -0800 Subject: [Python-ideas] new pickle semantics/API In-Reply-To: <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com> References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com> <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com> Message-ID: <20070125130548.59F4.JCARLSON@uci.edu> "tomer filiba" wrote: > i'm having great trouble in RPyC with pickling object proxies. > several users have asked for this feature, but no matter how hard > i try to "bend the truth", pickle always complains. it uses > type(obj) for the dispatching, which "uncovers" the object > is actually a proxy, rather than a real object. [snip] > also note that it makes the copy module much simpler: > def copy(obj): > state = obj.__getstate__() > return type(obj).__setstate__(state) I presume you mean... def copy(obj): typ, state = obj.__getstate__() return typ.__setstate__(state) > - - - - - - - - > executive summary: > simplifying object serialization and copying by revising > __getstate__ and __setstate__, so that they return a > "simplified" version of the object. > > this new mechanism should become an official API to > getting or setting the "contents" of objects (either builtin or > user-defined). > > having this unified mechanism, pickling proxy objects would > work as expected. > > if there's interest, i'll write a pep-like document to explain > all the semantics. Overall, I like the idea; I'm a big fan of simplifying object persistence and/or serialization. A part of me also likes how the objects can choose to lie about their types. But another part of me says; the basic objects that you specified already have a format that is unambiguous, repr(obj). They also are able to be reconstructed from their component parts via eval(repr(obj)), or even via the 'unrepr' function in the ConfigObj module. It doesn't handle circular referencse. Even better, it has 3 native representations; repr(a).encode('zlib'), repr(a), pprint.pprint(a); each offering a different amount of user readability. I digress. I believe the biggest problem with the proposal, as specified, is that changing the semantics of __getstate__ and __setstate__ is a bad idea. Add a new pair of methods and ask the twisted people what they think. My only criticism will then be the strawman repr/unrepr. - Josiah From tomerfiliba at gmail.com Thu Jan 25 22:55:19 2007 From: tomerfiliba at gmail.com (tomer filiba) Date: Thu, 25 Jan 2007 23:55:19 +0200 Subject: [Python-ideas] new pickle semantics/API In-Reply-To: <43aa6ff70701251321y8885892j28916f83b9a4d8c0@mail.gmail.com> References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com> <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com> <43aa6ff70701251321y8885892j28916f83b9a4d8c0@mail.gmail.com> Message-ID: <1d85506f0701251355x76a1c6f8yef7196304dab1054@mail.gmail.com> On 1/25/07, Collin Winter wrote: > How will e.g. classes be simplified? Can I simplify a dictionary with > function objects for values? > well, pickle just saves them as a global name (modulename.classname). so types would generally just return themselves as primitives, and let the actual simplifier do the trick. it may choose to save the type's dict, or just a global name. that's up to the serializer-dependent simplifier. it's good you mentioned that, because it reminded me of something i forgot. for instance, code objects will be serialized by value, so you could actually pickle functions and classes. this means pyc files could become just a pickle of the module, i.e.: import foo pickle.dump(foo, open("foo.pyc", "w")) but again, that's up to the serializer. an enhanced pickle could do that. -tomer From tomerfiliba at gmail.com Thu Jan 25 23:16:15 2007 From: tomerfiliba at gmail.com (tomer filiba) Date: Fri, 26 Jan 2007 00:16:15 +0200 Subject: [Python-ideas] new pickle semantics/API In-Reply-To: <20070125130548.59F4.JCARLSON@uci.edu> References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com> <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com> <20070125130548.59F4.JCARLSON@uci.edu> Message-ID: <1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com> On 1/25/07, Josiah Carlson wrote: > Overall, I like the idea; I'm a big fan of simplifying object > persistence and/or serialization. A part of me also likes how the > objects can choose to lie about their types. > > But another part of me says; the basic objects that you specified > already have a format that is unambiguous, repr(obj). They also are > able to be reconstructed from their component parts via eval(repr(obj)), > or even via the 'unrepr' function in the ConfigObj module. It doesn't > handle circular referencse. well, repr is fine for most simple things, but you don't use repr to serialize objects, right? it's not powerful/introspective enough. besides repr is meant to be readable, while __getstate__ can return any object. imagine this: class complex: def __repr__(self): return "(%f+%fj)" % (self.real, self.imag) def __getstate__(self): return self.__class__, (self.real, self.imag) repr is made for humans of course, while serialization is made for machines. they serves different purposes, so they need different APIs. > Even better, it has 3 native representations; repr(a).encode('zlib'), > repr(a), pprint.pprint(a); each offering a different amount of user > readability. I digress. you may have digressed, but that's a good point -- that's exactly why i do NOT specify how objects are encoded as a stream of bytes. all i'm after is the state of the object (which is expressed in terms of other, more primitive objects). you can think of repr as a textual serializer to some extent, that can use the proposed __getstate__ API. pprint is yet another form of serializer. > I believe the biggest problem with the proposal, as specified, is that > changing the semantics of __getstate__ and __setstate__ is a bad idea. > Add a new pair of methods and ask the twisted people what they think. > My only criticism will then be the strawman repr/unrepr. i'll try to come up with new names... but i don't have any ideas at the moment. -tomer From collinw at gmail.com Thu Jan 25 23:22:19 2007 From: collinw at gmail.com (Collin Winter) Date: Thu, 25 Jan 2007 16:22:19 -0600 Subject: [Python-ideas] new pickle semantics/API In-Reply-To: <1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com> References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com> <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com> <20070125130548.59F4.JCARLSON@uci.edu> <1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com> Message-ID: <43aa6ff70701251422v79c4818as141770b5c4dc0775@mail.gmail.com> On 1/25/07, tomer filiba wrote: > On 1/25/07, Josiah Carlson wrote: > > I believe the biggest problem with the proposal, as specified, is that > > changing the semantics of __getstate__ and __setstate__ is a bad idea. > > Add a new pair of methods and ask the twisted people what they think. > > My only criticism will then be the strawman repr/unrepr. > > i'll try to come up with new names... but i don't have any ideas > at the moment. The "__getstate__" and "__setstate__" names don't really work for me either, especially since __setstate__ creates a new object, as opposed to changing the state of an existing object. Since this proposal is all about simplification, how about something like "__simplify__" and "__expand__", respectively? Collin Winter From tomerfiliba at gmail.com Thu Jan 25 23:41:36 2007 From: tomerfiliba at gmail.com (tomer filiba) Date: Fri, 26 Jan 2007 00:41:36 +0200 Subject: [Python-ideas] new pickle semantics/API In-Reply-To: <43aa6ff70701251422v79c4818as141770b5c4dc0775@mail.gmail.com> References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com> <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com> <20070125130548.59F4.JCARLSON@uci.edu> <1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com> <43aa6ff70701251422v79c4818as141770b5c4dc0775@mail.gmail.com> Message-ID: <1d85506f0701251441k4c960736yc329c1016e331d9c@mail.gmail.com> On 1/26/07, Collin Winter wrote: > The "__getstate__" and "__setstate__" names don't really work for me > either, especially since __setstate__ creates a new object, as opposed > to changing the state of an existing object. Since this proposal is > all about simplification, how about something like "__simplify__" and > "__expand__", respectively? well, i like __simplify__, but __expand__ seems wrong to me. some other suggestions i looked up: __complicate__ (:-)) __rebuild__ __reconstruct__ (too long?) __restore__ (but that's semantically the same as __setstate__) i'd vote for __rebuild__. any other ideas? -tomer From phd at phd.pp.ru Thu Jan 25 23:48:31 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Fri, 26 Jan 2007 01:48:31 +0300 Subject: [Python-ideas] new pickle semantics/API In-Reply-To: <1d85506f0701251441k4c960736yc329c1016e331d9c@mail.gmail.com> References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com> <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com> <20070125130548.59F4.JCARLSON@uci.edu> <1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com> <43aa6ff70701251422v79c4818as141770b5c4dc0775@mail.gmail.com> <1d85506f0701251441k4c960736yc329c1016e331d9c@mail.gmail.com> Message-ID: <20070125224831.GA7310@phd.pp.ru> On Fri, Jan 26, 2007 at 12:41:36AM +0200, tomer filiba wrote: > well, i like __simplify__, but __expand__ seems wrong to me. > some other suggestions i looked up: > > __complicate__ (:-)) > __rebuild__ > __reconstruct__ (too long?) > __restore__ (but that's semantically the same as __setstate__) > > i'd vote for __rebuild__. any other ideas? __(un)pickle__ __(de)serialize__ Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From collinw at gmail.com Fri Jan 26 00:00:47 2007 From: collinw at gmail.com (Collin Winter) Date: Thu, 25 Jan 2007 17:00:47 -0600 Subject: [Python-ideas] new pickle semantics/API In-Reply-To: <1d85506f0701251355x76a1c6f8yef7196304dab1054@mail.gmail.com> References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com> <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com> <43aa6ff70701251321y8885892j28916f83b9a4d8c0@mail.gmail.com> <1d85506f0701251355x76a1c6f8yef7196304dab1054@mail.gmail.com> Message-ID: <43aa6ff70701251500o102615a5pfb3664b47979d03e@mail.gmail.com> On 1/25/07, tomer filiba wrote: > On 1/25/07, Collin Winter wrote: > > How will e.g. classes be simplified? Can I simplify a dictionary with > > function objects for values? [snip] > it's good you mentioned that, because it reminded me of something > i forgot. for instance, code objects will be serialized by value, > so you could actually pickle functions and classes. Are you intending to simplify code objects to the co_* attributes? co_code is a string of interpreter-specific bytecode, which would be pretty much useless outside of a copy() function. Whatever the scheme, it will need to take into account cell objects and globals. Collin Winter From collinw at gmail.com Fri Jan 26 00:02:26 2007 From: collinw at gmail.com (Collin Winter) Date: Thu, 25 Jan 2007 17:02:26 -0600 Subject: [Python-ideas] new pickle semantics/API In-Reply-To: <1d85506f0701251441k4c960736yc329c1016e331d9c@mail.gmail.com> References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com> <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com> <20070125130548.59F4.JCARLSON@uci.edu> <1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com> <43aa6ff70701251422v79c4818as141770b5c4dc0775@mail.gmail.com> <1d85506f0701251441k4c960736yc329c1016e331d9c@mail.gmail.com> Message-ID: <43aa6ff70701251502n1ca075e5ne15e2e8c2fbdb1a2@mail.gmail.com> On 1/25/07, tomer filiba wrote: > On 1/26/07, Collin Winter wrote: > > The "__getstate__" and "__setstate__" names don't really work for me > > either, especially since __setstate__ creates a new object, as opposed > > to changing the state of an existing object. Since this proposal is > > all about simplification, how about something like "__simplify__" and > > "__expand__", respectively? > > well, i like __simplify__, but __expand__ seems wrong to me. > some other suggestions i looked up: > > __complicate__ (:-)) > __rebuild__ > __reconstruct__ (too long?) > __restore__ (but that's semantically the same as __setstate__) > > i'd vote for __rebuild__. any other ideas? I like __rebuild__. Collin Winter From jcarlson at uci.edu Fri Jan 26 00:29:02 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 25 Jan 2007 15:29:02 -0800 Subject: [Python-ideas] new pickle semantics/API In-Reply-To: <1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com> References: <20070125130548.59F4.JCARLSON@uci.edu> <1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com> Message-ID: <20070125151500.59F9.JCARLSON@uci.edu> "tomer filiba" wrote: > On 1/25/07, Josiah Carlson wrote: > > Overall, I like the idea; I'm a big fan of simplifying object > > persistence and/or serialization. A part of me also likes how the > > objects can choose to lie about their types. > > > > But another part of me says; the basic objects that you specified > > already have a format that is unambiguous, repr(obj). They also are > > able to be reconstructed from their component parts via eval(repr(obj)), > > or even via the 'unrepr' function in the ConfigObj module. It doesn't > > handle circular referencse. > > well, repr is fine for most simple things, but you don't use repr to > serialize objects, right? it's not powerful/introspective enough. > besides repr is meant to be readable, while __getstate__ can return > any object. imagine this: I use repr to serialize objects all the time. ConfigObj is great when I want to handle python-based configuration information, and/or I don't want to worry about the security implications of 'eval(arbitrary string)', or 'import module'. With a proper __repr__ method, I can even write towards your API: class mylist(object): def __repr__(self): state = ... return 'mylist.__setstate__(%r)'%(state,) > class complex: > def __repr__(self): > return "(%f+%fj)" % (self.real, self.imag) I would use 'return "(%r+%rj)"% (self.real, self.imag)', but it doesn't much matter. > repr is made for humans of course, while serialization is > made for machines. they serves different purposes, > so they need different APIs. I happen to disagree. The only reason to use a different representation or API is if there are size and/or performance benefits to offering a machine readable vs. human readable format. I'm know that there are real performance advantages to using (c)Pickle over repr/unrepr, but I use it also so that I can change settings with notepad (as has been necessary on occasion). > > Even better, it has 3 native representations; repr(a).encode('zlib'), > > repr(a), pprint.pprint(a); each offering a different amount of user > > readability. I digress. > > you may have digressed, but that's a good point -- that's exactly > why i do NOT specify how objects are encoded as a stream of bytes. > > all i'm after is the state of the object (which is expressed in terms of > other, more primitive objects). Right, but as 'primative objects' go, you cant get significantly more primitive than producing a string that can be naively understood by someone familliar with Python *and* the built-in Python parser. Nevermind that it works *today* with all of the types you specified earlier (with the exception of file objects - which you discover on parsing/reproducing the object). > you can think of repr as a textual serializer to some extent, that > can use the proposed __getstate__ API. pprint is yet another > form of serializer. Well, pprint is more or less a pretty repr. > > I believe the biggest problem with the proposal, as specified, is that > > changing the semantics of __getstate__ and __setstate__ is a bad idea. > > Add a new pair of methods and ask the twisted people what they think. > > My only criticism will then be the strawman repr/unrepr. > > i'll try to come up with new names... but i don't have any ideas > at the moment. Like Colin, I also like __rebuild__. - Josiah From collinw at gmail.com Fri Jan 26 00:39:58 2007 From: collinw at gmail.com (Collin Winter) Date: Thu, 25 Jan 2007 17:39:58 -0600 Subject: [Python-ideas] new pickle semantics/API In-Reply-To: <20070125151500.59F9.JCARLSON@uci.edu> References: <20070125130548.59F4.JCARLSON@uci.edu> <1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com> <20070125151500.59F9.JCARLSON@uci.edu> Message-ID: <43aa6ff70701251539u5f0f0131n20ed52828a6b1fb@mail.gmail.com> On 1/25/07, Josiah Carlson wrote: > "tomer filiba" wrote: > > repr is made for humans of course, while serialization is > > made for machines. they serves different purposes, > > so they need different APIs. > > I happen to disagree. The only reason to use a different representation > or API is if there are size and/or performance benefits to offering a > machine readable vs. human readable format. [snip] > I'm know that there are real performance advantages to using (c)Pickle > over repr/unrepr, but I use it also so that I can change settings with > notepad (as has been necessary on occasion). [snip] > > all i'm after is the state of the object (which is expressed in terms of > > other, more primitive objects). > > Right, but as 'primative objects' go, you cant get significantly more > primitive than producing a string that can be naively understood by > someone familliar with Python *and* the built-in Python parser. > Nevermind that it works *today* with all of the types you specified > earlier (with the exception of file objects - which you discover on > parsing/reproducing the object). You use pickle because it's more general than repr/unrepr, not because it's faster or the result is smaller. Assuming you're talking about the ConfigObj's unrepr mode (http://www.voidspace.org.uk/python/configobj.html#unrepr-mode), """ The types that unrepr can work with are : strings, lists tuples None, True, False dictionaries, integers, floats longs and complex numbers You can't store classes, types or instances. """ Pickle -- and the simplification API Tomer is proposing -- is far more general than that. repr/unrepr is in no way a substitute. Collin Winter From jcarlson at uci.edu Fri Jan 26 01:14:42 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 25 Jan 2007 16:14:42 -0800 Subject: [Python-ideas] new pickle semantics/API In-Reply-To: <43aa6ff70701251539u5f0f0131n20ed52828a6b1fb@mail.gmail.com> References: <20070125151500.59F9.JCARLSON@uci.edu> <43aa6ff70701251539u5f0f0131n20ed52828a6b1fb@mail.gmail.com> Message-ID: <20070125160432.59FC.JCARLSON@uci.edu> "Collin Winter" wrote: > On 1/25/07, Josiah Carlson wrote: > > "tomer filiba" wrote: > > > repr is made for humans of course, while serialization is > > > made for machines. they serves different purposes, > > > so they need different APIs. > > > > I happen to disagree. The only reason to use a different representation > > or API is if there are size and/or performance benefits to offering a > > machine readable vs. human readable format. > [snip] > > I'm know that there are real performance advantages to using (c)Pickle > > over repr/unrepr, but I use it also so that I can change settings with > > notepad (as has been necessary on occasion). > [snip] > > > all i'm after is the state of the object (which is expressed in terms of > > > other, more primitive objects). > > > > Right, but as 'primative objects' go, you cant get significantly more > > primitive than producing a string that can be naively understood by > > someone familliar with Python *and* the built-in Python parser. > > Nevermind that it works *today* with all of the types you specified > > earlier (with the exception of file objects - which you discover on > > parsing/reproducing the object). > > You use pickle because it's more general than repr/unrepr, not because > it's faster or the result is smaller. Assuming you're talking about > the ConfigObj's unrepr mode > (http://www.voidspace.org.uk/python/configobj.html#unrepr-mode), *I* use pickle when I want speed. I use repr and unrepr (from configobj) when I want to be able to change things by hand. None of the objects I transfer between processes/disk/whatever are ever more than base Python types. As for whatever *others* do, I don't know, I haven't done a survey. They may do as you say. > Pickle -- and the simplification API Tomer is proposing -- is far more > general than that. repr/unrepr is in no way a substitute. I never claimed it was a substitute, I stated quite clearly up front; "My only criticism will then be the strawman repr/unrepr." The alternate strawman is repr/eval, can be made to support basically everything except for self-referential objects. It still has that horrible security issue, but that also exists in pickle, and could still exist in the simplification/rebuilding rutines specified by Tomer. - Josiah From cvrebert at gmail.com Fri Jan 26 04:36:33 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Thu, 25 Jan 2007 19:36:33 -0800 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: References: <20070118084211.5977.JCARLSON@uci.edu> <45B303AA.1010707@gmail.com> <20070121162511.59B9.JCARLSON@uci.edu> Message-ID: <45B97741.1040406@gmail.com> So, basically the same as my proposal except without syntax changes and with the default argument value semantics applied to all arguments. I'm okay with this, however some possible performance issues might exist with re-evaluating expensive default arg vals on every call where they're required. This is basically why my proposal required new syntax, so that people could use the old "eval once at definition-time" semantics on expensive default values to get better performance. Also, since people (including me) don't really like the <>-syntax, I've come up with some real syntax possibilities for my proposal: def foo(bar=new baz): def foo(bar=fresh baz): def foo(bar=separate baz): def foo(bar=another baz): def foo(bar=unique baz): I personally like 'fresh'. It seems accurate and not too long. I welcome other thoughts on better syntax for this. If the performance issues aren't significant, I'm all for your proposal. It'd be nice to not to have to add new syntax. - Chris Rebert Jan Kanis wrote: > I don't like new syntax for something like this, but I think the default > argument values can be fixed with semantic changes (which should not > break the most common current uses): > > What I think should happen is compile a function like this > > def popo(x=[]): > x.append(666) > print x > > as if it had read > > def popo(x=__default_argument_marker__): > if x == __default_argument_marker__: > x = [] > x.append(666) > print x > > This way, every execution of popo gets its own list. Of course, > __default_argument_marker__ is just a way to tell the python runtime > that no argument was provided, it should not be exposed to the language. > > If a variable is used in the default argument, it becomes a closure > variable: > > d = createMyListOfLists() > n = getDefaultIndex() > > def foo(x=d[n]): > x.append(666) > print x > > > this is compiled as if it had read > > d = createMyListOfLists() > n = getDefaultIndex() > > def foo(x=__default_argument_marker__): > if x == __default_argument_marker__: > x = d[n] # d and n are closure variables > x.append(666) > print x > > > d and n are looked up in foo's parent scope, which in this example is > the global scope. Of course the bytecode compiler should make sure d and > n don't name-clash with any variables used in the body of foo. > > When you use variables as default value instead of literals, I think > most of the time you intend to have the function do something to the > same object the variable is bound to, instead of the function creating > it's own copy every time it's called. This behaviour still works with > these semantics: > >>>> a = [] >>>> def foo(x=[[],a]): >>>> x[0].append(123) >>>> x[1].append(123) >>>> print x >>>> foo() > [[123], [123]] >>>> foo() > [[123], [123, 123]] >>>> foo() > [[123], [123, 123, 123]] > > foo is compiled as if it had read: > > def foo(x=__default_argument_marker__): > if x == __default_argument_marker__: > x = [[],a] # a is a closure variable > x[0].append(123) > x[1].append(123) > print x > > An other difference between this proposal and the current situation is > that it would be possible to change the value of a default argument > after the function is defined. However I don't think that would really > be a problem, and this behaviour is just the same as that of other > closure variables. Besides, this (what I perceive as a) problem with > closure variables is fixable on its own. > > Jan > > > On Mon, 22 Jan 2007 01:49:51 +0100, Josiah Carlson > wrote: > >> >> Chris Rebert wrote: >>> Josiah Carlson wrote: >>> > As provided by Calvin Spealman, the above can be fixed with: >>> > >>> > def popo(x=None): >>> > x = x if x is not None else [] >>> > x.append(666) >>> > print x >>> > >>> > I would also mention that forcing users to learn about mutable >>> arguments >>> > and procedural programming is not a bad thing. Learning the "gotcha" >>> > of mutable default arguments is a very useful lesson, and to remove >>> that >>> > lesson, I believe, wouldn't necessarily help new users to Python, >>> or new >>> > programmers in general. >>> > >>> > - Josiah >> >> Maybe you are taking me a bit too seriously, but hopefully this will add >> some levity; I'm a poo-poo head. Moving on... >> >> >>> First, your 'fix' misses the point: though the proposed feature isn't >>> necessary, and code can be written without using it, it allows mutable >>> default argument values to be expressed more clearly and succinctly than >>> the idiom your 'fix' uses. >> >> As I stated, it wasn't my fix. And using previously existing syntax >> that adds 1 line to a function to support a particular desired result, I >> think, is perfectly reasonable. Had the conditional syntax been >> available for the last decade, those "gotchas" pages would have said >> "mutable default arguments are tricky, always use the following, and it >> will probably be the right thing" and moved on. >> >>> Second, Python isn't (inherently) about >>> teaching new programmers about programming, and what is good for newbies >>> isn't necessarily good for experienced programmers. >> >> Indeed, and what *may* be good for *certain* experienced programmers, >> may not be good for other experienced programmers, or for the language >> in general. And personally, I am not sure that I could be convinced >> that a syntax to support what can be emulated by a single line is even >> worthy of addition. In the case of decorators, or even the py3k support >> for argument annotation, there are certain operations that can be made >> *significantly* easier. In this case, I'm not convinced that the extra >> syntax baggage is worthwhile. >> >> Nevermind that it would be one more incompatible syntax that would make >> it difficult to write for 2.5/2.6 and 3.x . >> >> >> - Josiah >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas > > > From tomerfiliba at gmail.com Fri Jan 26 10:29:50 2007 From: tomerfiliba at gmail.com (tomer filiba) Date: Fri, 26 Jan 2007 11:29:50 +0200 Subject: [Python-ideas] new pickle semantics/API In-Reply-To: <20070125160432.59FC.JCARLSON@uci.edu> References: <20070125151500.59F9.JCARLSON@uci.edu> <43aa6ff70701251539u5f0f0131n20ed52828a6b1fb@mail.gmail.com> <20070125160432.59FC.JCARLSON@uci.edu> Message-ID: <1d85506f0701260129s4410881coeac2660671aaa7d1@mail.gmail.com> On 1/26/07, Josiah Carlson wrote: > *I* use pickle when I want speed. I use repr and unrepr (from configobj) > when I want to be able to change things by hand. None of the objects I > transfer between processes/disk/whatever are ever more than base Python > types. > > As for whatever *others* do, I don't know, I haven't done a survey. > They may do as you say. well, since this proposal originated from an RPC point-of-view, i, for once, also need to transfer full-blown user-defined objects and classes. for that, i need to be able to get the full state of the object, and the means to reconstruct it later. pickling the "primitives" was never a problem, because they have a well defined interface. although cyclic references call for something stronger than repr... and this is what i mean by "repr is for humans". > I never claimed it was a substitute, I stated quite clearly up front; > "My only criticism will then be the strawman repr/unrepr." The > alternate strawman is repr/eval, can be made to support basically > everything except for self-referential objects. It still has that > horrible security issue, but that also exists in pickle, and could still > exist in the simplification/rebuilding rutines specified by Tomer. well, pickle is unsafe for one main reason -- it's omnipotent. it performs arbitrary imports and object instantiation, which are equivalent to eval of arbitrary strings. BUT, it has nothing to do with the way it gets of sets the *state* of objects. to solve that, we can have a "capability-based pickle": dumping objects was never a security issue, it's the loading part that's dangerous. we can add a new function, loadsec(), that takes both the string to load and a set of classes it may use to __rebuilt__ the object. capabilities = {"list" : list, "str" : str, "os.stat_result" : os.stat_result} loadsec(data, capabilities) that way, you can control the objects that will be instantiated, which you trust, so no arbitrary code may be executed behind the scenes. for the "classic" unsafe load(), we can pass a magical dict-like thing that imports names via __getitem__ if i had a way to control what pickle.loads has access to, i wouldn't need to write my own serializer.. http://sebulbasvn.googlecode.com/svn/trunk/rpyc/core/marshal/brine.py > > you may have digressed, but that's a good point -- that's exactly > > why i do NOT specify how objects are encoded as a stream of bytes. > > > > all i'm after is the state of the object (which is expressed in terms of > > other, more primitive objects). > > Right, but as 'primative objects' go, you cant get significantly more > primitive than producing a string that can be naively understood by > someone familliar with Python *and* the built-in Python parser. but that's not the issue. when i send my objects across a socket back and forth, i want something that is fast, compact, and safe. i don't care for anyone sniffing my wire to "easily understand" what i'm sending... i mean, it's not like i encrypt the data, but readability doesn't count here. again, __simplify__ and __rebuild__ offer a mechanism for serializer-implementors, which may choose different encoding schemes for their internal purposes. this API isn't meant for the end user. i'll write a pre-pep to try to clarify it all in an orderly manner. -tomer From tomerfiliba at gmail.com Fri Jan 26 16:13:12 2007 From: tomerfiliba at gmail.com (tomer filiba) Date: Fri, 26 Jan 2007 17:13:12 +0200 Subject: [Python-ideas] simplification pep-like thing Message-ID: <1d85506f0701260713k1fe0d548u8103e86a7dfffdf7@mail.gmail.com> this text needs more thought and rephrasing, but i think that it covers all the details. -------------------------------------------------------------------------- Abstract ========= The simplification API provides a mechanism to get the state of an object, as well as reconstructing an object with a given state. This new API would affect pickle, copy_reg, and copy modules, as well as the writers of serializers. The idea is to separate the state-gathering from the encoding: * State-gathering is getting the state of the object * Encoding is the converting the state (which is an object) into a sequence of bytes (textual or binary) This is somewhat similar to the ISerializable interface of .NET, which defines only GetObjectData(); the actual encoding is done by a Formatter class, which has different implementations for SOAP-formatting, binary-formatting, and other formats. Motivation ========== There are many generic and niche serializers out there, including pickle, banana/jelly, Cerealizer, brine, and lots of serializers targeting XML output. Currently, all serializers have their own methods of retrieving the contents, or state of the object. This API attempts to solve this issue by providing standard means for getting object state, and creating objects with their state restored. Another issue is making the serialization process "proxy-friendly". Many frameworks use object proxies to indirectly refer to another object (for instance, RPC proxies, FFI, etc.). In this case, it's desirable to simplify the referenced object rather than the proxy, and this API addresses this issue too. Simplification ============== Simplification is the process of converting a 'complex' object into its "atomic" components. You may think of these atomic components as the *contents* of the object. This proposal does not state what "atomic" means -- this is open to the decision of the class. The only restriction imposed is, collections of any kind must be simplified as tuples. Moreover, the simplification process may be recursive: object X may simplify itself in terms of object Y, which in turn may go further simplification. Simplification Protocol ======================= This proposal introduces two new special methods: def __simplify__(self): return type, state @classmethod def __rebuild__(cls, state): return new_instance_of_cls_with_given_state __simplify__ takes no arguments (bar 'self'), and returns a tuple of '(type, state)', representing the contents of 'self': * 'type' is expected to be a class or a builtin type, although it can be any object that has a '__rebuild__' method. This 'type' will be used later to reconstruct the object (using 'type.__rebuild__(state)'). * 'state' is any other that represents the inner state of 'self', in a simplified form. This can be an atomic value, or yet another complex object, that may go further simplification. __rebuild__ is a expected to be classmethod that takes the state returned by __simplify__, and returns a new instance of 'cls' with the given state. If a type does not wish to be simplified, it may throw a TypeError in its __simplify__ method; however, this is not recommended. Types that want to be treated as atomic elements, such as file, should just return themselves, and let the serializer handle them. Default Simplification ===================== All the built in types would grow a __simplify__ and __rebuild__ methods, which would follow these guidelines: Primitive types (int, str, float, ...) are considered atomic. Composite types (I think 'complex' is the only type), are broken down into their components. For the complex type, that would be a tuple of (real, imaginary). Container types (tuples, lists, sets, dicts) represent themselves as tuples of items. For example, dicts would be simplified according to this pseudocode: def PyDict_Simplifiy(PyObject * self): return PyDictType, tuple(self.items()) Built in types would be considered atomic. User-defined classes can be simplified into their metaclass, __bases__, and __dict__. The type 'object' would simplify instances by returning their __dict__ and any __slots__ the instance may have. This is the default behavior; classes that desire a different behavior would override __simplify__ and __rebuild__. Example of default behavior: >>> class Foo(object): ... def __init__(self): ... self.a = 5 ... self.b = "spam" ... >>> f = Foo() >>> cls, state = f.__simplify__() >>> cls >>> state {"a" : 5, "b" : "spam"} >>> shallow_copy = cls.__rebuild__(state) >>> state.__simplify__() (, (("a", 5), ("b", "spam"))) Example of customized behavior >>> class Bar(object): ... def __init__(self): ... self.a = 5 ... self.b = "spam" ... def __simplify__(self): ... return Bar, 17.5 ... @clasmethod ... def __rebuild__(cls, state) ... self = cls.__new__(cls) ... if state == 17.5: ... self.a = 5 ... self.b = "spam" ... return self ... >>> b = Bar() >>> b.__simplify__() (, 17.5) Code objects ============= I wish that modules, classes and functions would also be simplifiable, however, there are some issues with that: * How to serialize code objects? These can be simplified as tuple of their co_* attributes, but these attributes are very implementation- specific. * How to serialize cell variables, or other globals? It would be nice if .pyc files where generated like so: import foo pickle.dump(foo) It would also allow sending of code between machines, just like any other object. Copying ======== Shallow copy, as well as deep copy, can be implemented using the semantics of this new API. The copy module should be rewritten accordingly: def copy(obj): cls, state = obj.__simplify__() return cls.__rebuild__(state) deepcopy() can be implemented similarly. Deprecation =========== With the proposed API, copy_reg, __reduce__, __reduce_ex__, and possibly other modules become deprecated. Apart from that, the pickle and copy modules need to be updated accordingly. C API ====== The proposal introduces two new C-API functions: PyObject * PyObject_Simplify(PyObject * self); PyObject * PyObject_Rebuild(PyObject * type, PyObject * state); Although this is only a suggestion. I'd like to hear ideas from someone with more experience in the core. I do not see a need for a convenience routine such as simplify(obj) <--> obj.__simplify__(), since this API is not designed for everyday usage. This is the case with __reduce__ today. Object Proxying =============== Because __simplify__ returns '(type, state)', it may choose to "lie" about it's actual type. This means that when the state is reconstructed, another type is used. Object proxies will use this mechanism to serialize the referred object, rather than the proxy. class Proxy(object): [...] def __simplify__(self): # this returns a tuple with the *real* type and state return self.value.__simplify__() Serialization ============ Serialization is the process of converting fully-simplified objects into byte sequences (strings). Fully simplified objects are created by a recursive simplifier, that simplifies the entire object graph into atomic components. Then, the serializer would convert the atomic components into strings. Note that this proposal does not define how atomic objects are to be converted to strings, or how a 'recursive simplifier' should work. These issues are to be resolved by the implementation of the serializer. For instance, file objects are atomic; one serializer may be able to handle them, by storing them as (filename, file-mode, file-position), while another may not be, so it would raise an exception. Recursive Simplifier =================== This code demonstrates the general idea of how recursive simplifiers may be implemented: def recursive_simplifier(obj): cls, state = obj.__simplify__() # simplify all the elements inside tuples if type(state) is tuple: nested_state = [] for item in state: nested_state.append(recursive_simplifier(item)) return cls, nested_state # see if the object is atomic; if not, dig deeper if (cls, state) == state.__simplify__(): # 'state' is an atomic object, no need to go further return cls, state else: # this object is not atomic, so dig deeper return cls, recusrive_simplifier(state) -tomer From aahz at pythoncraft.com Fri Jan 26 19:16:30 2007 From: aahz at pythoncraft.com (Aahz) Date: Fri, 26 Jan 2007 10:16:30 -0800 Subject: [Python-ideas] simplification pep-like thing In-Reply-To: <1d85506f0701260713k1fe0d548u8103e86a7dfffdf7@mail.gmail.com> References: <1d85506f0701260713k1fe0d548u8103e86a7dfffdf7@mail.gmail.com> Message-ID: <20070126181630.GA4818@panix.com> On Fri, Jan 26, 2007, tomer filiba wrote: > > Default Simplification > ===================== > All the built in types would grow a __simplify__ and __rebuild__ > methods, which would follow these guidelines: > > Primitive types (int, str, float, ...) are considered atomic. > > Composite types (I think 'complex' is the only type), are > broken down into their components. For the complex type, > that would be a tuple of (real, imaginary). > > Container types (tuples, lists, sets, dicts) represent themselves > as tuples of items. For example, dicts would be simplified > according to this pseudocode: > > def PyDict_Simplifiy(PyObject * self): > return PyDictType, tuple(self.items()) > > Built in types would be considered atomic. User-defined classes > can be simplified into their metaclass, __bases__, and __dict__. This seems to contradict the following: > For instance, file objects are atomic; one serializer may be able to > handle them, by storing them as (filename, file-mode, file-position), > while another may not be, so it would raise an exception. Where do files fit into all this? -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "I disrespectfully agree." --SJM From jan.kanis at phil.uu.nl Sat Jan 27 01:36:22 2007 From: jan.kanis at phil.uu.nl (Jan Kanis) Date: Sat, 27 Jan 2007 01:36:22 +0100 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: References: <20070118084211.5977.JCARLSON@uci.edu> <45B303AA.1010707@gmail.com> <20070121162511.59B9.JCARLSON@uci.edu> Message-ID: On Thu, 25 Jan 2007 15:41:54 +0100, Jim Jewett wrote: > On 1/24/07, Jan Kanis wrote: >> I don't like new syntax for something like this, but I think the default >> argument values can be fixed with semantic changes (which should not >> break >> the most common current uses): >> >> What I think should happen is compile a function like this >> >> def popo(x=[]): >> x.append(666) >> print x >> >> as if it had read >> >> def popo(x=__default_argument_marker__): >> if x == __default_argument_marker__: >> x = [] >> x.append(666) >> print x > > How is this different from the x=None idiom of today? > > def f(inlist=None): > if inlist is None: > inlist=[] > The __default_argument_marker__ is not really a part of my proposal. You can replace it with None everywhere if you want to. The reason I used it is because using None can clash when the caller passes None in as explicit value like this: def foo(x, y=None): y = getApropreateDefaultValue() if y == None else y x.insert(y) foo(bar, None) Now if you want to have foo do bar.insert(None), calling foo(bar, None) won't work. However, I guess the risk of running into such a case in real code is neglegible, and my urge to write correct code won from writing understandable code. Just pretend I used None everywhere. (and that the compiler can magically distinguish between a default-argument None and a caller-provided None, if you wish.) > The if (either 2 lines or against PEP8) is a bit ugly, but Calvin > pointed out that you can now write it as > > def f(inlist=None): > inlist = inlist if (inlist is not None) else [] > > I see below that you give it slightly different semantics, but I'm > not entirely sure how to tell when those different semantics should > apply (always? when the variable name is marked with __*__? When a > specific non-None singleton appears?), or why you would ever want > them. Please just ignore the __default_argument_marker__ thing. I hope we agree that the problem we're trying to solve is that while def f(inlist=None): inlist = inlist if (inlist is not None) else [] works in the current python, it's non-intuitive and ugly, and it would be nice to have python do 'the right thing' if we can find a nice way to make it do that. Oh, and the changed semantics would allways be uses. >> When you use variables as default value instead of literals, I think >> most >> of the time you intend to have the function do something to the same >> object the variable is bound to, instead of the function creating it's >> own >> copy every time it's called. This behaviour still works with these >> semantics: > >> >>> a = [] >> >>>def foo(x=[[],a]): >> >>> x[0].append(123) >> >>> x[1].append(123) >> >>> print x >> >>>foo() >> [[123], [123]] >> >>> foo() >> [[123], [123, 123]] >> >>> foo() >> [[123], [123, 123, 123]] > > So you're saying that x[1] should be persistent because it (also) has > a name (as 'a'), but x[0] should be recreated fresh on each call > because it doesn't? I think what python does currently can be improved upon. I think that if someone defines a function like def f(x=[]): ... he'll most likely want x to be an empty list every time the function is called. But just having python do an automagic copy(x) or deepcopy(x) is not going to work, becaus it can be expensive, isn't always nescessary, and is sometimes plainly impossible, eg: def f(x=sys.stdout): ... So, sometimes we want a new thing on every call, and sometimes we don't. And sometimes we want to specify the default value with a literal, and sometimes with a variable. My assumption is that these two differences coincide most of the time. I also think my approach is more intuitive to people new to python and not familiar with this quirk/wart. You make it sound as if doing something different with a named variable and a literal is strange, but this is exactly what happens in every normal python expression: >>> a = [] >>> b = [[], a] >>> id(a) 12976416 >>> id(b[0]) 12994640 >>> id(b[1]) 12976416 >>> # let's execute the b = ... statement again (comparable to 'call a function again') >>> b = [[], a] >>> id(b[0]) 12934800 >>> id(b[1]) 12976416 b[0] gets recreated, while b[1] is not. So, I think my solution of evaluating the default argument on every call and letting any variables in the expression be closure variables accomplishes everything we want: * most of the time python does 'the right thing' wrt to copying or not copying, and does this in a way that is very regular wrt the rest of the language (and therefore easy to learn) * if we don't want a copy while python would do this, that can easily be accomplished by first creating the object and then using the variable name that references the object as default * if we do want a copy, just do an explicit copy.copy(x). On Fri, 26 Jan 2007 04:36:33 +0100, Chris Rebert wrote: > So, basically the same as my proposal except without syntax changes and > with the default argument value semantics applied to all > arguments. I'm okay with this, however some possible performance issues > might exist with re-evaluating expensive default arg vals on every call > where they're required. This is basically why my proposal required new > syntax, so that people could use the old "eval once at definition-time" > semantics on expensive default values to get better performance. > [snip] > If the performance issues aren't significant, I'm all for your proposal. > It'd be nice to not to have to add new syntax. Well, I wasn't thinking about it as basically the same as your proposal, but thinking about it again I think it is. (I was thinking more along the lines of having this do what looks intuitive to me, by applying the normal python language rules in the IMO 'right' way.) on performance issues: If you're using the x=None ... if x==None: x = [] trick the object gets evaluated and recreated on every call anyway, so there's no change. If you aren't using a literal as default, nothing gets re-evaluated, so no problem either. The only time it could become a problem is with code like this: def foo(x=createExpensiveThing()): return x.bar() If the function modifies x, you'll probably want to use a fresh x every call anyway (using x=None). If you do want the x to be persistant, chances are you already have a reference to it somewhere, but if you don't you'll have to create one. The reference doesn't get re-evaluated, so there's no performance issue, only possibly a namespace clutter issue. If the function doesn't modify x it may give rise to a performance issue, but that can easily be solved by creating the default before defining the function and using the variable. Rejecting this proposition because of this seems like premature optimisation to me. Another slight performance loss may be the fact that variables used in a default value will sometimes become closure variables, which are slightly slower than locals. However these variables are not used in the functions body, and it only is an issue if we're defining a function inside another function. I think this point is neglegible. - Jan From jcarlson at uci.edu Sat Jan 27 06:30:00 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri, 26 Jan 2007 21:30:00 -0800 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: References: Message-ID: <20070126212657.5A06.JCARLSON@uci.edu> "Jan Kanis" wrote: > I hope we agree > that the problem we're trying to solve is that while > > def f(inlist=None): > inlist = inlist if (inlist is not None) else [] > > works in the current python, it's non-intuitive and ugly, and it would be > nice to have python do 'the right thing' if we can find a nice way to make > it do that. I'm going to have to disagree on the 'non-intuitive and ugly' claim. We are just going to have to agree to disagree. - Josiah From tomerfiliba at gmail.com Sat Jan 27 22:30:51 2007 From: tomerfiliba at gmail.com (tomer filiba) Date: Sat, 27 Jan 2007 23:30:51 +0200 Subject: [Python-ideas] my take on mutable default arguments Message-ID: <1d85506f0701271330o1df7628fy2e82cedf97bada68@mail.gmail.com> i thought this code be solved nicely with a decorator... it needs some more work, but it would make a good cookbook recipe: .>>> from copy import deepcopy. .>>> .>>> def defaults(**kwdefs): ... def deco(func): ... def wrapper(*args, **kwargs): ... for k,v in kwdefs.iteritems(): ... if k not in kwargs: ... kwargs[k] = deepcopy(v) ... return func(*args, **kwargs) ... return wrapper ... return deco ... .>>> @defaults(x = []) ... def foo(a, x): ... x.append(a) ... print x ... .>>> foo(5) [5] .>>> foo(5) [5] .>>> foo(5) [5] maybe it should be done by copying func_defaults... then it could be written as @copydefaults def f(a, b = 5, c = []): ... -tomer From tomerfiliba at gmail.com Sat Jan 27 22:34:02 2007 From: tomerfiliba at gmail.com (tomer filiba) Date: Sat, 27 Jan 2007 23:34:02 +0200 Subject: [Python-ideas] simplification pep-like thing In-Reply-To: <1d85506f0701260713k1fe0d548u8103e86a7dfffdf7@mail.gmail.com> References: <1d85506f0701260713k1fe0d548u8103e86a7dfffdf7@mail.gmail.com> Message-ID: <1d85506f0701271334v21a43faen96db5ae3b3085aeb@mail.gmail.com> Aahz: >> Built in types would be considered atomic. User-defined classes >> can be simplified into their metaclass, __bases__, and __dict__. > > This seems to contradict the following: > >> For instance, file objects are atomic; one serializer may be able to >> handle them, by storing them as (filename, file-mode, file-position), >> while another may not be, so it would raise an exception. > > Where do files fit into all this? it doesn't... it was just meant to show how different serializers may implement their converters for the atomic values. it's not a practical suggestion. -tomer From tomerfiliba at gmail.com Sat Jan 27 22:42:50 2007 From: tomerfiliba at gmail.com (tomer filiba) Date: Sat, 27 Jan 2007 23:42:50 +0200 Subject: [Python-ideas] my take on mutable default arguments In-Reply-To: <1d85506f0701271330o1df7628fy2e82cedf97bada68@mail.gmail.com> References: <1d85506f0701271330o1df7628fy2e82cedf97bada68@mail.gmail.com> Message-ID: <1d85506f0701271342w2dc4c937l5cfc9b829a1becb2@mail.gmail.com> i think this is better: .>>> from copy import deepcopy .>>> .>>> def copydefaults(func): ... defaults = func.func_defaults ... def wrapper(*args, **kwargs): ... func.func_defaults = deepcopy(defaults) ... return func(*args, **kwargs) ... return wrapper ... .>>> @copydefaults ... def f(a, x = []): ... x.append(a) ... print x ... .>>> .>>> f(1) [1] .>>> f(2) [2] .>>> f(3) [3] -tomer From cvrebert at gmail.com Sat Jan 27 23:12:34 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Sat, 27 Jan 2007 14:12:34 -0800 Subject: [Python-ideas] my take on mutable default arguments In-Reply-To: <1d85506f0701271330o1df7628fy2e82cedf97bada68@mail.gmail.com> References: <1d85506f0701271330o1df7628fy2e82cedf97bada68@mail.gmail.com> Message-ID: <45BBCE52.5030207@gmail.com> As pointed out by Calvin Spealman in an earlier email: Calvin Spealman wrote: > Deep copy of the originally created > default argument can be expensive and would not work in any useful way > with non-literals as defaults, such as function calls or subscript > lookups or even simple attributes. However, your decorator is useful in several common cases. - Chris Rebert tomer filiba wrote: > i thought this code be solved nicely with a decorator... it needs some > more work, but it would make a good cookbook recipe: > > .>>> from copy import deepcopy. > .>>> > .>>> def defaults(**kwdefs): > ... def deco(func): > ... def wrapper(*args, **kwargs): > ... for k,v in kwdefs.iteritems(): > ... if k not in kwargs: > ... kwargs[k] = deepcopy(v) > ... return func(*args, **kwargs) > ... return wrapper > ... return deco > ... > .>>> @defaults(x = []) > ... def foo(a, x): > ... x.append(a) > ... print x > ... > .>>> foo(5) > [5] > .>>> foo(5) > [5] > .>>> foo(5) > [5] > > maybe it should be done by copying func_defaults... then it could > be written as > > @copydefaults > def f(a, b = 5, c = []): > ... > > > -tomer > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From tomerfiliba at gmail.com Sat Jan 27 23:22:32 2007 From: tomerfiliba at gmail.com (tomer filiba) Date: Sun, 28 Jan 2007 00:22:32 +0200 Subject: [Python-ideas] my take on mutable default arguments In-Reply-To: <45BBCE52.5030207@gmail.com> References: <1d85506f0701271330o1df7628fy2e82cedf97bada68@mail.gmail.com> <45BBCE52.5030207@gmail.com> Message-ID: <1d85506f0701271422j5650fcdajefb0ab33215c1de3@mail.gmail.com> On 1/28/07, Chris Rebert wrote: > As pointed out by Calvin Spealman in an earlier email: > > Calvin Spealman wrote: > > Deep copy of the originally created > > default argument can be expensive and would not work in any useful way > > with non-literals as defaults, such as function calls or subscript > > lookups or even simple attributes. first, please refer to my second version (@copydefaults). second, theoretically all objects are copyable, as long as they define __copy__ or __deepcopy__. it's up to you. third, you'll use this decorator only when you want "fresh copies" of the default values... not "just for fun", and since you'll need these fresh copies anyhow, creating the value by hand ("if x is None: x = []") will not yield a dramatic performance gain, as most default objects are "primitive" anyway (empty lists and stuff, not dicts of 10000 keys) the main issue is readability and auto-documentation. using "def f(x = [])" is better documenting than "def f(x = None)" and fourth, my apologies, but such a decorator already existed (and even with the same semantics that i used): http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/303440 -tomer From cvrebert at gmail.com Sun Jan 28 20:22:44 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Sun, 28 Jan 2007 11:22:44 -0800 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments Message-ID: <45BCF804.7040207@gmail.com> The following is a proto-PEP based on the discussion in the thread "fixing mutable default argument values". Comments would be greatly appreciated. - Chris Rebert Title: Fixing Non-constant Default Arguments Abstract This PEP proposes new semantics for default arguments to remove boilerplate code associated with non-constant default argument values, allowing them to be expressed more clearly and succinctly. Motivation Currently, to write functions using non-constant default arguments, one must use the idiom: def foo(non_const=None): if non_const is None: non_const = some_expr #rest of function or equivalent code. Naive programmers desiring mutable default arguments often make the mistake of writing the following: def foo(mutable=some_expr_producing_mutable): #rest of function However, this does not work as intended, as 'some_expr_producing_mutable' is evaluated only *once* at definition-time, rather than once per call at call-time. This results in all calls to 'foo' using the same default value, which can result in unintended consequences. This necessitates the previously mentioned idiom. This unintuitive behavior is such a frequent stumbling block for newbies that it is present in at least 3 lists of Python's problems [0] [1] [2]. There are currently few, if any, known good uses of the current behavior of mutable default arguments. The most common one is to preserve function state between calls. However, as one of the lists [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables. Therefore, since the current semantics aren't useful for non-constant default values and an idiom is necessary to work around this deficiency, why not change the semantics so that people can write what they mean more directly, without the annoying boilerplate? Rationale Originally, it was proposed that all default argument values be deep-copied from the original (evaluated at definition-time) at each invocation of the function where the default value was required. However, this doesn't take into account default values that are not literals, e.g. function calls, subscripts, attribute accesses. Thus, the new idea was to re-evaluate the default arguments at each call where they were needed. There was some concern over the possible performance hit this could cause, and whether there should be new syntax so that code could use the existing semantics for performance reasons. Some of the proposed syntaxes were: def foo(bar=): #code def foo(bar=new baz): #code def foo(bar=fresh baz): #code def foo(bar=separate baz): #code def foo(bar=another baz): #code def foo(bar=unique baz): #code where the new keyword (or angle brackets) would indicate that the parameter's default argument should use the new semantics. Other parameters would continue to use the old semantics. It was generally agreed that the angle-bracket syntax was particularly ugly, leading to the proposal of the other syntaxes. However, having 2 different sets of semantics could be confusing and leaving in the old semantics just for performance might be premature optimization. Refactorings to deal with the possible performance hit are discussed below. Specification The current semantics for default arguments are replaced by the following semantics: - Whenever a function is called, and the caller does not provide a value for a parameter with a default expression, the parameter's default expression shall be evaluated in the function's scope. The resulting value shall be assigned to a local variable in the function's scope with the same name as the parameter. - The default argument expressions shall be evaluated before the body of the function. - The evaluation of default argument expressions shall proceed in the same order as that of the parameter list in the function's definition. Given these semantics, it makes more sense to refer to default argument expressions rather than default argument values, as the expression is re-evaluated at each call, rather than just once at definition-time. Therefore, we shall do so hereafter. Demonstrative examples of new semantics: #default argument expressions can refer to #variables in the enclosing scope... CONST = "hi" def foo(a=CONST): print a >>> foo() hi >>> CONST="bye" >>> foo() bye #...or even other arguments def ncopies(container, n=len(container)): return [container for i in range(n)] >>> ncopies([1, 2], 5) [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]] >>> ncopies([1, 2, 3]) [[1, 2, 3], [1, 2, 3], [1, 2, 3]] >>> #ncopies grabbed n from [1, 2, 3]'s length (3) #default argument expressions are arbitrary expressions def my_sum(lst): cur_sum = lst[0] for i in lst[1:]: cur_sum += i return cur_sum def bar(b=my_sum((["b"] * (2 * 3))[:4])): print b >>> bar() bbbb #default argument expressions are re-evaluated at every call... from random import randint def baz(c=randint(1,3)): print c >>> baz() 2 >>> baz() 3 #...but only when they're required def silly(): print "spam" return 42 def qux(d=silly()): pass >>> qux() spam >>> qux(17) >>> qux(d=17) >>> qux(*[17]) >>> qux(**{'d':17}) >>> #no output because silly() never called because d's value was specified in the calls #Rule 3 count = 0 def next(): global count count += 1 return count - 1 def frobnicate(g=next(), h=next(), i=next()): print g, h, i >>> frobnicate() 0 1 2 >>> #g, h, and i's default argument expressions are evaluated in the same order as the parameter definition Backwards Compatibility This change in semantics breaks all code which uses mutable default argument values. Such code can be refactored from: def foo(bar=mutable): #code to def stateify(state): def _wrap(func): def _wrapper(*args, **kwds): kwds['bar'] = state return func(*args, **kwds) return _wrapper return _wrap @stateify(mutable) def foo(bar): #code or state = mutable def foo(bar=state): #code or class Baz(object): def __init__(self): self.state = mutable def foo(self, bar=self.state): #code The changes in this PEP are backwards-compatible with all code whose default argument values are immutable, including code using the idiom mentioned in the 'Motivation' section. However, such values will now be recomputed for each call for which they are required. This may cause performance degradation. If such recomputation is significantly expensive, the same refactorings mentioned above can be used. In relation to Python 3.0, this PEP's proposal is compatible with those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend on the acceptance of either of those PEPs. Reference Implementation All code of the form: def foo(bar=some_expr, baz=other_expr): #body Should act as if it had read (in pseudo-Python): def foo(bar=_undefined, baz=_undefined): if bar is _undefined: bar = some_expr if baz is _undefined: baz = other_expr #body where _undefined is the value given to a parameter when the caller didn't specify a value for it. This is not intended to be a literal translation, but rather a demonstration as to how Python's internal argument-handling machinery should be changed. References [0] 10 Python pitfalls http://zephyrfalcon.org/labs/python_pitfalls.html [1] Python Gotchas http://www.ferg.org/projects/python_gotchas.html#contents_item_6 [2] When Pythons Attack http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2 [3] Keyword-Only Arguments http://www.python.org/dev/peps/pep-3102/ [4] Function Annotations http://www.python.org/dev/peps/pep-3107/ From gagsl-py at yahoo.com.ar Sun Jan 28 01:42:37 2007 From: gagsl-py at yahoo.com.ar (Gabriel Genellina) Date: Sat, 27 Jan 2007 21:42:37 -0300 Subject: [Python-ideas] my take on mutable default arguments References: <1d85506f0701271330o1df7628fy2e82cedf97bada68@mail.gmail.com><45BBCE52.5030207@gmail.com> <1d85506f0701271422j5650fcdajefb0ab33215c1de3@mail.gmail.com> Message-ID: "tomer filiba" escribi? en el mensaje news:1d85506f0701271422j5650fcdajefb0ab33215c1de3 at mail.gmail.com... > On 1/28/07, Chris Rebert > wrote: >> As pointed out by Calvin Spealman in an earlier email: >> >> Calvin Spealman wrote: >> > Deep copy of the originally created >> > default argument can be expensive and would not work in any useful way >> > with non-literals as defaults, such as function calls or subscript >> > lookups or even simple attributes. > > first, please refer to my second version (@copydefaults). > > second, theoretically all objects are copyable, as long as they > define __copy__ or __deepcopy__. it's up to you. No, I think he's pointing another problem. For function calls, subscripts or attributes, it's important *when* you execute it. By example: def foo1(x, prompt=sys.ps2): print prompt,x def foo2(x, prompt=None): if prompt is None: prompt=sys.ps2 print prompt,x @copydefaults def foo3(x, prompt=sys.ps2): print prompt,x If the intent of the author is "use sys.ps2 unless I pass an explicit prompt" neither foo1 nor foo3 will work if sys.ps2 is changed. > second, theoretically all objects are copyable, as long as they > define __copy__ or __deepcopy__. it's up to you. Not all objects are copyable, or not as one would expect: @copydefaults def foo4(x, outfile=sys.stdout): "Use sys.stdout unless an explicit outfile is given" outfile.write(x) foo4 doesn't even work, it raises: ValueError: I/O operation on closed file. Even if you manage to special-case files, or "fix" how they are copied, it won't work as intended because sys.stdout could have been reassigned after the function was defined. So you have to use the old idiom anyway: def foo5(x, outfile=None): if outfile is None: outfile = sys.stdout outfile.write(x) > third, you'll use this decorator only when you want "fresh copies" of the > default values... not "just for fun", [...] Some way to say "I want this default argument to be re-evaluated when I don't provide an explicit value" would be useful, but this proposal does not help on this. -- Gabriel Genellina From rnd at onego.ru Mon Jan 29 08:38:39 2007 From: rnd at onego.ru (Roman Susi) Date: Mon, 29 Jan 2007 09:38:39 +0200 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BCF804.7040207@gmail.com> References: <45BCF804.7040207@gmail.com> Message-ID: <45BDA47F.4040603@onego.ru> Hello! I'd liked to say outright that this bad idea which complicates matters more than provides solutions. Right now it is enough to know that the part from def to ":" is executed at definition time. This is what incremental dynamic semantics is about. So, the suggestion is good only as separated feature, but is IMHO wrong if considered in the language design as a whole. So things like def foo(non_const=None): non_const = non_const or [] are good becuase explicitely tell you that the mutable object is to be created at call-time, not def-time. And I do not like PEP 3107 neither: its overly complex. If there is a need for Python type checking, I'd suggested to make a special superset which could be used to write compiled extensions as well (Pyrex comes to mind). Regards, Roman P.S. However, I may be wrong. In that case my syntax suggestion would be this: def foo(non_const or []): ... where [] is executed at runtime BECAUSE at def time non_const is somehow True and that is enough to leave [] alone. I have not checked, but I believe it is backward compatible. Anyway, could you summarize both contr-argument and this syntax proposal in the PEP? Chris Rebert wrote: >The following is a proto-PEP based on the discussion in the thread >"fixing mutable default argument values". Comments would be greatly >appreciated. >- Chris Rebert > >Title: Fixing Non-constant Default Arguments > >Abstract > > This PEP proposes new semantics for default arguments to remove >boilerplate code associated with non-constant default argument values, >allowing them to be expressed more clearly and succinctly. > > >Motivation > > Currently, to write functions using non-constant default arguments, >one must use the idiom: > > def foo(non_const=None): > if non_const is None: > non_const = some_expr > #rest of function > >or equivalent code. Naive programmers desiring mutable default arguments >often make the mistake of writing the following: > > def foo(mutable=some_expr_producing_mutable): > #rest of function > >However, this does not work as intended, as >'some_expr_producing_mutable' is evaluated only *once* at >definition-time, rather than once per call at call-time. This results >in all calls to 'foo' using the same default value, which can result in >unintended consequences. This necessitates the previously mentioned >idiom. This unintuitive behavior is such a frequent stumbling block for >newbies that it is present in at least 3 lists of Python's problems [0] >[1] [2]. > There are currently few, if any, known good uses of the current >behavior of mutable default arguments. The most common one is to >preserve function state between calls. However, as one of the lists [2] >comments, this purpose is much better served by decorators, classes, or >(though less preferred) global variables. > Therefore, since the current semantics aren't useful for >non-constant default values and an idiom is necessary to work around >this deficiency, why not change the semantics so that people can write >what they mean more directly, without the annoying boilerplate? > > >Rationale > > Originally, it was proposed that all default argument values be >deep-copied from the original (evaluated at definition-time) at each >invocation of the function where the default value was required. >However, this doesn't take into account default values that are not >literals, e.g. function calls, subscripts, attribute accesses. Thus, >the new idea was to re-evaluate the default arguments at each call where >they were needed. There was some concern over the possible performance >hit this could cause, and whether there should be new syntax so that >code could use the existing semantics for performance reasons. Some of >the proposed syntaxes were: > > def foo(bar=): > #code > > def foo(bar=new baz): > #code > > def foo(bar=fresh baz): > #code > > def foo(bar=separate baz): > #code > > def foo(bar=another baz): > #code > > def foo(bar=unique baz): > #code > >where the new keyword (or angle brackets) would indicate that the >parameter's default argument should use the new semantics. Other >parameters would continue to use the old semantics. It was generally >agreed that the angle-bracket syntax was particularly ugly, leading to >the proposal of the other syntaxes. However, having 2 different sets of >semantics could be confusing and leaving in the old semantics just for >performance might be premature optimization. Refactorings to deal with >the possible performance hit are discussed below. > > >Specification > > The current semantics for default arguments are replaced by the >following semantics: > - Whenever a function is called, and the caller does not provide a > value for a parameter with a default expression, the parameter's > default expression shall be evaluated in the function's scope. The > resulting value shall be assigned to a local variable in the > function's scope with the same name as the parameter. > - The default argument expressions shall be evaluated before the > body of the function. > - The evaluation of default argument expressions shall proceed in > the same order as that of the parameter list in the function's > definition. >Given these semantics, it makes more sense to refer to default argument >expressions rather than default argument values, as the expression is >re-evaluated at each call, rather than just once at definition-time. >Therefore, we shall do so hereafter. > >Demonstrative examples of new semantics: > #default argument expressions can refer to > #variables in the enclosing scope... > CONST = "hi" > def foo(a=CONST): > print a > > >>> foo() > hi > >>> CONST="bye" > >>> foo() > bye > > #...or even other arguments > def ncopies(container, n=len(container)): > return [container for i in range(n)] > > >>> ncopies([1, 2], 5) > [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]] > >>> ncopies([1, 2, 3]) > [[1, 2, 3], [1, 2, 3], [1, 2, 3]] > >>> #ncopies grabbed n from [1, 2, 3]'s length (3) > > #default argument expressions are arbitrary expressions > def my_sum(lst): > cur_sum = lst[0] > for i in lst[1:]: cur_sum += i > return cur_sum > > def bar(b=my_sum((["b"] * (2 * 3))[:4])): > print b > > >>> bar() > bbbb > > #default argument expressions are re-evaluated at every call... > from random import randint > def baz(c=randint(1,3)): > print c > > >>> baz() > 2 > >>> baz() > 3 > > #...but only when they're required > def silly(): > print "spam" > return 42 > > def qux(d=silly()): > pass > > >>> qux() > spam > >>> qux(17) > >>> qux(d=17) > >>> qux(*[17]) > >>> qux(**{'d':17}) > >>> #no output because silly() never called because d's value was >specified in the calls > > #Rule 3 > count = 0 > def next(): > global count > count += 1 > return count - 1 > > def frobnicate(g=next(), h=next(), i=next()): > print g, h, i > > >>> frobnicate() > 0 1 2 > >>> #g, h, and i's default argument expressions are evaluated in >the same order as the parameter definition > > >Backwards Compatibility > > This change in semantics breaks all code which uses mutable default >argument values. Such code can be refactored from: > > def foo(bar=mutable): > #code > >to > > def stateify(state): > def _wrap(func): > def _wrapper(*args, **kwds): > kwds['bar'] = state > return func(*args, **kwds) > return _wrapper > return _wrap > > @stateify(mutable) > def foo(bar): > #code > >or > > state = mutable > def foo(bar=state): > #code > >or > > class Baz(object): > def __init__(self): > self.state = mutable > > def foo(self, bar=self.state): > #code > >The changes in this PEP are backwards-compatible with all code whose >default argument values are immutable, including code using the idiom >mentioned in the 'Motivation' section. However, such values will now be >recomputed for each call for which they are required. This may cause >performance degradation. If such recomputation is significantly >expensive, the same refactorings mentioned above can be used. > > In relation to Python 3.0, this PEP's proposal is compatible with >those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend >on the acceptance of either of those PEPs. > > >Reference Implementation > > All code of the form: > > def foo(bar=some_expr, baz=other_expr): > #body > > Should act as if it had read (in pseudo-Python): > > def foo(bar=_undefined, baz=_undefined): > if bar is _undefined: > bar = some_expr > if baz is _undefined: > baz = other_expr > #body > >where _undefined is the value given to a parameter when the caller >didn't specify a value for it. This is not intended to be a literal >translation, but rather a demonstration as to how Python's internal >argument-handling machinery should be changed. > > >References > > [0] 10 Python pitfalls > http://zephyrfalcon.org/labs/python_pitfalls.html > > [1] Python Gotchas > http://www.ferg.org/projects/python_gotchas.html#contents_item_6 > > [2] When Pythons Attack > http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2 > > [3] Keyword-Only Arguments > http://www.python.org/dev/peps/pep-3102/ > > [4] Function Annotations > http://www.python.org/dev/peps/pep-3107/ >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >http://mail.python.org/mailman/listinfo/python-ideas > > >!DSPAM:45bcf8295417644621095! > > > From jimjjewett at gmail.com Mon Jan 29 15:42:40 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 29 Jan 2007 09:42:40 -0500 Subject: [Python-ideas] simplification pep-like thing In-Reply-To: <1d85506f0701260713k1fe0d548u8103e86a7dfffdf7@mail.gmail.com> References: <1d85506f0701260713k1fe0d548u8103e86a7dfffdf7@mail.gmail.com> Message-ID: On 1/26/07, tomer filiba wrote: > This proposal does not state what "atomic" means -- this is > open to the decision of the class. The only restriction imposed is, > collections of any kind must be simplified as tuples. It is worth stating the minimum that a compliant serializer must be able to treat atomically. For instance, is it always sufficient to reduce state to instances of (builtin, not subclasses of) string, tuple, float, or int? > Simplification Protocol > ======================= > This proposal introduces two new special methods: > def __simplify__(self): > return type, state If I understand correctly, this returns the actual type object, rather than its name, or the source code to rebuild that type. Are there any requirements on what sort of type objects the serializer must be able to support? > I do not see a need for a convenience routine such as > simplify(obj) <--> obj.__simplify__(), since this API is not designed > for everyday usage. This is the case with __reduce__ today. reduce is nasty for beginners. In fairness, I think much of the problem is that reduce and __reduce__ (as well as __reduce_ex__) both exist, but are unrelated. So adding simplify probably isn't required, but reserving the name might be. > Note that this proposal does not define how atomic objects are to be > converted to strings, or how a 'recursive simplifier' should work. These > issues are to be resolved by the implementation of the serializer. Is there some minimum requirement on is-stability? For example, would the following relationships be preserved after deserialization? >>> a1=object() >>> a2=a1 >>> b=object() >>> a1 is a2 True >>> a1 is b False -jJ From tomerfiliba at gmail.com Mon Jan 29 16:54:45 2007 From: tomerfiliba at gmail.com (tomer filiba) Date: Mon, 29 Jan 2007 17:54:45 +0200 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments Message-ID: <1d85506f0701290754n688a622ft835c4203c749f31f@mail.gmail.com> there really is no need to have something like that built into the language. most default arguments are immutable anyhow, and i'd guess most of the mutable defaults can be addressed with the suggested @copydefaults decorator. as for uncopyable or stateful defaults (dev = sys.stdout), which require reevaluation, you can just use this modified version of my first suggested decorator: >>> def reeval(**kwdefs): ... def deco(func): ... def wrapper(*args, **kwargs): ... for k, v in kwdefs.iteritems(): ... if k not in kwargs: ... kwargs[k] = v() # <--- this is the big change ... return func(*args, **kwargs) ... return wrapper ... return deco ... the defaults are now provided as *functions*, which are evaluated at the time of calling the function. >>> @reeval(device = lambda: sys.stdout) ... def say(text, device): ... device.write("%s: %s\n" % (device.name, text)) ... device.flush() this means you can do things like -- >>> say("hello1") : hello1 >>> >>> say("hello2", device = sys.stderr) : hello2 >>> >>> sys.stdout = sys.stderr >>> say("hello3") : hello3 >>> >>> sys.stdout = sys.__stdout__ >>> say("hello4") : hello4 decorators are powerful enough for this not-so-common case. it would be nice to have a collection of useful decorators as part of stdlib (such as these ones, but including memoize and many others)... but that's a different issue. maybe we should really start a list of useful decorators to be included as an stdlib module. -tomer -------------- next part -------------- An HTML attachment was scrubbed... URL: From jan.kanis at phil.uu.nl Mon Jan 29 17:03:52 2007 From: jan.kanis at phil.uu.nl (Jan Kanis) Date: Mon, 29 Jan 2007 17:03:52 +0100 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BCF804.7040207@gmail.com> References: <45BCF804.7040207@gmail.com> Message-ID: Well, I (obviously) like the idea, but your pep misses some important points (and some not so important). The important one is about the scoping of variables in default expressions. The pep says nothing about them. If I read the pep correctly, any variables in default expressions are handled the same as variables in the body of a function. This means the compiler decides if they should be local, lexical or global. If there is an assignment to a variable, the compiler makes it a local, else it finds the right enclosing scope (lexical or global). In the current python, this works fine: >>> a = 123 >>> def foo(b=a): a = 2 print a, b >>> foo() 2 123 >>> a = 42 >>> foo() 2 123 In the pep, the a in the default expression would be handled just like any other a in the function body, which means it wil become a _local_ variable. Calling the function would then result in an UnboundLocalError. Just like this in current python: >>> a = 123 >>> def foo(b=None): b = a if b==None else b a = 2 print a >>> foo() Traceback (most recent call last): File "", line 1, in foo() File "", line 2, in foo b = a if b==None else b UnboundLocalError: local variable 'a' referenced before assignment The solution, I think, as I wrote in my previous messages, is to have the compiler explicitly make variables in default expressions lexical or global variables. This would still break the foo() in my example above, because you can't assign to a lexical variable, or to a global that isn't declared global. Therefore I think the compiler should distinguish between the a in the default expression and the a in the function body, and treat them as two different variables. You can think about it as if the compiler silently renames one of them (without this rename being visible to python code. AFAIK the bytecode is stack based and closure vars are put in some kind of anonymous cell, which means they both don't actually have a name anyway.) see below for more comments regarding this and other things On Sun, 28 Jan 2007 20:22:44 +0100, Chris Rebert wrote: > The following is a proto-PEP based on the discussion in the thread > "fixing mutable default argument values". Comments would be greatly > appreciated. > - Chris Rebert > > Title: Fixing Non-constant Default Arguments > > Abstract > > This PEP proposes new semantics for default arguments to remove > boilerplate code associated with non-constant default argument values, > allowing them to be expressed more clearly and succinctly. > > > Motivation > > Currently, to write functions using non-constant default arguments, > one must use the idiom: > > def foo(non_const=None): > if non_const is None: > non_const = some_expr > #rest of function > > or equivalent code. Naive programmers desiring mutable default arguments > often make the mistake of writing the following: > > def foo(mutable=some_expr_producing_mutable): > #rest of function > > However, this does not work as intended, as > 'some_expr_producing_mutable' is evaluated only *once* at > definition-time, rather than once per call at call-time. This results > in all calls to 'foo' using the same default value, which can result in > unintended consequences. This necessitates the previously mentioned > idiom. This unintuitive behavior is such a frequent stumbling block for > newbies that it is present in at least 3 lists of Python's problems [0] > [1] [2]. Also, I just found out that python's own documentation refers to this with an "Important warning: The default value is evaluated only once. This makes a difference when the default is a mutable object such as a list, dictionary, or instances of most classes. ..." (http://docs.python.org/tut/node6.html#SECTION006710000000000000000) this indicates imo that also the python doc writers don't think of the current situation as optimal. > There are currently few, if any, known good uses of the current > behavior of mutable default arguments. The most common one is to > preserve function state between calls. However, as one of the lists [2] > comments, this purpose is much better served by decorators, classes, or > (though less preferred) global variables. > Therefore, since the current semantics aren't useful for > non-constant default values and an idiom is necessary to work around > this deficiency, why not change the semantics so that people can write > what they mean more directly, without the annoying boilerplate? > > > Rationale > > Originally, it was proposed that all default argument values be > deep-copied from the original (evaluated at definition-time) at each > invocation of the function where the default value was required. > However, this doesn't take into account default values that are not > literals, e.g. function calls, subscripts, attribute accesses. Thus, > the new idea was to re-evaluate the default arguments at each call where > they were needed. There was some concern over the possible performance > hit this could cause, and whether there should be new syntax so that > code could use the existing semantics for performance reasons. Some of > the proposed syntaxes were: > > def foo(bar=): > #code > > def foo(bar=new baz): > #code > > def foo(bar=fresh baz): > #code > > def foo(bar=separate baz): > #code > > def foo(bar=another baz): > #code > > def foo(bar=unique baz): > #code > > where the new keyword (or angle brackets) would indicate that the > parameter's default argument should use the new semantics. Other > parameters would continue to use the old semantics. It was generally > agreed that the angle-bracket syntax was particularly ugly, leading to > the proposal of the other syntaxes. However, having 2 different sets of > semantics could be confusing and leaving in the old semantics just for > performance might be premature optimization. Refactorings to deal with > the possible performance hit are discussed below. > > > Specification > > The current semantics for default arguments are replaced by the > following semantics: > - Whenever a function is called, and the caller does not provide a > value for a parameter with a default expression, the parameter's > default expression shall be evaluated in the function's scope. The > resulting value shall be assigned to a local variable in the > function's scope with the same name as the parameter. Include something saying that any variables in a default expression shall be lexical variables, with their scope being the first outer scope that defines a variable with the same name (they should just use the same rules as other lexical/closure variables), and that if the function body defines a local variable with the same name as a variable in a default expression, those variables shall be handled as two separate variables. > - The default argument expressions shall be evaluated before the > body of the function. > - The evaluation of default argument expressions shall proceed in > the same order as that of the parameter list in the function's > definition. > Given these semantics, it makes more sense to refer to default argument > expressions rather than default argument values, as the expression is > re-evaluated at each call, rather than just once at definition-time. > Therefore, we shall do so hereafter. > > Demonstrative examples of new semantics: > #default argument expressions can refer to > #variables in the enclosing scope... > CONST = "hi" > def foo(a=CONST): > print a > > >>> foo() > hi > >>> CONST="bye" > >>> foo() > bye > > #...or even other arguments > def ncopies(container, n=len(container)): > return [container for i in range(n)] > > >>> ncopies([1, 2], 5) > [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]] > >>> ncopies([1, 2, 3]) > [[1, 2, 3], [1, 2, 3], [1, 2, 3]] > >>> #ncopies grabbed n from [1, 2, 3]'s length (3) I'm not sure if this can be combined elegantly with what I said about variables being lexical variables. The first argument to ncopies, 'container', is clearly a local variable to ncopies. The 'container' in the second arg default expr should, if my comments above are accepted, be a lexical variable referring to the 'container' in the global scope. The best way to combine the two features seems to be to let 'container' be a local var if any of the preceding args is named 'container', and let it be a lexically scoped variable otherwise. However, I'm not convinced this complexity is worth it and the vars in default expressions shouldn't just always be lexical vars. > > #default argument expressions are arbitrary expressions > def my_sum(lst): > cur_sum = lst[0] > for i in lst[1:]: cur_sum += i > return cur_sum > > def bar(b=my_sum((["b"] * (2 * 3))[:4])): > print b > > >>> bar() > bbbb > > #default argument expressions are re-evaluated at every call... > from random import randint > def baz(c=randint(1,3)): > print c > > >>> baz() > 2 > >>> baz() > 3 > > #...but only when they're required > def silly(): > print "spam" > return 42 > > def qux(d=silly()): > pass > > >>> qux() > spam > >>> qux(17) > >>> qux(d=17) > >>> qux(*[17]) > >>> qux(**{'d':17}) > >>> #no output because silly() never called because d's value was > specified in the calls > > #Rule 3 > count = 0 > def next(): > global count > count += 1 > return count - 1 > > def frobnicate(g=next(), h=next(), i=next()): > print g, h, i > > >>> frobnicate() > 0 1 2 > >>> #g, h, and i's default argument expressions are evaluated in > the same order as the parameter definition > > > Backwards Compatibility > > This change in semantics breaks all code which uses mutable default > argument values. Such code can be refactored from: Wow, let's not scare everyone away just yet. This should read: "This change in semantics breaks code which uses mutable default argument expressions and depends on those expressions being evaluated only once, or code that assigns new incompatible values in a parent scope to variables used in default expressions" > > def foo(bar=mutable): > #code if 'mutable' is just a single variable, this isn't gonna break, unless the global scope decides to do something like this: def foo(bar=mutable): #code mutable = incompatible_mutable # ... foo() > > to > > def stateify(state): > def _wrap(func): > def _wrapper(*args, **kwds): > kwds['bar'] = state > return func(*args, **kwds) > return _wrapper > return _wrap > > @stateify(mutable) > def foo(bar): > #code > > or > > state = mutable > def foo(bar=state): > #code > > or > > class Baz(object): > def __init__(self): > self.state = mutable > > def foo(self, bar=self.state): > #code Minor point: the stateify decorator looks a bit scary to me as it uses three levels of nested functions. (that's inherent to decorators, but still.) Suggest you name the class and global var solutions first, and the decorator as last, just to prevent people from stopping reading the pep and voting '-1' right when they hit the decorator solution. > > The changes in this PEP are backwards-compatible with all code whose > default argument values are immutable ...or don't depend on being evaluated only once, and don't modify in a parent scope the variables in default expressions in an incompatible way, (hmm, the 'or' and 'and' may need some disambiguation parentheses...) > including code using the idiom > mentioned in the 'Motivation' section. However, such values will now be > recomputed for each call for which they are required. This may cause > performance degradation. If such recomputation is significantly > expensive, the same refactorings mentioned above can be used. > > In relation to Python 3.0, this PEP's proposal is compatible with > those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend > on the acceptance of either of those PEPs. > > > Reference Implementation > > All code of the form: > > def foo(bar=some_expr, baz=other_expr): > #body > > Should act as if it had read (in pseudo-Python): > > def foo(bar=_undefined, baz=_undefined): > if bar is _undefined: > bar = some_expr > if baz is _undefined: > baz = other_expr > #body and, if there are any variables occuring in the function body and in some_expr or other_expr, rename those in the function body to something that doesn't name-clash. > > where _undefined is the value given to a parameter when the caller > didn't specify a value for it. This is not intended to be a literal > translation, but rather a demonstration as to how Python's internal > argument-handling machinery should be changed. > > > References > > [0] 10 Python pitfalls > http://zephyrfalcon.org/labs/python_pitfalls.html > > [1] Python Gotchas > http://www.ferg.org/projects/python_gotchas.html#contents_item_6 > > [2] When Pythons Attack > http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2 > > [3] Keyword-Only Arguments > http://www.python.org/dev/peps/pep-3102/ > > [4] Function Annotations > http://www.python.org/dev/peps/pep-3107/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas I'm not quite sure if your decision not to include the default-expr-vars-become-lexical part was intentional or not. If it is, can you tell me why you'd want that? Else you can incorporate my comments in the pep. Another minor point: I'm personally not too fond of too many 'shall's close together. It makes me think of lots of bureaucracy and design-by-committee. I think several peps don't have a shall in them, but maybe it is the right language for this one. What's the right language to use in peps? Well, that's about it I think. - Jan From jimjjewett at gmail.com Mon Jan 29 23:51:04 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 29 Jan 2007 17:51:04 -0500 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BCF804.7040207@gmail.com> References: <45BCF804.7040207@gmail.com> Message-ID: On 1/28/07, Chris Rebert wrote: > Naive programmers desiring mutable default arguments > often make the mistake of writing the following: > def foo(mutable=some_expr_producing_mutable): > #rest of function Yes, it is an ugly gotcha, but so is the alternative. If it is changed, then just as many naive programmers (though perhaps not exactly the same ones) will make the opposite mistake -- and so will some experienced programmers who are used to current semantics. In a dynamic language, it makes perfect sense to reevaluate the entire call signature with each call -- but in almost any language, it makes perfect sense for the signature to be a constant. Usually, it doesn't matter which you assume, which is why this problem doesn't get ironed out in the first few minutes of writing python. > There are currently few, if any, known good uses of the current > behavior of mutable default arguments. The most common one is to > preserve function state between calls. However, as one of the lists [2] > comments, this purpose is much better served by decorators, classes, or > (though less preferred) global variables. I disagree. This is particularly wrong for someone coming from a functional background. A class plus an instantiation seems far too heavyweight for what ought to be a simple function. I'm not talking (only) about the runtime; the number of methods and lines of code is the real barrier for me. I'll sometimes do it (or use a global) anyhow, but it feels wrong. If I had felt that need more often when I was first learning python (or before I knew about the __call__ workaround), I might have just written the language off as no less bloated than java. You see the problem with globals, but decorators are in some sense worse -- a function cannot see its own decorations. At best, it can *assume* that repeating its own name (despite Don't Repeat Yourself) will get another reference to self, but this isn't always true. Programmers used to creating functions outside of toplevel (or class-level) will be more aware of this, and see the suggestion as an indication that python is inherently buggy. > def foo(bar=new baz): > #code This would be less bad. That said, I fear many new programmers would fail to understand when they needed new and when they didn't, so that in practice, it would be just optional random noise. > Demonstrative examples of new semantics: > #default argument expressions can refer to > #variables in the enclosing scope... > CONST = "hi" > def foo(a=CONST): > print a This would work if there were any easy way to create a new scope. In Lisp, it makes sense. In python, it would probably be better to just find a way for functions to refer to their own decorations reliably. > Backwards Compatibility > > This change in semantics breaks all code which uses mutable default > argument values. Such code can be refactored from: > > def foo(bar=mutable): > #code > > to [a decorator option uses 3 levels of nested functions, which aren't even generic enough for reuse, unless you give up introspection.] No. It could be done with a (more complex) decorator, if that decorator came with the stdlib (probably in functools), but that is heavy backwards-incompatibility to change a corner case (most defaults aren't mutable) in a taste-dependent manner. -jJ From jimjjewett at gmail.com Tue Jan 30 00:11:11 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 29 Jan 2007 18:11:11 -0500 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BDA47F.4040603@onego.ru> References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru> Message-ID: On 1/29/07, Roman Susi wrote: > P.S. However, I may be wrong. In that case my syntax suggestion would be > this: > def foo(non_const or []): > ... > > where [] is executed at runtime BECAUSE at def time non_const is somehow > True and that is enough to leave [] alone. It would also be possible to treat literals (like "[]") as "do it over each time", and more general expressions (like "list()") as they are treated today. Though like Roman, I think this would still be worse than the status quo. -jJ From rhamph at gmail.com Tue Jan 30 01:07:00 2007 From: rhamph at gmail.com (Adam Olsen) Date: Mon, 29 Jan 2007 17:07:00 -0700 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru> Message-ID: On 1/29/07, Jim Jewett wrote: > On 1/29/07, Roman Susi wrote: > > > P.S. However, I may be wrong. In that case my syntax suggestion would be > > this: > > > def foo(non_const or []): > > ... > > > > where [] is executed at runtime BECAUSE at def time non_const is somehow > > True and that is enough to leave [] alone. > > It would also be possible to treat literals (like "[]") as "do it over > each time", and more general expressions (like "list()") as they are > treated today. > > Though like Roman, I think this would still be worse than the status quo. Another, more bizarre approach would be to require default arguments be hashable. This would prevent lists from being used, protecting people from mistakes. None of the options given sit right with me though. -- Adam Olsen, aka Rhamphoryncus From cvrebert at gmail.com Tue Jan 30 04:17:15 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Mon, 29 Jan 2007 19:17:15 -0800 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru> Message-ID: <45BEB8BB.7010806@gmail.com> Jim Jewett wrote: > On 1/29/07, Roman Susi wrote: > >> P.S. However, I may be wrong. In that case my syntax suggestion would be >> this: > >> def foo(non_const or []): >> ... >> >> where [] is executed at runtime BECAUSE at def time non_const is somehow >> True and that is enough to leave [] alone. > > It would also be possible to treat literals (like "[]") as "do it over > each time", and more general expressions (like "list()") as they are > treated today. As discussed in the proto-PEP, there are many cases where non-literal default values need to be re-evaluated at each call. A solution to this problem needs to take this into account. Also, the proto-PEP's proposal tries to have the behavior of default arguments be as uniform as possible and not "magically" decide the correct behavior, especially by using something as arbitrary as what expression is used for the default argument. - Chris Rebert From gsakkis at rutgers.edu Tue Jan 30 05:16:50 2007 From: gsakkis at rutgers.edu (George Sakkis) Date: Mon, 29 Jan 2007 23:16:50 -0500 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BDA47F.4040603@onego.ru> References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru> Message-ID: <91ad5bf80701292016m63bf1b28w84e2c9f62fb7ea0e@mail.gmail.com> > Chris Rebert wrote: > > >The following is a proto-PEP based on the discussion in the thread > >"fixing mutable default argument values". Comments would be greatly > >appreciated. As I see it, the main objection to this is the inversal of the current default semantics: every default argument is treated as if it were mutable, or re-evaluatable more generally. Although mutable default arguments are useful some times and would be nice to have, they are most likely less common than the immutable ones, so the latter should be the default. I'm sure that the python-devs and the BDFL would have thought about it quite a bit when the current semantics were decided, and it's unlikely they'll change their mind now without very good reasons. OTOH, a proposal that leaves the current semantics as is and adds a mechanism to specify default arguments to be evaluated at call-time rather than definition-time would have more chances. Here's a proof-of-concept solution that specifies explicitly the re-evaluatable default expressions as "deferred": import inspect class Deferred(object): def __init__(self, expr): self.expr = expr def eval_deferreds(func): varnames,_,_,defaults = inspect.getargspec(func) num_varnames = len(varnames); num_defaults = len(defaults) def wrapper(*args, **kwds): if len(args) >= num_varnames: # defaults not used here return func(*args,**kwds) f_locals = dict(zip(varnames,args)) used_defaults = min(num_defaults, num_varnames-len(args)) for var,default in zip(varnames[-used_defaults:], defaults[-used_defaults:]): if var in kwds: # passed as keyword argument; don't use the default value = kwds[var] elif not isinstance(default, Deferred): # non re-evaluatable default value = default else: # evaluatable default in f_locals value = eval(default.expr, func.func_globals, f_locals) f_locals[var] = value f_locals.update(kwds) # add any extra keyword arguments return func(**f_locals) return wrapper #======= example ============================== W = 1 # some global @eval_deferreds def f(x, y=Deferred('x**2+W'), z=Deferred('[]')): z.append(x) z.append(y) return z from collections import deque print f(3) # [3,10] W=3; print f(4) # [4,19] print f(4,5) # [4,5] print f(-1, z=deque()) # deque([-1,4]) Regards, George From collinw at gmail.com Tue Jan 30 05:36:51 2007 From: collinw at gmail.com (Collin Winter) Date: Mon, 29 Jan 2007 22:36:51 -0600 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BCF804.7040207@gmail.com> References: <45BCF804.7040207@gmail.com> Message-ID: <43aa6ff70701292036t7f0a2a96hc013ecd47957de4d@mail.gmail.com> On 1/28/07, Chris Rebert wrote: > Rationale [snip] > There was some concern over the possible performance > hit this could cause, and whether there should be new syntax so that > code could use the existing semantics for performance reasons. Some of > the proposed syntaxes were: > > def foo(bar=): > #code > > def foo(bar=new baz): > #code > > def foo(bar=fresh baz): > #code > > def foo(bar=separate baz): > #code > > def foo(bar=another baz): > #code > > def foo(bar=unique baz): > #code > > where the new keyword (or angle brackets) would indicate that the > parameter's default argument should use the new semantics. Syntax changes are a huge stick to wield against such a small problem. I realize the boilerplate is annoying, but Tomer has pointed out a number of decorator-based solutions [1] that could easily be adapted to any unusual needs you have. Also, you haven't talked at all about how all this might be accomplished. You say > Given these semantics, it makes more sense to refer to default argument > expressions rather than default argument values, as the expression is > re-evaluated at each call, rather than just once at definition-time. but how do you intend to capture these "default argument expressions"? Emit additional bytecode for functions making use of these special default arguments? Collin Winter From cvrebert at gmail.com Tue Jan 30 05:54:19 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Mon, 29 Jan 2007 20:54:19 -0800 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <43aa6ff70701292036t7f0a2a96hc013ecd47957de4d@mail.gmail.com> References: <45BCF804.7040207@gmail.com> <43aa6ff70701292036t7f0a2a96hc013ecd47957de4d@mail.gmail.com> Message-ID: <45BECF7B.6090503@gmail.com> Collin Winter wrote: > On 1/28/07, Chris Rebert wrote: > Syntax changes are a huge stick to wield against such a small problem. > I realize the boilerplate is annoying, but Tomer has pointed out a > number of decorator-based solutions [1] that could easily be adapted > to any unusual needs you have. One of his decorators assumes that the default value merely needs to be copied across calls. However, as the PEP says, there are cases where this isn't sufficient or just doesn't work. His second decorator involves using lambdas (ick!), and doesn't allow you to refer to other arguments in determining the default value (think 'self' as a hint for how this could be useful). Also, I just think it's plain nicer/prettier if decorators don't have to be used for this. > Also, you haven't talked at all about how all this might be > accomplished. You say > >> Given these semantics, it makes more sense to refer to default argument >> expressions rather than default argument values, as the expression is >> re-evaluated at each call, rather than just once at definition-time. > > but how do you intend to capture these "default argument expressions"? > Emit additional bytecode for functions making use of these special > default arguments? The "Reference Implementation" section of the PEP discusses this. I don't personally know Python's internals, but I imagine this proposal would just change how default arguments are compiled and might entail some changes to the interpreter's argument-processing machinery. As Jan Kanis pointed out, the PEP does need to elaborate on exactly what kind of variables the default arguments use/are. I'm working on revising this in the next draft. - Chris Rebert From cvrebert at gmail.com Tue Jan 30 06:09:29 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Mon, 29 Jan 2007 21:09:29 -0800 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: References: <45BCF804.7040207@gmail.com> Message-ID: <45BED309.10301@gmail.com> Jim Jewett wrote: > On 1/28/07, Chris Rebert wrote: > >> Naive programmers desiring mutable default arguments >> often make the mistake of writing the following: > >> def foo(mutable=some_expr_producing_mutable): >> #rest of function > > Yes, it is an ugly gotcha, but so is the alternative. > > If it is changed, then just as many naive programmers (though perhaps > not exactly the same ones) will make the opposite mistake -- and so > will some experienced programmers who are used to current semantics. Anyone (ab)using the current semantics with the above construct is or ought to be aware that such usage is unpythonic. Also, due to the hairiness/obscurity of the construct, hardly anyone uses it, so the impact of removing it should be relatively minor. Additionally, I see making the opposite mistake as usually only maybe causing a performance problem, as opposed to causing the program to work incorrectly. As discussed in my proposal, this might be premature optimization. If it is a serious issue (which hasn't been indicated by the discussion so far), we can add syntax for the old/new semantics, which I also mentioned in the proposal. >> There are currently few, if any, known good uses of the current >> behavior of mutable default arguments. The most common one is to >> preserve function state between calls. However, as one of the lists [2] >> comments, this purpose is much better served by decorators, classes, or >> (though less preferred) global variables. > > I disagree. This is particularly wrong for someone coming from a > functional background. I assume you disagree with the "purpose is much better served by decorators, classes, or local variables" part as opposed to the "default mutable arguments with current semantics have few good uses" part. Please correct me if I'm in error. > A class plus an instantiation seems far too heavyweight for what ought > to be a simple function. I'm not talking (only) about the runtime; > the number of methods and lines of code is the real barrier for me. > I'll sometimes do it (or use a global) anyhow, but it feels wrong. If > I had felt that need more often when I was first learning python (or > before I knew about the __call__ workaround), I might have just > written the language off as no less bloated than java. I'm sorry, but when you have a function sharing state between calls, that just screams to me that you should make it a method of an object so you don't have to store the state in some roundabout way, such as in mutable default arguments. If performance is your concern, the decorator version might perform better (I don't really know), or in the extreme case, you could use a global variable, which is definitely faster. Speed and elegance often come in inverse proportions. Also, I've revised the refactoring in question so that you no longer need to instanciate the class, which at least makes it marginally better. > You see the problem with globals, but decorators are in some sense > worse -- a function cannot see its own decorations. At best, it can > *assume* that repeating its own name (despite Don't Repeat Yourself) > will get another reference to self, but this isn't always true. I really don't see how the decorator in the PEP is any worse than other decorators in this regard. The problem you describe currently applies to all decorated functions, though functools.wraps might help mitigate this situation. > Programmers used to creating functions outside of toplevel (or > class-level) will be more aware of this, and see the suggestion as an > indication that python is inherently buggy. > >> def foo(bar=new baz): >> #code > > This would be less bad. > > That said, I fear many new programmers would fail to understand when > they needed new and when they didn't, so that in practice, it would be > just optional random noise. This is part of the reason I'm trying to avoid adding new syntax. However, I assert that at least 'new' is clearer than the' x=None; if x is None: x=expr' idiom in that it expresses one's intent more clearly. Also, this would at least be a prettier way to spell the idiom even if the reason still needed explaining. Alternatively, we could make the new semantics the default and have syntax to use the old semantics via 'once' or some other keyword. This does nicely emphasize that such semantics would be purely for optimization reasons. I think I'll add this to the PEP. >> Demonstrative examples of new semantics: >> #default argument expressions can refer to >> #variables in the enclosing scope... >> CONST = "hi" >> def foo(a=CONST): >> print a > > This would work if there were any easy way to create a new scope. In > Lisp, it makes sense. In python, it would probably be better to just > find a way for functions to refer to their own decorations reliably. This is outside of the scope of my PEP. However, the below improvement should help and you could always use one of the other refactorings to work around this issue. >> Backwards Compatibility >> >> This change in semantics breaks all code which uses mutable default >> argument values. Such code can be refactored from: >> >> def foo(bar=mutable): >> #code >> >> to > > [a decorator option uses 3 levels of nested functions, which aren't even > generic enough for reuse, unless you give up introspection.] > > No. It could be done with a (more complex) decorator, if that > decorator came with the stdlib (probably in functools) Agreed, the decorator could be better. I've just enhanced it using functools.wraps, which should help with the introspection issues you raise. Other improvements to the refactoring code in the PEP are welcomed. - Chris Rebert From collinw at gmail.com Tue Jan 30 06:22:10 2007 From: collinw at gmail.com (Collin Winter) Date: Mon, 29 Jan 2007 23:22:10 -0600 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BECF7B.6090503@gmail.com> References: <45BCF804.7040207@gmail.com> <43aa6ff70701292036t7f0a2a96hc013ecd47957de4d@mail.gmail.com> <45BECF7B.6090503@gmail.com> Message-ID: <43aa6ff70701292122o6248a9afi3a139a106fe717f5@mail.gmail.com> On 1/29/07, Chris Rebert wrote: > Collin Winter wrote: > > On 1/28/07, Chris Rebert wrote: > > Syntax changes are a huge stick to wield against such a small problem. > > I realize the boilerplate is annoying, but Tomer has pointed out a > > number of decorator-based solutions [1] that could easily be adapted > > to any unusual needs you have. > > One of his decorators assumes that the default value merely needs to be > copied across calls. However, as the PEP says, there are cases where > this isn't sufficient or just doesn't work. Tomer wasn't proposing that any of those decorators be included in the standard library. Those cases where one of decorators isn't sufficient as written? Modify it, use it in your code, post it to the cookbook. Not every n-line function should be turned into syntax. > His second decorator > involves using lambdas (ick!), and doesn't allow you to refer to other > arguments in determining the default value (think 'self' as a hint for > how this could be useful). You can't do that as things are, so that's not a strike against the decorator-based approach. > > Also, you haven't talked at all about how all this might be > > accomplished. You say > > > >> Given these semantics, it makes more sense to refer to default argument > >> expressions rather than default argument values, as the expression is > >> re-evaluated at each call, rather than just once at definition-time. > > > > but how do you intend to capture these "default argument expressions"? > > Emit additional bytecode for functions making use of these special > > default arguments? > > The "Reference Implementation" section of the PEP discusses this. I > don't personally know Python's internals, but I imagine this proposal > would just change how default arguments are compiled and might entail > some changes to the interpreter's argument-processing machinery. As you work on this, think about how you'll explain the new semantics in the documentation. Rewriting http://docs.python.org/ref/calls.html and http://docs.python.org/ref/function.html -- paying particular attention to paragraphs 2-5 in the former -- would be a good start. Collin Winter From cvrebert at gmail.com Tue Jan 30 06:23:25 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Mon, 29 Jan 2007 21:23:25 -0800 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BDA47F.4040603@onego.ru> References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru> Message-ID: <45BED64D.10801@gmail.com> Roman Susi wrote: > Hello! > > I'd liked to say outright that this bad idea which complicates matters > more than provides solutions. > Right now it is enough to know that the part from def to ":" is executed > at definition time. This is what > incremental dynamic semantics is about. So, the suggestion is good only > as separated feature, but is IMHO wrong > if considered in the language design as a whole. You're entitled to your own opinion on the PEP. > So things like > > def foo(non_const=None): > non_const = non_const or [] > > are good becuase explicitely tell you that the mutable object is to be > created at call-time, not def-time. The 'new' (or similar) keyword (might) indicate the new semantics, or alternatively, 'old' (or a similar) keyword (might) indicate the old semantics. If the new semantics become the default (as the PEP proposes), then this point is moot anyway as it will be explicit by way of the language definition. > And I do not like PEP 3107 neither: its overly complex. > > If there is a need for Python type checking, I'd suggested to make a > special superset which could be used > to write compiled extensions as well (Pyrex comes to mind). That's not part of my proposal, nor does my PEP depend on that one. I merely mention PEP 3107 when considering compatibility with other 3100-series PEPs. > P.S. However, I may be wrong. In that case my syntax suggestion would be > this: > > def foo(non_const or []): > ... > > where [] is executed at runtime BECAUSE at def time non_const is somehow > True and that is enough to leave [] alone. > I have not checked, but I believe it is backward compatible. > Anyway, could you summarize both contr-argument and this syntax proposal > in the PEP? I don't quite understand exactly how this would work and would like more details on it, but once you've explained it, of course I'd be happy to include it in the next draft. - Chris Rebert From cvrebert at gmail.com Tue Jan 30 06:27:53 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Mon, 29 Jan 2007 21:27:53 -0800 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru> Message-ID: <45BED759.6060109@gmail.com> Adam Olsen wrote: > [snip] > Another, more bizarre approach would be to require default arguments > be hashable. This would prevent lists from being used, protecting > people from mistakes. If my PEP ends up being rejected, I would fully support this. It wouldn't solve the more general problem, but at least it would prevent a common newbie mistake. - Chris Rebert From cvrebert at gmail.com Tue Jan 30 06:40:35 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Mon, 29 Jan 2007 21:40:35 -0800 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <91ad5bf80701292016m63bf1b28w84e2c9f62fb7ea0e@mail.gmail.com> References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru> <91ad5bf80701292016m63bf1b28w84e2c9f62fb7ea0e@mail.gmail.com> Message-ID: <45BEDA53.9010103@gmail.com> George Sakkis wrote: > As I see it, the main objection to this is the inversal of the current > default semantics: every default argument is treated as if it were > mutable, or re-evaluatable more generally. Although mutable default > arguments are useful some times and would be nice to have, they are > most likely less common than the immutable ones, so the latter should > be the default. Why? Yes, there _might_ be performance issues (which have yet to be demonstrated or deeply speculated upon), but re-evaluating immutable default arguments wouldn't affect a program's correct operation. > I'm sure that the python-devs and the BDFL would have > thought about it quite a bit when the current semantics were decided, ...which was probably a while ago. They might reconsider the issue now that some time has passed and they've seen how their decision has worked out. But yes, your analysis is a definite possibility. > and it's unlikely they'll change their mind now without very good > reasons. I hope to provide those reasons in my PEP. > OTOH, a proposal that leaves the current semantics as is and adds a > mechanism to specify default arguments to be evaluated at call-time > rather than definition-time would have more chances. My PEP does discuss the possibility of adding new syntax for the new semantics and leaving the old semantics as the default. The final proposal will take into account the community's opinion as to the specifics of how the syntax/semantics ought to be changed. > Here's a > proof-of-concept solution that specifies explicitly the re-evaluatable > default expressions as "deferred": > > > import inspect > > class Deferred(object): > def __init__(self, expr): > self.expr = expr > > def eval_deferreds(func): > varnames,_,_,defaults = inspect.getargspec(func) > num_varnames = len(varnames); num_defaults = len(defaults) > def wrapper(*args, **kwds): > if len(args) >= num_varnames: # defaults not used here > return func(*args,**kwds) > f_locals = dict(zip(varnames,args)) > used_defaults = min(num_defaults, num_varnames-len(args)) > for var,default in zip(varnames[-used_defaults:], > defaults[-used_defaults:]): > if var in kwds: # passed as keyword argument; don't use the > default > value = kwds[var] > elif not isinstance(default, Deferred): # non re-evaluatable > default > value = default > else: # evaluatable default in f_locals > value = eval(default.expr, func.func_globals, f_locals) > f_locals[var] = value > f_locals.update(kwds) # add any extra keyword arguments > return func(**f_locals) > return wrapper > > > #======= example ============================== > > W = 1 # some global > > @eval_deferreds > def f(x, y=Deferred('x**2+W'), z=Deferred('[]')): > z.append(x) > z.append(y) > return z > > from collections import deque > print f(3) # [3,10] > W=3; print f(4) # [4,19] > print f(4,5) # [4,5] > print f(-1, z=deque()) # deque([-1,4]) > While that is some pretty nifty/fancy coding, the use of strings for the default values does seem a bit kludgey. However, if my proposal does not end up getting approved, I'll be sure to recommend that some of the great decorators mentioned on this thread get added to the standard library. - Chris Rebert From jcarlson at uci.edu Tue Jan 30 06:50:01 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 29 Jan 2007 21:50:01 -0800 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BECF7B.6090503@gmail.com> References: <43aa6ff70701292036t7f0a2a96hc013ecd47957de4d@mail.gmail.com> <45BECF7B.6090503@gmail.com> Message-ID: <20070129212157.5A2B.JCARLSON@uci.edu> Chris Rebert wrote: > > Collin Winter wrote: > > On 1/28/07, Chris Rebert wrote: > > Syntax changes are a huge stick to wield against such a small problem. > > I realize the boilerplate is annoying, but Tomer has pointed out a > > number of decorator-based solutions [1] that could easily be adapted > > to any unusual needs you have. > > One of his decorators assumes that the default value merely needs to be > copied across calls. However, as the PEP says, there are cases where > this isn't sufficient or just doesn't work. His second decorator > involves using lambdas (ick!), [snip] Using the features of a language to attempt to compensate for that same language's (argued) shortcomings are a valid _and encouraged_ approach. Your poo-pooing of Python conditionals, decorators, and lambdas to solve this particular problem, to me, seems like you want a *particular solution*. I don't see a problem with the current default argument semantics. Why? Because in the case where I would want to receive a mutable parameter, like in the case below that wouldn't work with Python's standard semantics... def append_10(lst=[]): lst.append(10) return lst I would presumably change it to... def append_10(lst=None): if lst is None: lst = [] lst.append(10) return lst Or some variant thereof. Now, here's the thing; if I want a mutable argument, then None is a nonsensical value to pass, generally, as it is not mutable. So I don't buy the whole "but then None would no longer be a valid argument to pass" bull that was offered as a reason why the above isn't a reasonable translation (I can't remember who offered it). I'm also not convinced by either of the 3 pages that talk about Python "gotchas". You get bitten by it, you learn it, understand it, and move on. If they can't figure it out, I'm not sure I want them writing Python software anyways; I certainly wouldn't want them to work on any of the Python software I work on and use. - Josiah From collinw at gmail.com Tue Jan 30 06:52:59 2007 From: collinw at gmail.com (Collin Winter) Date: Mon, 29 Jan 2007 23:52:59 -0600 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BEDA53.9010103@gmail.com> References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru> <91ad5bf80701292016m63bf1b28w84e2c9f62fb7ea0e@mail.gmail.com> <45BEDA53.9010103@gmail.com> Message-ID: <43aa6ff70701292152q158092d3xf1b37b4ff4ff7438@mail.gmail.com> On 1/29/07, Chris Rebert wrote: > George Sakkis wrote: > > As I see it, the main objection to this is the inversal of the current > > default semantics: every default argument is treated as if it were > > mutable, or re-evaluatable more generally. Although mutable default > > arguments are useful some times and would be nice to have, they are > > most likely less common than the immutable ones, so the latter should > > be the default. > > Why? Yes, there _might_ be performance issues (which have yet to be > demonstrated or deeply speculated upon), but re-evaluating immutable > default arguments wouldn't affect a program's correct operation. If the underlying intent of your proposal -- that all default arguments be re-evaluated with every call -- were to be approved, there would undoubtedly be a serious performance impact. The alternative is an ugly, narrow-use syntax that seeks to eliminate two lines of boilerplate per default argument, boilerplate that can already be replaced with decorators. As for whether these effects have "yet to be demonstrated", as you say, the burden is on you, the PEP author, to investigate and resolve, mitigate or justify any and all performance changes. Collin Winter From cvrebert at gmail.com Tue Jan 30 08:09:37 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Mon, 29 Jan 2007 23:09:37 -0800 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: References: <45BCF804.7040207@gmail.com> Message-ID: <45BEEF31.6070108@gmail.com> Wow, that's a lot to think about. Yes, the exact nature of the variables in default arguments does need clarification. I'll probably go with something like your proposal, but I need to consider a few things, particularly the 'ncopies' situation. I added a reference to the Python documentation you mentioned. Thanks! >> def foo(bar=mutable): >> #code > > if 'mutable' is just a single variable, this isn't gonna break, unless > the global scope decides to do something like this: > > def foo(bar=mutable): > #code > > mutable = incompatible_mutable > # ... > foo() Actually, I was considering the case where mutable is a constant (e.g. a list), in which case it *will* break since the expr 'mutable' will be re-evaluated at every call, so modifying it won't have the same effect on future calls that it used to. I included your rephrasing under "Backwards Compatibility". I shuffled around the refactorings as you suggested. My decision not to include the default-expr-vars-become-lexical part was completely unintentional, as you can guess from above. I also reworded the sentences using 'shall'. I don't really get why that matters, but what the heck. Thanks for your useful comments and suggestions. - Chris Rebert Jan Kanis wrote: > Well, I (obviously) like the idea, but your pep misses some important > points (and some not so important). > > The important one is about the scoping of variables in default > expressions. The pep says nothing about them. > If I read the pep correctly, any variables in default expressions are > handled the same as variables in the body of a function. This means the > compiler decides if they should be local, lexical or global. If there is > an assignment to a variable, the compiler makes it a local, else it > finds the right enclosing scope (lexical or global). In the current > python, this works fine: > > >>> a = 123 > >>> def foo(b=a): > a = 2 > print a, b > > >>> foo() > 2 123 > >>> a = 42 > >>> foo() > 2 123 > > In the pep, the a in the default expression would be handled just like > any other a in the function body, which means it wil become a _local_ > variable. Calling the function would then result in an > UnboundLocalError. Just like this in current python: > > >>> a = 123 > >>> def foo(b=None): > b = a if b==None else b > a = 2 > print a > > >>> foo() > > Traceback (most recent call last): > File "", line 1, in > foo() > File "", line 2, in foo > b = a if b==None else b > UnboundLocalError: local variable 'a' referenced before assignment > > The solution, I think, as I wrote in my previous messages, is to have > the compiler explicitly make variables in default expressions lexical or > global variables. > This would still break the foo() in my example above, because you can't > assign to a lexical variable, or to a global that isn't declared global. > Therefore I think the compiler should distinguish between the a in the > default expression and the a in the function body, and treat them as two > different variables. You can think about it as if the compiler silently > renames one of them (without this rename being visible to python code. > AFAIK the bytecode is stack based and closure vars are put in some kind > of anonymous cell, which means they both don't actually have a name > anyway.) > > see below for more comments regarding this and other things > > > On Sun, 28 Jan 2007 20:22:44 +0100, Chris Rebert > wrote: > >> The following is a proto-PEP based on the discussion in the thread >> "fixing mutable default argument values". Comments would be greatly >> appreciated. >> - Chris Rebert >> >> Title: Fixing Non-constant Default Arguments >> >> Abstract >> >> This PEP proposes new semantics for default arguments to remove >> boilerplate code associated with non-constant default argument values, >> allowing them to be expressed more clearly and succinctly. >> >> >> Motivation >> >> Currently, to write functions using non-constant default arguments, >> one must use the idiom: >> >> def foo(non_const=None): >> if non_const is None: >> non_const = some_expr >> #rest of function >> >> or equivalent code. Naive programmers desiring mutable default arguments >> often make the mistake of writing the following: >> >> def foo(mutable=some_expr_producing_mutable): >> #rest of function >> >> However, this does not work as intended, as >> 'some_expr_producing_mutable' is evaluated only *once* at >> definition-time, rather than once per call at call-time. This results >> in all calls to 'foo' using the same default value, which can result in >> unintended consequences. This necessitates the previously mentioned >> idiom. This unintuitive behavior is such a frequent stumbling block for >> newbies that it is present in at least 3 lists of Python's problems [0] >> [1] [2]. > > Also, I just found out that python's own documentation refers to this > with an "Important warning: The default value is evaluated only once. > This makes a difference when the default is a mutable object such as a > list, dictionary, or instances of most classes. ..." > (http://docs.python.org/tut/node6.html#SECTION006710000000000000000) > this indicates imo that also the python doc writers don't think of the > current situation as optimal. > >> There are currently few, if any, known good uses of the current >> behavior of mutable default arguments. The most common one is to >> preserve function state between calls. However, as one of the lists [2] >> comments, this purpose is much better served by decorators, classes, or >> (though less preferred) global variables. >> Therefore, since the current semantics aren't useful for >> non-constant default values and an idiom is necessary to work around >> this deficiency, why not change the semantics so that people can write >> what they mean more directly, without the annoying boilerplate? >> >> >> Rationale >> >> Originally, it was proposed that all default argument values be >> deep-copied from the original (evaluated at definition-time) at each >> invocation of the function where the default value was required. >> However, this doesn't take into account default values that are not >> literals, e.g. function calls, subscripts, attribute accesses. Thus, >> the new idea was to re-evaluate the default arguments at each call where >> they were needed. There was some concern over the possible performance >> hit this could cause, and whether there should be new syntax so that >> code could use the existing semantics for performance reasons. Some of >> the proposed syntaxes were: >> >> def foo(bar=): >> #code >> >> def foo(bar=new baz): >> #code >> >> def foo(bar=fresh baz): >> #code >> >> def foo(bar=separate baz): >> #code >> >> def foo(bar=another baz): >> #code >> >> def foo(bar=unique baz): >> #code >> >> where the new keyword (or angle brackets) would indicate that the >> parameter's default argument should use the new semantics. Other >> parameters would continue to use the old semantics. It was generally >> agreed that the angle-bracket syntax was particularly ugly, leading to >> the proposal of the other syntaxes. However, having 2 different sets of >> semantics could be confusing and leaving in the old semantics just for >> performance might be premature optimization. Refactorings to deal with >> the possible performance hit are discussed below. >> >> >> Specification >> >> The current semantics for default arguments are replaced by the >> following semantics: >> - Whenever a function is called, and the caller does not provide a >> value for a parameter with a default expression, the parameter's >> default expression shall be evaluated in the function's scope. The >> resulting value shall be assigned to a local variable in the >> function's scope with the same name as the parameter. > > Include something saying that any variables in a default expression > shall be lexical variables, with their scope being the first outer scope > that defines a variable with the same name (they should just use the > same rules as other lexical/closure variables), and that if the function > body defines a local variable with the same name as a variable in a > default expression, those variables shall be handled as two separate > variables. > >> - The default argument expressions shall be evaluated before the >> body of the function. >> - The evaluation of default argument expressions shall proceed in >> the same order as that of the parameter list in the function's >> definition. >> Given these semantics, it makes more sense to refer to default argument >> expressions rather than default argument values, as the expression is >> re-evaluated at each call, rather than just once at definition-time. >> Therefore, we shall do so hereafter. >> >> Demonstrative examples of new semantics: >> #default argument expressions can refer to >> #variables in the enclosing scope... >> CONST = "hi" >> def foo(a=CONST): >> print a >> >> >>> foo() >> hi >> >>> CONST="bye" >> >>> foo() >> bye >> >> #...or even other arguments >> def ncopies(container, n=len(container)): >> return [container for i in range(n)] >> >> >>> ncopies([1, 2], 5) >> [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]] >> >>> ncopies([1, 2, 3]) >> [[1, 2, 3], [1, 2, 3], [1, 2, 3]] >> >>> #ncopies grabbed n from [1, 2, 3]'s length (3) > > I'm not sure if this can be combined elegantly with what I said about > variables being lexical variables. The first argument to ncopies, > 'container', is clearly a local variable to ncopies. The 'container' in > the second arg default expr should, if my comments above are accepted, > be a lexical variable referring to the 'container' in the global scope. > The best way to combine the two features seems to be to let 'container' > be a local var if any of the preceding args is named 'container', and > let it be a lexically scoped variable otherwise. However, I'm not > convinced this complexity is worth it and the vars in default > expressions shouldn't just always be lexical vars. > >> >> #default argument expressions are arbitrary expressions >> def my_sum(lst): >> cur_sum = lst[0] >> for i in lst[1:]: cur_sum += i >> return cur_sum >> >> def bar(b=my_sum((["b"] * (2 * 3))[:4])): >> print b >> >> >>> bar() >> bbbb >> >> #default argument expressions are re-evaluated at every call... >> from random import randint >> def baz(c=randint(1,3)): >> print c >> >> >>> baz() >> 2 >> >>> baz() >> 3 >> >> #...but only when they're required >> def silly(): >> print "spam" >> return 42 >> >> def qux(d=silly()): >> pass >> >> >>> qux() >> spam >> >>> qux(17) >> >>> qux(d=17) >> >>> qux(*[17]) >> >>> qux(**{'d':17}) >> >>> #no output because silly() never called because d's value was >> specified in the calls >> >> #Rule 3 >> count = 0 >> def next(): >> global count >> count += 1 >> return count - 1 >> >> def frobnicate(g=next(), h=next(), i=next()): >> print g, h, i >> >> >>> frobnicate() >> 0 1 2 >> >>> #g, h, and i's default argument expressions are evaluated in >> the same order as the parameter definition >> >> >> Backwards Compatibility >> >> This change in semantics breaks all code which uses mutable default >> argument values. Such code can be refactored from: > > Wow, let's not scare everyone away just yet. This should read: > "This change in semantics breaks code which uses mutable default > argument expressions and depends on those expressions being evaluated > only once, or code that assigns new incompatible values in a parent > scope to variables used in default expressions" > >> >> def foo(bar=mutable): >> #code > > if 'mutable' is just a single variable, this isn't gonna break, unless > the global scope decides to do something like this: > > def foo(bar=mutable): > #code > > mutable = incompatible_mutable > # ... > foo() > >> >> to >> >> def stateify(state): >> def _wrap(func): >> def _wrapper(*args, **kwds): >> kwds['bar'] = state >> return func(*args, **kwds) >> return _wrapper >> return _wrap >> >> @stateify(mutable) >> def foo(bar): >> #code >> >> or >> >> state = mutable >> def foo(bar=state): >> #code >> >> or >> >> class Baz(object): >> def __init__(self): >> self.state = mutable >> >> def foo(self, bar=self.state): >> #code > > Minor point: the stateify decorator looks a bit scary to me as it uses > three levels of nested functions. (that's inherent to decorators, but > still.) Suggest you name the class and global var solutions first, and > the decorator as last, just to prevent people from stopping reading the > pep and voting '-1' right when they hit the decorator solution. > >> >> The changes in this PEP are backwards-compatible with all code whose >> default argument values are immutable > > ...or don't depend on being evaluated only once, and don't modify in a > parent scope the variables in default expressions in an incompatible way, > > (hmm, the 'or' and 'and' may need some disambiguation parentheses...) > >> including code using the idiom >> mentioned in the 'Motivation' section. However, such values will now be >> recomputed for each call for which they are required. This may cause >> performance degradation. If such recomputation is significantly >> expensive, the same refactorings mentioned above can be used. >> >> In relation to Python 3.0, this PEP's proposal is compatible with >> those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend >> on the acceptance of either of those PEPs. >> >> >> Reference Implementation >> >> All code of the form: >> >> def foo(bar=some_expr, baz=other_expr): >> #body >> >> Should act as if it had read (in pseudo-Python): >> >> def foo(bar=_undefined, baz=_undefined): >> if bar is _undefined: >> bar = some_expr >> if baz is _undefined: >> baz = other_expr >> #body > > and, if there are any variables occuring in the function body and in > some_expr or other_expr, rename those in the function body to something > that doesn't name-clash. > >> >> where _undefined is the value given to a parameter when the caller >> didn't specify a value for it. This is not intended to be a literal >> translation, but rather a demonstration as to how Python's internal >> argument-handling machinery should be changed. >> >> >> References >> >> [0] 10 Python pitfalls >> http://zephyrfalcon.org/labs/python_pitfalls.html >> >> [1] Python Gotchas >> http://www.ferg.org/projects/python_gotchas.html#contents_item_6 >> >> [2] When Pythons Attack >> >> http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2 >> >> [3] Keyword-Only Arguments >> http://www.python.org/dev/peps/pep-3102/ >> >> [4] Function Annotations >> http://www.python.org/dev/peps/pep-3107/ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas > > I'm not quite sure if your decision not to include the > default-expr-vars-become-lexical part was intentional or not. If it is, > can you tell me why you'd want that? Else you can incorporate my > comments in the pep. > Another minor point: I'm personally not too fond of too many 'shall's > close together. It makes me think of lots of bureaucracy and > design-by-committee. I think several peps don't have a shall in them, > but maybe it is the right language for this one. What's the right > language to use in peps? > > Well, that's about it I think. > > - Jan > From jimjjewett at gmail.com Tue Jan 30 08:19:11 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 30 Jan 2007 02:19:11 -0500 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BED309.10301@gmail.com> References: <45BCF804.7040207@gmail.com> <45BED309.10301@gmail.com> Message-ID: On 1/30/07, Chris Rebert wrote: > Jim Jewett wrote: > > On 1/28/07, Chris Rebert wrote: > >> Naive programmers desiring mutable default arguments > >> often make the mistake of writing the following: > >> def foo(mutable=some_expr_producing_mutable): > >> #rest of function > > Yes, it is an ugly gotcha, but so is the alternative. > > If it is changed, then just as many naive programmers (though perhaps > > not exactly the same ones) will make the opposite mistake -- and so > > will some experienced programmers who are used to current semantics. > Anyone (ab)using the current semantics with the above construct is or > ought to be aware that such usage is unpythonic. Are you worried about naive programmers or not? > Also, due to the > hairiness/obscurity of the construct, hardly anyone uses it, so the > impact of removing it should be relatively minor. I just did found 181 such uses in my own installation. 29 were either 3rd-party or test code, but that still leaves over 150 uses in the standard library. It is possible that some of them would also work with your semantics, or may even be bugs today -- but you would have to go through them one-by-one as part of the PEP process. > Additionally, I see > making the opposite mistake as usually only maybe causing a performance > problem, as opposed to causing the program to work incorrectly. That is true only when the argument is a cache. Generally, resetting the storage will make the program work incorrectly, but the program won't fail on the first data tested -- which means the bug is less likely to be caught. > >> There are currently few, if any, known good uses of the current > >> behavior of mutable default arguments. The most common one is to > >> preserve function state between calls. However, as one of the lists [2] > >> comments, this purpose is much better served by decorators, classes, or > >> (though less preferred) global variables. > > I disagree. This is particularly wrong for someone coming from a > > functional background. > I assume you disagree with the "purpose is much better served by > decorators, classes, or local variables" part as opposed to the "default > mutable arguments with current semantics have few good uses" part. > Please correct me if I'm in error. I disagree with both. There are many uses for preserving function state between calls. C uses static local variables. Object Oriented languages often use objects. > > A class plus an instantiation seems far too heavyweight for what ought > > to be a simple function. I'm not talking (only) about the runtime; > > the number of methods and lines of code is the real barrier for me. > > I'll sometimes do it (or use a global) anyhow, but it feels wrong. If > > I had felt that need more often when I was first learning python (or > > before I knew about the __call__ workaround), I might have just > > written the language off as no less bloated than java. > I'm sorry, but when you have a function sharing state between calls, > that just screams to me that you should make it a method of an object That's because you drank the OO koolaid. Python tries not to enforce any particular programming style. Its object model happens to be pretty good, but there are still times when OO isn't the answer. > so > you don't have to store the state in some roundabout way, such as in > mutable default arguments. If performance is your concern, the decorator > version might perform better (I don't really know), or in the extreme > case, you could use a global variable, which is definitely faster. I think there is something wrong with your benchmark. The default argument is accessible straight from the function object itself; even a cell variable shouldn't be that fast, let alone global variables. That said, speed is *not* my concern -- code size is. If I have to add several lines of boilerplate, the code becomes much less readable. That is roughly the same justification you have for not liking the "if arg is None: arg=foo()" idiom. The difference is that you're suggesting adding far more than a line or two. > > You see the problem with globals, but decorators are in some sense > > worse -- a function cannot see its own decorations. At best, it can > > *assume* that repeating its own name (despite Don't Repeat Yourself) > > will get another reference to self, but this isn't always true. > > Programmers used to creating functions outside of toplevel (or > > class-level) will be more aware of this, and see the suggestion as an > > indication that python is inherently buggy. > I really don't see how the decorator in the PEP is any worse than other > decorators in this regard. The problem you describe currently applies to > all decorated functions, though functools.wraps might help mitigate this > situation. Either you effectively hide the real function so as to create a lexical closure (plenty of extra work, both mentally and computationally) or you accept buggy code. The bug possibility isn't inherent to decorations; only to functions reading their *own* decorations. > Alternatively, we could make the new > semantics the default and have syntax to use the old semantics via > 'once' or some other keyword. This does nicely emphasize that such > semantics would be purely for optimization reasons. Using a mutable default is almost never for optimization reasons. -jJ From cvrebert at gmail.com Tue Jan 30 08:25:58 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Mon, 29 Jan 2007 23:25:58 -0800 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <20070129212157.5A2B.JCARLSON@uci.edu> References: <43aa6ff70701292036t7f0a2a96hc013ecd47957de4d@mail.gmail.com> <45BECF7B.6090503@gmail.com> <20070129212157.5A2B.JCARLSON@uci.edu> Message-ID: <45BEF306.5030801@gmail.com> Josiah Carlson wrote: > Using the features of a language to attempt to compensate for that same > language's (argued) shortcomings are a valid _and encouraged_ approach. > Your poo-pooing of Python conditionals, decorators, and lambdas to solve > this particular problem, to me, seems like you want a *particular > solution*. No, it's just that why use several different decorators when a slight change in semantics renders them all obsolete? Why have to use a decorator/lambda/conditional and remember when to use which? I'm trying to generalize the pattern of "reevaluate/copy default values each time they're required" into new behavior for default arguments. Indeed, as I stated in another email (no idea which one, there's been too many), I wholeheartedly support adding some of those decorators to the standard library should my PEP get rejected. > I don't see a problem with the current default argument semantics. Why? > Because in the case where I would want to receive a mutable parameter, > like in the case below that wouldn't work with Python's standard > semantics... > > def append_10(lst=[]): > lst.append(10) > return lst > > I would presumably change it to... > > def append_10(lst=None): > if lst is None: lst = [] > lst.append(10) > return lst > > Or some variant thereof. Now, here's the thing; if I want a mutable > argument, then None is a nonsensical value to pass, generally, as it is > not mutable. So I don't buy the whole "but then None would no longer be a > valid argument to pass" bull that was offered as a reason why the above > isn't a reasonable translation (I can't remember who offered it). Nowhere has it been proposed to forbid passing None as an argument. The basic core of the proposal is to have Python compile the code such that the boilerplate is either removed entirely (if the new semantics are the default) by no longer being necessary, or dramatically shortened via the addition of a new keyword (which would indicate the new semantics). Yes, you can manually transform your code to use the idiom you mention. But why should you have to? Why not have Python do the heavy-lifting for you? As stated in the "Reference Implementation" section of the PEP, the 'translation' provided is not intended to be a literal translation of the code at compile-time, but rather explain how Python's argument-handling machinery should be changed. The 'translation' looks similar to the existing idiom because that's what it aims to replace. But note that it's *Python* that's doing this 'translation', not the programmer! That's the gain! The programmer no longer needs to write the boilerplate. > I'm also not convinced by either of the 3 pages that talk about Python > "gotchas". You get bitten by it, you learn it, understand it, and move > on. If they can't figure it out, I'm not sure I want them writing > Python software anyways; I certainly wouldn't want them to work on any > of the Python software I work on and use. I'm not here to argue that people shouldn't be made to understand the difference between mutable and immutable values. They definitely should know the difference. I'm just advocating a language change to make certain kinds of functions less verbose. If you're worried about losing this 'teaching tool' (which another respondent was worried about), it will still exist in the form of: x = [[0]*4]*4 x[3][1] = 7 x[0][1] == 7 #TRUE! - Chris Rebert From jan.kanis at phil.uu.nl Tue Jan 30 11:36:26 2007 From: jan.kanis at phil.uu.nl (Jan Kanis) Date: Tue, 30 Jan 2007 11:36:26 +0100 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BEDA53.9010103@gmail.com> References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru> <91ad5bf80701292016m63bf1b28w84e2c9f62fb7ea0e@mail.gmail.com> <45BEDA53.9010103@gmail.com> Message-ID: On Tue, 30 Jan 2007 06:40:35 +0100, Chris Rebert wrote: > George Sakkis wrote: >> I'm sure that the python-devs and the BDFL would have >> thought about it quite a bit when the current semantics were decided, > > ...which was probably a while ago. They might reconsider the issue now > that some time has passed and they've seen how their decision has worked > out. But yes, your analysis is a definite possibility. Just looked it up, and python has had lexical variables since version 2.1, and default arguments since long before that. (forever?) Without lexical variables it's near impossible to implement re-evaluating default arguments, because the variables those default expressions refer to may no longer be available at function call time. So, The BDFL and other python devs didn't really have a choice but to have the default expressions evaluate at definition time (or implement lexical scopes, which is what has happened by now). Jan From jan.kanis at phil.uu.nl Tue Jan 30 12:21:10 2007 From: jan.kanis at phil.uu.nl (Jan Kanis) Date: Tue, 30 Jan 2007 12:21:10 +0100 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BEEF31.6070108@gmail.com> References: <45BCF804.7040207@gmail.com> <45BEEF31.6070108@gmail.com> Message-ID: On Tue, 30 Jan 2007 08:09:37 +0100, Chris Rebert wrote: > I also reworded the sentences using 'shall'. I don't really get why that > matters, but what the heck. Well, it doesn't. Just makes it nicer to read. (As long as the text is still clear, that is.) > Jan Kanis wrote: >> Well, I (obviously) like the idea, but your pep misses some important >> points (and some not so important). >> The important one is about the scoping of variables in default >> expressions. The pep says nothing about them. >> If I read the pep correctly, any variables in default expressions are >> handled the same as variables in the body of a function. This means the >> compiler decides if they should be local, lexical or global. If there >> is an assignment to a variable, the compiler makes it a local, else it >> finds the right enclosing scope (lexical or global). In the current >> python, this works fine: >> >>> a = 123 >> >>> def foo(b=a): >> a = 2 >> print a, b >> >>> foo() >> 2 123 >> >>> a = 42 >> >>> foo() >> 2 123 >> In the pep, the a in the default expression would be handled just like >> any other a in the function body, which means it wil become a _local_ >> variable. Calling the function would then result in an >> UnboundLocalError. Just like this in current python: >> >>> a = 123 >> >>> def foo(b=None): >> b = a if b==None else b >> a = 2 >> print a >> >>> foo() >> Traceback (most recent call last): >> File "", line 1, in >> foo() >> File "", line 2, in foo >> b = a if b==None else b >> UnboundLocalError: local variable 'a' referenced before assignment >> The solution, I think, as I wrote in my previous messages, is to have >> the compiler explicitly make variables in default expressions lexical >> or global variables. >> This would still break the foo() in my example above, because you can't >> assign to a lexical variable, or to a global that isn't declared >> global. Therefore I think the compiler should distinguish between the a >> in the default expression and the a in the function body, and treat >> them as two different variables. You can think about it as if the >> compiler silently renames one of them (without this rename being >> visible to python code. AFAIK the bytecode is stack based and closure >> vars are put in some kind of anonymous cell, which means they both >> don't actually have a name anyway.) Also, I've been thinking about the part that I suggested about having python distinguish between lexically scoped variables in default expressions and local variables that may have the same name. To clarify my proposal, a demonstration: Current python: a = 4 b = 5 def foo(x = [b]*a): x = copy.deepcopy(x) # one possible workaround to have the same list on every call b = 'bla' # no name clash of this b and b in default arg expr x += [b] print x foo() # prints [5, 5, 5, 5, 'bla'] foo() # prints [5, 5, 5, 5, 'bla'] proposed semantics: a = 4 b = 5 def foo(x = [b]*a): # deepcopy() no longer nescessary b = 'bla' # no name clash of this b and b in default arg expr, the compiler distinguishes them x += [b] print x foo() # prints [5, 5, 5, 5, 'bla'] foo() # prints [5, 5, 5, 5, 'bla'] This would work the same as this code in the current python. Imagine it as the compiler silently transforming the above code to this code: a = 4 b = 5 def foo(x = _undefined): if x == _undefined: x = [b]*a _b = 'bla' x += [_b] print x foo() # prints [5, 5, 5, 5, 'bla'] foo() # prints [5, 5, 5, 5, 'bla'] note: the use of deepcopy assumes that whatever is passed to foo is deepcopyable. With the new semantics this function can be used without these kind of assumptions. However, I've been thinking about weather it is a good idea to have python distinguish between the local and lexical 'b' in the above code, and do the what you can think of as implicit renaming. Making the distinction makes the whole proposal more backward compatible, however I think that if backward compatibility were not an issue, it would be best not to make the distinction and just have the programmer choose a different name for the local variable. This would improve readability as there are no two different variables with the same name in the same piece of code, and decrease language complexity as there is no rule that's best explained in terms of variable name rewriting. Not making the distinction does cause the default arguments proposal to be less backward compatible, but I don't know if that is really a big problem. I assume this pep wil be a py3k pep anyway, so there are going to be incompatibilities anyway. Also, the compiler is perfectly able to see at compile time if a variable used in a default expression and as a local, so it is able to raise a very descriptive error. All in all, I think this part of my proposal is not such a good idea, unless there's going to be some kind of transitional implementation of this pep in py 2.x meta-note: I hope it is clear what I'm trying to say. Basically, I'm clarifying, arguing against, and retracting a part of my own proposal I made in an earlier mail. - Jan From rrr at ronadam.com Tue Jan 30 12:35:34 2007 From: rrr at ronadam.com (Ron Adam) Date: Tue, 30 Jan 2007 05:35:34 -0600 Subject: [Python-ideas] Import and '..', '../..' in serach path. Message-ID: <45BF2D86.2080801@ronadam.com> In order to resolve a path conflict where I'm working on several copies of the same package. I found it useful to add the following near the top of modules in a package or sub package. Module in package: import sys sys.path = ['..'] + sys.path import package.module # Imports module in "this!" package. Note: There could still be conflicts if a module with the same name is in the same directory as the package. But that's much less likely than one in the rest of the path. Module in sub-package: import sys sys.path = ['../..'] + sys.path import package.subpackage.module # finds "self" (subpackage) reliably. By explicitly adding the packages parent directory to the *front* of sys.path it resolves cases where imports using absolute imports, import modules from another package because they are found first in the search path. Adding this tip to the documentation some where would be nice. (providing there is no major surprising side effects.) Of course I may have missed some obvious way to do this. If so, it wasn't in an obvious place to be found. I looked. ;-) ---------------------------------- It might be useful to have a built-in function to do this. A function could also check for __init__ files and raise errors if they are missing. set_package_name(dotted.name) # Replaces import sys & path modification Where dotted.name is the full package + sub-package name the current module is located in. The function would search upwards to get the root package directory and add that to the *front* of sys.path. Module in package: set_package_name('package') # Add parent directory to front of sys.path import packagename.module # Finds module in "this!" package reliably. Module in subpackage: set_package_name('package.subpackage') import package.subpackage.module # Finds "self" (subpackage) reliably. ---------------------------------- It may also be able to modify the import behavior to allow relative imports to work when the module is run as script. set_package_name('package') from . import module1 # Imports modules from "this" package. from . import module2 Currently an exception is raised you try to run a module with relative references as a script. ValueError: Attempted relative import in non-package I think it is very handy to be able to run tests as scripts and keep them in a sub-package. Especially while I'm writing them. Another benefit of using relative imports with an absolute specified package name, is if you rename a package or relocate a submodule, you only have one line to change. Cheers, Ron From rrr at ronadam.com Tue Jan 30 13:13:31 2007 From: rrr at ronadam.com (Ron Adam) Date: Tue, 30 Jan 2007 06:13:31 -0600 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <20070129212157.5A2B.JCARLSON@uci.edu> References: <43aa6ff70701292036t7f0a2a96hc013ecd47957de4d@mail.gmail.com> <45BECF7B.6090503@gmail.com> <20070129212157.5A2B.JCARLSON@uci.edu> Message-ID: <45BF366B.9040107@ronadam.com> Josiah Carlson wrote: > Chris Rebert wrote: >> Collin Winter wrote: >>> On 1/28/07, Chris Rebert wrote: >>> Syntax changes are a huge stick to wield against such a small problem. >>> I realize the boilerplate is annoying, but Tomer has pointed out a >>> number of decorator-based solutions [1] that could easily be adapted >>> to any unusual needs you have. >> One of his decorators assumes that the default value merely needs to be >> copied across calls. However, as the PEP says, there are cases where >> this isn't sufficient or just doesn't work. His second decorator >> involves using lambdas (ick!), > [snip] > > Using the features of a language to attempt to compensate for that same > language's (argued) shortcomings are a valid _and encouraged_ approach. > Your poo-pooing of Python conditionals, decorators, and lambdas to solve > this particular problem, to me, seems like you want a *particular > solution*. > > I don't see a problem with the current default argument semantics. Why? > Because in the case where I would want to receive a mutable parameter, > like in the case below that wouldn't work with Python's standard > semantics... > > def append_10(lst=[]): > lst.append(10) > return lst > > I would presumably change it to... > > def append_10(lst=None): > if lst is None: lst = [] > lst.append(10) > return lst Strings sometimes work nice. Then you have a hint as to what should go there. def append_10(lst='[]'): if lst == '[]': lst = [] lst.append(10) return lst > Or some variant thereof. Now, here's the thing; if I want a mutable > argument, then None is a nonsensical value to pass, generally, as it is > not mutable. So I don't buy the whole "but then None would no longer be a > valid argument to pass" bull that was offered as a reason why the above > isn't a reasonable translation (I can't remember who offered it). > > I'm also not convinced by either of the 3 pages that talk about Python > "gotchas". You get bitten by it, you learn it, understand it, and move > on. If they can't figure it out, I'm not sure I want them writing > Python software anyways; I certainly wouldn't want them to work on any > of the Python software I work on and use. > > > - Josiah I agree, What seems like a gotcha is some situations can be a feature in others. At the most, documenting things like this should be more visible. Like maybe adding a [Gotcha! Mutable arguments...] info box's in the documents. I like those little info box's in manuals, especially if they are written with a bit of humor. Cheers, Ron From jan.kanis at phil.uu.nl Tue Jan 30 13:24:43 2007 From: jan.kanis at phil.uu.nl (Jan Kanis) Date: Tue, 30 Jan 2007 13:24:43 +0100 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: References: <45BCF804.7040207@gmail.com> <45BED309.10301@gmail.com> Message-ID: On Tue, 30 Jan 2007 08:19:11 +0100, Jim Jewett wrote: > On 1/30/07, Chris Rebert wrote: >> Also, due to the >> hairiness/obscurity of the construct, hardly anyone uses it, so the >> impact of removing it should be relatively minor. > > I just did found 181 such uses in my own installation. 29 were either > 3rd-party or test code, but that still leaves over 150 uses in the > standard library. It is possible that some of them would also work > with your semantics, or may even be bugs today -- but you would have > to go through them one-by-one as part of the PEP process. Well, it seems that numbers are gonna be needed anyway to convince people. But that's certainly possible. What exactly did you search for, and what did you find? I did some preliminary grepping through the standardlib myself. regex = "def.*\(.*=.*\):", searching through all .py files in /Lib: 2455 matches in total in 474 files out of 998 I manually looked through the first bunch of files, those contained 145 matches. (the first files sorted alphabetically up until and including cookielib.py) By far the most of them had literals as default value. A huge amount of them used the =None idiom, I estimate perhaps one third. 10 matches used variables but will not break under the pep's semantics. I found 1 (one) file that used the current semantics to keep state between calls, in two inner functions. This can easily be changed. So, that leaves 145-12=133 uses of default values that are just constants. If I have time and figure out the right regexes I'll try and come up with some more numbers on the entire stdlib, and the ammount of uses of =None. - Jan From eopadoan at altavix.com Tue Jan 30 13:43:09 2007 From: eopadoan at altavix.com (Eduardo "EdCrypt" O. Padoan) Date: Tue, 30 Jan 2007 10:43:09 -0200 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: References: <45BCF804.7040207@gmail.com> <45BED309.10301@gmail.com> Message-ID: On 1/30/07, Eduardo EdCrypt O. Padoan wrote: > > If I have time and figure out the right regexes I'll try and come up with > > some more numbers on the entire stdlib, and the ammount of uses of =None. > > > > Some uses of the spam=[] and ham=None in Python projects, including > Python itself: > > http://www.google.com/codesearch?q=def.*%5C(.*%3D%5C%5B%5C%5D.*%5C)%3A%20lang%3APython&hl=en&btnG=Search+Code > http://www.google.com/codesearch?hl=en&lr=&q=def.*%5C%28.*%3DNone.*%5C%29%3A+lang%3APython&btnG=Search > > In this second search, I need a way to search that, in the body of the > function, we have something like "if foo is not None: foo = []" (and > foo = {} too) > > -- > EduardoOPadoan (eopadoan->altavix::com) > Bookmarks: http://del.icio.us/edcrypt > Blog: http://edcrypt.blogspot.com > Jabber: edcrypt at jabber dot org > ICQ: 161480283 > GTalk: eduardo dot padoan at gmail dot com > MSN: eopadoan at altavix dot com > -- EduardoOPadoan (eopadoan->altavix::com) Bookmarks: http://del.icio.us/edcrypt Blog: http://edcrypt.blogspot.com Jabber: edcrypt at jabber dot org ICQ: 161480283 GTalk: eduardo dot padoan at gmail dot com MSN: eopadoan at altavix dot com From jan.kanis at phil.uu.nl Tue Jan 30 14:18:35 2007 From: jan.kanis at phil.uu.nl (Jan Kanis) Date: Tue, 30 Jan 2007 14:18:35 +0100 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: <20070126212657.5A06.JCARLSON@uci.edu> References: <20070126212657.5A06.JCARLSON@uci.edu> Message-ID: On Mon, 29 Jan 2007 08:38:39 +0100, Roman Susi wrote: > This is what > incremental dynamic semantics is about. So, the suggestion is good only > as separated feature, but is IMHO wrong > if considered in the language design as a whole. wtf is incremental dynamic semantics, in this context? I did some googling but all I found referred to techniques related to programming environments. This proposal is just about a change in the language. On Sat, 27 Jan 2007 06:30:00 +0100, Josiah Carlson wrote: > "Jan Kanis" wrote: >> I hope we agree >> that the problem we're trying to solve is that while [snip] > I'm going to have to disagree on the 'non-intuitive and ugly' claim. We > are just going to have to agree to disagree. > On Mon, 29 Jan 2007 08:38:39 +0100, Roman Susi wrote: > Hello! > > I'd liked to say outright that this bad idea which complicates matters > more than provides solutions. > Right now it is enough to know that the part from def to ":" is executed > at definition time. Well, it's good to be clear on where the disagreements lie. However I'm not yet ready to let it rest at that without some more arguments. As Chris pointed out in his first mail, this 'wart' is mentioned on several lists of python misfeatures: [0][1][2]. I'd like to add to this that even the python documentation finds this issue severe enough to issue an "Important warning"[4]. It seems clear that this behaviour is a gotcha, at least for newbies. This could be excused if there is a good reason to spend the additional time learning this behaviour, but some of the links state, and my assumption is, that there are very few situations where re-evaluating causes a problem and which isn't easily fixable. The semantics which I'd like to have are even easier than the current semantics: everything in a function, be it before or after the colon, is executed when the function is called. Of course, as Collin Winters pointed out, the burden of proof of showing that these semantics aren't going to be a problem is still on the pep proponents. On the other hand, are there really any good reasons to choose the current semantics of evaluation at definition time? What I've heard basically boils down to two arguments: - "let's not change anything", i.e. resist change because it is change, which I don't think is a very pythonic argument. - Arguments based on the assumption that people actually do make lots of use of the fact that default arguments are shared between function invocations, many of which will result in (much) more code if it has to be transformed to using one of the alternative idioms. If this is true, it is a valid argument. I guess there's still some stdlib grepping to do to decide this. So, are there any _other_ arguments in favour of the current semantics?? - Jan [0] 10 Python pitfalls (http://zephyrfalcon.org/labs/python_pitfalls.html) [1] Python Gotchas (http://www.ferg.org/projects/python_gotchas.html#contents_item_6) [2] When Pythons Attack (http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2) [4] Python manual - 4. More control flow tools (http://docs.python.org/tut/node6.html#SECTION006710000000000000000) From veloso at verylowsodium.com Tue Jan 30 16:48:54 2007 From: veloso at verylowsodium.com (Greg Falcon) Date: Tue, 30 Jan 2007 10:48:54 -0500 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: References: <20070126212657.5A06.JCARLSON@uci.edu> Message-ID: <3cdcefb80701300748y3c92d544g6414bb5438d28305@mail.gmail.com> On 1/30/07, Jan Kanis wrote: > On the other hand, are there really any good reasons to choose the current > semantics of evaluation at definition time? While I sympathize with the programmer that falls for this common Python gotcha, and would not have minded if Python's semantics were different from the start (though the current behavior is cleaner and more consistent), making such a radical change to such a core part of the language semantics now is a very bad idea for many reasons. > What I've heard basically > boils down to two arguments: > - "let's not change anything", i.e. resist change because it is change, > which I don't think is a very pythonic argument. The argument here is not "let's not change anything because it's change," but rather "let's not break large amounts of existing code without a very good reason." As has been stated here by others, making obsolete a common two-line idiom is not a compelling enough reason to do so. Helping out beginning Python programmers, while well-intentioned, doesn't feel like enough of a motivation either. Notice that the main challenge for the novice programmer is not to learn how default arguments work -- novices can learn to recognize and write the idiom easily enough -- but rather to learn how variables and objects work in general. >>> a=b=['foo'] >>> c=d=42 >>> a+=['bar'] >>> c+=1 >>> b ['foo', 'bar'] >>> d 42 At some point in his Python career, a novice is going to have to understand why b "changed" but d didn't. Fixing the default argument "wart" doesn't remove the necessity to understand the nature of mutable objects and variable bindings in Python; it just postpones the problem. This is a fact worth keeping in mind when deciding whether the sweeping change in semantics is worth the costs. > - Arguments based on the assumption that people actually do make lots of > use of the fact that default arguments are shared between function > invocations, many of which will result in (much) more code if it has to be > transformed to using one of the alternative idioms. If this is true, it is > a valid argument. I guess there's still some stdlib grepping to do to > decide this. Though it's been decried here as unPythonic, I can't be the only person who uses the idiom def foo(..., cache={}): for making a cache when the function in question does not rise to the level of deserving to be a class object instead. I don't apologize for finding it less ugly than using a global variable. I know I'm not the only user of the idiom because I didn't invent it -- I learned it from the Python community. And the fact that people have already found usages of the current default argument behavior in the standard library is an argument against the "unPythonic" claim. I'm reminded of GvR's post on what happened when he made strings non-iterable in a local build (iterable strings being another "wart" that people thought needed fixing): http://mail.python.org/pipermail/python-3000/2006-April/000824.html > So, are there any _other_ arguments in favour of the current semantics?? Yes. First, consistency. What do the three following Python constructs have in common? 1) lambda x=foo(): None 2) (x for x in foo()) 3) def bar(x=foo()): pass Answer: all three evaluate foo() immediately, choosing not to defer the evaluation to when the resulting object is invoked, even though they all reasonably could. It's especially notable that the recently-added feature (generator expressions) follows existing precedent. This was not accidental, but rather a considered design decision. Two paragraphs from PEP 289 could apply equally well to your proposal: | Various use cases were proposed for binding all free variables when | the generator is defined. And some proponents felt that the resulting | expressions would be easier to understand and debug if bound | immediately. | However, Python takes a late binding approach to lambda expressions | and has no precedent for automatic, early binding. It was felt that | introducing a new paradigm would unnecessarily introduce complexity. In fact, the situation here is worse. PEP 289 is arguing against early binding of free variables as being complex. You're not proposing an early binding, but rather a whole new meaning of the "=" token, "save this expression for conditional evaluation later." It's never meant anything like that before. Second, the a tool can't fix all usages of the old idiom. When things break, they can break in subtle or confusing ways. Consider my module "greeter": == begin greeter.py == import sys def say_hi(out = sys.stdout): print >> out, "Hi!" del sys # don't want to leak greeter.sys to the outside world == end greeter.py == Nothing I've done here is strange or unidiomatic, and yet your proposed change breaks it, and it's unclear how an automated tool should fix it. What's worse about the breakage is that it doesn't break when greeter is imported, or even when greeter.say_hi is called with an argument. It might take a while before getting a very surprising error "global name 'sys' is not defined". Third, the old idiom is less surprising. def foo(x=None): if x is None: x= may take arbitrarily long to complete. It may have side effects. It may throw an exception. It is evaluated inside the function call, but only evaluated when the default value is used (or the function is passed None). There is nothing surprising about any of that. Now: def foo(x=): pass Everything I said before applies. The expression can take a long time, have side effects, throw an exception. It is conditionally evaluated inside the function call. Only now, all of that is terribly confusing and surprising (IMO). Greg F From duda.piotr at gmail.com Tue Jan 30 18:05:40 2007 From: duda.piotr at gmail.com (Piotr Duda) Date: Tue, 30 Jan 2007 18:05:40 +0100 Subject: [Python-ideas] Mutable default arguments - another approach Message-ID: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com> I think that this problem can be solved by the following change of default argument behavior: default arguments are evaluated in definition time (like in current implementation), but right after being evaluated, result object is checked if it's mutable (for example by checking of presence __copy__ special method or being instance of built in (sub)class list/dict/set), if object is mutable, argument is marked by COPY_DEF_ARG flag. There are two reasons for this check being done there: 1. performance 2. it can be controlled by "from __future__ import ..." statement in per-file manner Then if default argument is needed in function call, first COPY_DEF_ARG flag is checked if not set, default argument behaves exactly like in current implementation, if the flag is set, it's shallow copy is used instead. Adding following classes/functions to stdlib allow reproduce old behavior as well as add some new. class IterateDefaultArg(object): def __init__(self, iterator): self.__iterator = iterator def __copy__(self): return self.__iterator.next() class DefaultArgWrapper(object): def __init__(self, generatorfunc): self.__generatorfunc = generatorfunc def __call__(self, *args, **kwargs): return DefaultArgObject(self.__generatorfunc(*args, **kwargs)) @DefaultArgWrapper def nocopyarg(obj): while 1: yield obj With this current definition like: def foo(cache = {}): ... need to be replaced by: def foo(cache = nocopyarg({})): ... If one want to use deep copy instead copy, it might be done like this: @DefaultArgWrapper def deepcopyarg(obj): from copy import deepcopy while 1: yield deepcopy(obj) def foo(x = deepcopyarg()): ... P.S. sorry for my bad English -- ???????? ?????? From jcarlson at uci.edu Tue Jan 30 18:16:03 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 30 Jan 2007 09:16:03 -0800 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BEF306.5030801@gmail.com> References: <20070129212157.5A2B.JCARLSON@uci.edu> <45BEF306.5030801@gmail.com> Message-ID: <20070129233754.5A32.JCARLSON@uci.edu> Chris Rebert wrote: > > Josiah Carlson wrote: > > Using the features of a language to attempt to compensate for that same > > language's (argued) shortcomings are a valid _and encouraged_ approach. > > Your poo-pooing of Python conditionals, decorators, and lambdas to solve > > this particular problem, to me, seems like you want a *particular > > solution*. > > No, it's just that why use several different decorators when a slight > change in semantics renders them all obsolete? Why have to use a > decorator/lambda/conditional and remember when to use which? I'm trying > to generalize the pattern of "reevaluate/copy default values each time > they're required" into new behavior for default arguments. Indeed, as I > stated in another email (no idea which one, there's been too many), I > wholeheartedly support adding some of those decorators to the standard > library should my PEP get rejected. The conditional works in all cases. The lambda works in all cases. The set of decorators seems to need to be tuned depending on the situation. Me, I would just use a conditional or even a plain if statement, as it is already sufficiently universal to handle every case. Discussions about 'but then I can't mask global names with local names' are going to fall on deaf ears; it's seen as poor form, and discussing what one can do using poor Python form is not topical. > > I don't see a problem with the current default argument semantics. Why? > > Because in the case where I would want to receive a mutable parameter, > > like in the case below that wouldn't work with Python's standard > > semantics... > > > > def append_10(lst=[]): > > lst.append(10) > > return lst > > > > I would presumably change it to... > > > > def append_10(lst=None): > > if lst is None: lst = [] > > lst.append(10) > > return lst > > > > Or some variant thereof. Now, here's the thing; if I want a mutable > > argument, then None is a nonsensical value to pass, generally, as it is > > not mutable. So I don't buy the whole "but then None would no longer be a > > valid argument to pass" bull that was offered as a reason why the above > > isn't a reasonable translation (I can't remember who offered it). > > Nowhere has it been proposed to forbid passing None as an argument. And I never claimed as much. There was a previous post by someone (I can't remember who offered it), saying that replacing 'lst=[]' with 'lst=None' would make it so that a user couldn't signal *something special* with a 'None' argument. I was trying to get across that such reasoning doesn't make sense in this particular context. > The > basic core of the proposal is to have Python compile the code such that > the boilerplate is either removed entirely (if the new semantics are the > default) by no longer being necessary, or dramatically shortened via the > addition of a new keyword (which would indicate the new semantics). Yes, > you can manually transform your code to use the idiom you mention. But > why should you have to? Why not have Python do the heavy-lifting for you? Heavy lifting? One line change and one line addition is heavy lifting? There has got to be an episode of Monty Python embedded in this thread somewhere, because I can't help laughing at your claim. > > I'm also not convinced by either of the 3 pages that talk about Python > > "gotchas". You get bitten by it, you learn it, understand it, and move > > on. If they can't figure it out, I'm not sure I want them writing > > Python software anyways; I certainly wouldn't want them to work on any > > of the Python software I work on and use. > > I'm not here to argue that people shouldn't be made to understand the > difference between mutable and immutable values. They definitely should > know the difference. I'm just advocating a language change to make > certain kinds of functions less verbose. My point is that the change results in *certain* functions being trivially less verbose. I've spent more time working around re's call semantics than I have dealing with (incorrect assumptions about Python's) default argument semantics. As I don't see a need to change re, I don't see a need to change default argument semantics. - Josiah From jimjjewett at gmail.com Tue Jan 30 18:22:01 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 30 Jan 2007 12:22:01 -0500 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: References: <45BCF804.7040207@gmail.com> <45BED309.10301@gmail.com> Message-ID: On 1/30/07, Jan Kanis wrote: > On Tue, 30 Jan 2007 08:19:11 +0100, Jim Jewett > wrote: > > On 1/30/07, Chris Rebert wrote: > >> Also, due to the > >> hairiness/obscurity of the construct, hardly anyone uses it, so the > >> impact of removing it should be relatively minor. > > I just did found 181 such uses in my own installation. 29 were either > > 3rd-party or test code, but that still leaves over 150 uses in the > > standard library. It is possible that some of them would also work > > with your semantics, or may even be bugs today -- but you would have > > to go through them one-by-one as part of the PEP process. > But that's certainly possible. What exactly did you search for, and what > did you find? In idle, edit menu, find in files, regex 'def.*=(\\[|\\{)(\\]|\\})' In other words, it caught "={}" or "=[]" on a def line. It did not catch other (possibly mutable) default args, nor did it catch multi-line definitions, nor did it even catch extra whitespace. I didn't notice any false positives where the "={}" was really in a comment. I didn't even look at defaults of None (let alone other singletons) to see if they were part of this idiom. They mostly seemed to be passing state, but the __init__ methods in particular may have been using a class-level variable by accident. (such as the cnf argument to Tk widgets.) I'll try to check more closely later. -jJ From george.sakkis at gmail.com Tue Jan 30 18:22:54 2007 From: george.sakkis at gmail.com (George Sakkis) Date: Tue, 30 Jan 2007 12:22:54 -0500 Subject: [Python-ideas] Mutable default arguments - another approach In-Reply-To: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com> References: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com> Message-ID: <91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com> On 1/30/07, Piotr Duda wrote: > I think that this problem can be solved by the following change of > default argument behavior: > default arguments are evaluated in definition time (like in current > implementation), but right after being evaluated, result object is > checked if it's mutable (for example by checking of presence __copy__ > special method or being instance of built in (sub)class > list/dict/set) > > (snipped) > AFAIK there's no reliable way of deciding whether an arbitrary object is mutable or not, so the rest of the post is irrelevant. Besides, mutable default arguments is just a specific use case; the general problem discussed here is call-time vs definition-time semantics for arbitrary default argument expressions (function calls, attribute lookups, etc.). George From duda.piotr at gmail.com Tue Jan 30 19:18:46 2007 From: duda.piotr at gmail.com (Piotr Duda) Date: Tue, 30 Jan 2007 19:18:46 +0100 Subject: [Python-ideas] Mutable default arguments - another approach In-Reply-To: <91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com> References: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com> <91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com> Message-ID: <3df8f2650701301018i22fff8deq86829a292dceb8b2@mail.gmail.com> 2007/1/30, George Sakkis : > On 1/30/07, Piotr Duda wrote: > > > I think that this problem can be solved by the following change of > > default argument behavior: > > default arguments are evaluated in definition time (like in current > > implementation), but right after being evaluated, result object is > > checked if it's mutable (for example by checking of presence __copy__ > > special method or being instance of built in (sub)class > > list/dict/set) > > > > (snipped) > > > > AFAIK there's no reliable way of deciding whether an arbitrary object > is mutable or not, so the rest of the post is irrelevant. Besides, > mutable default arguments is just a specific use case; the general > problem discussed here is call-time vs definition-time semantics for > arbitrary default argument expressions (function calls, attribute > lookups, etc.). AFAIK there's no reliable way of deciding whether an arbitrary object has any feature (except it's type at c level), in worst case you can treat all object as mutable, but that itself doesn't make rest of my post irrelevant (have you even read it?). Read my previous post again, and give me examples of what you cannot do regarding default arguments. -- ???????? ?????? From rnd at onego.ru Tue Jan 30 20:29:34 2007 From: rnd at onego.ru (Roman Susi) Date: Tue, 30 Jan 2007 21:29:34 +0200 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: References: <20070126212657.5A06.JCARLSON@uci.edu> Message-ID: <45BF9C9E.8010206@onego.ru> Jan Kanis wrote: > On Mon, 29 Jan 2007 08:38:39 +0100, Roman Susi wrote: > >> This is what >> incremental dynamic semantics is about. So, the suggestion is good only >> as separated feature, but is IMHO wrong >> if considered in the language design as a whole. > > > wtf is incremental dynamic semantics, in this context? I did some > googling but all I found referred to techniques related to programming > environments. This proposal is just about a change in the language. Oh, I thought this is very fundamental and well-known as it is the second line of Guido's definition of Python: "Python is an interpreted, object-oriented, high-level programming language with dynamic semantics. ..." As I understand it, it means that statements are read sequencially and the semantics of names depends on the flow control. The proposal (as it seems to me) wants to make change to the language inconsistent with nature dynamic semantics. Its like put implicit: if RUN_FIRST_TIME: do this else: do that for the same line of code (def statement's first line). > [snip] > Well, it's good to be clear on where the disagreements lie. However I'm > not yet ready to let it rest at that without some more arguments. > > As Chris pointed out in his first mail, this 'wart' is mentioned on > several lists of python misfeatures: [0][1][2]. I'd like to add to this > that even the python documentation finds this issue severe enough to > issue an "Important warning"[4]. > > It seems clear that this behaviour is a gotcha, at least for newbies. > This could be excused if there is a good reason to spend the additional > time learning this behaviour, but some of the links state, and my But as somebody already said the alternative is even worse... Its quite easier to mention 3-5 Python warts up front to newbies than to introduce subtle exception for semantics and noise words such as "new" which do not have any other use elsewhere. > assumption is, that there are very few situations where re-evaluating > causes a problem and which isn't easily fixable. > The semantics which I'd like to have are even easier than the current > semantics: everything in a function, be it before or after the colon, > is executed when the function is called. > Of course, as Collin Winters pointed out, the burden of proof of > showing that these semantics aren't going to be a problem is still on > the pep proponents. > > So, are there any _other_ arguments in favour of the current semantics?? Repeat the mantra: Python's dynamic semantics will not change Python's dynamic semantics will not change Python's dynamic semantics will not change ;-) > > - Jan Roman From brett at python.org Tue Jan 30 20:32:21 2007 From: brett at python.org (Brett Cannon) Date: Tue, 30 Jan 2007 11:32:21 -0800 Subject: [Python-ideas] Import and '..', '../..' in serach path. In-Reply-To: <45BF2D86.2080801@ronadam.com> References: <45BF2D86.2080801@ronadam.com> Message-ID: On 1/30/07, Ron Adam wrote: > > In order to resolve a path conflict where I'm working on several copies of the > same package. I found it useful to add the following near the top of modules in > a package or sub package. > > > Module in package: > > import sys > sys.path = ['..'] + sys.path > > import package.module # Imports module in "this!" package. > > > Note: There could still be conflicts if a module with the same name is in the > same directory as the package. But that's much less likely than one in the rest > of the path. > > > Module in sub-package: > > import sys > sys.path = ['../..'] + sys.path > > import package.subpackage.module # finds "self" (subpackage) reliably. > > > By explicitly adding the packages parent directory to the *front* of sys.path it > resolves cases where imports using absolute imports, import modules from another > package because they are found first in the search path. Why aren't you using relative imports (e.g., ``from . import module``)? That should be doing exactly what you want. That uses __path__ which is set to the path of the package. -Brett From rnd at onego.ru Tue Jan 30 20:48:57 2007 From: rnd at onego.ru (Roman Susi) Date: Tue, 30 Jan 2007 21:48:57 +0200 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BED64D.10801@gmail.com> References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru> <45BED64D.10801@gmail.com> Message-ID: <45BFA129.1030805@onego.ru> Chris Rebert wrote: > Roman Susi wrote: > >> Hello! >> >> I'd liked to say outright that this bad idea which complicates matters [skip] > >> P.S. However, I may be wrong. In that case my syntax suggestion would >> be this: >> >> def foo(non_const or []): >> ... >> >> where [] is executed at runtime BECAUSE at def time non_const is >> somehow True and that is enough to leave [] alone. >> I have not checked, but I believe it is backward compatible. >> Anyway, could you summarize both contr-argument and this syntax >> proposal in the PEP? > > > I don't quite understand exactly how this would work and would like more > details on it, but once you've explained it, of course I'd be happy to > include it in the next draft. Simple. def foo(non_const or []): ... is equivalent to def foo(non_const=None): if non_const is None: none_const = [] ... And this will be as before: def foo(non_const=[]): ... Also, I thing that programmers should not use subtle difference between None and other False values, so something like def foo(non_const=None): non_const = none_const or [] is also valid. Another approach (if you want to pursue the feature) could be complication to name binding protocol. a = [] will be as before, but default value assignment could trigger some extra method. So, you can explicitly regulate your instance reaction to default-value assignment: class MyMutableList: ... def __default__(self, old_default): return old_default.copy() Regards, Roman > - Chris Rebert > From rnd at onego.ru Tue Jan 30 21:03:25 2007 From: rnd at onego.ru (Roman Susi) Date: Tue, 30 Jan 2007 22:03:25 +0200 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BED309.10301@gmail.com> References: <45BCF804.7040207@gmail.com> <45BED309.10301@gmail.com> Message-ID: <45BFA48D.2020501@onego.ru> Chris Rebert wrote: > Jim Jewett wrote: > >>On 1/28/07, Chris Rebert wrote: /skip/ >> >>> def foo(bar=new baz): >>> #code >> >>This would be less bad. >> >>That said, I fear many new programmers would fail to understand when >>they needed new and when they didn't, so that in practice, it would be >>just optional random noise. > > > This is part of the reason I'm trying to avoid adding new syntax. > However, I assert that at least 'new' is clearer than the' x=None; if x But if you are concerned with the current None, there could be some other, new False value serving the same need, like: def foo(x, y, z, bar=Missing, qux=Missing): if baz is Missing: baz = [] #code or even: def foo(x, y, z, bar=, qux=): if baz is Missing: baz = [] #code at least, it doesn't require decorators, is backward compatible (hopefully no grammar conflicts in there), reads as English. > is None: x=expr' idiom in that it expresses one's intent more clearly. > Also, this would at least be a prettier way to spell the idiom even if > the reason still needed explaining. Alternatively, we could make the new > semantics the default and have syntax to use the old semantics via > 'once' or some other keyword. This does nicely emphasize that such > semantics would be purely for optimization reasons. I think I'll add > this to the PEP. > > >>>Demonstrative examples of new semantics: >>> #default argument expressions can refer to >>> #variables in the enclosing scope... >>> CONST = "hi" >>> def foo(a=CONST): >>> print a >> >>This would work if there were any easy way to create a new scope. In >>Lisp, it makes sense. In python, it would probably be better to just >>find a way for functions to refer to their own decorations reliably. > > > This is outside of the scope of my PEP. However, the below improvement > should help and you could always use one of the other refactorings to > work around this issue. > > >>>Backwards Compatibility >>> >>> This change in semantics breaks all code which uses mutable default >>>argument values. Such code can be refactored from: >>> >>> def foo(bar=mutable): >>> #code >>> >>>to >> >> [a decorator option uses 3 levels of nested functions, which aren't even >> generic enough for reuse, unless you give up introspection.] >> >>No. It could be done with a (more complex) decorator, if that >>decorator came with the stdlib (probably in functools) > > > Agreed, the decorator could be better. I've just enhanced it using > functools.wraps, which should help with the introspection issues you > raise. Other improvements to the refactoring code in the PEP are welcomed. > > - Chris Rebert From g.brandl at gmx.net Tue Jan 30 21:17:55 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 30 Jan 2007 21:17:55 +0100 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BFA48D.2020501@onego.ru> References: <45BCF804.7040207@gmail.com> <45BED309.10301@gmail.com> <45BFA48D.2020501@onego.ru> Message-ID: Roman Susi schrieb: > Chris Rebert wrote: >> Jim Jewett wrote: >> >>>On 1/28/07, Chris Rebert wrote: > > /skip/ > >>> >>>> def foo(bar=new baz): >>>> #code >>> >>>This would be less bad. >>> >>>That said, I fear many new programmers would fail to understand when >>>they needed new and when they didn't, so that in practice, it would be >>>just optional random noise. >> >> >> This is part of the reason I'm trying to avoid adding new syntax. >> However, I assert that at least 'new' is clearer than the' x=None; if x > > But if you are concerned with the current None, there could be some > other, new False value serving the same need, like: > > def foo(x, y, z, bar=Missing, qux=Missing): > if baz is Missing: > baz = [] > #code > > or even: > > def foo(x, y, z, bar=, qux=): > if baz is Missing: > baz = [] > #code why not even def foo(x, y, z, bar=..., qux=...): if bar is Ellipsis: # foo ;) Georg From rnd at onego.ru Tue Jan 30 21:33:31 2007 From: rnd at onego.ru (Roman Susi) Date: Tue, 30 Jan 2007 22:33:31 +0200 Subject: [Python-ideas] Mutable default arguments - another approach In-Reply-To: <91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com> References: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com> <91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com> Message-ID: <45BFAB9B.4090904@onego.ru> George Sakkis wrote: > On 1/30/07, Piotr Duda wrote: > > >>I think that this problem can be solved by the following change of >>default argument behavior: >>default arguments are evaluated in definition time (like in current >>implementation), but right after being evaluated, result object is >>checked if it's mutable (for example by checking of presence __copy__ >>special method or being instance of built in (sub)class >>list/dict/set) >> >>(snipped) >> > > > AFAIK there's no reliable way of deciding whether an arbitrary object > is mutable or not, so the rest of the post is irrelevant. Besides, > mutable default arguments is just a specific use case; the general > problem discussed here is call-time vs definition-time semantics for > arbitrary default argument expressions (function calls, attribute > lookups, etc.). > > George Maybe Python interpreter could be better at deciding if the programmer which wrote the code is novice or experienced. For example, from __present__ import _I_AM_PYTHON_EXPERT at the beginning of each module will do ;-) Or, alternatively, Python interpreter will read user's directory for some obscure configuration file where programmer will put True's into originally: I_KNOW_WHAT_MUTABLE_MEANS = False I_AM_AWARE_OF_MUTABLE_DEFAULT_ARGUMENT = False ... (And of course, there will be secret I_AM_GUIDO_VAN_ROSSUM = True which will override all the previous settings.) AFAIK, no language has this feature yet (not sure about Perl). Regards, Roman From rrr at ronadam.com Tue Jan 30 21:38:53 2007 From: rrr at ronadam.com (Ron Adam) Date: Tue, 30 Jan 2007 14:38:53 -0600 Subject: [Python-ideas] Import and '..', '../..' in serach path. In-Reply-To: References: <45BF2D86.2080801@ronadam.com> Message-ID: <45BFACDD.5040602@ronadam.com> Brett Cannon wrote: > On 1/30/07, Ron Adam wrote: >> >> In order to resolve a path conflict where I'm working on several >> copies of the >> same package. I found it useful to add the following near the top of >> modules in >> a package or sub package. >> >> >> Module in package: >> >> import sys >> sys.path = ['..'] + sys.path >> >> import package.module # Imports module in "this!" package. >> >> >> Note: There could still be conflicts if a module with the same name is >> in the >> same directory as the package. But that's much less likely than one >> in the rest >> of the path. >> >> >> Module in sub-package: >> >> import sys >> sys.path = ['../..'] + sys.path >> >> import package.subpackage.module # finds "self" (subpackage) >> reliably. >> >> >> By explicitly adding the packages parent directory to the *front* of >> sys.path it >> resolves cases where imports using absolute imports, import modules >> from another >> package because they are found first in the search path. > > Why aren't you using relative imports (e.g., ``from . import > module``)? That should be doing exactly what you want. That uses > __path__ which is set to the path of the package. > > -Brett But if you try to run a module in a package, vs from a package, with relative from . imports, you will get: ValueError: Attempted relative import in non-package And there is no __path__ attribute to look at since no package has been imported yet. So you need to import the package first, and then you may not import the correct package even then if there is another package or module with the same name. For example if you are running a test module from test sub-package that has a "if __name__=='__main__': _test()" at the bottom. Something that is not uncommon. It may attempt to import modules from a different package. In my case it was a package with the same name in a different SVN branch. The tests actually ran, but with errors because it wasn't exactly the same. It wasn't obvious what was going on either. Cheers, Ron From gsakkis at rutgers.edu Tue Jan 30 23:11:18 2007 From: gsakkis at rutgers.edu (George Sakkis) Date: Tue, 30 Jan 2007 17:11:18 -0500 Subject: [Python-ideas] Mutable default arguments - another approach In-Reply-To: <45BFAB9B.4090904@onego.ru> References: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com> <91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com> <45BFAB9B.4090904@onego.ru> Message-ID: <91ad5bf80701301411l72943aediad759282697630c8@mail.gmail.com> On 1/30/07, Roman Susi wrote: > George Sakkis wrote: > > On 1/30/07, Piotr Duda wrote: > > > > > >>I think that this problem can be solved by the following change of > >>default argument behavior: > >>default arguments are evaluated in definition time (like in current > >>implementation), but right after being evaluated, result object is > >>checked if it's mutable (for example by checking of presence __copy__ > >>special method or being instance of built in (sub)class > >>list/dict/set) > >> > >>(snipped) > >> > > > > > > AFAIK there's no reliable way of deciding whether an arbitrary object > > is mutable or not, so the rest of the post is irrelevant. Besides, > > mutable default arguments is just a specific use case; the general > > problem discussed here is call-time vs definition-time semantics for > > arbitrary default argument expressions (function calls, attribute > > lookups, etc.). > > > > George > > Maybe Python interpreter could be better at deciding if the programmer > which wrote the code is novice or experienced. > > For example, > > from __present__ import _I_AM_PYTHON_EXPERT > > at the beginning of each module will do ;-) That would be a good start, but certainly not the end; how about programmers who *read* other people's code ? Their experience level may well be quite different, and the code should be understandable by these poor souls too. The python source files would have to be able to sense when they're being read or printed, determine the experience level of the reader and transform themselves to accomodate his level. Anyone wanna write the PEP for Python-7000 ? ;-) George From brett at python.org Tue Jan 30 23:21:49 2007 From: brett at python.org (Brett Cannon) Date: Tue, 30 Jan 2007 14:21:49 -0800 Subject: [Python-ideas] Import and '..', '../..' in serach path. In-Reply-To: <45BFACDD.5040602@ronadam.com> References: <45BF2D86.2080801@ronadam.com> <45BFACDD.5040602@ronadam.com> Message-ID: On 1/30/07, Ron Adam wrote: > Brett Cannon wrote: > > On 1/30/07, Ron Adam wrote: > >> > >> In order to resolve a path conflict where I'm working on several > >> copies of the > >> same package. I found it useful to add the following near the top of > >> modules in > >> a package or sub package. > >> > >> > >> Module in package: > >> > >> import sys > >> sys.path = ['..'] + sys.path > >> > >> import package.module # Imports module in "this!" package. > >> > >> > >> Note: There could still be conflicts if a module with the same name is > >> in the > >> same directory as the package. But that's much less likely than one > >> in the rest > >> of the path. > >> > >> > >> Module in sub-package: > >> > >> import sys > >> sys.path = ['../..'] + sys.path > >> > >> import package.subpackage.module # finds "self" (subpackage) > >> reliably. > >> > >> > >> By explicitly adding the packages parent directory to the *front* of > >> sys.path it > >> resolves cases where imports using absolute imports, import modules > >> from another > >> package because they are found first in the search path. > > > > Why aren't you using relative imports (e.g., ``from . import > > module``)? That should be doing exactly what you want. That uses > > __path__ which is set to the path of the package. > > > > -Brett > > > But if you try to run a module in a package, vs from a package, with relative > from . imports, you will get: > > ValueError: Attempted relative import in non-package > > And there is no __path__ attribute to look at since no package has been imported > yet. Yeah, I have been bitten by that. You need to have that initial package import. I got nailed by this when using '-m' and a module contained within a package for executing tests. > > So you need to import the package first, and then you may not import the correct > package even then if there is another package or module with the same name. > Well, just don't do that. =) You really shouldn't have multiple things with the same name in sys.path. But as you noticed, you can control it with sys.path if you really need to. I just don't see this as a common enough problem to warrant documenting it somewhere (and I have no clue where that "somewhere" would be). -Brett From jcarlson at uci.edu Tue Jan 30 23:31:36 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 30 Jan 2007 14:31:36 -0800 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BFA48D.2020501@onego.ru> References: <45BED309.10301@gmail.com> <45BFA48D.2020501@onego.ru> Message-ID: <20070130142937.5A38.JCARLSON@uci.edu> Roman Susi wrote: > def foo(x, y, z, bar=Missing, qux=Missing): > if baz is Missing: > baz = [] > #code With the proper definition of Missing, the above is fine, and is more or less creating a new 'None' value. > or even: > > def foo(x, y, z, bar=, qux=): > if baz is Missing: > baz = [] > #code > > at least, it doesn't require decorators, is backward compatible > (hopefully no grammar conflicts in there), reads as English. The above with a missing value for a default *is not* backwards compatible with previous Pythons. New syntax is, by definition, not backwards compatible. - Josiah From cvrebert at gmail.com Wed Jan 31 00:08:53 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Tue, 30 Jan 2007 15:08:53 -0800 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: <45BF9C9E.8010206@onego.ru> References: <20070126212657.5A06.JCARLSON@uci.edu> <45BF9C9E.8010206@onego.ru> Message-ID: <47c890dc0701301508yeaab646t85efb2cf9f69e4df@mail.gmail.com> On 1/30/07, Roman Susi wrote: > The proposal (as it seems to me) wants to make change to the language > inconsistent with nature dynamic semantics. Its like put implicit: > > if RUN_FIRST_TIME: > do this > else: > do that > > for the same line of code (def statement's first line). No, my proto-PEP has never contained semantics like that anywhere in it. > > Well, it's good to be clear on where the disagreements lie. However I'm > > not yet ready to let it rest at that without some more arguments. > > > > As Chris pointed out in his first mail, this 'wart' is mentioned on > > several lists of python misfeatures: [0][1][2]. I'd like to add to this > > that even the python documentation finds this issue severe enough to > > issue an "Important warning"[4]. > > > > It seems clear that this behaviour is a gotcha, at least for newbies. > > This could be excused if there is a good reason to spend the additional > > time learning this behaviour, but some of the links state, and my > > But as somebody already said the alternative is even worse... > Its quite easier to mention 3-5 Python warts up front to newbies than to > introduce subtle exception for semantics and noise words such as "new" > which do not have any other use elsewhere. Sidenote: There could instead be a new keyword 'once' to indicate the old semantics. I think the PEP related to adding a switch statement proposes the same keyword for a similar use. - Chris Rebert From jan.kanis at phil.uu.nl Wed Jan 31 01:28:31 2007 From: jan.kanis at phil.uu.nl (Jan Kanis) Date: Wed, 31 Jan 2007 01:28:31 +0100 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <20070130142937.5A38.JCARLSON@uci.edu> References: <45BED309.10301@gmail.com> <45BFA48D.2020501@onego.ru> <20070130142937.5A38.JCARLSON@uci.edu> Message-ID: On Tue, 30 Jan 2007 23:31:36 +0100, Josiah Carlson wrote: > Roman Susi wrote: >> >> def foo(x, y, z, bar=, qux=): >> if baz is Missing: >> baz = [] >> #code >> >> at least, it doesn't require decorators, is backward compatible >> (hopefully no grammar conflicts in there), reads as English. > > The above with a missing value for a default *is not* backwards > compatible with previous Pythons. New syntax is, by definition, not > backwards compatible. > > - Josiah As a matter of fact, backward-compatible syntax changes are certainly possible. (ever wondered how C++ got it's syntax?) Any valid current python is still going to behave exactly the same if this syntax were to be accepted. Talking about backward compatibility, I think it is safe to ignore any text files that don't get accepted by the python interpreter. This syntax change would certainly not break any existing production python code. (note: the above statements do not entail in any way that I am in favour of this syntax change) - Jan From jan.kanis at phil.uu.nl Wed Jan 31 01:33:47 2007 From: jan.kanis at phil.uu.nl (Jan Kanis) Date: Wed, 31 Jan 2007 01:33:47 +0100 Subject: [Python-ideas] Mutable default arguments - another approach In-Reply-To: <91ad5bf80701301411l72943aediad759282697630c8@mail.gmail.com> References: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com> <91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com> <45BFAB9B.4090904@onego.ru> <91ad5bf80701301411l72943aediad759282697630c8@mail.gmail.com> Message-ID: Nah, this is all old school. If this pep is gonna succeed it needs some buzzwords like Web 2.0: The interpreter should check with google what kinds of programming-related search terms the programmer has used and for how long. Using google desktop search, it can also include the kind of documents the programmer has and when he's read them. With GvR working where he is, I don't think implementing this is going to be a big problem. ;-) On Tue, 30 Jan 2007 23:11:18 +0100, George Sakkis wrote: > On 1/30/07, Roman Susi wrote: > >> Maybe Python interpreter could be better at deciding if the programmer >> which wrote the code is novice or experienced. >> >> For example, >> >> from __present__ import _I_AM_PYTHON_EXPERT >> >> at the beginning of each module will do ;-) > > That would be a good start, but certainly not the end; how about > programmers who *read* other people's code ? Their experience level > may well be quite different, and the code should be understandable by > these poor souls too. The python source files would have to be able to > sense when they're being read or printed, determine the experience > level of the reader and transform themselves to accomodate his level. > Anyone wanna write the PEP for Python-7000 ? ;-) > > George > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From jan.kanis at phil.uu.nl Wed Jan 31 02:16:37 2007 From: jan.kanis at phil.uu.nl (Jan Kanis) Date: Wed, 31 Jan 2007 02:16:37 +0100 Subject: [Python-ideas] fixing mutable default argument values In-Reply-To: <45BF9C9E.8010206@onego.ru> References: <20070126212657.5A06.JCARLSON@uci.edu> <45BF9C9E.8010206@onego.ru> Message-ID: On Tue, 30 Jan 2007 20:29:34 +0100, Roman Susi wrote: > Jan Kanis wrote: >> On Mon, 29 Jan 2007 08:38:39 +0100, Roman Susi wrote: >> >>> This is what >>> incremental dynamic semantics is about. So, the suggestion is good only >>> as separated feature, but is IMHO wrong >>> if considered in the language design as a whole. >> >> >> wtf is incremental dynamic semantics, in this context? I did some >> googling but all I found referred to techniques related to programming >> environments. This proposal is just about a change in the language. > > Oh, I thought this is very fundamental and well-known as it is the > second line of Guido's definition of Python: > > "Python is an interpreted, object-oriented, high-level programming > language with dynamic semantics. ..." > > As I understand it, it means that statements are read sequencially and > the semantics of names depends on the flow control. I never made a link with this description of python and the term 'incremental dynamic semantics'. Guess I learned something new today. > The proposal (as it seems to me) wants to make change to the language > inconsistent with nature dynamic semantics. Its like put implicit: > > if RUN_FIRST_TIME: > do this > else: > do that > > for the same line of code (def statement's first line). > > >> [snip] > I'm certainly no proponent of doing something like this. Do note that the discussion is between evaluating default expressions at definition time or at call time, and the pep entails evaluating default expressions on /every/ function call, just like the body of the function is evaluated on every call. There's already other code being evaluated at both def time and call time, so the proposal does not introduce anything new to pythons evaluation model. IMO having the default exprs evaluate at call time is in no way against pythons language design. > >> Well, it's good to be clear on where the disagreements lie. However I'm >> not yet ready to let it rest at that without some more arguments. >> >> As Chris pointed out in his first mail, this 'wart' is mentioned on >> several lists of python misfeatures: [0][1][2]. I'd like to add to this >> that even the python documentation finds this issue severe enough to >> issue an "Important warning"[4]. >> >> It seems clear that this behaviour is a gotcha, at least for newbies. >> This could be excused if there is a good reason to spend the additional >> time learning this behaviour, but some of the links state, and my > > But as somebody already said the alternative is even worse... > Its quite easier to mention 3-5 Python warts up front to newbies than to > introduce subtle exception for semantics and noise words such as "new" > which do not have any other use elsewhere. But it's much better to just eliminate the warts without introducing subtle exceptions. The pep proposes evaluating default exprs on every function call, just like the function body is. No new exceptions are introduced. The fact that newbies often expect default values to be fresh on every call seems to entail that they won't be surprised a lot when the idiom of using default values as caches won't work, if the pep gets accepted. Old time pythoneers who know what's new in 3.0 won't be surprised either. I'm not in favour of introducing new noise words or other new syntax, I just want to 'fix' pythons current semantics. >> assumption is, that there are very few situations where re-evaluating >> causes a problem and which isn't easily fixable. >> The semantics which I'd like to have are even easier than the current >> semantics: everything in a function, be it before or after the colon, >> is executed when the function is called. >> Of course, as Collin Winters pointed out, the burden of proof of >> showing that these semantics aren't going to be a problem is still on >> the pep proponents. >> > >> So, are there any _other_ arguments in favour of the current semantics?? > > Repeat the mantra: > > Python's dynamic semantics will not change > Python's dynamic semantics will not change > Python's dynamic semantics will not change > > ;-) If anything, the proposal is going to _improve_ pythons dynamicness and late-bindingness. Oh, and it's nice to know repeating mantras now count as 'arguments' - Jan From jcarlson at uci.edu Wed Jan 31 02:35:08 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 30 Jan 2007 17:35:08 -0800 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: References: <20070130142937.5A38.JCARLSON@uci.edu> Message-ID: <20070130173337.5A4D.JCARLSON@uci.edu> "Jan Kanis" wrote: > > On Tue, 30 Jan 2007 23:31:36 +0100, Josiah Carlson > wrote: > > Roman Susi wrote: > >> > >> def foo(x, y, z, bar=, qux=): > >> if baz is Missing: > >> baz = [] > >> #code > >> > >> at least, it doesn't require decorators, is backward compatible > >> (hopefully no grammar conflicts in there), reads as English. > > > > The above with a missing value for a default *is not* backwards > > compatible with previous Pythons. New syntax is, by definition, not > > backwards compatible. > > > > - Josiah > > As a matter of fact, backward-compatible syntax changes are certainly > possible. (ever wondered how C++ got it's syntax?) Any valid current > python is still going to behave exactly the same if this syntax were to be > accepted. Talking about backward compatibility, I think it is safe to > ignore any text files that don't get accepted by the python interpreter. > This syntax change would certainly not break any existing production > python code. > (note: the above statements do not entail in any way that I am in favour > of this syntax change) Fowards compatible then. That is to say, writing for the new proposed system will break compatibility with older Pythons. On the other hand, using currently-available language syntax and semantics is compatible among the entire range of Pythons available today. - Josiah From jan.kanis at phil.uu.nl Wed Jan 31 02:31:06 2007 From: jan.kanis at phil.uu.nl (Jan Kanis) Date: Wed, 31 Jan 2007 02:31:06 +0100 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <20070129233754.5A32.JCARLSON@uci.edu> References: <20070129212157.5A2B.JCARLSON@uci.edu> <45BEF306.5030801@gmail.com> <20070129233754.5A32.JCARLSON@uci.edu> Message-ID: On Tue, 30 Jan 2007 18:16:03 +0100, Josiah Carlson wrote: > > Chris Rebert wrote: >> >> > Because in the case where I would want to receive a mutable parameter, >> > like in the case below that wouldn't work with Python's standard >> > semantics... >> > >> > def append_10(lst=[]): >> > lst.append(10) >> > return lst >> > >> > I would presumably change it to... >> > >> > def append_10(lst=None): >> > if lst is None: lst = [] >> > lst.append(10) >> > return lst >> > >> > Or some variant thereof. Now, here's the thing; if I want a mutable >> > argument, then None is a nonsensical value to pass, generally, as it >> is >> > not mutable. So I don't buy the whole "but then None would no longer >> be a >> > valid argument to pass" bull that was offered as a reason why the >> above >> > isn't a reasonable translation (I can't remember who offered it). >> >> Nowhere has it been proposed to forbid passing None as an argument. > > And I never claimed as much. There was a previous post by someone (I > can't remember who offered it), saying that replacing 'lst=[]' with > 'lst=None' would make it so that a user couldn't signal *something > special* with a 'None' argument. I was trying to get across that such > reasoning doesn't make sense in this particular context. The above example code was mine. I was just trying to explain that the proposed semantics should not mistake a caller-provided argument value of None for a no-argument-provided situation and evaluate the default expression. If the proposal were to be implemented naively by literally rewriting the python code in the way that I used to explain the proposal and compiling the rewritten code, this bug could happen. - Jan From cvrebert at gmail.com Wed Jan 31 02:54:13 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Tue, 30 Jan 2007 17:54:13 -0800 Subject: [Python-ideas] Mutable default arguments - another approach In-Reply-To: <3df8f2650701301018i22fff8deq86829a292dceb8b2@mail.gmail.com> References: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com> <91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com> <3df8f2650701301018i22fff8deq86829a292dceb8b2@mail.gmail.com> Message-ID: <45BFF6C5.8020909@gmail.com> Piotr Duda wrote: > Read my previous post again, and give me examples of what you cannot > do regarding default arguments. First, I'd like to register my dislike for implicitly determining when a default argument will be copied on every call. Second, as per your request: def foo(bar=None): if bar is None: bar = function_call() #code def foo(bar=None): if bar is None: bar = something.method_call() #code #These were just a few. There are others. These cannot be rewritten using simple copying decorators. Various decorators have been presented that account for some of these (and other) situations. However, IMHO they're not too elegant/clean. Thus, a slightly more sophisticated solution is required. - Chris Rebert From cvrebert at gmail.com Wed Jan 31 02:59:05 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Tue, 30 Jan 2007 17:59:05 -0800 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <43aa6ff70701292152q158092d3xf1b37b4ff4ff7438@mail.gmail.com> References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru> <91ad5bf80701292016m63bf1b28w84e2c9f62fb7ea0e@mail.gmail.com> <45BEDA53.9010103@gmail.com> <43aa6ff70701292152q158092d3xf1b37b4ff4ff7438@mail.gmail.com> Message-ID: <45BFF7E9.1080305@gmail.com> Collin Winter wrote: >> On 1/29/07, Chris Rebert wrote: >> Why? Yes, there _might_ be performance issues (which have yet to be >> demonstrated or deeply speculated upon), but re-evaluating immutable >> default arguments wouldn't affect a program's correct operation. > > If the underlying intent of your proposal -- that all default > arguments be re-evaluated with every call -- were to be approved, > there would undoubtedly be a serious performance impact. [snip] > As for whether these effects have "yet to be demonstrated", as you > say, the burden is on you, the PEP author, to investigate and resolve, > mitigate or justify any and all performance changes. > > Collin Winter > You do have a point there, it is my responsibility. I therefore shall endeavor to find out how much of a performance hit my suggested change would have. - Chris Rebert From cvrebert at gmail.com Wed Jan 31 03:05:13 2007 From: cvrebert at gmail.com (Chris Rebert) Date: Tue, 30 Jan 2007 18:05:13 -0800 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BFA48D.2020501@onego.ru> References: <45BCF804.7040207@gmail.com> <45BED309.10301@gmail.com> <45BFA48D.2020501@onego.ru> Message-ID: <45BFF959.4030906@gmail.com> Roman Susi wrote: > Chris Rebert wrote: >> Jim Jewett wrote: >> >>> On 1/28/07, Chris Rebert wrote: > > /skip/ > >>>> def foo(bar=new baz): >>>> #code >>> This would be less bad. >>> >>> That said, I fear many new programmers would fail to understand when >>> they needed new and when they didn't, so that in practice, it would be >>> just optional random noise. >> >> This is part of the reason I'm trying to avoid adding new syntax. >> However, I assert that at least 'new' is clearer than the' x=None; if x > > But if you are concerned with the current None, there could be some > other, new False value serving the same need, like: > > def foo(x, y, z, bar=Missing, qux=Missing): > if baz is Missing: > baz = [] > #code > > or even: > > def foo(x, y, z, bar=, qux=): > if baz is Missing: > baz = [] > #code > > at least, it doesn't require decorators, is backward compatible > (hopefully no grammar conflicts in there), reads as English. Those examples are only slightly better than using None. The convention of using None to indicate use of a mutable/non-constant default value is about as clear as your examples. The point wasn't that None wasn't descriptive/specific enough, but rather why have to write the 'bar=None' and then the 'if bar is None' when 'bar=[]' could be sufficient? Also, this indicates exactly what the default value is, as opposed to None/Missing, which is just an opaque indicator for 'some mutable/non-constant default value'. Thus, I see no need for 'Missing' or a similar constant. - Chris Rebert From rrr at ronadam.com Wed Jan 31 05:17:34 2007 From: rrr at ronadam.com (Ron Adam) Date: Tue, 30 Jan 2007 22:17:34 -0600 Subject: [Python-ideas] Import and '..', '../..' in serach path. In-Reply-To: References: <45BF2D86.2080801@ronadam.com> <45BFACDD.5040602@ronadam.com> Message-ID: <45C0185E.2090408@ronadam.com> Brett Cannon wrote: > On 1/30/07, Ron Adam wrote: >> Brett Cannon wrote: >> > On 1/30/07, Ron Adam wrote: >> >> >> >> In order to resolve a path conflict where I'm working on several >> >> copies of the >> >> same package. I found it useful to add the following near the top of >> >> modules in >> >> a package or sub package. >> >> >> >> >> >> Module in package: >> >> >> >> import sys >> >> sys.path = ['..'] + sys.path >> >> >> >> import package.module # Imports module in "this!" >> package. >> >> >> >> >> >> Note: There could still be conflicts if a module with the same name is >> >> in the >> >> same directory as the package. But that's much less likely than one >> >> in the rest >> >> of the path. >> >> >> >> >> >> Module in sub-package: >> >> >> >> import sys >> >> sys.path = ['../..'] + sys.path >> >> >> >> import package.subpackage.module # finds "self" (subpackage) >> >> reliably. >> >> >> >> >> >> By explicitly adding the packages parent directory to the *front* of >> >> sys.path it >> >> resolves cases where imports using absolute imports, import modules >> >> from another >> >> package because they are found first in the search path. >> > >> > Why aren't you using relative imports (e.g., ``from . import >> > module``)? That should be doing exactly what you want. That uses >> > __path__ which is set to the path of the package. >> > >> > -Brett >> >> >> But if you try to run a module in a package, vs from a package, with >> relative >> from . imports, you will get: >> >> ValueError: Attempted relative import in non-package >> >> And there is no __path__ attribute to look at since no package has >> been imported >> yet. > > Yeah, I have been bitten by that. You need to have that initial > package import. I got nailed by this when using '-m' and a module > contained within a package for executing tests. And you can't set __path__ manually, that only changes the Exception to an Import Error. >> So you need to import the package first, and then you may not import >> the correct >> package even then if there is another package or module with the same >> name. >> > > Well, just don't do that. =) You really shouldn't have multiple > things with the same name in sys.path. But as you noticed, you can > control it with sys.path if you really need to. > > I just don't see this as a common enough problem to warrant > documenting it somewhere (and I have no clue where that "somewhere" > would be). I expected that someone would say that. How about adding a note at the bottom of PEP 328. That was one of the places I looked for a way to resolve this. Or possibly posting a message on python-ideas. Consider it done ;-) A note on PEP 328 might be a good idea still. I have no idea what the best wording would be. The import tutorial is another place where a note clarifying relative imports can't be used for stand alone scripts is needed. The nice thing about adjusting sys.path directly is you can then move back and forth from different same named versions in development without doing renames. If you do a rename, you also have to remember to rename all the submodule absolute references in the package as well. Another alternative is to hide the ones you don't want by renaming them. Then you have to do renames to switch back to them. That makes it more difficult to do side by side comparisons as well. Altering PYTHONPATH is another way to do it, but you also need to switch it back if you switch versions. But if you forget to edit it back, and use a version not on the path, it will import modules from the version on the path. In this case the package I'm working on may replace pydoc.py module in pythons library. (no promises though) The name matters so that site.py can locate the help function. It's convenient for me to be able to easily and dependably compare different working versions. A few things to consider... ----- According to PEP 328: You may use relative imports freely. In Python 2.6, any import statement that results in an intra-package import will raise DeprecationWarning (this also applies to from <> import that fails to use the relative import syntax). In Python 2.7, import will always be an absolute import (and the __future__ directive will no longer be needed). ----- So It will probably come up more often as more people start using absolute imports and the '.' style relative imports. Or as you put it, more people get "bitten". Take a look at how much __name__ == '__main__' is used in the standard library. Those modules won't be able to use relative imports. They will need to be updated and can use only absolute imports. Some people will probably want a way to have relative imports work with modules meant to be run as scripts (in packages). An optional function (or property) may be a good way to resolve both of these package self reference situations without any changes to the default behavior. Cheers, Ron From rnd at onego.ru Wed Jan 31 09:04:33 2007 From: rnd at onego.ru (Roman Susi) Date: Wed, 31 Jan 2007 10:04:33 +0200 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <20070130173337.5A4D.JCARLSON@uci.edu> References: <20070130142937.5A38.JCARLSON@uci.edu> <20070130173337.5A4D.JCARLSON@uci.edu> Message-ID: <45C04D91.8030906@onego.ru> Josiah Carlson wrote: > "Jan Kanis" wrote: > >>On Tue, 30 Jan 2007 23:31:36 +0100, Josiah Carlson >>wrote: >> >>>Roman Susi wrote: >>> >>>> def foo(x, y, z, bar=, qux=): >>>> if baz is Missing: >>>> baz = [] >>>> #code >>>> >>>>at least, it doesn't require decorators, is backward compatible >>>>(hopefully no grammar conflicts in there), reads as English. >>> >>>The above with a missing value for a default *is not* backwards >>>compatible with previous Pythons. New syntax is, by definition, not >>>backwards compatible. >>> >>> - Josiah >> >>As a matter of fact, backward-compatible syntax changes are certainly >>possible. (ever wondered how C++ got it's syntax?) Any valid current >>python is still going to behave exactly the same if this syntax were to be >>accepted. Talking about backward compatibility, I think it is safe to >>ignore any text files that don't get accepted by the python interpreter. >>This syntax change would certainly not break any existing production >>python code. >>(note: the above statements do not entail in any way that I am in favour >>of this syntax change) > > > Fowards compatible then. That is to say, writing for the new proposed > system will break compatibility with older Pythons. On the other hand, > using currently-available language syntax and semantics is compatible > among the entire range of Pythons available today. >From wikipedia: "... a product is said to be backward compatible (or downward compatible) when it is able to take the place of an older product, by interoperating with other products that were designed for the older product." "Forward compatibility (sometimes confused with extensibility) is the ability of a system to accept input intended for later versions of itself." So, the right term is that Python 3000 will be in the syntactic aspect we discuss backward compatible because it will accept old syntax too. Regards, Roman > - Josiah > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > > !DSPAM:45bff1845411635716952! > From rnd at onego.ru Wed Jan 31 09:24:04 2007 From: rnd at onego.ru (Roman Susi) Date: Wed, 31 Jan 2007 10:24:04 +0200 Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments In-Reply-To: <45BFF959.4030906@gmail.com> References: <45BCF804.7040207@gmail.com> <45BED309.10301@gmail.com> <45BFA48D.2020501@onego.ru> <45BFF959.4030906@gmail.com> Message-ID: <45C05224.6090407@onego.ru> Chris Rebert wrote: > Roman Susi wrote: > >> Chris Rebert wrote: >> >>> Jim Jewett wrote: >>> >>>> On 1/28/07, Chris Rebert wrote: >> >> >> /skip/ >> >>>>> def foo(bar=new baz): >>>>> #code >>>> >>>> This would be less bad. >>>> >>>> That said, I fear many new programmers would fail to understand when >>>> they needed new and when they didn't, so that in practice, it would be >>>> just optional random noise. >>> >>> >>> This is part of the reason I'm trying to avoid adding new syntax. >>> However, I assert that at least 'new' is clearer than the' x=None; if x >> >> >> But if you are concerned with the current None, there could be some >> other, new False value serving the same need, like: >> >> def foo(x, y, z, bar=Missing, qux=Missing): >> if baz is Missing: >> baz = [] >> #code >> >> or even: >> >> def foo(x, y, z, bar=, qux=): >> if baz is Missing: >> baz = [] >> #code >> >> at least, it doesn't require decorators, is backward compatible >> (hopefully no grammar conflicts in there), reads as English. > > > Those examples are only slightly better than using None. The convention > of using None to indicate use of a mutable/non-constant default value is > about as clear as your examples. > > The point wasn't that None wasn't descriptive/specific enough, but > rather why have to write the 'bar=None' and then the 'if bar is None' > when 'bar=[]' could be sufficient? Also, this indicates exactly what the > default value is, as opposed to None/Missing, which is just an opaque > indicator for 'some mutable/non-constant default value'. > > Thus, I see no need for 'Missing' or a similar constant. And I think it is overkill to add new syntax there. For example, if the function or method is near the external (human or network) interface border it is good to check ALL arguments anyway. So the extra if baz is Missing: baz = [] doesn't spoil the impression: def foo(x, y, z, baz=, qux=): ... if baz is Missing: baz = [] ... if type(baz) != type([]) or len(baz) > 1000 or min(baz) < 0 or ...: raise ... #code What I'd wanted to say is that =None or =Missing is not just "indicator for 'some mutable/non-constant default value'." but may also mean that the argument is subject to further constraints and thus funtion code has checks for that. So, instead of having clear indicator of missing argument we have some fresh instance of a list or dict. Also, in many situation another approach is used nowadays: def foo(x, y, z, **args): ... baz = args.get("baz", []) qux = args.get("qux", {}) Which has the effect proto-PEP author would like to achieve. Of course, no automatic tools are going to make type checks with that helpful function signature. Also, while its offtopic in this post, I'd really liked Python 3000 to have more unification for dict vs attribute access because there are use cases (Storage in web.py, Namespace in mxTools - I am sure many other projects have similar things) which single purpose is to have both dict and attribute syntax in one data object interchangeably: def foo(**args): ... baz = args.baz qux = args["qux"] Regards, Roman > > - Chris Rebert From duda.piotr at gmail.com Wed Jan 31 14:19:06 2007 From: duda.piotr at gmail.com (Piotr Duda) Date: Wed, 31 Jan 2007 14:19:06 +0100 Subject: [Python-ideas] Mutable default arguments - another approach In-Reply-To: <45BFF6C5.8020909@gmail.com> References: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com> <91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com> <3df8f2650701301018i22fff8deq86829a292dceb8b2@mail.gmail.com> <45BFF6C5.8020909@gmail.com> Message-ID: <3df8f2650701310519g28f09105t6e367cdd5a59981a@mail.gmail.com> 2007/1/31, Chris Rebert : > Piotr Duda wrote: > > Read my previous post again, and give me examples of what you cannot > > do regarding default arguments. > > First, I'd like to register my dislike for implicitly determining when a > default argument will be copied on every call. In my solution you can always declare this explicitly. Only problem here is to determine which objects are copied, simplest rule here is to copy all object except instances of built-in immutable types. > > Second, as per your request: > > def foo(bar=None): > if bar is None: bar = function_call() > #code > > def foo(bar=None): > if bar is None: bar = something.method_call() > #code In my solution you can do both. Assuming there are all classes/methods defined in my first post, use: @DefaultArgWrapper def calldefarg(callable): #this function should be defined in stdlib while 1: yield callable() first example def foo(bar = calldefarg(function_call)): #code here i assume that function_call is available at definition call, if it isn't then use this def foo(bar = calldefarg(lambda: function_call())): #code and the second example def foo(bar = calldefarg(something.method_call)): #code or if something isn't available at definition time def foo(bar = calldefarg(lambda: something.method_call())): #code there's one restriction, that 'something' cannot depend on other function argument(s). > #These were just a few. There are others. > > These cannot be rewritten using simple copying decorators. Various > decorators have been presented that account for some of these (and > other) situations. However, IMHO they're not too elegant/clean. Thus, a > slightly more sophisticated solution is required. Are you referring here to my solution? -- ???????? ?????? From brett at python.org Wed Jan 31 20:48:53 2007 From: brett at python.org (Brett Cannon) Date: Wed, 31 Jan 2007 11:48:53 -0800 Subject: [Python-ideas] Import and '..', '../..' in serach path. In-Reply-To: <45C0185E.2090408@ronadam.com> References: <45BF2D86.2080801@ronadam.com> <45BFACDD.5040602@ronadam.com> <45C0185E.2090408@ronadam.com> Message-ID: On 1/30/07, Ron Adam wrote: > Brett Cannon wrote: [SNIP] > > I just don't see this as a common enough problem to warrant > > documenting it somewhere (and I have no clue where that "somewhere" > > would be). > > > I expected that someone would say that. > > How about adding a note at the bottom of PEP 328. That was one of the places I > looked for a way to resolve this. Or possibly posting a message on > python-ideas. Consider it done ;-) > > A note on PEP 328 might be a good idea still. I have no idea what the best > wording would be. The import tutorial is another place where a note clarifying > relative imports can't be used for stand alone scripts is needed. > Sure, but that is not my PEP. That is something to bug Thomas Wouters about. > > The nice thing about adjusting sys.path directly is you can then move back and > forth from different same named versions in development without doing renames. > If you do a rename, you also have to remember to rename all the submodule > absolute references in the package as well. Another alternative is to hide the > ones you don't want by renaming them. Then you have to do renames to switch > back to them. That makes it more difficult to do side by side comparisons as well. > Right but the point of relative imports is to help prevent renaming. It's a catch-22 situation of what matters more to you. > Altering PYTHONPATH is another way to do it, but you also need to switch it back > if you switch versions. But if you forget to edit it back, and use a version > not on the path, it will import modules from the version on the path. > > In this case the package I'm working on may replace pydoc.py module in pythons > library. (no promises though) The name matters so that site.py can locate the > help function. It's convenient for me to be able to easily and dependably > compare different working versions. > > > A few things to consider... > > ----- > According to PEP 328: > > You may use relative imports freely. In Python 2.6, any import statement that > results in an intra-package import will raise DeprecationWarning (this also > applies to from <> import that fails to use the relative import syntax). In > Python 2.7, import will always be an absolute import (and the __future__ > directive will no longer be needed). > ----- > > So It will probably come up more often as more people start using absolute > imports and the '.' style relative imports. Or as you put it, more people get > "bitten". > Probably. > Take a look at how much __name__ == '__main__' is used in the standard library. > Those modules won't be able to use relative imports. They will need to be > updated and can use only absolute imports. > It's possible. Maybe this will finally push forward the idea of having a proper 'main' function or something. > Some people will probably want a way to have relative imports work with modules > meant to be run as scripts (in packages). > > An optional function (or property) may be a good way to resolve both of these > package self reference situations without any changes to the default behavior. > Yeah, some built-in or something might do it that lets you decorate what function is to be the 'main' function and maybe do some magic to make it resolve properly. -Brett From rrr at ronadam.com Wed Jan 31 23:45:45 2007 From: rrr at ronadam.com (Ron Adam) Date: Wed, 31 Jan 2007 16:45:45 -0600 Subject: [Python-ideas] Import and '..', '../..' in serach path. In-Reply-To: References: <45BF2D86.2080801@ronadam.com> <45BFACDD.5040602@ronadam.com> <45C0185E.2090408@ronadam.com> Message-ID: <45C11C19.5030703@ronadam.com> Brett Cannon wrote: > On 1/30/07, Ron Adam wrote: >> Brett Cannon wrote: > [SNIP] >> > I just don't see this as a common enough problem to warrant >> > documenting it somewhere (and I have no clue where that "somewhere" >> > would be). >> >> >> I expected that someone would say that. >> >> How about adding a note at the bottom of PEP 328. That was one of the >> places I >> looked for a way to resolve this. Or possibly posting a message on >> python-ideas. Consider it done ;-) >> >> A note on PEP 328 might be a good idea still. I have no idea what the >> best >> wording would be. The import tutorial is another place where a note >> clarifying >> relative imports can't be used for stand alone scripts is needed. >> > > Sure, but that is not my PEP. That is something to bug Thomas Wouters > about. Ok, Maybe later. I looked into it further and I think it needs more thought in any case. >> The nice thing about adjusting sys.path directly is you can then move >> back and >> forth from different same named versions in development without doing >> renames. >> If you do a rename, you also have to remember to rename all the submodule >> absolute references in the package as well. Another alternative is to >> hide the >> ones you don't want by renaming them. Then you have to do renames to >> switch >> back to them. That makes it more difficult to do side by side >> comparisons as well. > Right but the point of relative imports is to help prevent renaming. > It's a catch-22 situation of what matters more to you. Right, thats what I tried first. It would have been nice if that worked. >> Take a look at how much __name__ == '__main__' is used in the standard >> library. >> Those modules won't be able to use relative imports. They will need >> to be >> updated and can use only absolute imports. > It's possible. Maybe this will finally push forward the idea of > having a proper 'main' function or something. > Yeah, some built-in or something might do it that lets you decorate > what function is to be the 'main' function and maybe do some magic to > make it resolve properly. It's not the main function that is the problem, it's the imports at the top of the file before the main function. If they import ok, then any main function will work also. I did some experimenting and just adding '../..' in front of the path has some problems of its own. It's relative to the directory you run it from instead of the directory the script is in. So it may or may not allow the script to run. What is needed is to append the actual root package location to the front os sys.path. If absolute imports are used for intra-package imports, then there is not chance of shadowing the library by doing this. This is what I came up with, but there may be an easier way. (and shorter) def pkg_dir(file): import os file = os.path.abspath(file) path = os.path.dirname(file) # Next line should check for (.pyo, .pyc) files too. while os.path.exists(os.path.join(path, '__init__.py')): path = os.path.abspath(os.path.join(path, os.pardir)) return path # Library imports import sys sys.path = [pkg_dir(__file__)] + sys.path # Absolute package imports here ... tests(): ... if __name__ == '__main__': tests() This does what you would expect it to do. It always finds it's own packages modules and it's not sensitive to what location you run it from. But it's bit bulky to be putting everywhere. The real problem isn't that you might get errors, it is that you *may not*. For example, if you decide to modify an existing package, and so copy it to a new location for editing. Then you do your edits, and after everything looks good you run the tests in the tests sub module. It passes, Yay! So then you copy it back to it's original location. Unfortunately the tests may have imported the unedited modules in the live package. So the edited package you replaced it with may have errors. How serious that is depends on what the package is used for. Those mars satellites can get pretty expensive after you loose 2 or 3 of them. ;-) I admit that's pretty unlikely because the package would probably go though some pretty rigorous testing in that case. But still, there is the chance of something like that happening in more casual circumstances. I don't think this is that far fetched. And the other benefit is it just makes it easier to run modules in different circumstances without problems. On the path, directly, indirectly... while editing. That is really what I want at the moment. It should *just work*. Cheers, Ron