From jcarlson at uci.edu  Mon Jan  1 23:03:56 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Mon, 01 Jan 2007 14:03:56 -0800
Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations:
	overloadable ->
In-Reply-To: <ca471dc20701011320t41e538c8yf217a50fdc9900bd@mail.gmail.com>
References: <4599791F.9020505@acm.org>
	<ca471dc20701011320t41e538c8yf217a50fdc9900bd@mail.gmail.com>
Message-ID: <20070101133749.82F5.JCARLSON@uci.edu>


To Tony and Kay, my short answer is: use __returns__ .

Moving to python-ideas as per Guido's request:

"Guido van Rossum" <guido at python.org> wrote:
> This is sufficiently controversial that I believe it ought to go to
> python-ideas first. If it comes to a PEP it should be a separate one
> from PEP 3107.

> On 1/1/07, Talin <talin at acm.org> wrote:
> > Tony Lownds wrote:
> > >> From: Tony Lownds <tony at pagedna.com>
> > > What do people here think?
> > 1) Normally, we don't name operators based on their shape - we don't
> > call '/' the __slash__ operator, for example, nor do we call '|' the
> > "__vbar__" operator.

Certainly, but those two operators, and basically every other operator
used in Python have long-standing semantics in basically every language
that Python is even remotely related to.


> > 2) I think that all operators should have a "suggested semantic". When
> > someone overloads the '+' operator, its a good bet that the meaning of
> > the overload has something to do with addition or accumulation in a
> > general sense. This won't *always* be true, but it will be true often
> > enough.

I don't buy your "suggested semantic" argument.  And even if I did,
operator overloading allows people to choose the semantics of operations
for themselves; suggesting a semantic for an operation would be a set of
documentation that would never be read, and if it was read, ignored.


> > But an arbitrary operator with no guidelines as to what it means is
> > anyone's guess; It means that when we see a '->' operator embedded in
> > the code, we have no idea what is being said.

Ahh, but in this case there is precisely one place where the '->'
operator is planned on being found for Py3k:

    def <name>(<arglist with or without annotations>) -> <annotation>:
        <body>

In that sense, we don't need a fcn_obj.__becomes__(<annotation>) method,
and that wasn't what the discussion was about, it was "what should the
attribute be called for this already agreed upon *function annotation*?".

Now, because this *particular* annotation was created to allow for the
annotation of "returns", I agree with Kay's last suggestion, the
attribute should be called __returns__, as that is *exactly* what the
annotation was meant to convey.


> >  From an HCI perspective, punctuation symbols improve code readability
> > only if their meanings are familiar to the reader; An operator whose
> > meaning is constantly changing is a hindrance to readability rather than
> > a help.

Claiming "we want a particular operation to always refer to the same
method/attribute" is only applicable if a particular operation has a
chance of meaning more than one thing.  Currently it means *exactly* one
thing in the context of Python, 'this function returns X', so in my
opinion, your argument isn't applicable.

If you can manage to convince more people in python-ideas of arbitrary
operations (as per your previous message(s) on the subject), and/or you
can convince Guido to say "I would like more operations in Python", then
your argument is applicable.

However, I don't believe that you will be able to convince Guido that
the large set of operations that you have previously posted about would
be a good idea, and I certainly don't believe it would happen with
sufficient time to make it into the first Py3k release.


 - Josiah



From talin at acm.org  Mon Jan  1 23:25:22 2007
From: talin at acm.org (Talin)
Date: Mon, 01 Jan 2007 14:25:22 -0800
Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations:
	overloadable ->
In-Reply-To: <20070101133749.82F5.JCARLSON@uci.edu>
References: <4599791F.9020505@acm.org>
	<ca471dc20701011320t41e538c8yf217a50fdc9900bd@mail.gmail.com>
	<20070101133749.82F5.JCARLSON@uci.edu>
Message-ID: <45998A52.9060801@acm.org>

Josiah Carlson wrote:
> To Tony and Kay, my short answer is: use __returns__ .
> 
> Moving to python-ideas as per Guido's request:
> 
> "Guido van Rossum" <guido at python.org> wrote:
>> This is sufficiently controversial that I believe it ought to go to
>> python-ideas first. If it comes to a PEP it should be a separate one
>> from PEP 3107.
> 
>> On 1/1/07, Talin <talin at acm.org> wrote:
>>> Tony Lownds wrote:
>>>>> From: Tony Lownds <tony at pagedna.com>
>>>> What do people here think?
>>> 1) Normally, we don't name operators based on their shape - we don't
>>> call '/' the __slash__ operator, for example, nor do we call '|' the
>>> "__vbar__" operator.
> 
> Certainly, but those two operators, and basically every other operator
> used in Python have long-standing semantics in basically every language
> that Python is even remotely related to.
> 
> 
>>> 2) I think that all operators should have a "suggested semantic". When
>>> someone overloads the '+' operator, its a good bet that the meaning of
>>> the overload has something to do with addition or accumulation in a
>>> general sense. This won't *always* be true, but it will be true often
>>> enough.
> 
> I don't buy your "suggested semantic" argument.  And even if I did,
> operator overloading allows people to choose the semantics of operations
> for themselves; suggesting a semantic for an operation would be a set of
> documentation that would never be read, and if it was read, ignored.
> 
> 
>>> But an arbitrary operator with no guidelines as to what it means is
>>> anyone's guess; It means that when we see a '->' operator embedded in
>>> the code, we have no idea what is being said.
> 
> Ahh, but in this case there is precisely one place where the '->'
> operator is planned on being found for Py3k:
> 
>     def <name>(<arglist with or without annotations>) -> <annotation>:
>         <body>
> 
> In that sense, we don't need a fcn_obj.__becomes__(<annotation>) method,
> and that wasn't what the discussion was about, it was "what should the
> attribute be called for this already agreed upon *function annotation*?".
> 
> Now, because this *particular* annotation was created to allow for the
> annotation of "returns", I agree with Kay's last suggestion, the
> attribute should be called __returns__, as that is *exactly* what the
> annotation was meant to convey.
> 
> 
>>>  From an HCI perspective, punctuation symbols improve code readability
>>> only if their meanings are familiar to the reader; An operator whose
>>> meaning is constantly changing is a hindrance to readability rather than
>>> a help.
> 
> Claiming "we want a particular operation to always refer to the same
> method/attribute" is only applicable if a particular operation has a
> chance of meaning more than one thing.  Currently it means *exactly* one
> thing in the context of Python, 'this function returns X', so in my
> opinion, your argument isn't applicable.
> 
> If you can manage to convince more people in python-ideas of arbitrary
> operations (as per your previous message(s) on the subject), and/or you
> can convince Guido to say "I would like more operations in Python", then
> your argument is applicable.
> 
> However, I don't believe that you will be able to convince Guido that
> the large set of operations that you have previously posted about would
> be a good idea, and I certainly don't believe it would happen with
> sufficient time to make it into the first Py3k release.
> 
> 
>  - Josiah

Sorry, I'm a habitual generalizer :)

Anyway, I'm not going to push on this one any further, neither here nor 
in python-ideas. There are more important things to work on.

-- Talin


From tony at PageDNA.com  Mon Jan  1 23:28:11 2007
From: tony at PageDNA.com (Tony Lownds)
Date: Mon, 1 Jan 2007 14:28:11 -0800
Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations:
	overloadable ->
In-Reply-To: <20070101133749.82F5.JCARLSON@uci.edu>
References: <4599791F.9020505@acm.org>
	<ca471dc20701011320t41e538c8yf217a50fdc9900bd@mail.gmail.com>
	<20070101133749.82F5.JCARLSON@uci.edu>
Message-ID: <571A7E3A-297F-40DA-86C0-E5E07F234B13@PageDNA.com>


On Jan 1, 2007, at 2:03 PM, Josiah Carlson wrote:

>
> To Tony and Kay, my short answer is: use __returns__ .
>

Thanks

Talin's arguments for a consistent semantic make me think
that __implies__ would be a bad idea (among other things).

Is anyone against "->" overloadable as __returns__ /__rreturns__
with no semantics for existing types?

-Tony






From tony at PageDNA.com  Tue Jan  2 00:17:36 2007
From: tony at PageDNA.com (Tony Lownds)
Date: Mon, 1 Jan 2007 15:17:36 -0800
Subject: [Python-ideas] PEP 3107 Function Annotations: interoperability
	(again)
Message-ID: <D041783D-7517-456E-9440-07FECA65D73C@PageDNA.com>

In discussing the Function Annotations PEP on python-list,  
interoperability
between schemes came up again:

http://mail.python.org/pipermail/python-list/2006-December/420645.html

John Roth wrote:
> Third, it's half of a proposal. Type checking isn't the only use
> for metadata about functions/methods, classes, properties
> and other objects, and the notion that there are only going to
> be a small number of non-intersecting libraries out there is
> an abdication of responsibility to think this thing through.

This issue came up before in
http://mail.python.org/pipermail/python-3000/2006-August/002796.html
and a rather long thread followed. Here is the paragraph in the PEP that
needs updating, at the least:

    There is no worry that these libraries will assign semantics at
    random, or that a variety of libraries will appear, each with
    varying semantics and interpretations of what, say, a tuple of
    strings means. The difficulty inherent in writing annotation
    interpreting libraries will keep their number low and their
    authorship in the hands of people who, frankly, know what they're
    doing.

The notion that libraries don't intersect should be stripped from the
PEP. The question in my mind is whether this PEP needs to take
responsibility for interoperability.

I contend that people who design an annotation-consuming library
that ends up intersecting several others will likely be capable of  
finding
a solution even if not ideal without a central mechanism, and will be  
far
better situated to define a solution for a central mechanism.

Any thoughts?

Thanks
-Tony



From brett at python.org  Tue Jan  2 01:17:27 2007
From: brett at python.org (Brett Cannon)
Date: Mon, 1 Jan 2007 16:17:27 -0800
Subject: [Python-ideas] PEP 3107 Function Annotations: interoperability
	(again)
In-Reply-To: <D041783D-7517-456E-9440-07FECA65D73C@PageDNA.com>
References: <D041783D-7517-456E-9440-07FECA65D73C@PageDNA.com>
Message-ID: <bbaeab100701011617s2f7c5e56pb5146518e85503ae@mail.gmail.com>

On 1/1/07, Tony Lownds <tony at pagedna.com> wrote:
>
> In discussing the Function Annotations PEP on python-list,
> interoperability
> between schemes came up again:
>
> http://mail.python.org/pipermail/python-list/2006-December/420645.html
>
> John Roth wrote:
> > Third, it's half of a proposal. Type checking isn't the only use
> > for metadata about functions/methods, classes, properties
> > and other objects, and the notion that there are only going to
> > be a small number of non-intersecting libraries out there is
> > an abdication of responsibility to think this thing through.
>
> This issue came up before in
> http://mail.python.org/pipermail/python-3000/2006-August/002796.html
> and a rather long thread followed. Here is the paragraph in the PEP that
> needs updating, at the least:
>
>     There is no worry that these libraries will assign semantics at
>     random, or that a variety of libraries will appear, each with
>     varying semantics and interpretations of what, say, a tuple of
>     strings means. The difficulty inherent in writing annotation
>     interpreting libraries will keep their number low and their
>     authorship in the hands of people who, frankly, know what they're
>     doing.
>
> The notion that libraries don't intersect should be stripped from the
> PEP. The question in my mind is whether this PEP needs to take
> responsibility for interoperability.
>
> I contend that people who design an annotation-consuming library
> that ends up intersecting several others will likely be capable of
> finding
> a solution even if not ideal without a central mechanism, and will be
> far
> better situated to define a solution for a central mechanism.
>
> Any thoughts?



Until extensive usage happens with annotations we shouldn't try to shoehorn
consumers of the annotations into a specific solution.  As you said,
something will probably organically grow and that can be supported in
another PEP later on (probably with stdlib support).  People tend to want to
design up front but with something like this that is meant to be highly
customizable by the people wanting to use it you just don't get that option
without ample feedback from users.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20070101/0f1a6cee/attachment.html>

From guido at python.org  Tue Jan  2 01:20:40 2007
From: guido at python.org (Guido van Rossum)
Date: Mon, 1 Jan 2007 16:20:40 -0800
Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations:
	overloadable ->
In-Reply-To: <571A7E3A-297F-40DA-86C0-E5E07F234B13@PageDNA.com>
References: <4599791F.9020505@acm.org>
	<ca471dc20701011320t41e538c8yf217a50fdc9900bd@mail.gmail.com>
	<20070101133749.82F5.JCARLSON@uci.edu>
	<571A7E3A-297F-40DA-86C0-E5E07F234B13@PageDNA.com>
Message-ID: <ca471dc20701011620w28e097d1qc402fa39eb54b6ec@mail.gmail.com>

On 1/1/07, Tony Lownds <tony at pagedna.com> wrote:
> Is anyone against "->" overloadable as __returns__ /__rreturns__
> with no semantics for existing types?

I suggest that you question *really* *hard* whether it's worth it.
We'd be introducing two very different syntactic uses for '->'. I
chose this operator to signify function return annotation specifically
because it *does't* have another meaning in Python. (My first choice
would've been ':', which is mostly a delimiter that derives its
meaning from context, like ',', but that would've introduced too much
syntactic ambiguity.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido at python.org  Tue Jan  2 01:40:33 2007
From: guido at python.org (Guido van Rossum)
Date: Mon, 1 Jan 2007 16:40:33 -0800
Subject: [Python-ideas] PEP 3107 Function Annotations: interoperability
	(again)
In-Reply-To: <D041783D-7517-456E-9440-07FECA65D73C@PageDNA.com>
References: <D041783D-7517-456E-9440-07FECA65D73C@PageDNA.com>
Message-ID: <ca471dc20701011640r6cd0b6b8o4cf3d3e523af2886@mail.gmail.com>

Well, it's your PEP, isn't it? Just submit a change to the PEP editor.

If it were me, any suggestion of how annotations could/should be used
should be stricken from the PEP, including the example

  def foo(a: int, b: dict, c: int = 5):

The only place where an example like this could be mentioned would be
a motivational section which could suffice by referring to Collin's
existing type annotation library and explaining how it could be made
more elegant by attaching the types directly to the arguments. The
interop issue needn't be mentioned -- I imagine it's not easy to
ensure interop between something like Collin's current library and
some other library that makes extensive use of decorators, either.

A later PEP, based on actual experience, could propose an interop
convention to be used by frameworks that expect to be needing it. To
me the current fuss about interop sounds like over-analyzing the
situation based on zero data points. Remember YAGNI.

--Guido

On 1/1/07, Tony Lownds <tony at pagedna.com> wrote:
> In discussing the Function Annotations PEP on python-list,
> interoperability
> between schemes came up again:
>
> http://mail.python.org/pipermail/python-list/2006-December/420645.html
>
> John Roth wrote:
> > Third, it's half of a proposal. Type checking isn't the only use
> > for metadata about functions/methods, classes, properties
> > and other objects, and the notion that there are only going to
> > be a small number of non-intersecting libraries out there is
> > an abdication of responsibility to think this thing through.
>
> This issue came up before in
> http://mail.python.org/pipermail/python-3000/2006-August/002796.html
> and a rather long thread followed. Here is the paragraph in the PEP that
> needs updating, at the least:
>
>     There is no worry that these libraries will assign semantics at
>     random, or that a variety of libraries will appear, each with
>     varying semantics and interpretations of what, say, a tuple of
>     strings means. The difficulty inherent in writing annotation
>     interpreting libraries will keep their number low and their
>     authorship in the hands of people who, frankly, know what they're
>     doing.
>
> The notion that libraries don't intersect should be stripped from the
> PEP. The question in my mind is whether this PEP needs to take
> responsibility for interoperability.
>
> I contend that people who design an annotation-consuming library
> that ends up intersecting several others will likely be capable of
> finding
> a solution even if not ideal without a central mechanism, and will be
> far
> better situated to define a solution for a central mechanism.
>
> Any thoughts?
>
> Thanks
> -Tony
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From jcarlson at uci.edu  Tue Jan  2 02:39:17 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Mon, 01 Jan 2007 17:39:17 -0800
Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations:
	overloadable ->
In-Reply-To: <ca471dc20701011620w28e097d1qc402fa39eb54b6ec@mail.gmail.com>
References: <571A7E3A-297F-40DA-86C0-E5E07F234B13@PageDNA.com>
	<ca471dc20701011620w28e097d1qc402fa39eb54b6ec@mail.gmail.com>
Message-ID: <20070101172715.82F8.JCARLSON@uci.edu>


"Guido van Rossum" <guido at python.org> wrote:
> 
> On 1/1/07, Tony Lownds <tony at pagedna.com> wrote:
> > Is anyone against "->" overloadable as __returns__ /__rreturns__
> > with no semantics for existing types?
> 
> I suggest that you question *really* *hard* whether it's worth it.
> We'd be introducing two very different syntactic uses for '->'. I
> chose this operator to signify function return annotation specifically
> because it *does't* have another meaning in Python. (My first choice
> would've been ':', which is mostly a delimiter that derives its
> meaning from context, like ',', but that would've introduced too much
> syntactic ambiguity.

Just to clarify, the only place '->' is allowable is in the case of
function annotations...

    def <name>(<args...>) -> <annotation>:
        <body>

Given a reading of version 53169 of PEP 3107, I see no reason to even
offer a __returns__ attribute, automatic method call, etc.  PEP 3107
already defines the annotation to be specified as a dictionary of named
argument names, with a key of 'return' for the annotation of the return
annotation.  Is there any reason why this isn't sufficient?

Even in the context of the signature PEP 362, there is no need for
__returns__, as an annotation-consuming library would check the
signature object for argument annotations, and the func_annotations
dictionary for the 'return' key for any return annotations.


 - Josiah



From tony at PageDNA.com  Tue Jan  2 03:53:48 2007
From: tony at PageDNA.com (Tony Lownds)
Date: Mon, 1 Jan 2007 18:53:48 -0800
Subject: [Python-ideas] Attribute Docstrings and Annotations
Message-ID: <A0197878-AEBF-4C5D-9468-8642CF21A701@PageDNA.com>

I'd like to propose annotations and docstrings on attributes for  
Python 3000
with the following Grammar and properties:

expr_stmt: test (':' test ['=' (yield_expr|testlist)] |
                  augassign (yield_expr|testlist) |
                  [',' testlist] ('=' (yield_expr|testlist))*
                 )

* annotations can appear for attributes that are not defined yet
* code to generate and populate __annotations__ and __attrdoc__
   would appear for all modules and class bodies, not for functions.
* attribute annotations allow a name as target only
* attribute annotations without assignment are illegal for functions
* attribute annotations with assignment should probably be illegal  
for functions
* docstring annotations only apply to the first target, and only if  
it is a name
* docstring annotations do not apply to augmented assignments
* docstring and annotations on functions do not get populated in  
__annotations__
* the class docstring is not reused as a function docstring

The basic rationale for annotations on attributes is completeness  
with PEP3107.
I'm proposing attribute docstrings as well because I think it's  
preferable to have
a spot for documentation that isn't tied to annotations, like  
functions' __doc__.
Also attribute docstrings as specified look similar to function  
docstrings; both
have statements consisting of only a string that is taken as  
documentation.

Here is an interactive session and the code that might be generated.  
What do you
think?

Thanks
-Tony

 >>> class X:
...     "class docstring"
...     foo: 1 = 1
...     bar: 2
...     "attribute docstring"
...     attr = None
...     "another attribute docstring"
...     fields = __slots__ = ['fields', 'attr']
...     "docstring ignored"
...     x, y = 1, 2
...
 >>> X.__attrdoc__
{'fields': 'another attribute docstring', 'attr': 'attribute docstring'}
 >>> X.__annotations__
{'foo': 1, 'bar': 2}
 >>> X.foo
1
 >>> X.bar
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
AttributeError: type object 'X' has no attribute 'bar'
 >>> def f():
...     x: 1
...
   File "<stdin>", line 2
SyntaxError: annotation without assignment not allowed in function



   2           0 LOAD_NAME                0 (__name__)
               3 STORE_NAME               1 (__module__)
               6 BUILD_MAP                0
               9 STORE_NAME               2 (__annotations__)
              12 BUILD_MAP                0
              15 STORE_NAME               3 (__attrdoc__)

   3          18 LOAD_CONST               0 ('class docstring')
              21 STORE_NAME               4 (__doc__)

   5          24 LOAD_CONST               1 ('attribute docstring')
              27 LOAD_NAME                3 (__attrdoc__)
              30 LOAD_CONST               2 ('attr')
              33 STORE_SUBSCR
              34 LOAD_NAME                5 (None)
              37 STORE_NAME               6 (attr)

   6          40 LOAD_CONST               3 (1)
              48 STORE_NAME               7 (foo)
              44 LOAD_CONST               3 (1)
              51 LOAD_NAME                2 (__annotations__)
              54 LOAD_CONST               4 ('foo')
              57 STORE_SUBSCR

   7          58 LOAD_CONST               5 (2)
              61 LOAD_NAME                2 (__annotations__)
              64 LOAD_CONST               6 ('bar')
              67 STORE_SUBSCR

   9          68 LOAD_CONST               7 ('another attribute  
docstring')
              71 LOAD_NAME                3 (__attrdoc__)
              74 LOAD_CONST               8 ('fields')
              77 STORE_SUBSCR
              78 LOAD_CONST               8 ('fields')
              81 LOAD_CONST               2 ('attr')
              84 BUILD_LIST               2
              87 DUP_TOP
              88 STORE_NAME               8 (fields)
              91 STORE_NAME               9 (__slots__)
              94 LOAD_LOCALS
              95 RETURN_VALUE




From tony at pagedna.com  Tue Jan  2 04:28:07 2007
From: tony at pagedna.com (Tony Lownds)
Date: Mon, 1 Jan 2007 19:28:07 -0800
Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations:
	overloadable ->
In-Reply-To: <20070101172715.82F8.JCARLSON@uci.edu>
References: <571A7E3A-297F-40DA-86C0-E5E07F234B13@PageDNA.com>
	<ca471dc20701011620w28e097d1qc402fa39eb54b6ec@mail.gmail.com>
	<20070101172715.82F8.JCARLSON@uci.edu>
Message-ID: <78364BAD-1C96-4D15-927E-983DB909FBF5@pagedna.com>


On Jan 1, 2007, at 5:39 PM, Josiah Carlson wrote:

> Given a reading of version 53169 of PEP 3107, I see no reason to even
> offer a __returns__ attribute, automatic method call, etc.  PEP 3107
> already defines the annotation to be specified as a dictionary of  
> named
> argument names, with a key of 'return' for the annotation of the  
> return
> annotation.  Is there any reason why this isn't sufficient?
>
> Even in the context of the signature PEP 362, there is no need for
> __returns__, as an annotation-consuming library would check the
> signature object for argument annotations, and the func_annotations
> dictionary for the 'return' key for any return annotations.

Something got lost in the movement between lists... __returns__ wasn't
going to supplant or alter func_annotations or PEP 362 in any way.

__returns__ was proposed as the special method attached to a new
operator (->). That operator could be used to write expressions that  
look
like a function signature.

Here's Kay's original email.

http://mail.python.org/pipermail/python-list/2007-January/420774.html

-Tony



From jcarlson at uci.edu  Tue Jan  2 07:21:21 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Mon, 01 Jan 2007 22:21:21 -0800
Subject: [Python-ideas] Attribute Docstrings and Annotations
In-Reply-To: <A0197878-AEBF-4C5D-9468-8642CF21A701@PageDNA.com>
References: <A0197878-AEBF-4C5D-9468-8642CF21A701@PageDNA.com>
Message-ID: <20070101221716.8306.JCARLSON@uci.edu>


Tony Lownds <tony at PageDNA.com> wrote:
> 
> I'd like to propose annotations and docstrings on attributes for  
> Python 3000
> with the following Grammar and properties:
> 
> expr_stmt: test (':' test ['=' (yield_expr|testlist)] |
>                   augassign (yield_expr|testlist) |
>                   [',' testlist] ('=' (yield_expr|testlist))*
>                  )
[snip]
>  >>> class X:
> ...     "class docstring"
> ...     foo: 1 = 1
> ...     bar: 2
> ...     "attribute docstring"
> ...     attr = None
> ...     "another attribute docstring"
> ...     fields = __slots__ = ['fields', 'attr']
> ...     "docstring ignored"
> ...     x, y = 1, 2
> ...

I have never needed attribute annotations, and I've never heard any core
Python developer talk about it being useful to ahve them.  -1 for the
feature in any form.

The syntax as described is ugly.  -100 for the feature if it has the
syntax provided.

 - Josiah



From jcarlson at uci.edu  Tue Jan  2 07:34:41 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Mon, 01 Jan 2007 22:34:41 -0800
Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations:
	overloadable ->
In-Reply-To: <78364BAD-1C96-4D15-927E-983DB909FBF5@pagedna.com>
References: <20070101172715.82F8.JCARLSON@uci.edu>
	<78364BAD-1C96-4D15-927E-983DB909FBF5@pagedna.com>
Message-ID: <20070101222204.8309.JCARLSON@uci.edu>


Tony Lownds <tony at pagedna.com> wrote:
> 
> 
> On Jan 1, 2007, at 5:39 PM, Josiah Carlson wrote:
> 
> > Given a reading of version 53169 of PEP 3107, I see no reason to even
> > offer a __returns__ attribute, automatic method call, etc.  PEP 3107
> > already defines the annotation to be specified as a dictionary of  
> > named
> > argument names, with a key of 'return' for the annotation of the  
> > return
> > annotation.  Is there any reason why this isn't sufficient?
> >
> > Even in the context of the signature PEP 362, there is no need for
> > __returns__, as an annotation-consuming library would check the
> > signature object for argument annotations, and the func_annotations
> > dictionary for the 'return' key for any return annotations.
> 
> Something got lost in the movement between lists... __returns__ wasn't
> going to supplant or alter func_annotations or PEP 362 in any way.
> 
> __returns__ was proposed as the special method attached to a new
> operator (->). That operator could be used to write expressions that  
> look
> like a function signature.
> 
> Here's Kay's original email.
> 
> http://mail.python.org/pipermail/python-list/2007-January/420774.html

-1 on the -> operator as specified in the email you link.  There is no
reason to add aribtrary operators for call-site annotations.  PEP 3107
has already defined the syntax for function definition site annotations
as follows...

    def name(argument=default:annotation) -> annotation:
        body

While PEP 3107 has not been formally accepted (according to the web page),
the ideas contained received general approval from more or less everyone
involved in the py3k discussion, or at least more or less everyone who
has written and/or consumed annotations in the past.


The mail you link, talks about an arbitrary -> operator for call-site
annotations.  Without seeing actual use-cases where a -> operator would
be useful in the real-world, I can't help but be -1 on the -> operator
and -1 on the __returns__ method.


Use-case(s) before syntax and semantics,
 - Josiah



From kay.schluehr at gmx.net  Tue Jan  2 13:44:34 2007
From: kay.schluehr at gmx.net (Kay Schluehr)
Date: Tue, 02 Jan 2007 13:44:34 +0100
Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations:
 overloadable ->
In-Reply-To: <20070101222204.8309.JCARLSON@uci.edu>
References: <20070101172715.82F8.JCARLSON@uci.edu>
	<78364BAD-1C96-4D15-927E-983DB909FBF5@pagedna.com>
	<20070101222204.8309.JCARLSON@uci.edu>
Message-ID: <459A53B2.2030107@gmx.net>

Josiah Carlson schrieb:

>Tony Lownds <tony at pagedna.com> wrote:
>  
>
>>On Jan 1, 2007, at 5:39 PM, Josiah Carlson wrote:
>>
>>    
>>
>>>Given a reading of version 53169 of PEP 3107, I see no reason to even
>>>offer a __returns__ attribute, automatic method call, etc.  PEP 3107
>>>already defines the annotation to be specified as a dictionary of  
>>>named
>>>argument names, with a key of 'return' for the annotation of the  
>>>return
>>>annotation.  Is there any reason why this isn't sufficient?
>>>
>>>Even in the context of the signature PEP 362, there is no need for
>>>__returns__, as an annotation-consuming library would check the
>>>signature object for argument annotations, and the func_annotations
>>>dictionary for the 'return' key for any return annotations.
>>>      
>>>
>>Something got lost in the movement between lists... __returns__ wasn't
>>going to supplant or alter func_annotations or PEP 362 in any way.
>>
>>__returns__ was proposed as the special method attached to a new
>>operator (->). That operator could be used to write expressions that  
>>look
>>like a function signature.
>>
>>Here's Kay's original email.
>>
>>http://mail.python.org/pipermail/python-list/2007-January/420774.html
>>    
>>
>
>-1 on the -> operator as specified in the email you link.  There is no
>reason to add aribtrary operators for call-site annotations.  PEP 3107
>has already defined the syntax for function definition site annotations
>as follows...
>
>    def name(argument=default:annotation) -> annotation:
>        body
>  
>

If a function consumes or returns another function, it is quite natural 
to check against the function type ( at least for certain usefull 
annotation handlers ). Some experts call those "higher order functions". 
Programmers have found those in Pythons __builtins__ module. Others have 
guessed that decorators could be HOF. I hope this is convenient enough 
for motivation.

With current syntax one ends up defining function type annotations like this

Function(int,str).returns(str)
Function(int,str)*str
(Function(int,str), str)
"(int,str)->str"
...

Maybe one can update PEP 8 for a recommendation, if there is just too 
much syntax angst for choosing the form

    Function(int, str)->str

where wild hordes of Pythonistas might think about this as an 
implication - and are perfectly correct about it of course, at least 
according to Curry-Howard and any person, reading at least an 
introductory paper about type theory.

The latter is also the reason why Tonys __implies__ proposal makes 
perfect sense to me.

Regards and a happy new year,

Kay



From tony at pagedna.com  Tue Jan  2 17:33:00 2007
From: tony at pagedna.com (Tony Lownds)
Date: Tue, 2 Jan 2007 08:33:00 -0800
Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations:
	overloadable ->
In-Reply-To: <20070101222204.8309.JCARLSON@uci.edu>
References: <20070101172715.82F8.JCARLSON@uci.edu>
	<78364BAD-1C96-4D15-927E-983DB909FBF5@pagedna.com>
	<20070101222204.8309.JCARLSON@uci.edu>
Message-ID: <338E39B3-446B-4322-816B-9783ED882393@pagedna.com>


On Jan 1, 2007, at 10:34 PM, Josiah Carlson wrote:
> -1 on the -> operator as specified in the email you link.  There is no
> reason to add aribtrary operators for call-site annotations.

What was the arbitrary operator? -> is not arbitrary, and there was a  
reason
given to add it.

> PEP 3107
> has already defined the syntax for function definition site  
> annotations
> as follows...
>
>     def name(argument=default:annotation) -> annotation:
>         body
>

You mean

def name(argument:annotation=default)

> The mail you link, talks about an arbitrary -> operator for call-site
> annotations.  Without seeing actual use-cases where a -> operator  
> would
> be useful in the real-world, I can't help but be -1 on the -> operator
> and -1 on the __returns__ method.

That email had a use case... The gain is being able to write

   def wrap(text: str, split: Function(str)-> list):

instead of

   def wrap(text: str, split: Function(str) == list):

or

   def wrap(text: str, split: Function(str, returns=list)):

or one of the other methods that Kay came up with.

-Tony




From tony at pagedna.com  Tue Jan  2 17:46:29 2007
From: tony at pagedna.com (Tony Lownds)
Date: Tue, 2 Jan 2007 08:46:29 -0800
Subject: [Python-ideas] Attribute Docstrings and Annotations
In-Reply-To: <20070101221716.8306.JCARLSON@uci.edu>
References: <A0197878-AEBF-4C5D-9468-8642CF21A701@PageDNA.com>
	<20070101221716.8306.JCARLSON@uci.edu>
Message-ID: <77616156-108A-40EC-9CC3-1F59082CCD4C@pagedna.com>


On Jan 1, 2007, at 10:21 PM, Josiah Carlson wrote:

>
> Tony Lownds <tony at PageDNA.com> wrote:
>>
>> I'd like to propose annotations and docstrings on attributes for
>> Python 3000
>> with the following Grammar and properties:
>>
>> expr_stmt: test (':' test ['=' (yield_expr|testlist)] |
>>                   augassign (yield_expr|testlist) |
>>                   [',' testlist] ('=' (yield_expr|testlist))*
>>                  )
> [snip]
>>>>> class X:
>> ...     "class docstring"
>> ...     foo: 1 = 1
>> ...     bar: 2
>> ...     "attribute docstring"
>> ...     attr = None
>> ...     "another attribute docstring"
>> ...     fields = __slots__ = ['fields', 'attr']
>> ...     "docstring ignored"
>> ...     x, y = 1, 2
>> ...
>
> I have never needed attribute annotations, and I've never heard any  
> core
> Python developer talk about it being useful to ahve them.  -1 for the
> feature in any form.
>
> The syntax as described is ugly.  -100 for the feature if it has the
> syntax provided.
>

It's the same syntax as function annotations...

def f(name: annotation = value):
       ^^^^^^^^^^^^^^^^^^^^^^^^

class F:
     name: annotation = value
     ^^^^^^^^^^^^^^^^^^^^^^^^

The syntax was presented on Guido's blog, too.

http://www.artima.com/weblogs/viewpost.jsp?thread=87182

-Tony



From jcarlson at uci.edu  Tue Jan  2 18:21:31 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Tue, 02 Jan 2007 09:21:31 -0800
Subject: [Python-ideas] Attribute Docstrings and Annotations
In-Reply-To: <77616156-108A-40EC-9CC3-1F59082CCD4C@pagedna.com>
References: <20070101221716.8306.JCARLSON@uci.edu>
	<77616156-108A-40EC-9CC3-1F59082CCD4C@pagedna.com>
Message-ID: <20070102090907.831E.JCARLSON@uci.edu>


Tony Lownds <tony at pagedna.com> wrote:
> On Jan 1, 2007, at 10:21 PM, Josiah Carlson wrote:
> > I have never needed attribute annotations, and I've never heard any  
> > core
> > Python developer talk about it being useful to have them.  -1 for the
> > feature in any form.
> > The syntax as described is ugly.  -100 for the feature if it has the
> > syntax provided.
> >
> 
> It's the same syntax as function annotations...
> 
> def f(name: annotation = value):
>        ^^^^^^^^^^^^^^^^^^^^^^^^

I don't particularly like the look of function annotations either, but I
don't have a better syntax (aside from swapping the annotation and value).
In this case, I really don't like attribute annotations because it
looks to me like a bunch of line noise without meaning, or some C-like
conditional expression gone wrong.


> class F:
>      name: annotation = value
>      ^^^^^^^^^^^^^^^^^^^^^^^^
> 
> The syntax was presented on Guido's blog, too.
> 
> http://www.artima.com/weblogs/viewpost.jsp?thread=87182

I didn't like it when Guido posted it on his blog either.  I would also
point out that Guido lists attribute annotations as a "maybe".  Perhaps
he has become 100% on them, I don't know, but I'm still -1.

In any case, you still haven't provided any use-cases, or an example
where developers have been asking for the feature and could show that
*not* having the feature was constraining them in some significant way.


 - Josiah



From tony at pagedna.com  Tue Jan  2 18:40:44 2007
From: tony at pagedna.com (Tony Lownds)
Date: Tue, 2 Jan 2007 09:40:44 -0800
Subject: [Python-ideas] Attribute Docstrings and Annotations
In-Reply-To: <20070102090907.831E.JCARLSON@uci.edu>
References: <20070101221716.8306.JCARLSON@uci.edu>
	<77616156-108A-40EC-9CC3-1F59082CCD4C@pagedna.com>
	<20070102090907.831E.JCARLSON@uci.edu>
Message-ID: <FBCBA61F-1F0E-4542-A524-517CC137281A@pagedna.com>


On Jan 2, 2007, at 9:21 AM, Josiah Carlson wrote:

>
> Tony Lownds <tony at pagedna.com> wrote:
>> On Jan 1, 2007, at 10:21 PM, Josiah Carlson wrote:
>>> I have never needed attribute annotations, and I've never heard any
>>> core
>>> Python developer talk about it being useful to have them.  -1 for  
>>> the
>>> feature in any form.
>>> The syntax as described is ugly.  -100 for the feature if it has the
>>> syntax provided.
>>>
>>
>> It's the same syntax as function annotations...
>>
>> def f(name: annotation = value):
>>        ^^^^^^^^^^^^^^^^^^^^^^^^
>
> I don't particularly like the look of function annotations either,  
> but I
> don't have a better syntax (aside from swapping the annotation and  
> value).
> In this case, I really don't like attribute annotations because it
> looks to me like a bunch of line noise without meaning, or some C-like
> conditional expression gone wrong.
>
>
>> class F:
>>      name: annotation = value
>>      ^^^^^^^^^^^^^^^^^^^^^^^^
>>
>> The syntax was presented on Guido's blog, too.
>>
>> http://www.artima.com/weblogs/viewpost.jsp?thread=87182
>
> I didn't like it when Guido posted it on his blog either.  I would  
> also
> point out that Guido lists attribute annotations as a "maybe".   
> Perhaps
> he has become 100% on them, I don't know, but I'm still -1.
>
> In any case, you still haven't provided any use-cases, or an example
> where developers have been asking for the feature and could show that
> *not* having the feature was constraining them in some significant  
> way.

As long as it's clear that the syntax as proposed has SOME existing  
thought
behind it, I'm happy to leave the proposal alone.

-Tony




From jcarlson at uci.edu  Tue Jan  2 19:29:06 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Tue, 02 Jan 2007 10:29:06 -0800
Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations:
	overloadable ->
In-Reply-To: <338E39B3-446B-4322-816B-9783ED882393@pagedna.com>
References: <20070101222204.8309.JCARLSON@uci.edu>
	<338E39B3-446B-4322-816B-9783ED882393@pagedna.com>
Message-ID: <20070102090653.831B.JCARLSON@uci.edu>


Tony Lownds <tony at pagedna.com> wrote:
> On Jan 1, 2007, at 10:34 PM, Josiah Carlson wrote:
> > -1 on the -> operator as specified in the email you link.  There is no
> > reason to add aribtrary operators for call-site annotations.
> 
> What was the arbitrary operator? -> is not arbitrary, and there was a  
> reason given to add it.

I misunderstood portions of the post.

> > PEP 3107
> > has already defined the syntax for function definition site  
> > annotations
> > as follows...
> >
> >     def name(argument=default:annotation) -> annotation:
> >         body
> 
> You mean
> 
> def name(argument:annotation=default)

Thank you for the correction.  Maybe I'm the only one, but I would
prefer the default value to come before the annotation.


> > The mail you link, talks about an arbitrary -> operator for call-site
> > annotations.  Without seeing actual use-cases where a -> operator  
> > would
> > be useful in the real-world, I can't help but be -1 on the -> operator
> > and -1 on the __returns__ method.
> 
> That email had a use case... The gain is being able to write
> 
>    def wrap(text: str, split: Function(str)-> list):
> 
> instead of
> 
>    def wrap(text: str, split: Function(str) == list):
> 
> or
> 
>    def wrap(text: str, split: Function(str, returns=list)):
> 
> or one of the other methods that Kay came up with.

Kay's mail specified:

    maptype = Function( Function(object)->object, list) -> list

That looks to me like the -> operator being used anywhere the user
desires.

With your examples above, I see the use-case, but I'm still -1 (for the
same reasons).

For other reasons to be -1 on ->, we can look at how it would be abused
to violate the one obvious way to do things...

    Stack() -> 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8

    t = Tree()
    t -> (t -> (1,2), t -> (3,4))

Or for a concrete example...

    class mylist(list):
        def __returns__(self, arg):
            self.append(arg)
            return self

    mylist() -> 1 -> 2 -> 3 -> 4 -> 5

Right now those things aren't done because the punctuation for doing it
isn't as natural as -> would be.


 - Josiah



From guido at python.org  Tue Jan  2 19:41:46 2007
From: guido at python.org (Guido van Rossum)
Date: Tue, 2 Jan 2007 10:41:46 -0800
Subject: [Python-ideas] Attribute Docstrings and Annotations
In-Reply-To: <FBCBA61F-1F0E-4542-A524-517CC137281A@pagedna.com>
References: <20070101221716.8306.JCARLSON@uci.edu>
	<77616156-108A-40EC-9CC3-1F59082CCD4C@pagedna.com>
	<20070102090907.831E.JCARLSON@uci.edu>
	<FBCBA61F-1F0E-4542-A524-517CC137281A@pagedna.com>
Message-ID: <ca471dc20701021041t2fdef7d4xbfc2ffa6c168f096@mail.gmail.com>

Yes, I blogged about it, but in the discussion that followed on
python-3000 it became clear that the "typed attribute" notation is not
a favorite of many folks, and I'm no longer in favor of it myself. The
use cases are a lot weaker than for signature annotations. So let's
drop it.

On 1/2/07, Tony Lownds <tony at pagedna.com> wrote:
>
> On Jan 2, 2007, at 9:21 AM, Josiah Carlson wrote:
>
> >
> > Tony Lownds <tony at pagedna.com> wrote:
> >> On Jan 1, 2007, at 10:21 PM, Josiah Carlson wrote:
> >>> I have never needed attribute annotations, and I've never heard any
> >>> core
> >>> Python developer talk about it being useful to have them.  -1 for
> >>> the
> >>> feature in any form.
> >>> The syntax as described is ugly.  -100 for the feature if it has the
> >>> syntax provided.
> >>>
> >>
> >> It's the same syntax as function annotations...
> >>
> >> def f(name: annotation = value):
> >>        ^^^^^^^^^^^^^^^^^^^^^^^^
> >
> > I don't particularly like the look of function annotations either,
> > but I
> > don't have a better syntax (aside from swapping the annotation and
> > value).
> > In this case, I really don't like attribute annotations because it
> > looks to me like a bunch of line noise without meaning, or some C-like
> > conditional expression gone wrong.
> >
> >
> >> class F:
> >>      name: annotation = value
> >>      ^^^^^^^^^^^^^^^^^^^^^^^^
> >>
> >> The syntax was presented on Guido's blog, too.
> >>
> >> http://www.artima.com/weblogs/viewpost.jsp?thread=87182
> >
> > I didn't like it when Guido posted it on his blog either.  I would
> > also
> > point out that Guido lists attribute annotations as a "maybe".
> > Perhaps
> > he has become 100% on them, I don't know, but I'm still -1.
> >
> > In any case, you still haven't provided any use-cases, or an example
> > where developers have been asking for the feature and could show that
> > *not* having the feature was constraining them in some significant
> > way.
>
> As long as it's clear that the syntax as proposed has SOME existing
> thought
> behind it, I'm happy to leave the proposal alone.
>
> -Tony
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From guido at python.org  Tue Jan  2 19:48:50 2007
From: guido at python.org (Guido van Rossum)
Date: Tue, 2 Jan 2007 10:48:50 -0800
Subject: [Python-ideas] [Python-3000] PEP 3107 Function Annotations:
	overloadable ->
In-Reply-To: <20070102090653.831B.JCARLSON@uci.edu>
References: <20070101222204.8309.JCARLSON@uci.edu>
	<338E39B3-446B-4322-816B-9783ED882393@pagedna.com>
	<20070102090653.831B.JCARLSON@uci.edu>
Message-ID: <ca471dc20701021048g1a3f1072ufdcb837c37666ecd@mail.gmail.com>

On 1/2/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> > > PEP 3107
> > > has already defined the syntax for function definition site
> > > annotations
> > > as follows...
> > >
> > >     def name(argument=default:annotation) -> annotation:
> > >         body
> >
> > You mean
> >
> > def name(argument:annotation=default)
>
> Thank you for the correction.  Maybe I'm the only one, but I would
> prefer the default value to come before the annotation.

You're about a year and a half too late with this objection. And
anyway, I strongly favor putting the annotation first; that's how it
is in other languages from which Python has borrowed before, e.g.
Modula-3.

Regarding the status of the PEP: I am mostly waiting for the wording
to improve, e.g. not use examples that suggest this will have type
checking semantics by default. I am +1 on the syntax used in the PEP,
since it is what I have been proposing for a long time.

I am against more additions like an '->' operator or attribute
annotations or anything else at this point, at least until there has
been actual experience with using the implementation as-is. (And yes,
I have seen the use case, and remain unconvinced. Not every use case
deserves new syntax.)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From jimjjewett at gmail.com  Tue Jan  2 23:58:20 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Tue, 2 Jan 2007 17:58:20 -0500
Subject: [Python-ideas] Attribute Docstrings and Annotations
In-Reply-To: <20070101221716.8306.JCARLSON@uci.edu>
References: <A0197878-AEBF-4C5D-9468-8642CF21A701@PageDNA.com>
	<20070101221716.8306.JCARLSON@uci.edu>
Message-ID: <fb6fbf560701021458r45c66f16r767a8d2a78d4cbd7@mail.gmail.com>

On 1/2/07, Josiah Carlson <jcarlson at uci.edu> wrote:

> I have never needed attribute annotations

For some classes, I document the intended use of certain attributes
with a comment.  Making this introspectable would be good.

I've snipped the rest because I agree it wouldn't be worth the
wordiness it would cause with any syntax I've seen so far.

-jJ


From cvrebert at gmail.com  Wed Jan  3 01:07:31 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Tue, 02 Jan 2007 16:07:31 -0800
Subject: [Python-ideas] new operators via backquoting
Message-ID: <459AF3C3.9020502@gmail.com>

In Haskell,  foo `baz` bar  means  (baz foo bar), which translates to 
baz(foo, bar) in Python. This allows Haskell programmers to use 
functions as infix operators.
If I recall correctly, in Py3k, enclosing something in backticks will no 
longer cause it to be repr()-ed, leaving the backtick without a meaning 
in Python.


Thus, I propose one of the following as the new use for the backtick (`):
[Note: In both, the characters between the backticks must be a valid 
Python identifier.]

(A)  `baz` is treated as an operator, named "baz", just as / is "div". 
foo `baz` bar  thus causes python to try to call  foo.__baz__(bar), and 
failing that, bar.__rbaz__(foo), and if both those fail, raise 
TypeError. This is, if I understand correctly, how the builtin operators 
work.

(B)  `baz` is a special way to call a callable. foo `baz` bar  is 
translated to  baz(foo, bar)  with the standard lookup rules for 
resolving "baz"


Example use cases, stolen from Haskell: The Craft of Functional Programming:
2 `max` 5  =>  5
7 `cons` tail  =>  ConsCell(val=7, next=tail)
matrix1 `crossproduct` matrix2  =>  cross-product of the matrices
[1, 2, 3] `zip` ['a', 'b', 'c']  =>  [[1, 'a'], [2, 'c'], [3, 'c']]

I believe that this would improve the readability of code, such as 
Numeric, without going off the deep end and offering programmable syntax.

- Chris Rebert


From rrr at ronadam.com  Wed Jan  3 01:14:38 2007
From: rrr at ronadam.com (Ron Adam)
Date: Tue, 02 Jan 2007 18:14:38 -0600
Subject: [Python-ideas] Attribute Docstrings and Annotations
In-Reply-To: <fb6fbf560701021458r45c66f16r767a8d2a78d4cbd7@mail.gmail.com>
References: <A0197878-AEBF-4C5D-9468-8642CF21A701@PageDNA.com>	<20070101221716.8306.JCARLSON@uci.edu>
	<fb6fbf560701021458r45c66f16r767a8d2a78d4cbd7@mail.gmail.com>
Message-ID: <459AF56E.50500@ronadam.com>

Jim Jewett wrote:
> On 1/2/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> 
>> I have never needed attribute annotations
> 
> For some classes, I document the intended use of certain attributes
> with a comment.  Making this introspectable would be good.
> 
> I've snipped the rest because I agree it wouldn't be worth the
> wordiness it would cause with any syntax I've seen so far.
> 
> -jJ

I feel the correct way to do this is to just put the attribute comment in the 
doc string of the object that contains the attribute.  Simple attributes are 
usually just that, simple values related to the object. Yes, it makes the doc 
string longer, but it also tends to makes for better documentation as well.

Why make things more complex than they need to be?

Ron


From brett at python.org  Wed Jan  3 01:33:58 2007
From: brett at python.org (Brett Cannon)
Date: Tue, 2 Jan 2007 16:33:58 -0800
Subject: [Python-ideas] new operators via backquoting
In-Reply-To: <459AF3C3.9020502@gmail.com>
References: <459AF3C3.9020502@gmail.com>
Message-ID: <bbaeab100701021633x6df8cb6bje00fae66166959c8@mail.gmail.com>

On 1/2/07, Chris Rebert <cvrebert at gmail.com> wrote:
>
> In Haskell,  foo `baz` bar  means  (baz foo bar), which translates to
> baz(foo, bar) in Python. This allows Haskell programmers to use
> functions as infix operators.
> If I recall correctly, in Py3k, enclosing something in backticks will no
> longer cause it to be repr()-ed, leaving the backtick without a meaning
> in Python.


Right.  I removed that back in August at the Google sprint.

Thus, I propose one of the following as the new use for the backtick (`):
> [Note: In both, the characters between the backticks must be a valid
> Python identifier.]
>
> (A)  `baz` is treated as an operator, named "baz", just as / is "div".
> foo `baz` bar  thus causes python to try to call  foo.__baz__(bar), and
> failing that, bar.__rbaz__(foo), and if both those fail, raise
> TypeError. This is, if I understand correctly, how the builtin operators
> work.



(B)  `baz` is a special way to call a callable. foo `baz` bar  is
> translated to  baz(foo, bar)  with the standard lookup rules for
> resolving "baz"
>
>
> Example use cases, stolen from Haskell: The Craft of Functional
> Programming:
> 2 `max` 5  =>  5
> 7 `cons` tail  =>  ConsCell(val=7, next=tail)
> matrix1 `crossproduct` matrix2  =>  cross-product of the matrices
> [1, 2, 3] `zip` ['a', 'b', 'c']  =>  [[1, 'a'], [2, 'c'], [3, 'c']]
>
> I believe that this would improve the readability of code, such as
> Numeric, without going off the deep end and offering programmable syntax.



Big -1 from me.  I hate this feature from Haskell.  It is a step towards
programmable syntax and I think that's just a messy.  And having it become a
magic method that is called instead of just some function that takes two
arguments really sends us down that road.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20070102/5ce7f5fe/attachment.html>

From jcarlson at uci.edu  Wed Jan  3 01:52:06 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Tue, 02 Jan 2007 16:52:06 -0800
Subject: [Python-ideas] new operators via backquoting
In-Reply-To: <459AF3C3.9020502@gmail.com>
References: <459AF3C3.9020502@gmail.com>
Message-ID: <20070102164348.8CC1.JCARLSON@uci.edu>


Chris Rebert <cvrebert at gmail.com> wrote:
> 
> In Haskell,  foo `baz` bar  means  (baz foo bar), which translates to 
> baz(foo, bar) in Python. This allows Haskell programmers to use 
> functions as infix operators.
> If I recall correctly, in Py3k, enclosing something in backticks will no 
> longer cause it to be repr()-ed, leaving the backtick without a meaning 
> in Python.
[snip]
> I believe that this would improve the readability of code, such as 
> Numeric, without going off the deep end and offering programmable syntax.

Let us break down what you are more or less proposing...

    foo `baz` bar

... is translated into ...

    foo.__baz__(bar)
    bar.__rbaz__(foo)

How is that any easier to read or understand than...

    baz(foo, bar)

I don't believe it is easier to understand, and would claim that it is
more difficult to understand, especially to users who have seen
backticks being used for repr.  The only people it could be more
undestandable to is those who have used Haskell and have seen such
operator use in the past.  I dont know how many of those users there are,
but I don't believe that the confusion would be worth it.

If you want to call a method on an object, call the method.  If you want
to call a function that does automatic method invocation, that is fine
too.  But what you are offering is more or less arbitrary infix
operators, which I can't see as being anything more than confusing to
new and seasoned Python users alike.


 - Josiah



From guido at python.org  Wed Jan  3 02:33:36 2007
From: guido at python.org (Guido van Rossum)
Date: Tue, 2 Jan 2007 17:33:36 -0800
Subject: [Python-ideas] new operators via backquoting
In-Reply-To: <459AF3C3.9020502@gmail.com>
References: <459AF3C3.9020502@gmail.com>
Message-ID: <ca471dc20701021733u4d240c03w5c111138973dffae@mail.gmail.com>

On 1/2/07, Chris Rebert <cvrebert at gmail.com> wrote:
> Thus, I propose one of the following as the new use for the backtick (`):

You're missing one of the main reasons for removing the backtick
syntax in the first place: the character itself causes trouble by
looking too much like a regular quote (depending on your font), is
routinely mangled by typesetting software (as every Python book author
can testify), and requires a four-finger chord on Swiss keyboards. No
new uses for it will be accepted in Python 3000 no matter how good the
idea.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From jimjjewett at gmail.com  Wed Jan  3 16:27:11 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Wed, 3 Jan 2007 10:27:11 -0500
Subject: [Python-ideas] new operators via backquoting
In-Reply-To: <ca471dc20701021733u4d240c03w5c111138973dffae@mail.gmail.com>
References: <459AF3C3.9020502@gmail.com>
	<ca471dc20701021733u4d240c03w5c111138973dffae@mail.gmail.com>
Message-ID: <fb6fbf560701030727v2bb9ac1h3d2a64a245aa82d6@mail.gmail.com>

Patch 1627052, assigned to Georg, adds backtick-reuse to PEP 3099
(rejected ideas).

On 1/2/07, Guido van Rossum <guido at python.org> wrote:
> On 1/2/07, Chris Rebert <cvrebert at gmail.com> wrote:
> > Thus, I propose one of the following as the new use for the backtick (`):
>
> You're missing one of the main reasons for removing the backtick
> syntax in the first place: the character itself causes trouble by
> looking too much like a regular quote (depending on your font), is
> routinely mangled by typesetting software (as every Python book author
> can testify), and requires a four-finger chord on Swiss keyboards. No
> new uses for it will be accepted in Python 3000 no matter how good the
> idea.
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From george.sakkis at gmail.com  Wed Jan  3 18:29:55 2007
From: george.sakkis at gmail.com (George Sakkis)
Date: Wed, 3 Jan 2007 12:29:55 -0500
Subject: [Python-ideas] new operators via backquoting
Message-ID: <91ad5bf80701030929i72d74cfeia4bfdcb66559a92f@mail.gmail.com>

Chris Rebert wrote:

> In Haskell,  foo `baz` bar  means  (baz foo bar), which translates to
> baz(foo, bar) in Python. This allows Haskell programmers to use
> functions as infix operators.
> If I recall correctly, in Py3k, enclosing something in backticks will no
> longer cause it to be repr()-ed, leaving the backtick without a meaning
> in Python.

Perhaps you're not aware of the "best hack of 2005", which works in
current Python and goes a step further by allowing you to even
"override" the operator's delimiter:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/384122

George


From cvrebert at gmail.com  Wed Jan  3 18:39:33 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Wed, 03 Jan 2007 09:39:33 -0800
Subject: [Python-ideas] new operators via backquoting
In-Reply-To: <91ad5bf80701030929i72d74cfeia4bfdcb66559a92f@mail.gmail.com>
References: <91ad5bf80701030929i72d74cfeia4bfdcb66559a92f@mail.gmail.com>
Message-ID: <459BEA55.1070908@gmail.com>

George Sakkis wrote:
> Perhaps you're not aware of the "best hack of 2005", which works in
> current Python and goes a step further by allowing you to even
> "override" the operator's delimiter:
> 
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/384122
> 
> George
> _______________________________________________

In light of this and Guido's pronouncement that the backtick will not 
have any meaning in Py3k, I withdraw my proposal.

- Chris Rebert


From collinw at gmail.com  Wed Jan  3 20:24:47 2007
From: collinw at gmail.com (Collin Winter)
Date: Wed, 3 Jan 2007 13:24:47 -0600
Subject: [Python-ideas] PEP 3107 Function Annotations: interoperability
	(again)
In-Reply-To: <bbaeab100701011617s2f7c5e56pb5146518e85503ae@mail.gmail.com>
References: <D041783D-7517-456E-9440-07FECA65D73C@PageDNA.com>
	<bbaeab100701011617s2f7c5e56pb5146518e85503ae@mail.gmail.com>
Message-ID: <43aa6ff70701031124h7137b946x77022e5e4ec34793@mail.gmail.com>

On 1/1/07, Brett Cannon <brett at python.org> wrote:
> Until extensive usage happens with annotations we shouldn't try to shoehorn
> consumers of the annotations into a specific solution.  As you said,
> something will probably organically grow and that can be supported in
> another PEP later on (probably with stdlib support).

I agree that a convention will develop organically, but I would add
that the emergence of that convention will be governed by two things:
who has the bigger market share, and who's first to market. When the
first big project like Zope or Twisted announces that "we will do
annotation interop like so...", everyone else will be pressured to
line up behind them. Until that happens, smaller projects like mine
will either a) not support annotation interop (because there's no
good, obvious solution), or b) pick an interop scheme at random
(again, because there's no good, obvious solution).

Thanks,
Collin Winter


From guido at python.org  Wed Jan  3 21:58:22 2007
From: guido at python.org (Guido van Rossum)
Date: Wed, 3 Jan 2007 12:58:22 -0800
Subject: [Python-ideas] PEP 3107 Function Annotations: interoperability
	(again)
In-Reply-To: <43aa6ff70701031124h7137b946x77022e5e4ec34793@mail.gmail.com>
References: <D041783D-7517-456E-9440-07FECA65D73C@PageDNA.com>
	<bbaeab100701011617s2f7c5e56pb5146518e85503ae@mail.gmail.com>
	<43aa6ff70701031124h7137b946x77022e5e4ec34793@mail.gmail.com>
Message-ID: <ca471dc20701031258y5499cfeds5b36224063ecc370@mail.gmail.com>

On 1/3/07, Collin Winter <collinw at gmail.com> wrote:
> On 1/1/07, Brett Cannon <brett at python.org> wrote:
> > Until extensive usage happens with annotations we shouldn't try to shoehorn
> > consumers of the annotations into a specific solution.  As you said,
> > something will probably organically grow and that can be supported in
> > another PEP later on (probably with stdlib support).
>
> I agree that a convention will develop organically, but I would add
> that the emergence of that convention will be governed by two things:
> who has the bigger market share, and who's first to market. When the
> first big project like Zope or Twisted announces that "we will do
> annotation interop like so...", everyone else will be pressured to
> line up behind them. Until that happens, smaller projects like mine
> will either a) not support annotation interop (because there's no
> good, obvious solution), or b) pick an interop scheme at random
> (again, because there's no good, obvious solution).

Actually, I think that's a pretty good way to arrive at a good
solution. I don't this is something you can analyze on paper ahead of
time; there aren't really any precedents I suspect (do you know of any
other language that has semantics-free signature annotations?). You
must've "built two and thrown one away" before you really know what
the constraints are on the solution.

I realize this is frustrating for you, but I really don't think it's
appropriate to pre-empty this with a standardization attempt
before-the-fact (remember ISO networking?). You could be first to
market yourself -- after all you already have a type checking package!

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From collinw at gmail.com  Wed Jan  3 22:55:25 2007
From: collinw at gmail.com (Collin Winter)
Date: Wed, 3 Jan 2007 15:55:25 -0600
Subject: [Python-ideas] PEP 3107 Function Annotations: interoperability
	(again)
In-Reply-To: <ca471dc20701031258y5499cfeds5b36224063ecc370@mail.gmail.com>
References: <D041783D-7517-456E-9440-07FECA65D73C@PageDNA.com>
	<bbaeab100701011617s2f7c5e56pb5146518e85503ae@mail.gmail.com>
	<43aa6ff70701031124h7137b946x77022e5e4ec34793@mail.gmail.com>
	<ca471dc20701031258y5499cfeds5b36224063ecc370@mail.gmail.com>
Message-ID: <43aa6ff70701031355g6535f02cnde7b13f95305bb36@mail.gmail.com>

On 1/3/07, Guido van Rossum <guido at python.org> wrote:
> On 1/3/07, Collin Winter <collinw at gmail.com> wrote:
> > I agree that a convention will develop organically, but I would add
> > that the emergence of that convention will be governed by two things:
> > who has the bigger market share, and who's first to market. When the
> > first big project like Zope or Twisted announces that "we will do
> > annotation interop like so...", everyone else will be pressured to
> > line up behind them. Until that happens, smaller projects like mine
> > will either a) not support annotation interop (because there's no
> > good, obvious solution), or b) pick an interop scheme at random
> > (again, because there's no good, obvious solution).

[snip]

> I realize this is frustrating for you, but I really don't think it's
> appropriate to pre-empty this with a standardization attempt
> before-the-fact (remember ISO networking?). You could be first to
> market yourself -- after all you already have a type checking package!

Unless someone comes up with a good interop solution before the Py3k
release, I doubt my typechecking package will support annotation
interop; I don't want to pick a mechanism,  only to have to roll
another release when Zope or Twisted or some other big project pushes
out their own solution.

I will support annotations, though; that way, I can at least gather
real-world usage data when users complain about not being able to use
more than one annotation consumer : )

Thanks,
Collin Winter


From guido at python.org  Wed Jan  3 23:06:29 2007
From: guido at python.org (Guido van Rossum)
Date: Wed, 3 Jan 2007 14:06:29 -0800
Subject: [Python-ideas] PEP 3107 Function Annotations: interoperability
	(again)
In-Reply-To: <43aa6ff70701031355g6535f02cnde7b13f95305bb36@mail.gmail.com>
References: <D041783D-7517-456E-9440-07FECA65D73C@PageDNA.com>
	<bbaeab100701011617s2f7c5e56pb5146518e85503ae@mail.gmail.com>
	<43aa6ff70701031124h7137b946x77022e5e4ec34793@mail.gmail.com>
	<ca471dc20701031258y5499cfeds5b36224063ecc370@mail.gmail.com>
	<43aa6ff70701031355g6535f02cnde7b13f95305bb36@mail.gmail.com>
Message-ID: <ca471dc20701031406u6b6049c0u12bd7058bc252f9b@mail.gmail.com>

On 1/3/07, Collin Winter <collinw at gmail.com> wrote:
> Unless someone comes up with a good interop solution before the Py3k
> release, I doubt my typechecking package will support annotation
> interop; I don't want to pick a mechanism,  only to have to roll
> another release when Zope or Twisted or some other big project pushes
> out their own solution.

> I will support annotations, though; that way, I can at least gather
> real-world usage data when users complain about not being able to use
> more than one annotation consumer : )

Those projects, due to sheer size, will be years behind you in their
adoption of Python 3000, so you still have an advantage if you change
your mind. Good luck!

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From matt at draisey.ca  Thu Jan  4 01:20:43 2007
From: matt at draisey.ca (Matt Draisey)
Date: Wed, 03 Jan 2007 19:20:43 -0500
Subject: [Python-ideas] Module local namespaces
Message-ID: <1167870043.1873.34.camel@della.draisey.ca>

A while ago, I developed a small PyGtk programme that could dynamically
reload all the working callbacks and logic while the GUI was still
running.  I could get away with this because of the flexible way Python
loads modules at runtime, but it ended up being a waste of time as
implementing it took more time that actually using it.  For sanity's
sake it quickly becomes clear you almost never want to rely on being
able to refer to a half initialized module.  And wouldn't it be nice if
Python enforced this.

My suggestion is that module importing occur in a temporary local
namespace that exists only until the end of the module code is executed,
then a small function could copy everything from the temporary namespace
into the module object.  The usual closure semantics would guarantee
that top-level functions could still call each other, but they would
effectively become immutable after the namespace wraps up.  The 'global'
keyword could be used at the top level in a module to force it to be
defined in the module immediately, and to ensure internal references to
the object go through the module object.

This would be a big change in module import semantics, but should have
remarkably few consequences, as it really is an enforcement mechanism
for good style.  The copying from the temporary namespace into the
module object would be a good place to insert a hook function to filter
what objects are actually published to the module.  You could by default
not copy any object indentified by a leading underscore.




From brett at python.org  Thu Jan  4 02:01:13 2007
From: brett at python.org (Brett Cannon)
Date: Wed, 3 Jan 2007 17:01:13 -0800
Subject: [Python-ideas] Module local namespaces
In-Reply-To: <1167870043.1873.34.camel@della.draisey.ca>
References: <1167870043.1873.34.camel@della.draisey.ca>
Message-ID: <bbaeab100701031701qc09709apa88c6d48cdd81322@mail.gmail.com>

On 1/3/07, Matt Draisey <matt at draisey.ca> wrote:
>
> A while ago, I developed a small PyGtk programme that could dynamically
> reload all the working callbacks and logic while the GUI was still
> running.  I could get away with this because of the flexible way Python
> loads modules at runtime, but it ended up being a waste of time as
> implementing it took more time that actually using it.  For sanity's
> sake it quickly becomes clear you almost never want to rely on being
> able to refer to a half initialized module.  And wouldn't it be nice if
> Python enforced this.


How are you having an issue with a partially initialized module?  The only
way I can see that happening is that you have a circular import dependency
where both modules want to execute code that the other one has.  And if that
happens the answer for that is "don't do it".

My suggestion is that module importing occur in a temporary local
> namespace that exists only until the end of the module code is executed,
> then a small function could copy everything from the temporary namespace
> into the module object.  The usual closure semantics would guarantee
> that top-level functions could still call each other, but they would
> effectively become immutable after the namespace wraps up.


But why would you want it to be immutable?  Being able to change the
function in a module and have all uses of it also change can be handy.

  The 'global'
> keyword could be used at the top level in a module to force it to be
> defined in the module immediately, and to ensure internal references to
> the object go through the module object.
>
> This would be a big change in module import semantics, but should have
> remarkably few consequences, as it really is an enforcement mechanism
> for good style.  The copying from the temporary namespace into the
> module object would be a good place to insert a hook function to filter
> what objects are actually published to the module.  You could by default
> not copy any object indentified by a leading underscore.


Private namespaces are not exactly a popular thing in Python.  =)

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20070103/4595d3f3/attachment.html>

From jcarlson at uci.edu  Thu Jan  4 02:17:16 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Wed, 03 Jan 2007 17:17:16 -0800
Subject: [Python-ideas] Module local namespaces
In-Reply-To: <1167870043.1873.34.camel@della.draisey.ca>
References: <1167870043.1873.34.camel@della.draisey.ca>
Message-ID: <20070103170636.8CDE.JCARLSON@uci.edu>


Matt Draisey <matt at draisey.ca> wrote:
[snip]

-1 .  The only thing that possibly should be fixed is that modules
should only be inserted into sys.modules after they have been imported
completely and correctly.

> This would be a big change in module import semantics, but should have
> remarkably few consequences, as it really is an enforcement mechanism
> for good style.

How is it an enforcement for good style?  If I can do exactly what I'm
doing now, then it isn't an enforcement mechanism in any way.

Regardless, the way to handle not allowing the rest of your program to
muck with partially initialized modules is to not use the standard
import machinery.  There are various ways of emulating module loading,
many of which can allow you to insert the module into sys.modules
*after* the module was loaded completely correctly.  I actually use one
of these methods to handle the dynamic loading and reloading of Python
macros in a source code editor, and none of the macros are available in
sys.modules .


> The copying from the temporary namespace into the
> module object would be a good place to insert a hook function to filter
> what objects are actually published to the module.  You could by default
> not copy any object indentified by a leading underscore.

You can already do this with the following code:

    __gl = globals()
    for name in __gl.keys():
        if name[:1] == '_' and len(name) > 1 and name.count('_') == 1:
            del __gl[name]
    del __gl

 - Josiah



From guido at python.org  Thu Jan  4 02:18:09 2007
From: guido at python.org (Guido van Rossum)
Date: Wed, 3 Jan 2007 17:18:09 -0800
Subject: [Python-ideas] Module local namespaces
In-Reply-To: <20070103170636.8CDE.JCARLSON@uci.edu>
References: <1167870043.1873.34.camel@della.draisey.ca>
	<20070103170636.8CDE.JCARLSON@uci.edu>
Message-ID: <ca471dc20701031718i7b64e03fg16be4d1445eacd1d@mail.gmail.com>

On 1/3/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> -1 .  The only thing that possibly should be fixed is that modules
> should only be inserted into sys.modules after they have been imported
> completely and correctly.

I agree, but there's the wrinkle that during recursive imports you
want partially-imported modules to be importable. I believe we once
agreed on a solution but I don't recall what it was. Does anybody
remember?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)


From jcarlson at uci.edu  Thu Jan  4 02:51:28 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Wed, 03 Jan 2007 17:51:28 -0800
Subject: [Python-ideas] Module local namespaces
In-Reply-To: <ca471dc20701031718i7b64e03fg16be4d1445eacd1d@mail.gmail.com>
References: <20070103170636.8CDE.JCARLSON@uci.edu>
	<ca471dc20701031718i7b64e03fg16be4d1445eacd1d@mail.gmail.com>
Message-ID: <20070103174954.8CE3.JCARLSON@uci.edu>


"Guido van Rossum" <guido at python.org> wrote:
> On 1/3/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> > -1 .  The only thing that possibly should be fixed is that modules
> > should only be inserted into sys.modules after they have been imported
> > completely and correctly.
> 
> I agree, but there's the wrinkle that during recursive imports you
> want partially-imported modules to be importable. I believe we once
> agreed on a solution but I don't recall what it was. Does anybody
> remember?

I don't remember what was agreed upon, but what about something like...

    try:
        newmodule = ...
        sys.modules[name] = newmodule
        handle_the_import(newmodule, ...)
    except:
        del sys.modules[name]
        raise

 - Josiah



From matt at draisey.ca  Thu Jan  4 03:25:16 2007
From: matt at draisey.ca (Matt Draisey)
Date: Wed, 03 Jan 2007 21:25:16 -0500
Subject: [Python-ideas] Module local namespaces
In-Reply-To: <20070103170636.8CDE.JCARLSON@uci.edu>
References: <1167870043.1873.34.camel@della.draisey.ca>
	<20070103170636.8CDE.JCARLSON@uci.edu>
Message-ID: <1167877516.1873.74.camel@della.draisey.ca>

On Wed, 2007-01-03 at 17:17 -0800, Josiah Carlson wrote:

> You can already do this with the following code:
> 
>     __gl = globals()
>     for name in __gl.keys():
>         if name[:1] == '_' and len(name) > 1 and name.count('_') == 1:
>             del __gl[name]
>     del __gl
> 
>  - Josiah

No, that is not what I meant.  I wasn't talking about deleting
temporaries but not publishing private objects.  Brett Cannon understood
when he said, "Private namespaces are not exactly a popular thing in
Python".  But functional programming style and generators are all the
rage and they often drag in private state via a closure.






From brett at python.org  Thu Jan  4 03:29:05 2007
From: brett at python.org (Brett Cannon)
Date: Wed, 3 Jan 2007 18:29:05 -0800
Subject: [Python-ideas] Module local namespaces
In-Reply-To: <20070103174954.8CE3.JCARLSON@uci.edu>
References: <20070103170636.8CDE.JCARLSON@uci.edu>
	<ca471dc20701031718i7b64e03fg16be4d1445eacd1d@mail.gmail.com>
	<20070103174954.8CE3.JCARLSON@uci.edu>
Message-ID: <bbaeab100701031829o76fcbdcbh9d59b8dce8256191@mail.gmail.com>

On 1/3/07, Josiah Carlson <jcarlson at uci.edu> wrote:
>
>
> "Guido van Rossum" <guido at python.org> wrote:
> > On 1/3/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> > > -1 .  The only thing that possibly should be fixed is that modules
> > > should only be inserted into sys.modules after they have been imported
> > > completely and correctly.
> >
> > I agree, but there's the wrinkle that during recursive imports you
> > want partially-imported modules to be importable. I believe we once
> > agreed on a solution but I don't recall what it was. Does anybody
> > remember?
>
> I don't remember what was agreed upon, but what about something like...
>
>     try:
>         newmodule = ...
>         sys.modules[name] = newmodule
>         handle_the_import(newmodule, ...)
>     except:
>         del sys.modules[name]
>         raise



My re-implementation does exactly that (I just sometimes postpone the module
creation to the top of the handle_the_import function).

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20070103/83dc1459/attachment.html>

From brett at python.org  Thu Jan  4 03:31:15 2007
From: brett at python.org (Brett Cannon)
Date: Wed, 3 Jan 2007 18:31:15 -0800
Subject: [Python-ideas] Module local namespaces
In-Reply-To: <1167877516.1873.74.camel@della.draisey.ca>
References: <1167870043.1873.34.camel@della.draisey.ca>
	<20070103170636.8CDE.JCARLSON@uci.edu>
	<1167877516.1873.74.camel@della.draisey.ca>
Message-ID: <bbaeab100701031831p15567bb6r9ebf3532890daa6c@mail.gmail.com>

On 1/3/07, Matt Draisey <matt at draisey.ca> wrote:
>
> On Wed, 2007-01-03 at 17:17 -0800, Josiah Carlson wrote:
>
> > You can already do this with the following code:
> >
> >     __gl = globals()
> >     for name in __gl.keys():
> >         if name[:1] == '_' and len(name) > 1 and name.count('_') == 1:
> >             del __gl[name]
> >     del __gl
> >
> >  - Josiah
>
> No, that is not what I meant.  I wasn't talking about deleting
> temporaries but not publishing private objects.  Brett Cannon understood
> when he said, "Private namespaces are not exactly a popular thing in
> Python".  But functional programming style and generators are all the
> rage and they often drag in private state via a closure.


True, but you can still get at everything in either (generators, for
instance, expose the paused execution frame and gives you access to
everything).  But I consider these more of an exception instead of the
rule.  The only way I can see myself approving something like this is if it
helped with security (which it might, but I don't know if it would be the
best solution).

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20070103/358a8ffa/attachment.html>

From eopadoan at altavix.com  Sat Jan  6 18:16:19 2007
From: eopadoan at altavix.com (Eduardo "EdCrypt" O. Padoan)
Date: Sat, 6 Jan 2007 15:16:19 -0200
Subject: [Python-ideas] [Python-Dev] features i'd like [Python 3000] ...
	#3: fix super()
In-Reply-To: <A353062D-A55D-4EF3-A6D0-B3DE1E7FB346@lag.net>
References: <4573909F.5050405@666.com> <457402E1.6060705@gmail.com>
	<A353062D-A55D-4EF3-A6D0-B3DE1E7FB346@lag.net>
Message-ID: <dea92f560701060916r1dc04e96x86e13020397d4922@mail.gmail.com>

> I was bitten by the urge to play with this today, and modified my
> previous "self" hack to handle "super" also, so that the following
> code works:
>
>      class D (C):
>          @method
>          def sum(n):
>              return super.sum(n * 2) - self.base
>
> Posted as "evil2.py" here:
>
>      http://www.lag.net/robey/code/surf/
>
> Because hacking "super" requires having the class object handy, this
> one needs a metaclass to do its magic, which is a shame.  I guess if
> it was implemented inside the cpython compiler, it would be less of a
> problem.

BTW, a "super-only" version of this decortor (that I think could be
called "implement") has  some more chances in Python. But think
belongs more to Python-Ideas list, ok?

-- 
EduardoOPadoan (eopadoan->altavix::com)
Bookmarks: http://del.icio.us/edcrypt
Blog: http://edcrypt.blogspot.com
Jabber: edcrypt at jabber dot org
ICQ: 161480283
GTalk: eduardo dot padoan at gmail dot com
MSN: eopadoan at altavix dot com


From eopadoan at altavix.com  Sat Jan  6 18:17:29 2007
From: eopadoan at altavix.com (Eduardo "EdCrypt" O. Padoan)
Date: Sat, 6 Jan 2007 15:17:29 -0200
Subject: [Python-ideas] [Python-Dev] features i'd like [Python 3000] ...
	#3: fix super()
In-Reply-To: <dea92f560701060916r1dc04e96x86e13020397d4922@mail.gmail.com>
References: <4573909F.5050405@666.com> <457402E1.6060705@gmail.com>
	<A353062D-A55D-4EF3-A6D0-B3DE1E7FB346@lag.net>
	<dea92f560701060916r1dc04e96x86e13020397d4922@mail.gmail.com>
Message-ID: <dea92f560701060917h628f204o3650f21d5ad03e4e@mail.gmail.com>

On 1/6/07, Eduardo EdCrypt O. Padoan <eopadoan at altavix.com> wrote:
> > I was bitten by the urge to play with this today, and modified my
> > previous "self" hack to handle "super" also, so that the following
> > code works:
> >
> >      class D (C):
> >          @method
> >          def sum(n):
> >              return super.sum(n * 2) - self.base
> >
> > Posted as "evil2.py" here:
> >
> >      http://www.lag.net/robey/code/surf/
> >
> > Because hacking "super" requires having the class object handy, this
> > one needs a metaclass to do its magic, which is a shame.  I guess if
> > it was implemented inside the cpython compiler, it would be less of a
> > problem.
>
> BTW, a "super-only" version of this decortor (that I think could be
> called "implement") has  some more chances in Python. But think
> belongs more to Python-Ideas list, ok?

Ops: But *I* think *it* belongs more to Python-Ideas list, ok?

-- 
EduardoOPadoan (eopadoan->altavix::com)
Bookmarks: http://del.icio.us/edcrypt
Blog: http://edcrypt.blogspot.com
Jabber: edcrypt at jabber dot org
ICQ: 161480283
GTalk: eduardo dot padoan at gmail dot com
MSN: eopadoan at altavix dot com


From tomerfiliba at gmail.com  Sun Jan 14 22:19:01 2007
From: tomerfiliba at gmail.com (tomer filiba)
Date: Sun, 14 Jan 2007 23:19:01 +0200
Subject: [Python-ideas] Fwd: multi-dispatch again
In-Reply-To: <1d85506f0701140428q2ac7cf7eyd22c7b1ef72ed7e8@mail.gmail.com>
References: <1d85506f0701140428q2ac7cf7eyd22c7b1ef72ed7e8@mail.gmail.com>
Message-ID: <1d85506f0701141319o67630ad3v5d0bb2a7d7357f10@mail.gmail.com>

Guido asked me to move it here. anyway, i might as well state my case
better.

in the formal definitions of object oriented programming, objects
are said to encapsulate state and behavior. behavior is largely dependent
on the type of the object, but the state is important nonetheless.

for example, file objects are stateful: they can be opened for reading-only,
writing-only, both, or be closed altogether. still, they are all instances
of the file type.

since generic functions/multi-dispatch come to help the programmer
by reducing boilerplate early type-checking code and providing a
more capable dispatch mechanism -- we can't overlook the state of the
object.

this early type-checking goes against the spirit of pure duck typing (which
i'm fond of), but is crucial when the code has side effects. in this case,
you can't just start executing the code and "hope it works", as the
resources (i.e., files) involved are modified. in this kind of code, you
want to check everything is alright *instead* of suddenly having
an AttributeError/TypeError somewhere.

here's an example:

@dispatch
def copy(src: file, dst: file):
    while True:
        buf = src.read(1000)
        if not buf: break
        dst.write(buf)

suppose now that dst is mistakenly opened for reading.
this means src would have already been modified, while
dst.write is bound to fail. if src is a socket, for instance,
this would be destructive.

so if we already go as far as having multiple dispatch, which imposes
constraints on the arguments a function accepts, we might as well
base it on the type and state of the object, rather than only on it's type.

we could say, for example:

@dispatch
def copy(src: file_for_reading, dst: file_for_writing):

file_for_reading is not a type -- it's a checker. it may be defined as

def file_for_reading(obj: file):
    return file.mode == "r" and not file.closed

types, by default, would check using isinstance(), but costume
checkers could check for stateful requirements too.

and a note about performance: this check is required whether
it's done explicitly or by the dispatch mechanism, and since
most functions don't require so many overloads, i don't think
it's an issue.

besides, we can have a different decorator for dispatching
by type or by checkers, i.e., @dispatch vs @stateful_dispatch,
or something. the simple @dispatch would use a dictionary,
while the stateful version would use a loop.


-tomer

---------- Forwarded message ----------
From: tomer filiba <tomerfiliba at gmail.com>
Date: Jan 14, 2007 2:28 PM
Subject: multi-dispatch again
To: Python-3000 at python.org

i just thought of a so-to-speak counter-example for ABCs... it's not really
a counter-example, but i believe it shows a deficiency in the concept.

theoretically speaking, objects are a made of type and state. ABCs,
isinstance()
and interfaces at general only check the type part. for example:

@dispatch
def log_to_file(text: str, device: file):
    file.write(text)

this will constrain the *type* of the device, but not its *state*.
practically speaking, i can pass a closed file, or a file open for
reading-only,
and it would pass silently.

basing multi-dispatch on types is of course a leap forward, but if we
already plan to take this leap, why not make it general enough to support
more complex use-cases?

this way we could rewrite the snippet above as

@dispatch
def log_to_file(text: str, device: open_file):
    file.write(text)

where open_file isn't a type, but rather a "checker" that may also examine
the
state. by default, type objects would check for inheritance (via a special
method),
but checkers could extend this behavior.

for efficiency purposes, we can have two decorators:
@type_dispatch - dispatches based on type only
@full_dispatch - dispatches based on type and state

bottom line -- we can't just look at the type of the object for dispatching,

overlooking its state. the state is meaningful, and we'd want the function
not to be called at all if the state of the object is wrong.


-tomer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20070114/7e887fb9/attachment.html>

From cvrebert at gmail.com  Mon Jan 15 02:19:09 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Sun, 14 Jan 2007 17:19:09 -0800
Subject: [Python-ideas] fixing mutable default argument values
Message-ID: <45AAD68D.5010502@gmail.com>

If A.M. Kuchling's list of Python Warts is any indication, Python has 
removed many of the warts it once had. However, the behavior of mutable 
default argument values is still a frequent stumbling-block for newbies. 
It is also present on at least 3 different lists of Python's 
deficiencies ([0][1][2]).

Example of current, unintuitive behavior (snipped from [0]):
 >>> def popo(x=[]):
...     x.append(666)
...     print x
...
 >>> popo()
[666]
 >>> popo()
[666, 666]
 >>> popo()
[666, 666, 666]

Whereas a newbie with experience with immutable default argument values 
would, by analogy, expect:
 >>> popo()
[666]
 >>> popo()
[666]
 >>> popo()
[666]

In scanning [0], [1], [2], and other similar lists, I have only found 
one mediocre use-case for this behavior: Using the default argument 
value to retain state between calls. However, as [2] comments, this 
purpose is much better served by decorators, classes, or (though less 
preferred) global variables. Other uses are alluded to be equally 
esoteric and unpythonic.

To work around this behavior, the following idiom is used:
def popo(x=None):
     if x is None:
         x = []
     x.append(666)
     print x

However, why should the programmer have to write this extra boilerplate 
code when the current, unusual behavior is only relied on by 1% of 
Python code?

Therefore, I propose that default arguments be handled as follows in Py3K:
1. The initial default value is evaluated at definition-time (as in the 
current behavior).
2. That in a function call where the caller has not specified a value 
for an optional argument, Python calls 
copy.deepcopy(initial_default_value), and fills in the optional argument 
with the resulting value.

This is fully backwards-compatible with the aforementioned workaround, 
and removes the need for the it, allowing one to write the first, 
simpler definition of popo().

Comments?

- Chris Rebert


[0] 10 Python pitfalls (http://zephyrfalcon.org/labs/python_pitfalls.html)
[1] Python Gotchas 
(http://www.ferg.org/projects/python_gotchas.html#contents_item_6)
[2] When Pythons Attack 
(http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2)


From ironfroggy at gmail.com  Mon Jan 15 03:58:00 2007
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Sun, 14 Jan 2007 21:58:00 -0500
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <45AAD68D.5010502@gmail.com>
References: <45AAD68D.5010502@gmail.com>
Message-ID: <76fd5acf0701141858w7e87a67h3eb477f099e022bf@mail.gmail.com>

There are a few problems here. Deep copy of the originally created
default argument can be expensive and would not work in any useful way
with non-literals as defaults, such as function calls or subscript
lookups or even simple attributes.

If any solution is possible, it would require a way to differentiate
between mutable and immutable objects, and evaluate the immutables for
every call. This brings on more problems as to when you do the initial
evaluation, if you then might do further evaluations depending on the
results of the first.

Any solution would be an unjust addition of expense to defaults.

On 1/14/07, Chris Rebert <cvrebert at gmail.com> wrote:
> If A.M. Kuchling's list of Python Warts is any indication, Python has
> removed many of the warts it once had. However, the behavior of mutable
> default argument values is still a frequent stumbling-block for newbies.
> It is also present on at least 3 different lists of Python's
> deficiencies ([0][1][2]).
>
> Example of current, unintuitive behavior (snipped from [0]):
>  >>> def popo(x=[]):
> ...     x.append(666)
> ...     print x
> ...
>  >>> popo()
> [666]
>  >>> popo()
> [666, 666]
>  >>> popo()
> [666, 666, 666]
>
> Whereas a newbie with experience with immutable default argument values
> would, by analogy, expect:
>  >>> popo()
> [666]
>  >>> popo()
> [666]
>  >>> popo()
> [666]
>
> In scanning [0], [1], [2], and other similar lists, I have only found
> one mediocre use-case for this behavior: Using the default argument
> value to retain state between calls. However, as [2] comments, this
> purpose is much better served by decorators, classes, or (though less
> preferred) global variables. Other uses are alluded to be equally
> esoteric and unpythonic.
>
> To work around this behavior, the following idiom is used:
> def popo(x=None):
>      if x is None:
>          x = []
>      x.append(666)
>      print x
>
> However, why should the programmer have to write this extra boilerplate
> code when the current, unusual behavior is only relied on by 1% of
> Python code?
>
> Therefore, I propose that default arguments be handled as follows in Py3K:
> 1. The initial default value is evaluated at definition-time (as in the
> current behavior).
> 2. That in a function call where the caller has not specified a value
> for an optional argument, Python calls
> copy.deepcopy(initial_default_value), and fills in the optional argument
> with the resulting value.
>
> This is fully backwards-compatible with the aforementioned workaround,
> and removes the need for the it, allowing one to write the first,
> simpler definition of popo().
>
> Comments?
>
> - Chris Rebert
>
>
> [0] 10 Python pitfalls (http://zephyrfalcon.org/labs/python_pitfalls.html)
> [1] Python Gotchas
> (http://www.ferg.org/projects/python_gotchas.html#contents_item_6)
> [2] When Pythons Attack
> (http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://ironfroggy-code.blogspot.com/


From cvrebert at gmail.com  Wed Jan 17 03:49:14 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Tue, 16 Jan 2007 18:49:14 -0800
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <76fd5acf0701141858w7e87a67h3eb477f099e022bf@mail.gmail.com>
References: <45AAD68D.5010502@gmail.com>
	<76fd5acf0701141858w7e87a67h3eb477f099e022bf@mail.gmail.com>
Message-ID: <45AD8EAA.6040902@gmail.com>

Calvin Spealman wrote:
> There are a few problems here. Deep copy of the originally created
> default argument can be expensive
For immutables, yes, it would make an unnecessary copy, though it seems 
less likely to have complex immutable objects than complex mutable 
objects as defaults.

> and would not work in any useful way
> with non-literals as defaults, such as function calls or subscript
> lookups or even simple attributes.
Good point, I hadn't considered that.

> If any solution is possible, it would require a way to differentiate
> between mutable and immutable objects, and evaluate the immutables for
> every call.
That doesn't seem to consider the other cases you just mentioned, though 
I see merit in your idea. How about there be some way to specify that a 
default argument value be re-evaluated at every call it's required for, 
while all other arguments have existing semantics?

Hypothetical syntax:

def foo(a, b=4, c=<Bar([q, w, e], 7)>):
     #b's default value is evaluated exactly once, at definition-time
     #c's default value is evaluated every time foo() is called and no 
value for c is given

Where the <>s indicate these special semantics.

> Any solution would be an unjust addition of expense to defaults.
That sounds a bit like premature optimization to me.

- Chris Rebert


From jjb5 at cornell.edu  Wed Jan 17 15:19:19 2007
From: jjb5 at cornell.edu (Joel Bender)
Date: Wed, 17 Jan 2007 09:19:19 -0500
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <45AD8EAA.6040902@gmail.com>
References: <45AAD68D.5010502@gmail.com>	<76fd5acf0701141858w7e87a67h3eb477f099e022bf@mail.gmail.com>
	<45AD8EAA.6040902@gmail.com>
Message-ID: <45AE3067.2080605@cornell.edu>

> Where the <>s indicate these special semantics.

The syntax bothers me, but launched an idea: How about separating 
parameter default value issues from instance specific definition time 
objects?

     def foo(a, b=4, c=None):
         local d = Bar([2,3,4])
         ...

The function code can decide if it wants to use c, provided by the 
caller, or d when no value for c is given.  The developer may decide 
that None is not a good value for 'no value for c', but that's a design 
decision.

You can do this now with:

     def foo(a, b=4, c=None):
         ...

     foo.func_dict['d'] = Bar([2,3,4])

But I would rather see this inside foo(), jamming variables into the 
func_dict bothers me too :-).

The new keyword would work for classes, but be a functional noop:

     class Snorf:
         local eggs = 3
         spam = 4


Joel


From ironfroggy at gmail.com  Thu Jan 18 05:10:30 2007
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Wed, 17 Jan 2007 23:10:30 -0500
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <45AE3067.2080605@cornell.edu>
References: <45AAD68D.5010502@gmail.com>
	<76fd5acf0701141858w7e87a67h3eb477f099e022bf@mail.gmail.com>
	<45AD8EAA.6040902@gmail.com> <45AE3067.2080605@cornell.edu>
Message-ID: <76fd5acf0701172010j554fa420q38f09151d46883af@mail.gmail.com>

On 1/17/07, Joel Bender <jjb5 at cornell.edu> wrote:
> > Where the <>s indicate these special semantics.
>
> The syntax bothers me, but launched an idea: How about separating
> parameter default value issues from instance specific definition time
> objects?
>
>      def foo(a, b=4, c=None):
>          local d = Bar([2,3,4])
>          ...
>
> The function code can decide if it wants to use c, provided by the
> caller, or d when no value for c is given.  The developer may decide
> that None is not a good value for 'no value for c', but that's a design
> decision.

I dont understand how that would be different than doing

   c = c if c is not None else Bar([2,3,4])

> You can do this now with:
>
>      def foo(a, b=4, c=None):
>          ...
>
>      foo.func_dict['d'] = Bar([2,3,4])

This would not really work in practice. See this:

>>> def f():
...     print a
...
>>> f.func_dict['a'] = 10
>>> f()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in f
NameError: global name 'a' is not defined

> But I would rather see this inside foo(), jamming variables into the
> func_dict bothers me too :-).
>
> The new keyword would work for classes, but be a functional noop:
>
>      class Snorf:
>          local eggs = 3
>          spam = 4
>
>
> Joel
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://ironfroggy-code.blogspot.com/


From jjb5 at cornell.edu  Thu Jan 18 14:44:12 2007
From: jjb5 at cornell.edu (Joel Bender)
Date: Thu, 18 Jan 2007 08:44:12 -0500
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <76fd5acf0701172010j554fa420q38f09151d46883af@mail.gmail.com>
References: <45AAD68D.5010502@gmail.com>	
	<76fd5acf0701141858w7e87a67h3eb477f099e022bf@mail.gmail.com>	
	<45AD8EAA.6040902@gmail.com> <45AE3067.2080605@cornell.edu>
	<76fd5acf0701172010j554fa420q38f09151d46883af@mail.gmail.com>
Message-ID: <45AF79AC.9090509@cornell.edu>

Calvin Spealman wrote:

> I dont understand how that would be different than doing
> 
>   c = c if c is not None else Bar([2,3,4])
> 

Because that would be calling Bar(), perhaps creating a new Bar object, 
every time foo() is called with None for c, which is not what a default 
argument values are about.  I'm proposing a way to create function local 
singleton objects, removing them from the parameter list.

     def foo(x):
         local history = []
         history.append(x)

Rather than:

     def foo(x, history=[]):
         history.append(x)

and then hoping that nobody calls foo() with a history parameter.


Joel



From jcarlson at uci.edu  Thu Jan 18 17:42:00 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Thu, 18 Jan 2007 08:42:00 -0800
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <45AF79AC.9090509@cornell.edu>
References: <76fd5acf0701172010j554fa420q38f09151d46883af@mail.gmail.com>
	<45AF79AC.9090509@cornell.edu>
Message-ID: <20070118084035.5974.JCARLSON@uci.edu>


Joel Bender <jjb5 at cornell.edu> wrote:
> 
> Calvin Spealman wrote:
> 
> > I dont understand how that would be different than doing
> > 
> >   c = c if c is not None else Bar([2,3,4])
> > 
> 
> Because that would be calling Bar(), perhaps creating a new Bar object, 
> every time foo() is called with None for c, which is not what a default 
> argument values are about.  I'm proposing a way to create function local 
> singleton objects, removing them from the parameter list.
[snip]
> and then hoping that nobody calls foo() with a history parameter.

With your proposal, you are seeking to attempt to fix a "problem" that
no one has complained about.  -1 .


 - Josiah



From jcarlson at uci.edu  Thu Jan 18 17:47:14 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Thu, 18 Jan 2007 08:47:14 -0800
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <45AAD68D.5010502@gmail.com>
References: <45AAD68D.5010502@gmail.com>
Message-ID: <20070118084211.5977.JCARLSON@uci.edu>


Chris Rebert <cvrebert at gmail.com> wrote:
> If A.M. Kuchling's list of Python Warts is any indication, Python has 
> removed many of the warts it once had. However, the behavior of mutable 
> default argument values is still a frequent stumbling-block for newbies. 
> It is also present on at least 3 different lists of Python's 
> deficiencies ([0][1][2]).
> 
> Example of current, unintuitive behavior (snipped from [0]):
>  >>> def popo(x=[]):
> ...     x.append(666)
> ...     print x
> ...
>  >>> popo()
> [666]
>  >>> popo()
> [666, 666]
>  >>> popo()
> [666, 666, 666]
[snip]
> Comments?

As provided by Calvin Spealman, the above can be fixed with:

    def popo(x=None):
        x = x if x is not None else []
        x.append(666)
        print x

I would also mention that forcing users to learn about mutable arguments
and procedural programming is not a bad thing.  Learning the "gotcha"
of mutable default arguments is a very useful lesson, and to remove that
lesson, I believe, wouldn't necessarily help new users to Python, or new
programmers in general.

 - Josiah



From jimjjewett at gmail.com  Thu Jan 18 20:00:12 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Thu, 18 Jan 2007 14:00:12 -0500
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <20070118084035.5974.JCARLSON@uci.edu>
References: <76fd5acf0701172010j554fa420q38f09151d46883af@mail.gmail.com>
	<45AF79AC.9090509@cornell.edu> <20070118084035.5974.JCARLSON@uci.edu>
Message-ID: <fb6fbf560701181100q361c7098if6c60b9c8afbc603@mail.gmail.com>

On 1/18/07, Josiah Carlson <jcarlson at uci.edu> wrote:

> Joel Bender <jjb5 at cornell.edu> wrote:
> > Calvin Spealman wrote:
> > > I dont understand how that would be different than doing

> > >   c = c if c is not None else Bar([2,3,4])

But he actually ones a variable that *does* keep state between calls
(like a mutable default arg), but can't be overridden.

> With your proposal, you are seeking to attempt to fix a "problem" that
> no one has complained about.  -1 .

Sure they have, and they've solved it (under different names) in
plenty of other languages.  In python, the only current solution seems
to be turning the function into a class (with self) or at least a
closure.  People have griped about this.

For What Its Worth, my personal opinion is that having to create an
object instead of a function is annoying, but not so bad (or so
frequent) that it is worth special syntax.

-jJ


From gsakkis at rutgers.edu  Thu Jan 18 21:39:01 2007
From: gsakkis at rutgers.edu (George Sakkis)
Date: Thu, 18 Jan 2007 15:39:01 -0500
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <fb6fbf560701181100q361c7098if6c60b9c8afbc603@mail.gmail.com>
References: <76fd5acf0701172010j554fa420q38f09151d46883af@mail.gmail.com>
	<45AF79AC.9090509@cornell.edu> <20070118084035.5974.JCARLSON@uci.edu>
	<fb6fbf560701181100q361c7098if6c60b9c8afbc603@mail.gmail.com>
Message-ID: <91ad5bf80701181239p1aa710afv4401861257d2eb4f@mail.gmail.com>

On 1/18/07, Jim Jewett <jimjjewett at gmail.com> wrote:

> On 1/18/07, Josiah Carlson <jcarlson at uci.edu> wrote:
>
> > Joel Bender <jjb5 at cornell.edu> wrote:
> > > Calvin Spealman wrote:
> > > > I dont understand how that would be different than doing
>
> > > >   c = c if c is not None else Bar([2,3,4])
>
> But he actually ones a variable that *does* keep state between calls
> (like a mutable default arg), but can't be overridden.
>
> > With your proposal, you are seeking to attempt to fix a "problem" that
> > no one has complained about.  -1 .
>
> Sure they have, and they've solved it (under different names) in
> plenty of other languages.  In python, the only current solution seems
> to be turning the function into a class (with self) or at least a
> closure.  People have griped about this.

User-defined function attributes is another handy solution.

> For What Its Worth, my personal opinion is that having to create an
> object instead of a function is annoying, but not so bad (or so
> frequent) that it is worth special syntax.

Function attributes fit the bill really good if writing a class is too
much overhead.

George


From jimjjewett at gmail.com  Thu Jan 18 23:12:52 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Thu, 18 Jan 2007 17:12:52 -0500
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <91ad5bf80701181239p1aa710afv4401861257d2eb4f@mail.gmail.com>
References: <76fd5acf0701172010j554fa420q38f09151d46883af@mail.gmail.com>
	<45AF79AC.9090509@cornell.edu> <20070118084035.5974.JCARLSON@uci.edu>
	<fb6fbf560701181100q361c7098if6c60b9c8afbc603@mail.gmail.com>
	<91ad5bf80701181239p1aa710afv4401861257d2eb4f@mail.gmail.com>
Message-ID: <fb6fbf560701181412k2a4463f3u9ce2e6aaebe8fb90@mail.gmail.com>

On 1/18/07, George Sakkis <gsakkis at rutgers.edu> wrote:
> On 1/18/07, Jim Jewett <jimjjewett at gmail.com> wrote:

> > But he actually ones a variable that *does* keep state between calls
> > (like a mutable default arg), but can't be overridden.

> > For What Its Worth, my personal opinion is that having to create an
> > object instead of a function is annoying, but not so bad (or so
> > frequent) that it is worth special syntax.

> Function attributes fit the bill really good if writing a class is too
> much overhead.

Not really, because Python doesn't have the equivalent of "this".  The
only way for a function to access its own attributes is to hardcode a
name and to assume the name will always refer to that same function
object.

In practice, it mostly works, but so does just using a global variable.

-jJ


From stephenemslie at gmail.com  Fri Jan 19 18:08:17 2007
From: stephenemslie at gmail.com (stephen emslie)
Date: Fri, 19 Jan 2007 17:08:17 +0000
Subject: [Python-ideas] tab completion in pdb
Message-ID: <51f97e530701190908l7c0edff0r866ce45fac914166@mail.gmail.com>

I make frequent use of the python's built-in debugger, which I think is
brilliant in its simplicity. However an important feature seems to be
missing: bash-like tab completion similar to that provided by the
rlcompleter module.

By default, Pdb and other instances of Cmd complete names for commands only.
However in the context of pdb, I think it is more useful to complete
identifiers and keywords in its current scope than to complete names of
commands (most of which have single letter abbreviations). I believe this
makes pdb a far more usable introspection tool.

Implementation:

I've attached a patch to pdb.py (on Python 2.4.4c1). The only real
difference to rlcompleter's default complete method is that because pdb
changes scope as you step through a program, rlcompleter's namespace is
updated to reflect the current local and global namespace.

This is my first attempt at a python patch. Any suggestions or improvements
are welcome.

Stephen Emslie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20070119/4acecaed/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pdb-python2.4.4-tabcomplete.patch
Type: text/x-patch
Size: 798 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20070119/4acecaed/attachment.bin>

From aahz at pythoncraft.com  Fri Jan 19 18:38:04 2007
From: aahz at pythoncraft.com (Aahz)
Date: Fri, 19 Jan 2007 09:38:04 -0800
Subject: [Python-ideas] tab completion in pdb
In-Reply-To: <51f97e530701190908l7c0edff0r866ce45fac914166@mail.gmail.com>
References: <51f97e530701190908l7c0edff0r866ce45fac914166@mail.gmail.com>
Message-ID: <20070119173804.GA8679@panix.com>

On Fri, Jan 19, 2007, stephen emslie wrote:
>
> I've attached a patch to pdb.py (on Python 2.4.4c1). The only real
> difference to rlcompleter's default complete method is that because pdb
> changes scope as you step through a program, rlcompleter's namespace is
> updated to reflect the current local and global namespace.
> 
> This is my first attempt at a python patch. Any suggestions or improvements
> are welcome.

Please go ahead and upload this patch to SourceForge to make sure it
doesn't get lost.  Thanks!
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

Help a hearing-impaired person: http://rule6.info/hearing.html


From cvrebert at gmail.com  Sun Jan 21 07:09:46 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Sat, 20 Jan 2007 22:09:46 -0800
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <20070118084211.5977.JCARLSON@uci.edu>
References: <45AAD68D.5010502@gmail.com> <20070118084211.5977.JCARLSON@uci.edu>
Message-ID: <45B303AA.1010707@gmail.com>

Josiah Carlson wrote:
> As provided by Calvin Spealman, the above can be fixed with:
> 
>     def popo(x=None):
>         x = x if x is not None else []
>         x.append(666)
>         print x
> 
> I would also mention that forcing users to learn about mutable arguments
> and procedural programming is not a bad thing.  Learning the "gotcha"
> of mutable default arguments is a very useful lesson, and to remove that
> lesson, I believe, wouldn't necessarily help new users to Python, or new
> programmers in general.
> 
>  - Josiah
> 
First, your 'fix' misses the point: though the proposed feature isn't 
necessary, and code can be written without using it, it allows mutable 
default argument values to be expressed more clearly and succinctly than 
the idiom your 'fix' uses. Second, Python isn't (inherently) about 
teaching new programmers about programming, and what is good for newbies 
isn't necessarily good for experienced programmers. And at any rate, the 
lesson would still exist in the form of having to use the new feature I 
proposed (in my strawman syntax, 
<DefaultArgumentValueThatGetsReEvaluated()>), and also in doing 
multiplication on 2D lists (e.g. x = [mutable]*42; x[7].mutate(); 
x[0].mutated == True).

Comments (from anyone) on my revised proposal (from the 2nd email)? I 
would particularly appreciate alternate syntax suggestions.

- Chris Rebert


From jcarlson at uci.edu  Mon Jan 22 01:49:51 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Sun, 21 Jan 2007 16:49:51 -0800
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <45B303AA.1010707@gmail.com>
References: <20070118084211.5977.JCARLSON@uci.edu> <45B303AA.1010707@gmail.com>
Message-ID: <20070121162511.59B9.JCARLSON@uci.edu>


Chris Rebert <cvrebert at gmail.com> wrote:
> Josiah Carlson wrote:
> > As provided by Calvin Spealman, the above can be fixed with:
> > 
> >     def popo(x=None):
> >         x = x if x is not None else []
> >         x.append(666)
> >         print x
> > 
> > I would also mention that forcing users to learn about mutable arguments
> > and procedural programming is not a bad thing.  Learning the "gotcha"
> > of mutable default arguments is a very useful lesson, and to remove that
> > lesson, I believe, wouldn't necessarily help new users to Python, or new
> > programmers in general.
> > 
> >  - Josiah

Maybe you are taking me a bit too seriously, but hopefully this will add
some levity; I'm a poo-poo head.  Moving on...


> First, your 'fix' misses the point: though the proposed feature isn't 
> necessary, and code can be written without using it, it allows mutable 
> default argument values to be expressed more clearly and succinctly than 
> the idiom your 'fix' uses.

As I stated, it wasn't my fix.  And using previously existing syntax
that adds 1 line to a function to support a particular desired result, I
think, is perfectly reasonable.  Had the conditional syntax been
available for the last decade, those "gotchas" pages would have said
"mutable default arguments are tricky, always use the following, and it
will probably be the right thing" and moved on.

> Second, Python isn't (inherently) about 
> teaching new programmers about programming, and what is good for newbies 
> isn't necessarily good for experienced programmers.

Indeed, and what *may* be good for *certain* experienced programmers,
may not be good for other experienced programmers, or for the language
in general.  And personally, I am not sure that I could be convinced
that a syntax to support what can be emulated by a single line is even
worthy of addition.  In the case of decorators, or even the py3k support
for argument annotation, there are certain operations that can be made
*significantly* easier.  In this case, I'm not convinced that the extra
syntax baggage is worthwhile.

Nevermind that it would be one more incompatible syntax that would make
it difficult to write for 2.5/2.6 and 3.x .


 - Josiah



From stephenemslie at gmail.com  Mon Jan 22 12:57:38 2007
From: stephenemslie at gmail.com (stephen emslie)
Date: Mon, 22 Jan 2007 11:57:38 +0000
Subject: [Python-ideas] tab completion in pdb
In-Reply-To: <20070119173804.GA8679@panix.com>
References: <51f97e530701190908l7c0edff0r866ce45fac914166@mail.gmail.com>
	<20070119173804.GA8679@panix.com>
Message-ID: <51f97e530701220357sdc7c1d8nab5dfebca2aae605@mail.gmail.com>

Thanks for taking a look. I've created a patch relative to pdb.py in svn and
submitted it to sourceforge here:

http://sourceforge.net/tracker/index.php?func=detail&aid=1641544&group_id=5470&atid=305470

On 1/19/07, Aahz <aahz at pythoncraft.com> wrote:
>
> On Fri, Jan 19, 2007, stephen emslie wrote:
> >
> > I've attached a patch to pdb.py (on Python 2.4.4c1). The only real
> > difference to rlcompleter's default complete method is that because pdb
> > changes scope as you step through a program, rlcompleter's namespace is
> > updated to reflect the current local and global namespace.
> >
> > This is my first attempt at a python patch. Any suggestions or
> improvements
> > are welcome.
>
> Please go ahead and upload this patch to SourceForge to make sure it
> doesn't get lost.  Thanks!
> --
> Aahz (aahz at pythoncraft.com)           <*>
> http://www.pythoncraft.com/
>
> Help a hearing-impaired person: http://rule6.info/hearing.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20070122/2931090e/attachment.html>

From ironfroggy at gmail.com  Tue Jan 23 07:10:21 2007
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Tue, 23 Jan 2007 01:10:21 -0500
Subject: [Python-ideas] oneof() and multi split and replace for stirngs
Message-ID: <76fd5acf0701222210p41f68c3asd79b03f0e508d6c7@mail.gmail.com>

I don't know if this would make sense to try to push to 2.6 or 3.0. I
was talking with some people about how the ability to split or replace
on multiple substrings would be added to python, without adding new
methods or having ugly tuple passing requirents like s.split(('foo',
'bar'), 4). This idea came to mind, so I wanted to toss it out there
for scrutination. It would be a builtin, but can be implemented in
python like this, basically:

class oneof(list):
    def __init__(self, *args):
        list.__init__(self)
        self.extend(args)
    def __eq__(self, o):
        return o in self

assert 'bar' == oneof('bar', 'baz')


In addition to the new type, .replace, .split, and other appropriate
functions would be updated to take this as the substring argument to
locate and would match any one of the substrings it contains. I've
asked a few people and gotten good responses on the general idea so
far, but what do you all think?

1) Would the multi-substring operations be welcomed?
2) Could this be a good way to add those to the API without breaking things?
3) What version would it target?
4) What all functions and methods should support this or generally
might gain value from some similar solution?

-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://ironfroggy-code.blogspot.com/


From jcarlson at uci.edu  Tue Jan 23 09:29:47 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Tue, 23 Jan 2007 00:29:47 -0800
Subject: [Python-ideas] oneof() and multi split and replace for stirngs
In-Reply-To: <76fd5acf0701222210p41f68c3asd79b03f0e508d6c7@mail.gmail.com>
References: <76fd5acf0701222210p41f68c3asd79b03f0e508d6c7@mail.gmail.com>
Message-ID: <20070123000433.59CC.JCARLSON@uci.edu>


"Calvin Spealman" <ironfroggy at gmail.com> wrote:
> I don't know if this would make sense to try to push to 2.6 or 3.0. I
> was talking with some people about how the ability to split or replace
> on multiple substrings would be added to python, without adding new
> methods or having ugly tuple passing requirents like s.split(('foo',
> 'bar'), 4). This idea came to mind, so I wanted to toss it out there
> for scrutination. It would be a builtin, but can be implemented in
> python like this, basically:

Whether it is a tuple being passed, or a "magic" container, I don't
think it matters; though I would lean towards a tuple because it is 5
less characters to type out, and one fewer data types to worry about.

> class oneof(list):
>     def __init__(self, *args):
>         list.__init__(self)
>         self.extend(args)
>     def __eq__(self, o):
>         return o in self
> 
> assert 'bar' == oneof('bar', 'baz')

If all you wanted to test was containment, it would be faster (in terms
of string searching) to use a set...and perhaps just use containment.

    class oneof(set):
        def __init__(self, *args):
            set.__init__(self)
            set.update(args)

    assert 'bar' in oneof('bar', 'baz')

The underlying string search implementation probably shouldn't concern
itself with specific types.  Though there are some ... gotchas if one
were to provide it with an iterator that isn't restartable.

> 1) Would the multi-substring operations be welcomed?

This has been discussed before in python-dev, I believe the general
consensus was that it would be convenient at times, but I also believe
the general consensus was "use re";

    def replace(options, becomes, source, count=0):
        pattern = '(%s)'%('|'.join(re.escape(i) for i in options))
        return re.sub(pattern, becomes, source, count)

> 2) Could this be a good way to add those to the API without breaking things?

No.  Adding a data structure that is so simple just to offer
multi-substring search, replace, split, etc., isn't worthwhile. Use a
tuple.

> 3) What version would it target?

If it were to be added, I would say 3.x.  The fewer API alterations in
2.5 -> 2.6, the better.

> 4) What all functions and methods should support this or generally
> might gain value from some similar solution?

If any, split.  Beyond that, it smells to me like a regular expression
job (though split still smells like a regular expression job to me).


 - Josiah



From ironfroggy at gmail.com  Tue Jan 23 09:44:59 2007
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Tue, 23 Jan 2007 03:44:59 -0500
Subject: [Python-ideas] oneof() and multi split and replace for stirngs
In-Reply-To: <76fd5acf0701230044t6fd28878m822bf482c156839@mail.gmail.com>
References: <76fd5acf0701222210p41f68c3asd79b03f0e508d6c7@mail.gmail.com>
	<20070123000433.59CC.JCARLSON@uci.edu>
	<76fd5acf0701230044t6fd28878m822bf482c156839@mail.gmail.com>
Message-ID: <76fd5acf0701230044j6381e1e3n97ab0df3e99be018@mail.gmail.com>

On 1/23/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> Whether it is a tuple being passed, or a "magic" container, I don't
> think it matters; though I would lean towards a tuple because it is 5
> less characters to type out, and one fewer data types to worry about.

I had talked to others and the concensus was against the tuples for
ugliness. s.split((a,b), c) wasn't a popular choice, but
s.split(oneof(a, b), c) reads better.

> This has been discussed before in python-dev, I believe the general
> consensus was that it would be convenient at times, but I also believe
> the general consensus was "use re";

It seems like we have a history of useful string operations being
moved away from "use re" to "dont use re", such that the slightly
recent startswith and endswith methods, and even split and replace
themselves. I would like to see less reasons for people to worry with
regular expressions until they actually need them. If we can provide a
better way to get the job done, that seems like a great idea.

ALSO

The oneof type isn't just a single use thing. Usecode may often make
use of it, and other types could benefit such as doing a lookup with
d[oneof(1,2,3)] (where order would matter for priority). I think this
semantic collection type would be very useful in a number of contexts
where we would currently just loop or duplicate code.

-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://ironfroggy-code.blogspot.com/


From jcarlson at uci.edu  Tue Jan 23 18:32:02 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Tue, 23 Jan 2007 09:32:02 -0800
Subject: [Python-ideas] oneof() and multi split and replace for stirngs
In-Reply-To: <76fd5acf0701230044j6381e1e3n97ab0df3e99be018@mail.gmail.com>
References: <76fd5acf0701230044t6fd28878m822bf482c156839@mail.gmail.com>
	<76fd5acf0701230044j6381e1e3n97ab0df3e99be018@mail.gmail.com>
Message-ID: <20070123090336.59CF.JCARLSON@uci.edu>


"Calvin Spealman" <ironfroggy at gmail.com> wrote:
> 
> On 1/23/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> > Whether it is a tuple being passed, or a "magic" container, I don't
> > think it matters; though I would lean towards a tuple because it is 5
> > less characters to type out, and one fewer data types to worry about.
> 
> I had talked to others and the concensus was against the tuples for
> ugliness. s.split((a,b), c) wasn't a popular choice, but
> s.split(oneof(a, b), c) reads better.

oneof = tuple

Now it reads better.  Stick with tuple.

> > This has been discussed before in python-dev, I believe the general
> > consensus was that it would be convenient at times, but I also believe
> > the general consensus was "use re";
> 
> It seems like we have a history of useful string operations being
> moved away from "use re" to "dont use re", such that the slightly
> recent startswith and endswith methods, and even split and replace
> themselves. I would like to see less reasons for people to worry with
> regular expressions until they actually need them. If we can provide a
> better way to get the job done, that seems like a great idea.

Well, startswith and endswith is still computable without re or the
methods, and turns out to be faster (for certain startswith and endswith
operations) to do...

    if st[:X] == sub:
or
    if st[-X:] == sub:

(more specifically, when sub is a constant and X is previously-known)

> The oneof type isn't just a single use thing. Usecode may often make
> use of it, and other types could benefit such as doing a lookup with
> d[oneof(1,2,3)] (where order would matter for priority). I think this
> semantic collection type would be very useful in a number of contexts
> where we would currently just loop or duplicate code.

I took the time to look for a thread discussing something like this
feature before.  And I found it.  Turns out I was wrong:
    http://mail.python.org/pipermail/python-dev/2005-September/056119.html

That thread discusses an enhancement to str.[r|l|]strip(), where one
could specify a list of strings to be trimmed from the end(s), not only
just a group of characters (does the equivalent of .startswith and
.endswith).

What was the outcome?  Use the X-line function that can be defined as Y. 
I'll stick with "use re".


 - Josiah



From rrr at ronadam.com  Tue Jan 23 21:23:02 2007
From: rrr at ronadam.com (Ron Adam)
Date: Tue, 23 Jan 2007 14:23:02 -0600
Subject: [Python-ideas] oneof() and multi split and replace for stirngs
In-Reply-To: <76fd5acf0701222210p41f68c3asd79b03f0e508d6c7@mail.gmail.com>
References: <76fd5acf0701222210p41f68c3asd79b03f0e508d6c7@mail.gmail.com>
Message-ID: <45B66EA6.5080108@ronadam.com>

Calvin Spealman wrote:
> I don't know if this would make sense to try to push to 2.6 or 3.0. I
> was talking with some people about how the ability to split or replace
> on multiple substrings would be added to python, without adding new
> methods or having ugly tuple passing requirents like s.split(('foo',
> 'bar'), 4). This idea came to mind, so I wanted to toss it out there
> for scrutination. It would be a builtin, but can be implemented in
> python like this, basically:
> 
> class oneof(list):
>     def __init__(self, *args):
>         list.__init__(self)
>         self.extend(args)
>     def __eq__(self, o):
>         return o in self
> 
> assert 'bar' == oneof('bar', 'baz')

          'bar' in 'bar baz'  ?



> In addition to the new type, .replace, .split, and other appropriate
> functions would be updated to take this as the substring argument to
> locate and would match any one of the substrings it contains. I've
> asked a few people and gotten good responses on the general idea so
> far, but what do you all think?

> 1) Would the multi-substring operations be welcomed?
> 2) Could this be a good way to add those to the API without breaking things?
> 3) What version would it target?
> 4) What all functions and methods should support this or generally
> might gain value from some similar solution?

It doesn't feel right to me to have this as a built in.  This falls in the 
category of mid level functionality, but I'm never sure where to draw the line 
between having simple objects to do more complex things on, vs more complex 
objects.  I tend to prefer the first case.

Having more functions to do more complex, but common, things to strings and 
lists of strings would be nice to have in the library.

Examples of this are fnmatch.py, and textwrap.py.  These both use re to do the 
work, but present an easier to use interface.

The functions in textwrap could be extended to accept 'lists of lines'. And 
functions to do justifying, right and full, might also be useful.

Having a simpler word matching alternative to do web style multi term searches 
on lists of strings, would be nice.  (Something I could use to handle search 
requests right now.)  It could be designed to be presentable to users like 
fnmatch.  The simpler pattern matching would work in many programming situations 
as well.  Like fnmatch, and textwrap, it would use re to do the actual work.

Functions like split_pattern(), replace_pattern(), and partition_pattern() could 
also be available in the same module to handle the cases you suggested.

Cheers,
   Ron



From jan.kanis at phil.uu.nl  Thu Jan 25 02:03:38 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Thu, 25 Jan 2007 02:03:38 +0100
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <20070121162511.59B9.JCARLSON@uci.edu>
References: <20070118084211.5977.JCARLSON@uci.edu> <45B303AA.1010707@gmail.com>
	<20070121162511.59B9.JCARLSON@uci.edu>
Message-ID: <op.tmoqkcmrd64u53@e500>

I don't like new syntax for something like this, but I think the default  
argument values can be fixed with semantic changes (which should not break  
the most common current uses):

What I think should happen is compile a function like this

def popo(x=[]):
     x.append(666)
     print x

as if it had read

def popo(x=__default_argument_marker__):
     if x == __default_argument_marker__:
         x = []
     x.append(666)
     print x

This way, every execution of popo gets its own list. Of course,  
__default_argument_marker__ is just a way to tell the python runtime that  
no argument was provided, it should not be exposed to the language.

If a variable is used in the default argument, it becomes a closure  
variable:

d = createMyListOfLists()
n = getDefaultIndex()

def foo(x=d[n]):
   x.append(666)
   print x


this is compiled as if it had read

d = createMyListOfLists()
n = getDefaultIndex()

def foo(x=__default_argument_marker__):
   if x == __default_argument_marker__:
     x = d[n]      # d and n are closure variables
   x.append(666)
   print x


d and n are looked up in foo's parent scope, which in this example is the  
global scope. Of course the bytecode compiler should make sure d and n  
don't name-clash with any variables used in the body of foo.

When you use variables as default value instead of literals, I think most  
of the time you intend to have the function do something to the same  
object the variable is bound to, instead of the function creating it's own  
copy every time it's called. This behaviour still works with these  
semantics:

>>> a = []
>>>def foo(x=[[],a]):
>>>   x[0].append(123)
>>>   x[1].append(123)
>>>   print x
>>>foo()
[[123], [123]]
>>> foo()
[[123], [123, 123]]
>>> foo()
[[123], [123, 123, 123]]

foo is compiled as if it had read:

def foo(x=__default_argument_marker__):
   if x == __default_argument_marker__:
     x = [[],a]    # a is a closure variable
   x[0].append(123)
   x[1].append(123)
   print x

An other difference between this proposal and the current situation is  
that it would be possible to change the value of a default argument after  
the function is defined. However I don't think that would really be a  
problem, and this behaviour is just the same as that of other closure  
variables. Besides, this (what I perceive as a) problem with closure  
variables is fixable on its own.

Jan


On Mon, 22 Jan 2007 01:49:51 +0100, Josiah Carlson <jcarlson at uci.edu>  
wrote:

>
> Chris Rebert <cvrebert at gmail.com> wrote:
>> Josiah Carlson wrote:
>> > As provided by Calvin Spealman, the above can be fixed with:
>> >
>> >     def popo(x=None):
>> >         x = x if x is not None else []
>> >         x.append(666)
>> >         print x
>> >
>> > I would also mention that forcing users to learn about mutable  
>> arguments
>> > and procedural programming is not a bad thing.  Learning the "gotcha"
>> > of mutable default arguments is a very useful lesson, and to remove  
>> that
>> > lesson, I believe, wouldn't necessarily help new users to Python, or  
>> new
>> > programmers in general.
>> >
>> >  - Josiah
>
> Maybe you are taking me a bit too seriously, but hopefully this will add
> some levity; I'm a poo-poo head.  Moving on...
>
>
>> First, your 'fix' misses the point: though the proposed feature isn't
>> necessary, and code can be written without using it, it allows mutable
>> default argument values to be expressed more clearly and succinctly than
>> the idiom your 'fix' uses.
>
> As I stated, it wasn't my fix.  And using previously existing syntax
> that adds 1 line to a function to support a particular desired result, I
> think, is perfectly reasonable.  Had the conditional syntax been
> available for the last decade, those "gotchas" pages would have said
> "mutable default arguments are tricky, always use the following, and it
> will probably be the right thing" and moved on.
>
>> Second, Python isn't (inherently) about
>> teaching new programmers about programming, and what is good for newbies
>> isn't necessarily good for experienced programmers.
>
> Indeed, and what *may* be good for *certain* experienced programmers,
> may not be good for other experienced programmers, or for the language
> in general.  And personally, I am not sure that I could be convinced
> that a syntax to support what can be emulated by a single line is even
> worthy of addition.  In the case of decorators, or even the py3k support
> for argument annotation, there are certain operations that can be made
> *significantly* easier.  In this case, I'm not convinced that the extra
> syntax baggage is worthwhile.
>
> Nevermind that it would be one more incompatible syntax that would make
> it difficult to write for 2.5/2.6 and 3.x .
>
>
>  - Josiah
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas




From jimjjewett at gmail.com  Thu Jan 25 15:41:54 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Thu, 25 Jan 2007 09:41:54 -0500
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <op.tmoqkcmrd64u53@e500>
References: <20070118084211.5977.JCARLSON@uci.edu> <45B303AA.1010707@gmail.com>
	<20070121162511.59B9.JCARLSON@uci.edu> <op.tmoqkcmrd64u53@e500>
Message-ID: <fb6fbf560701250641k45f24f40qfea54fa07e6b40c4@mail.gmail.com>

On 1/24/07, Jan Kanis <jan.kanis at phil.uu.nl> wrote:
> I don't like new syntax for something like this, but I think the default
> argument values can be fixed with semantic changes (which should not break
> the most common current uses):
>
> What I think should happen is compile a function like this
>
> def popo(x=[]):
>      x.append(666)
>      print x
>
> as if it had read
>
> def popo(x=__default_argument_marker__):
>      if x == __default_argument_marker__:
>          x = []
>      x.append(666)
>      print x

How is this different from the x=None idiom of today?

    def f(inlist=None):
        if inlist is None:
            inlist=[]

The if (either 2 lines or against PEP8) is a bit ugly, but Calvin
pointed out that you can now write it as

    def f(inlist=None):
        inlist = inlist if (inlist is not None) else []

I see  below that you give it slightly different semantics, but I'm
not entirely sure how to tell when those different semantics should
apply (always?  when the variable name is marked with __*__?  When a
specific non-None singleton appears?), or why you would ever want
them.

> When you use variables as default value instead of literals, I think most
> of the time you intend to have the function do something to the same
> object the variable is bound to, instead of the function creating it's own
> copy every time it's called. This behaviour still works with these
> semantics:

> >>> a = []
> >>>def foo(x=[[],a]):
> >>>   x[0].append(123)
> >>>   x[1].append(123)
> >>>   print x
> >>>foo()
> [[123], [123]]
> >>> foo()
> [[123], [123, 123]]
> >>> foo()
> [[123], [123, 123, 123]]

So you're saying that x[1] should be persistent because it (also) has
a name (as 'a'), but x[0] should be recreated fresh on each call
because it doesn't?

-jJ


From tomerfiliba at gmail.com  Thu Jan 25 21:48:00 2007
From: tomerfiliba at gmail.com (tomer filiba)
Date: Thu, 25 Jan 2007 22:48:00 +0200
Subject: [Python-ideas] new pickle semantics/API
In-Reply-To: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com>
References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com>
Message-ID: <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com>

---------- Forwarded message ----------
From: tomer filiba <tomerfiliba at gmail.com>
Date: Jan 24, 2007 10:45 PM
Subject: new pickle semantics/API
To: Python-3000 at python.org

i'm having great trouble in RPyC with pickling object proxies.
several users have asked for this feature, but no matter how hard
i try to "bend the truth", pickle always complains. it uses
type(obj) for the dispatching, which "uncovers" the object
is actually a proxy, rather than a real object.

recap: RPyC uses local proxies that refer to objects of a
remote interpreter (another process/machine).

if you'd noticed, every RPC framework has its own serializer.
for example banna/jelly in twisted and bunch of other
XML serializers, and what not.

for RPyC i wrote yet another serializer, but for different purposes,
so it's not relevant for the issue at hand.

what i want is a standard serialization *API*. the idea is that
any framework could make use of this API, and that it would
be generic enough to eliminate copy_reg and other misfortunes.
this also means the built in types should be familiarized with
this API.

- - - - - - - -

for example, currently the builtin types don't support __reduce__,
and require pickle to use it's own internal registry. moreover,
__reduce__ is very pickle-specific (i.e., it takes the protocol number).
what i'm after is an API for "simplifying" complex objects into
simpler parts.

here's the API i'm suggesting:

def __getstate__(self):
    # return a tuple of (type(self), obj), where obj is a simplified
    # version of self

@classmethod
def __setstate__(cls, state):
    # return an instance of cls, with the given state

well, you may already know these two, although their
semantics are different. but wait, there's more!

the idea is of having the following simple building blocks:
* integers (int/long)
* strings (str)
* arrays (tuples)

all picklable objects should be able to express themselves
as a collection of these building blocks. of course this will be
recursive, i.e., object X could simplify itself as object Y,
where object Y might go further simplification, until we are
left with building blocks only.

for example:
* int - return self
* float - string in the format "[+-]X.YYYe[+-]EEE"
* complex - two floats
* tuple - tuple of its simplified elements
* list - tuple of its simplified elements
* dict - a tuple of (key, value) tuples
* set - a tuple of its items
* file - raises TypeError("can't be simplified")

all in all, i choose to call that *simplification* rather than
*serialization*,
as serialization is more about converting the simplified objects into a
sequence of bytes. my suggestion leaves that out for the
implementers of specific serializers.

so this is how a typical serializer (e.g., pickle) would be implemented:
* define its version of a "recursive simplifier"
* optionally use a "memo" to remember objects that were already
visited (so they would be serialized by reference rather than by value)
* define its flavor of converting ints, strings, and arrays to bytes
(binary, textual, etc. etc.)

- - - - - - - -

the default implementation of __getstate__, in object.__getstate__,
will simply return self.__dict__ and any self.__slots__

this removes the need for __reduce__, __reduce_ex__, and copy_reg,
and simplifies pickle greatly. it does require, however, adding
support for simplification for all builtin types... but this doesn't
call for much code:
    def PyList_GetState(self):
        state = tuple(PyObject_GetState(item) for item in self)
        return PyListType, state

also note that it makes the copy module much simpler:
    def copy(obj):
        state = obj.__getstate__()
        return type(obj).__setstate__(state)

- - - - - - - -

executive summary:
simplifying object serialization and copying by revising
__getstate__ and __setstate__, so that they return a
"simplified" version of the object.

this new mechanism should become an official API to
getting or setting the "contents" of objects (either builtin or
user-defined).

having this unified mechanism, pickling proxy objects would
work as expected.

if there's interest, i'll write a pep-like document to explain
all the semantics.



-tomer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20070125/6985eb2b/attachment.html>

From tomerfiliba at gmail.com  Thu Jan 25 22:15:04 2007
From: tomerfiliba at gmail.com (tomer filiba)
Date: Thu, 25 Jan 2007 23:15:04 +0200
Subject: [Python-ideas] new pickle semantics/API
In-Reply-To: <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com>
References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com>
	<1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com>
Message-ID: <1d85506f0701251315p2a447195g401e272861a75bdc@mail.gmail.com>

there's a bug in the copy function that i wanted to fix:
    def copy(obj):
        cls, state = obj.__getstate__()
        return cls.__setstate__(state)

also, but there's a reason why __getstate__ returns
"(cls, state)" rather than just "state", and that's to keep
things agile. i don't want to be necessarily tightly-coupled
to a certain type.

the cls will be used to reconstruct the object (cls.__setstate__),
much like the function returned by __reduce__, so normally,
__getstate__ would just return self.__class__ for cls.

but there are times, especially when object proxies are
involved, that we may want to "lie" about the actual type,
i.e., use a different type to reconstruct the object with.

here's an example that shows why:
    class ListProxy:
        ...
        def __getstate__(self):
            return list, self._value

instances of ListProxy, when stored in a file (i.e., shelf),
want to be pickled by value. moreover, when they are
loaded from a file, they want to loaded as actual lists,
not proxies, as the proxied object is long lost.

so returning a tuple of (cls, state) gives more freedom to
frameworks and other utilities. of course, again, most code
would just return (self.__class__, state)


-tomer


From collinw at gmail.com  Thu Jan 25 22:21:54 2007
From: collinw at gmail.com (Collin Winter)
Date: Thu, 25 Jan 2007 15:21:54 -0600
Subject: [Python-ideas] new pickle semantics/API
In-Reply-To: <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com>
References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com>
	<1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com>
Message-ID: <43aa6ff70701251321y8885892j28916f83b9a4d8c0@mail.gmail.com>

On 1/25/07, tomer filiba <tomerfiliba at gmail.com> wrote:
> for example:
> * int - return self
> * float - string in the format "[+-]X.YYYe[+-]EEE"
> * complex - two floats
> * tuple - tuple of its simplified elements
> * list - tuple of its simplified elements
> * dict - a tuple of (key, value) tuples
> * set - a tuple of its items
> * file - raises TypeError("can't be simplified")
[snip]
> the default implementation of __getstate__, in object.__getstate__,
> will simply return self.__dict__ and any self.__slots__

How will e.g. classes be simplified? Can I simplify a dictionary with
function objects for values?

Collin Winter


From jcarlson at uci.edu  Thu Jan 25 22:43:54 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Thu, 25 Jan 2007 13:43:54 -0800
Subject: [Python-ideas] new pickle semantics/API
In-Reply-To: <1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com>
References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com>
	<1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com>
Message-ID: <20070125130548.59F4.JCARLSON@uci.edu>


"tomer filiba" <tomerfiliba at gmail.com> wrote:
> i'm having great trouble in RPyC with pickling object proxies.
> several users have asked for this feature, but no matter how hard
> i try to "bend the truth", pickle always complains. it uses
> type(obj) for the dispatching, which "uncovers" the object
> is actually a proxy, rather than a real object.
[snip]
> also note that it makes the copy module much simpler:
>     def copy(obj):
>         state = obj.__getstate__()
>         return type(obj).__setstate__(state)

I presume you mean...

    def copy(obj):
        typ, state = obj.__getstate__()
        return typ.__setstate__(state)

> - - - - - - - -
> executive summary:
> simplifying object serialization and copying by revising
> __getstate__ and __setstate__, so that they return a
> "simplified" version of the object.
> 
> this new mechanism should become an official API to
> getting or setting the "contents" of objects (either builtin or
> user-defined).
> 
> having this unified mechanism, pickling proxy objects would
> work as expected.
> 
> if there's interest, i'll write a pep-like document to explain
> all the semantics.

Overall, I like the idea; I'm a big fan of simplifying object
persistence and/or serialization.  A part of me also likes how the
objects can choose to lie about their types.

But another part of me says; the basic objects that you specified
already have a format that is unambiguous, repr(obj).  They also are
able to be reconstructed from their component parts via eval(repr(obj)),
or even via the 'unrepr' function in the ConfigObj module.  It doesn't
handle circular referencse.

Even better, it has 3 native representations; repr(a).encode('zlib'),
repr(a), pprint.pprint(a); each offering a different amount of user
readability.  I digress.


I believe the biggest problem with the proposal, as specified, is that
changing the semantics of __getstate__ and __setstate__ is a bad idea. 
Add a new pair of methods and ask the twisted people what they think. 
My only criticism will then be the strawman repr/unrepr.


 - Josiah



From tomerfiliba at gmail.com  Thu Jan 25 22:55:19 2007
From: tomerfiliba at gmail.com (tomer filiba)
Date: Thu, 25 Jan 2007 23:55:19 +0200
Subject: [Python-ideas] new pickle semantics/API
In-Reply-To: <43aa6ff70701251321y8885892j28916f83b9a4d8c0@mail.gmail.com>
References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com>
	<1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com>
	<43aa6ff70701251321y8885892j28916f83b9a4d8c0@mail.gmail.com>
Message-ID: <1d85506f0701251355x76a1c6f8yef7196304dab1054@mail.gmail.com>

On 1/25/07, Collin Winter <collinw at gmail.com> wrote:
> How will e.g. classes be simplified? Can I simplify a dictionary with
> function objects for values?
>

well, pickle just saves them as a global name (modulename.classname).
so types would generally just return themselves as primitives, and let
the actual simplifier do the trick.

it may choose to save the type's dict, or just a global name.
that's up to the serializer-dependent simplifier.

it's good you mentioned that, because it reminded me of something
i forgot. for instance, code objects will be serialized by value,
so you could actually pickle functions and classes.

this means pyc files could become just a pickle of the module, i.e.:
   import foo
   pickle.dump(foo, open("foo.pyc", "w"))

but again, that's up to the serializer. an enhanced pickle could do
that.


-tomer


From tomerfiliba at gmail.com  Thu Jan 25 23:16:15 2007
From: tomerfiliba at gmail.com (tomer filiba)
Date: Fri, 26 Jan 2007 00:16:15 +0200
Subject: [Python-ideas] new pickle semantics/API
In-Reply-To: <20070125130548.59F4.JCARLSON@uci.edu>
References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com>
	<1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com>
	<20070125130548.59F4.JCARLSON@uci.edu>
Message-ID: <1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com>

On 1/25/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> Overall, I like the idea; I'm a big fan of simplifying object
> persistence and/or serialization.  A part of me also likes how the
> objects can choose to lie about their types.
>
> But another part of me says; the basic objects that you specified
> already have a format that is unambiguous, repr(obj).  They also are
> able to be reconstructed from their component parts via eval(repr(obj)),
> or even via the 'unrepr' function in the ConfigObj module.  It doesn't
> handle circular referencse.

well, repr is fine for most simple things, but you don't use repr to
serialize objects, right? it's not powerful/introspective enough.
besides repr is meant to be readable, while __getstate__ can return
any object. imagine this:

class complex:
    def __repr__(self):
        return "(%f+%fj)" % (self.real, self.imag)
    def __getstate__(self):
        return self.__class__, (self.real, self.imag)

repr is made for humans of course, while serialization is
made for machines. they serves different purposes,
so they need different APIs.

> Even better, it has 3 native representations; repr(a).encode('zlib'),
> repr(a), pprint.pprint(a); each offering a different amount of user
> readability.  I digress.

you may have digressed, but that's a good point -- that's exactly
why i do NOT specify how objects are encoded as a stream of bytes.

all i'm after is the state of the object (which is expressed in terms of
other, more primitive objects).

you can think of repr as a textual serializer to some extent, that
can use the proposed __getstate__ API. pprint is yet another
form of serializer.

> I believe the biggest problem with the proposal, as specified, is that
> changing the semantics of __getstate__ and __setstate__ is a bad idea.
> Add a new pair of methods and ask the twisted people what they think.
> My only criticism will then be the strawman repr/unrepr.

i'll try to come up with new names... but i don't have any ideas
at the moment.


-tomer


From collinw at gmail.com  Thu Jan 25 23:22:19 2007
From: collinw at gmail.com (Collin Winter)
Date: Thu, 25 Jan 2007 16:22:19 -0600
Subject: [Python-ideas] new pickle semantics/API
In-Reply-To: <1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com>
References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com>
	<1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com>
	<20070125130548.59F4.JCARLSON@uci.edu>
	<1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com>
Message-ID: <43aa6ff70701251422v79c4818as141770b5c4dc0775@mail.gmail.com>

On 1/25/07, tomer filiba <tomerfiliba at gmail.com> wrote:
> On 1/25/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> > I believe the biggest problem with the proposal, as specified, is that
> > changing the semantics of __getstate__ and __setstate__ is a bad idea.
> > Add a new pair of methods and ask the twisted people what they think.
> > My only criticism will then be the strawman repr/unrepr.
>
> i'll try to come up with new names... but i don't have any ideas
> at the moment.

The "__getstate__" and "__setstate__" names don't really work for me
either, especially since __setstate__ creates a new object, as opposed
to changing the state of an existing object. Since this proposal is
all about simplification, how about something like "__simplify__" and
"__expand__", respectively?

Collin Winter


From tomerfiliba at gmail.com  Thu Jan 25 23:41:36 2007
From: tomerfiliba at gmail.com (tomer filiba)
Date: Fri, 26 Jan 2007 00:41:36 +0200
Subject: [Python-ideas] new pickle semantics/API
In-Reply-To: <43aa6ff70701251422v79c4818as141770b5c4dc0775@mail.gmail.com>
References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com>
	<1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com>
	<20070125130548.59F4.JCARLSON@uci.edu>
	<1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com>
	<43aa6ff70701251422v79c4818as141770b5c4dc0775@mail.gmail.com>
Message-ID: <1d85506f0701251441k4c960736yc329c1016e331d9c@mail.gmail.com>

On 1/26/07, Collin Winter <collinw at gmail.com> wrote:
> The "__getstate__" and "__setstate__" names don't really work for me
> either, especially since __setstate__ creates a new object, as opposed
> to changing the state of an existing object. Since this proposal is
> all about simplification, how about something like "__simplify__" and
> "__expand__", respectively?

well, i like __simplify__, but __expand__ seems wrong to me.
some other suggestions i looked up:

__complicate__ (:-))
__rebuild__
__reconstruct__ (too long?)
__restore__ (but that's semantically the same as __setstate__)

i'd vote for __rebuild__. any other ideas?


-tomer


From phd at phd.pp.ru  Thu Jan 25 23:48:31 2007
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Fri, 26 Jan 2007 01:48:31 +0300
Subject: [Python-ideas] new pickle semantics/API
In-Reply-To: <1d85506f0701251441k4c960736yc329c1016e331d9c@mail.gmail.com>
References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com>
	<1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com>
	<20070125130548.59F4.JCARLSON@uci.edu>
	<1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com>
	<43aa6ff70701251422v79c4818as141770b5c4dc0775@mail.gmail.com>
	<1d85506f0701251441k4c960736yc329c1016e331d9c@mail.gmail.com>
Message-ID: <20070125224831.GA7310@phd.pp.ru>

On Fri, Jan 26, 2007 at 12:41:36AM +0200, tomer filiba wrote:
> well, i like __simplify__, but __expand__ seems wrong to me.
> some other suggestions i looked up:
> 
> __complicate__ (:-))
> __rebuild__
> __reconstruct__ (too long?)
> __restore__ (but that's semantically the same as __setstate__)
> 
> i'd vote for __rebuild__. any other ideas?

   __(un)pickle__
   __(de)serialize__

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.


From collinw at gmail.com  Fri Jan 26 00:00:47 2007
From: collinw at gmail.com (Collin Winter)
Date: Thu, 25 Jan 2007 17:00:47 -0600
Subject: [Python-ideas] new pickle semantics/API
In-Reply-To: <1d85506f0701251355x76a1c6f8yef7196304dab1054@mail.gmail.com>
References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com>
	<1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com>
	<43aa6ff70701251321y8885892j28916f83b9a4d8c0@mail.gmail.com>
	<1d85506f0701251355x76a1c6f8yef7196304dab1054@mail.gmail.com>
Message-ID: <43aa6ff70701251500o102615a5pfb3664b47979d03e@mail.gmail.com>

On 1/25/07, tomer filiba <tomerfiliba at gmail.com> wrote:
> On 1/25/07, Collin Winter <collinw at gmail.com> wrote:
> > How will e.g. classes be simplified? Can I simplify a dictionary with
> > function objects for values?
[snip]
> it's good you mentioned that, because it reminded me of something
> i forgot. for instance, code objects will be serialized by value,
> so you could actually pickle functions and classes.

Are you intending to simplify code objects to the co_* attributes?
co_code is a string of interpreter-specific bytecode, which would be
pretty much useless outside of a copy() function.

Whatever the scheme, it will need to take into account cell objects and globals.

Collin Winter


From collinw at gmail.com  Fri Jan 26 00:02:26 2007
From: collinw at gmail.com (Collin Winter)
Date: Thu, 25 Jan 2007 17:02:26 -0600
Subject: [Python-ideas] new pickle semantics/API
In-Reply-To: <1d85506f0701251441k4c960736yc329c1016e331d9c@mail.gmail.com>
References: <1d85506f0701241245s45f5ca3bh931d3d7d8a8320ec@mail.gmail.com>
	<1d85506f0701251248i481fe47ew1724c34b842a0659@mail.gmail.com>
	<20070125130548.59F4.JCARLSON@uci.edu>
	<1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com>
	<43aa6ff70701251422v79c4818as141770b5c4dc0775@mail.gmail.com>
	<1d85506f0701251441k4c960736yc329c1016e331d9c@mail.gmail.com>
Message-ID: <43aa6ff70701251502n1ca075e5ne15e2e8c2fbdb1a2@mail.gmail.com>

On 1/25/07, tomer filiba <tomerfiliba at gmail.com> wrote:
> On 1/26/07, Collin Winter <collinw at gmail.com> wrote:
> > The "__getstate__" and "__setstate__" names don't really work for me
> > either, especially since __setstate__ creates a new object, as opposed
> > to changing the state of an existing object. Since this proposal is
> > all about simplification, how about something like "__simplify__" and
> > "__expand__", respectively?
>
> well, i like __simplify__, but __expand__ seems wrong to me.
> some other suggestions i looked up:
>
> __complicate__ (:-))
> __rebuild__
> __reconstruct__ (too long?)
> __restore__ (but that's semantically the same as __setstate__)
>
> i'd vote for __rebuild__. any other ideas?

I like __rebuild__.

Collin Winter


From jcarlson at uci.edu  Fri Jan 26 00:29:02 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Thu, 25 Jan 2007 15:29:02 -0800
Subject: [Python-ideas] new pickle semantics/API
In-Reply-To: <1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com>
References: <20070125130548.59F4.JCARLSON@uci.edu>
	<1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com>
Message-ID: <20070125151500.59F9.JCARLSON@uci.edu>


"tomer filiba" <tomerfiliba at gmail.com> wrote:
> On 1/25/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> > Overall, I like the idea; I'm a big fan of simplifying object
> > persistence and/or serialization.  A part of me also likes how the
> > objects can choose to lie about their types.
> >
> > But another part of me says; the basic objects that you specified
> > already have a format that is unambiguous, repr(obj).  They also are
> > able to be reconstructed from their component parts via eval(repr(obj)),
> > or even via the 'unrepr' function in the ConfigObj module.  It doesn't
> > handle circular referencse.
> 
> well, repr is fine for most simple things, but you don't use repr to
> serialize objects, right? it's not powerful/introspective enough.
> besides repr is meant to be readable, while __getstate__ can return
> any object. imagine this:

I use repr to serialize objects all the time.  ConfigObj is great when I
want to handle python-based configuration information, and/or I don't
want to worry about the security implications of 'eval(arbitrary string)',
or 'import module'.

With a proper __repr__ method, I can even write towards your API:

class mylist(object):
    def __repr__(self):
        state = ...
        return 'mylist.__setstate__(%r)'%(state,)

> class complex:
>     def __repr__(self):
>         return "(%f+%fj)" % (self.real, self.imag)

I would use 'return "(%r+%rj)"% (self.real, self.imag)', but it doesn't
much matter.


> repr is made for humans of course, while serialization is
> made for machines. they serves different purposes,
> so they need different APIs.

I happen to disagree.  The only reason to use a different representation
or API is if there are size and/or performance benefits to offering a
machine readable vs. human readable format.

I'm know that there are real performance advantages to using (c)Pickle
over repr/unrepr, but I use it also so that I can change settings with
notepad (as has been necessary on occasion).


> > Even better, it has 3 native representations; repr(a).encode('zlib'),
> > repr(a), pprint.pprint(a); each offering a different amount of user
> > readability.  I digress.
> 
> you may have digressed, but that's a good point -- that's exactly
> why i do NOT specify how objects are encoded as a stream of bytes.
> 
> all i'm after is the state of the object (which is expressed in terms of
> other, more primitive objects).

Right, but as 'primative objects' go, you cant get significantly more
primitive than producing a string that can be naively understood by
someone familliar with Python *and* the built-in Python parser. 
Nevermind that it works *today* with all of the types you specified
earlier (with the exception of file objects - which you discover on
parsing/reproducing the object).


> you can think of repr as a textual serializer to some extent, that
> can use the proposed __getstate__ API. pprint is yet another
> form of serializer.

Well, pprint is more or less a pretty repr.


> > I believe the biggest problem with the proposal, as specified, is that
> > changing the semantics of __getstate__ and __setstate__ is a bad idea.
> > Add a new pair of methods and ask the twisted people what they think.
> > My only criticism will then be the strawman repr/unrepr.
> 
> i'll try to come up with new names... but i don't have any ideas
> at the moment.

Like Colin, I also like __rebuild__.

 - Josiah



From collinw at gmail.com  Fri Jan 26 00:39:58 2007
From: collinw at gmail.com (Collin Winter)
Date: Thu, 25 Jan 2007 17:39:58 -0600
Subject: [Python-ideas] new pickle semantics/API
In-Reply-To: <20070125151500.59F9.JCARLSON@uci.edu>
References: <20070125130548.59F4.JCARLSON@uci.edu>
	<1d85506f0701251416m24957ff5qa61703ca83b40e55@mail.gmail.com>
	<20070125151500.59F9.JCARLSON@uci.edu>
Message-ID: <43aa6ff70701251539u5f0f0131n20ed52828a6b1fb@mail.gmail.com>

On 1/25/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> "tomer filiba" <tomerfiliba at gmail.com> wrote:
> > repr is made for humans of course, while serialization is
> > made for machines. they serves different purposes,
> > so they need different APIs.
>
> I happen to disagree.  The only reason to use a different representation
> or API is if there are size and/or performance benefits to offering a
> machine readable vs. human readable format.
[snip]
> I'm know that there are real performance advantages to using (c)Pickle
> over repr/unrepr, but I use it also so that I can change settings with
> notepad (as has been necessary on occasion).
[snip]
> > all i'm after is the state of the object (which is expressed in terms of
> > other, more primitive objects).
>
> Right, but as 'primative objects' go, you cant get significantly more
> primitive than producing a string that can be naively understood by
> someone familliar with Python *and* the built-in Python parser.
> Nevermind that it works *today* with all of the types you specified
> earlier (with the exception of file objects - which you discover on
> parsing/reproducing the object).

You use pickle because it's more general than repr/unrepr, not because
it's faster or the result is smaller. Assuming you're talking about
the ConfigObj's unrepr mode
(http://www.voidspace.org.uk/python/configobj.html#unrepr-mode),

"""
The types that unrepr can work with are :

    strings, lists tuples
    None, True, False
    dictionaries, integers, floats
    longs and complex numbers

You can't store classes, types or instances.
"""

Pickle -- and the simplification API Tomer is proposing -- is far more
general than that. repr/unrepr is in no way a substitute.

Collin Winter


From jcarlson at uci.edu  Fri Jan 26 01:14:42 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Thu, 25 Jan 2007 16:14:42 -0800
Subject: [Python-ideas] new pickle semantics/API
In-Reply-To: <43aa6ff70701251539u5f0f0131n20ed52828a6b1fb@mail.gmail.com>
References: <20070125151500.59F9.JCARLSON@uci.edu>
	<43aa6ff70701251539u5f0f0131n20ed52828a6b1fb@mail.gmail.com>
Message-ID: <20070125160432.59FC.JCARLSON@uci.edu>


"Collin Winter" <collinw at gmail.com> wrote:
> On 1/25/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> > "tomer filiba" <tomerfiliba at gmail.com> wrote:
> > > repr is made for humans of course, while serialization is
> > > made for machines. they serves different purposes,
> > > so they need different APIs.
> >
> > I happen to disagree.  The only reason to use a different representation
> > or API is if there are size and/or performance benefits to offering a
> > machine readable vs. human readable format.
> [snip]
> > I'm know that there are real performance advantages to using (c)Pickle
> > over repr/unrepr, but I use it also so that I can change settings with
> > notepad (as has been necessary on occasion).
> [snip]
> > > all i'm after is the state of the object (which is expressed in terms of
> > > other, more primitive objects).
> >
> > Right, but as 'primative objects' go, you cant get significantly more
> > primitive than producing a string that can be naively understood by
> > someone familliar with Python *and* the built-in Python parser.
> > Nevermind that it works *today* with all of the types you specified
> > earlier (with the exception of file objects - which you discover on
> > parsing/reproducing the object).
> 
> You use pickle because it's more general than repr/unrepr, not because
> it's faster or the result is smaller. Assuming you're talking about
> the ConfigObj's unrepr mode
> (http://www.voidspace.org.uk/python/configobj.html#unrepr-mode),

*I* use pickle when I want speed.  I use repr and unrepr (from configobj)
when I want to be able to change things by hand.  None of the objects I
transfer between processes/disk/whatever are ever more than base Python
types.

As for whatever *others* do, I don't know, I haven't done a survey. 
They may do as you say.


> Pickle -- and the simplification API Tomer is proposing -- is far more
> general than that. repr/unrepr is in no way a substitute.

I never claimed it was a substitute, I stated quite clearly up front;
"My only criticism will then be the strawman repr/unrepr."  The
alternate strawman is repr/eval, can be made to support basically
everything except for self-referential objects.  It still has that
horrible security issue, but that also exists in pickle, and could still
exist in the simplification/rebuilding rutines specified by Tomer.


 - Josiah



From cvrebert at gmail.com  Fri Jan 26 04:36:33 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Thu, 25 Jan 2007 19:36:33 -0800
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <op.tmoqkcmrd64u53@e500>
References: <20070118084211.5977.JCARLSON@uci.edu> <45B303AA.1010707@gmail.com>
	<20070121162511.59B9.JCARLSON@uci.edu> <op.tmoqkcmrd64u53@e500>
Message-ID: <45B97741.1040406@gmail.com>

So, basically the same as my proposal except without syntax changes and 
with the <Foo()> default argument value semantics applied to all 
arguments. I'm okay with this, however some possible performance issues 
might exist with re-evaluating expensive default arg vals on every call 
where they're required. This is basically why my proposal required new 
syntax, so that people could use the old "eval once at definition-time" 
semantics on expensive default values to get better performance.

Also, since people (including me) don't really like the <>-syntax, I've 
come up with some real syntax possibilities for my proposal:

     def foo(bar=new baz):

     def foo(bar=fresh baz):

     def foo(bar=separate baz):

     def foo(bar=another baz):

     def foo(bar=unique baz):

I personally like 'fresh'. It seems accurate and not too long.
I welcome other thoughts on better syntax for this.

If the performance issues aren't significant, I'm all for your proposal. 
  It'd be nice to not to have to add new syntax.

- Chris Rebert

Jan Kanis wrote:
> I don't like new syntax for something like this, but I think the default 
> argument values can be fixed with semantic changes (which should not 
> break the most common current uses):
> 
> What I think should happen is compile a function like this
> 
> def popo(x=[]):
>     x.append(666)
>     print x
> 
> as if it had read
> 
> def popo(x=__default_argument_marker__):
>     if x == __default_argument_marker__:
>         x = []
>     x.append(666)
>     print x
> 
> This way, every execution of popo gets its own list. Of course, 
> __default_argument_marker__ is just a way to tell the python runtime 
> that no argument was provided, it should not be exposed to the language.
> 
> If a variable is used in the default argument, it becomes a closure 
> variable:
> 
> d = createMyListOfLists()
> n = getDefaultIndex()
> 
> def foo(x=d[n]):
>   x.append(666)
>   print x
> 
> 
> this is compiled as if it had read
> 
> d = createMyListOfLists()
> n = getDefaultIndex()
> 
> def foo(x=__default_argument_marker__):
>   if x == __default_argument_marker__:
>     x = d[n]      # d and n are closure variables
>   x.append(666)
>   print x
> 
> 
> d and n are looked up in foo's parent scope, which in this example is 
> the global scope. Of course the bytecode compiler should make sure d and 
> n don't name-clash with any variables used in the body of foo.
> 
> When you use variables as default value instead of literals, I think 
> most of the time you intend to have the function do something to the 
> same object the variable is bound to, instead of the function creating 
> it's own copy every time it's called. This behaviour still works with 
> these semantics:
> 
>>>> a = []
>>>> def foo(x=[[],a]):
>>>>   x[0].append(123)
>>>>   x[1].append(123)
>>>>   print x
>>>> foo()
> [[123], [123]]
>>>> foo()
> [[123], [123, 123]]
>>>> foo()
> [[123], [123, 123, 123]]
> 
> foo is compiled as if it had read:
> 
> def foo(x=__default_argument_marker__):
>   if x == __default_argument_marker__:
>     x = [[],a]    # a is a closure variable
>   x[0].append(123)
>   x[1].append(123)
>   print x
> 
> An other difference between this proposal and the current situation is 
> that it would be possible to change the value of a default argument 
> after the function is defined. However I don't think that would really 
> be a problem, and this behaviour is just the same as that of other 
> closure variables. Besides, this (what I perceive as a) problem with 
> closure variables is fixable on its own.
> 
> Jan
> 
> 
> On Mon, 22 Jan 2007 01:49:51 +0100, Josiah Carlson <jcarlson at uci.edu> 
> wrote:
> 
>>
>> Chris Rebert <cvrebert at gmail.com> wrote:
>>> Josiah Carlson wrote:
>>> > As provided by Calvin Spealman, the above can be fixed with:
>>> >
>>> >     def popo(x=None):
>>> >         x = x if x is not None else []
>>> >         x.append(666)
>>> >         print x
>>> >
>>> > I would also mention that forcing users to learn about mutable 
>>> arguments
>>> > and procedural programming is not a bad thing.  Learning the "gotcha"
>>> > of mutable default arguments is a very useful lesson, and to remove 
>>> that
>>> > lesson, I believe, wouldn't necessarily help new users to Python, 
>>> or new
>>> > programmers in general.
>>> >
>>> >  - Josiah
>>
>> Maybe you are taking me a bit too seriously, but hopefully this will add
>> some levity; I'm a poo-poo head.  Moving on...
>>
>>
>>> First, your 'fix' misses the point: though the proposed feature isn't
>>> necessary, and code can be written without using it, it allows mutable
>>> default argument values to be expressed more clearly and succinctly than
>>> the idiom your 'fix' uses.
>>
>> As I stated, it wasn't my fix.  And using previously existing syntax
>> that adds 1 line to a function to support a particular desired result, I
>> think, is perfectly reasonable.  Had the conditional syntax been
>> available for the last decade, those "gotchas" pages would have said
>> "mutable default arguments are tricky, always use the following, and it
>> will probably be the right thing" and moved on.
>>
>>> Second, Python isn't (inherently) about
>>> teaching new programmers about programming, and what is good for newbies
>>> isn't necessarily good for experienced programmers.
>>
>> Indeed, and what *may* be good for *certain* experienced programmers,
>> may not be good for other experienced programmers, or for the language
>> in general.  And personally, I am not sure that I could be convinced
>> that a syntax to support what can be emulated by a single line is even
>> worthy of addition.  In the case of decorators, or even the py3k support
>> for argument annotation, there are certain operations that can be made
>> *significantly* easier.  In this case, I'm not convinced that the extra
>> syntax baggage is worthwhile.
>>
>> Nevermind that it would be one more incompatible syntax that would make
>> it difficult to write for 2.5/2.6 and 3.x .
>>
>>
>>  - Josiah
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
> 
> 
> 


From tomerfiliba at gmail.com  Fri Jan 26 10:29:50 2007
From: tomerfiliba at gmail.com (tomer filiba)
Date: Fri, 26 Jan 2007 11:29:50 +0200
Subject: [Python-ideas] new pickle semantics/API
In-Reply-To: <20070125160432.59FC.JCARLSON@uci.edu>
References: <20070125151500.59F9.JCARLSON@uci.edu>
	<43aa6ff70701251539u5f0f0131n20ed52828a6b1fb@mail.gmail.com>
	<20070125160432.59FC.JCARLSON@uci.edu>
Message-ID: <1d85506f0701260129s4410881coeac2660671aaa7d1@mail.gmail.com>

On 1/26/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> *I* use pickle when I want speed.  I use repr and unrepr (from configobj)
> when I want to be able to change things by hand.  None of the objects I
> transfer between processes/disk/whatever are ever more than base Python
> types.
>
> As for whatever *others* do, I don't know, I haven't done a survey.
> They may do as you say.

well, since this proposal originated from an RPC point-of-view,
i, for once, also need to transfer full-blown user-defined objects
and classes. for that, i need to be able to get the full state of
the object, and the means to reconstruct it later.

pickling the "primitives" was never a problem, because they have
a well defined interface. although cyclic references call for something
stronger than repr... and this is what i mean by "repr is for humans".

> I never claimed it was a substitute, I stated quite clearly up front;
> "My only criticism will then be the strawman repr/unrepr."  The
> alternate strawman is repr/eval, can be made to support basically
> everything except for self-referential objects.  It still has that
> horrible security issue, but that also exists in pickle, and could still
> exist in the simplification/rebuilding rutines specified by Tomer.

well, pickle is unsafe for one main reason -- it's omnipotent.
it performs arbitrary imports and object instantiation, which are
equivalent to eval of arbitrary strings.

BUT, it has nothing to do with the way it gets of sets the *state*
of objects. to solve that, we can have a "capability-based pickle":
dumping objects was never a security issue, it's the loading part
that's dangerous.

we can add a new function, loadsec(), that takes both the string
to load and a set of classes it may use to __rebuilt__ the object.

capabilities = {"list" : list, "str" : str, "os.stat_result" : os.stat_result}
loadsec(data, capabilities)

that way, you can control the objects that will be instantiated,
which you trust, so no arbitrary code may be executed
behind the scenes.

for the "classic" unsafe load(), we can pass a magical dict-like
thing that imports names via __getitem__

if i had a way to control what pickle.loads has access to, i
wouldn't need to write my own serializer..
http://sebulbasvn.googlecode.com/svn/trunk/rpyc/core/marshal/brine.py

> > you may have digressed, but that's a good point -- that's exactly
> > why i do NOT specify how objects are encoded as a stream of bytes.
> >
> > all i'm after is the state of the object (which is expressed in terms of
> > other, more primitive objects).
>
> Right, but as 'primative objects' go, you cant get significantly more
> primitive than producing a string that can be naively understood by
> someone familliar with Python *and* the built-in Python parser.

but that's not the issue. when i send my objects across a socket
back and forth, i want something that is fast, compact, and safe.
i don't care for anyone sniffing my wire to "easily understand"
what i'm sending... i mean, it's not like i encrypt the data, but
readability doesn't count here.

again, __simplify__ and __rebuild__ offer a mechanism for
serializer-implementors, which may choose different encoding
schemes for their internal purposes. this API isn't meant for
the end user.

i'll write a pre-pep to try to clarify it all in an orderly manner.


-tomer


From tomerfiliba at gmail.com  Fri Jan 26 16:13:12 2007
From: tomerfiliba at gmail.com (tomer filiba)
Date: Fri, 26 Jan 2007 17:13:12 +0200
Subject: [Python-ideas] simplification pep-like thing
Message-ID: <1d85506f0701260713k1fe0d548u8103e86a7dfffdf7@mail.gmail.com>

this text needs more thought and rephrasing, but i think that
it covers all the details.

--------------------------------------------------------------------------

Abstract
=========
The simplification API provides a mechanism to get the state
of an object, as well as reconstructing an object with a given state.
This new API would affect pickle, copy_reg, and copy modules,
as well as the writers of serializers.

The idea is to separate the state-gathering from the encoding:
* State-gathering is getting the state of the object
* Encoding is the converting the state (which is an object) into
  a sequence of bytes (textual or binary)

This is somewhat similar to the ISerializable interface of .NET,
which defines only GetObjectData(); the actual encoding is done
by a Formatter class, which has different implementations for
SOAP-formatting, binary-formatting, and other formats.

Motivation
==========
There are many generic and niche serializers out there, including
pickle, banana/jelly, Cerealizer,  brine, and lots of serializers
targeting XML output. Currently, all serializers have their own
methods of retrieving the contents, or state of the object.

This API attempts to solve this issue by providing standard
means for getting object state, and creating objects with
their state restored.

Another issue is making the serialization process "proxy-friendly".
Many frameworks use object proxies to indirectly refer to another
object (for instance, RPC proxies, FFI, etc.). In this case, it's desirable
to simplify the referenced object rather than the proxy, and this API
addresses this issue too.

Simplification
==============
Simplification is the process of converting a 'complex' object
into its "atomic" components. You may think of these atomic
components as the *contents* of the object.

This proposal does not state what "atomic" means -- this is
open to the decision of the class. The only restriction imposed is,
collections of any kind must be simplified as tuples.

Moreover, the simplification process may be recursive: object X
may simplify itself in terms of object Y, which in turn may go
further simplification.

Simplification Protocol
=======================
This proposal introduces two new special methods:

    def __simplify__(self):
        return type, state

    @classmethod
    def __rebuild__(cls, state):
        return new_instance_of_cls_with_given_state

__simplify__ takes no arguments (bar 'self'), and returns a tuple
of '(type, state)', representing the contents of 'self':
* 'type' is expected to be a class or a builtin type, although
  it can be any object that has a '__rebuild__' method.
  This 'type' will be used later to reconstruct the object
  (using 'type.__rebuild__(state)').
* 'state' is any other that represents the inner state of 'self',
  in a simplified form. This can be an atomic value, or yet another
  complex object, that may go further simplification.

__rebuild__ is a expected to be classmethod that takes the
state returned by __simplify__, and returns a new instance of
'cls' with the given state.

If a type does not wish to be simplified, it may throw a TypeError
in its __simplify__ method; however, this is not recommended.
Types that want to be treated as atomic elements, such as file,
should just return themselves, and let the serializer handle them.

Default Simplification
=====================
All the built in types would grow a __simplify__ and __rebuild__
methods, which would follow these guidelines:

Primitive types (int, str, float, ...) are considered atomic.

Composite types (I think 'complex' is the only type), are
broken down into their components. For the complex type,
that would be a tuple of (real, imaginary).

Container types (tuples, lists, sets, dicts) represent themselves
as tuples of items. For example, dicts would be simplified
according to this pseudocode:

    def PyDict_Simplifiy(PyObject * self):
        return PyDictType, tuple(self.items())

Built in types would be considered atomic. User-defined classes
can be simplified into their metaclass, __bases__, and __dict__.

The type 'object' would simplify instances by returning their
__dict__ and any __slots__ the instance may have. This is
the default behavior; classes that desire a different behavior
would override __simplify__ and __rebuild__.

Example of default behavior:
    >>> class Foo(object):
    ...     def __init__(self):
    ...         self.a = 5
    ...         self.b = "spam"
    ...
    >>> f = Foo()
    >>> cls, state = f.__simplify__()
    >>> cls
    <class '__main__.Foo'>
    >>> state
    {"a" : 5, "b" : "spam"}
    >>> shallow_copy = cls.__rebuild__(state)
    >>> state.__simplify__()
    (<type 'dict'>, (("a", 5), ("b", "spam")))

Example of customized behavior
    >>> class Bar(object):
    ...     def __init__(self):
    ...         self.a = 5
    ...         self.b = "spam"
    ...     def __simplify__(self):
    ...          return Bar, 17.5
    ...     @clasmethod
    ...     def __rebuild__(cls, state)
    ...          self = cls.__new__(cls)
    ...          if state == 17.5:
    ...              self.a = 5
    ...              self.b = "spam"
    ...          return self
    ...
    >>> b = Bar()
    >>> b.__simplify__()
    (<class '__main__.Bar'>, 17.5)


Code objects
=============
I wish that modules, classes and functions would also be simplifiable,
however, there are some issues with that:
* How to serialize code objects? These can be simplified as tuple of
  their co_* attributes, but these attributes are very implementation-
  specific.
* How to serialize cell variables, or other globals?

It would be nice if .pyc files where generated like so:
    import foo
    pickle.dump(foo)

It would also allow sending of code between machines, just
like any other object.

Copying
========
Shallow copy, as well as deep copy, can be implemented using
the semantics of this new API. The copy module should be rewritten
accordingly:
    def copy(obj):
        cls, state = obj.__simplify__()
        return cls.__rebuild__(state)

deepcopy() can be implemented similarly.

Deprecation
===========
With the proposed API, copy_reg, __reduce__, __reduce_ex__,
and possibly other modules become deprecated.

Apart from that, the pickle and copy modules need to be updated
accordingly.

C API
======
The proposal introduces two new C-API functions:
PyObject * PyObject_Simplify(PyObject * self);
PyObject * PyObject_Rebuild(PyObject * type, PyObject * state);

Although this is only a suggestion. I'd like to hear ideas from someone
with more experience in the core.

I do not see a need for a convenience routine such as
simplify(obj) <--> obj.__simplify__(), since this API is not designed
for everyday usage. This is the case with __reduce__ today.

Object Proxying
===============
Because __simplify__ returns '(type, state)', it may choose to "lie"
about it's actual type. This means that when the state is reconstructed,
another type is used. Object proxies will use this mechanism to serialize
the referred object, rather than the proxy.

class Proxy(object):
    [...]
    def __simplify__(self):
        # this returns a tuple with the *real* type and state
        return self.value.__simplify__()

Serialization
============
Serialization is the process of converting fully-simplified objects into
byte sequences (strings). Fully simplified objects are created by a
recursive simplifier, that simplifies the entire object graph into atomic
components. Then, the serializer would convert the atomic components
into strings.

Note that this proposal does not define how atomic objects are to be
converted to strings, or how a 'recursive simplifier' should work. These
issues are to be resolved by the implementation of the serializer.

For instance, file objects are atomic; one serializer may be able to
handle them, by storing them as (filename, file-mode, file-position),
while another may not be, so it would raise an exception.

Recursive Simplifier
===================
This code demonstrates the general idea of how recursive simplifiers
may be implemented:

def recursive_simplifier(obj):
    cls, state = obj.__simplify__()

    # simplify all the elements inside tuples
    if type(state) is tuple:
        nested_state = []
        for item in state:
            nested_state.append(recursive_simplifier(item))
        return cls, nested_state

    # see if the object is atomic; if not, dig deeper
    if (cls, state) == state.__simplify__():
        # 'state' is an atomic object, no need to go further
        return cls, state
    else:
        # this object is not atomic, so dig deeper
        return cls, recusrive_simplifier(state)




-tomer


From aahz at pythoncraft.com  Fri Jan 26 19:16:30 2007
From: aahz at pythoncraft.com (Aahz)
Date: Fri, 26 Jan 2007 10:16:30 -0800
Subject: [Python-ideas] simplification pep-like thing
In-Reply-To: <1d85506f0701260713k1fe0d548u8103e86a7dfffdf7@mail.gmail.com>
References: <1d85506f0701260713k1fe0d548u8103e86a7dfffdf7@mail.gmail.com>
Message-ID: <20070126181630.GA4818@panix.com>

On Fri, Jan 26, 2007, tomer filiba wrote:
>
> Default Simplification
> =====================
> All the built in types would grow a __simplify__ and __rebuild__
> methods, which would follow these guidelines:
> 
> Primitive types (int, str, float, ...) are considered atomic.
> 
> Composite types (I think 'complex' is the only type), are
> broken down into their components. For the complex type,
> that would be a tuple of (real, imaginary).
> 
> Container types (tuples, lists, sets, dicts) represent themselves
> as tuples of items. For example, dicts would be simplified
> according to this pseudocode:
> 
>     def PyDict_Simplifiy(PyObject * self):
>         return PyDictType, tuple(self.items())
> 
> Built in types would be considered atomic. User-defined classes
> can be simplified into their metaclass, __bases__, and __dict__.

This seems to contradict the following:

> For instance, file objects are atomic; one serializer may be able to
> handle them, by storing them as (filename, file-mode, file-position),
> while another may not be, so it would raise an exception.

Where do files fit into all this?
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"I disrespectfully agree."  --SJM


From jan.kanis at phil.uu.nl  Sat Jan 27 01:36:22 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Sat, 27 Jan 2007 01:36:22 +0100
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <fb6fbf560701250641k45f24f40qfea54fa07e6b40c4@mail.gmail.com>
References: <20070118084211.5977.JCARLSON@uci.edu> <45B303AA.1010707@gmail.com>
	<20070121162511.59B9.JCARLSON@uci.edu> <op.tmoqkcmrd64u53@e500>
	<fb6fbf560701250641k45f24f40qfea54fa07e6b40c4@mail.gmail.com>
Message-ID: <op.tmsemwqxd64u53@e500>

On Thu, 25 Jan 2007 15:41:54 +0100, Jim Jewett <jimjjewett at gmail.com>  
wrote:

> On 1/24/07, Jan Kanis <jan.kanis at phil.uu.nl> wrote:
>> I don't like new syntax for something like this, but I think the default
>> argument values can be fixed with semantic changes (which should not  
>> break
>> the most common current uses):
>>
>> What I think should happen is compile a function like this
>>
>> def popo(x=[]):
>>      x.append(666)
>>      print x
>>
>> as if it had read
>>
>> def popo(x=__default_argument_marker__):
>>      if x == __default_argument_marker__:
>>          x = []
>>      x.append(666)
>>      print x
>
> How is this different from the x=None idiom of today?
>
>     def f(inlist=None):
>         if inlist is None:
>             inlist=[]
>

The __default_argument_marker__ is not really a part of my proposal. You  
can replace it with None everywhere if you want to. The reason I used it  
is because using None can clash when the caller passes None in as explicit  
value like this:

def foo(x, y=None):
   y = getApropreateDefaultValue() if y == None else y
   x.insert(y)

foo(bar, None)

Now if you want to have foo do bar.insert(None), calling foo(bar, None)  
won't work.

However, I guess the risk of running into such a case in real code is  
neglegible, and my urge to write correct code won from writing  
understandable code. Just pretend I used None everywhere. (and that the  
compiler can magically distinguish between a default-argument None and a  
caller-provided None, if you wish.)

> The if (either 2 lines or against PEP8) is a bit ugly, but Calvin
> pointed out that you can now write it as
>
>     def f(inlist=None):
>         inlist = inlist if (inlist is not None) else []
>
> I see  below that you give it slightly different semantics, but I'm
> not entirely sure how to tell when those different semantics should
> apply (always?  when the variable name is marked with __*__?  When a
> specific non-None singleton appears?), or why you would ever want
> them.

Please just ignore the __default_argument_marker__ thing. I hope we agree  
that the problem we're trying to solve is that while

def f(inlist=None):
   inlist = inlist if (inlist is not None) else []

works in the current python, it's non-intuitive and ugly, and it would be  
nice to have python do 'the right thing' if we can find a nice way to make  
it do that.

Oh, and the changed semantics would allways be uses.

>> When you use variables as default value instead of literals, I think  
>> most
>> of the time you intend to have the function do something to the same
>> object the variable is bound to, instead of the function creating it's  
>> own
>> copy every time it's called. This behaviour still works with these
>> semantics:
>
>> >>> a = []
>> >>>def foo(x=[[],a]):
>> >>>   x[0].append(123)
>> >>>   x[1].append(123)
>> >>>   print x
>> >>>foo()
>> [[123], [123]]
>> >>> foo()
>> [[123], [123, 123]]
>> >>> foo()
>> [[123], [123, 123, 123]]
>
> So you're saying that x[1] should be persistent because it (also) has
> a name (as 'a'), but x[0] should be recreated fresh on each call
> because it doesn't?

I think what python does currently can be improved upon. I think that if  
someone defines a function like def f(x=[]): ... he'll most likely want x  
to be an empty list every time the function is called. But just having  
python do an automagic copy(x) or deepcopy(x) is not going to work, becaus  
it can be expensive, isn't always nescessary, and is sometimes plainly  
impossible, eg: def f(x=sys.stdout): ...

So, sometimes we want a new thing on every call, and sometimes we don't.  
And sometimes we want to specify the default value with a literal, and  
sometimes with a variable. My assumption is that these two differences  
coincide most of the time. I also think my approach is more intuitive to  
people new to python and not familiar with this quirk/wart.

You make it sound as if doing something different with a named variable  
and a literal is strange, but this is exactly what happens in every normal  
python expression:

  >>> a = []
  >>> b = [[], a]
  >>> id(a)
  12976416
  >>> id(b[0])
  12994640
  >>> id(b[1])
  12976416
  >>> # let's execute the b = ... statement again (comparable to 'call a  
function again')
  >>> b = [[], a]
  >>> id(b[0])
  12934800
  >>> id(b[1])
  12976416

b[0] gets recreated, while b[1] is not.

So, I think my solution of evaluating the default argument on every call  
and letting any variables in the expression be closure variables  
accomplishes everything we want:
* most of the time python does 'the right thing' wrt to copying or not  
copying, and does this in a way that is very regular wrt the rest of the  
language (and therefore easy to learn)
* if we don't want a copy while python would do this, that can easily be  
accomplished by first creating the object and then using the variable name  
that references the object as default
* if we do want a copy, just do an explicit copy.copy(x).


On Fri, 26 Jan 2007 04:36:33 +0100, Chris Rebert <cvrebert at gmail.com>  
wrote:

> So, basically the same as my proposal except without syntax changes and  
> with the <Foo()> default argument value semantics applied to all  
> arguments. I'm okay with this, however some possible performance issues  
> might exist with re-evaluating expensive default arg vals on every call  
> where they're required. This is basically why my proposal required new  
> syntax, so that people could use the old "eval once at definition-time"  
> semantics on expensive default values to get better performance.
> [snip]
> If the performance issues aren't significant, I'm all for your proposal.  
>   It'd be nice to not to have to add new syntax.

Well, I wasn't thinking about it as basically the same as your proposal,  
but thinking about it again I think it is. (I was thinking more along the  
lines of having this do what looks intuitive to me, by applying the normal  
python language rules in the IMO 'right' way.)

on performance issues:

If you're using the x=None ... if x==None: x = [] trick the object gets  
evaluated and recreated on every call anyway, so there's no change.
If you aren't using a literal as default, nothing gets re-evaluated, so no  
problem either.
The only time it could become a problem is with code like this:

def foo(x=createExpensiveThing()):
	return x.bar()

If the function modifies x, you'll probably want to use a fresh x every  
call anyway (using x=None). If you do want the x to be persistant, chances  
are you already have a reference to it somewhere, but if you don't you'll  
have to create one. The reference doesn't get re-evaluated, so there's no  
performance issue, only possibly a namespace clutter issue.
If the function doesn't modify x it may give rise to a performance issue,  
but that can easily be solved by creating the default before defining the  
function and using the variable. Rejecting this proposition because of  
this seems like premature optimisation to me.

Another slight performance loss may be the fact that variables used in a  
default value will sometimes become closure variables, which are slightly  
slower than locals. However these variables are not used in the functions  
body, and it only is an issue if we're defining a function inside another  
function. I think this point is neglegible.

- Jan


From jcarlson at uci.edu  Sat Jan 27 06:30:00 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Fri, 26 Jan 2007 21:30:00 -0800
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <op.tmsemwqxd64u53@e500>
References: <fb6fbf560701250641k45f24f40qfea54fa07e6b40c4@mail.gmail.com>
	<op.tmsemwqxd64u53@e500>
Message-ID: <20070126212657.5A06.JCARLSON@uci.edu>


"Jan Kanis" <jan.kanis at phil.uu.nl> wrote:
> I hope we agree  
> that the problem we're trying to solve is that while
> 
> def f(inlist=None):
>    inlist = inlist if (inlist is not None) else []
> 
> works in the current python, it's non-intuitive and ugly, and it would be  
> nice to have python do 'the right thing' if we can find a nice way to make  
> it do that.

I'm going to have to disagree on the 'non-intuitive and ugly' claim.  We
are just going to have to agree to disagree.


 - Josiah



From tomerfiliba at gmail.com  Sat Jan 27 22:30:51 2007
From: tomerfiliba at gmail.com (tomer filiba)
Date: Sat, 27 Jan 2007 23:30:51 +0200
Subject: [Python-ideas] my take on mutable default arguments
Message-ID: <1d85506f0701271330o1df7628fy2e82cedf97bada68@mail.gmail.com>

i thought this code be solved nicely with a decorator... it needs some
more work, but it would make a good cookbook recipe:

.>>> from copy import deepcopy.
.>>>
.>>> def defaults(**kwdefs):
...     def deco(func):
...         def wrapper(*args, **kwargs):
...             for k,v in kwdefs.iteritems():
...                 if k not in kwargs:
...                     kwargs[k] = deepcopy(v)
...             return func(*args, **kwargs)
...         return wrapper
...     return deco
...
.>>> @defaults(x = [])
... def foo(a, x):
...     x.append(a)
...     print x
...
.>>> foo(5)
[5]
.>>> foo(5)
[5]
.>>> foo(5)
[5]

maybe it should be done by copying func_defaults... then it could
be written as

@copydefaults
def f(a, b = 5, c = []):
    ...


-tomer


From tomerfiliba at gmail.com  Sat Jan 27 22:34:02 2007
From: tomerfiliba at gmail.com (tomer filiba)
Date: Sat, 27 Jan 2007 23:34:02 +0200
Subject: [Python-ideas] simplification pep-like thing
In-Reply-To: <1d85506f0701260713k1fe0d548u8103e86a7dfffdf7@mail.gmail.com>
References: <1d85506f0701260713k1fe0d548u8103e86a7dfffdf7@mail.gmail.com>
Message-ID: <1d85506f0701271334v21a43faen96db5ae3b3085aeb@mail.gmail.com>

Aahz:
>> Built in types would be considered atomic. User-defined classes
>> can be simplified into their metaclass, __bases__, and __dict__.
>
> This seems to contradict the following:
>
>> For instance, file objects are atomic; one serializer may be able to
>> handle them, by storing them as (filename, file-mode, file-position),
>> while another may not be, so it would raise an exception.
>
> Where do files fit into all this?

it doesn't... it was just meant to show how different serializers may
implement their converters for the atomic values. it's not a practical
suggestion.


-tomer


From tomerfiliba at gmail.com  Sat Jan 27 22:42:50 2007
From: tomerfiliba at gmail.com (tomer filiba)
Date: Sat, 27 Jan 2007 23:42:50 +0200
Subject: [Python-ideas] my take on mutable default arguments
In-Reply-To: <1d85506f0701271330o1df7628fy2e82cedf97bada68@mail.gmail.com>
References: <1d85506f0701271330o1df7628fy2e82cedf97bada68@mail.gmail.com>
Message-ID: <1d85506f0701271342w2dc4c937l5cfc9b829a1becb2@mail.gmail.com>

i think this is better:

.>>> from copy import deepcopy
.>>>
.>>> def copydefaults(func):
...     defaults = func.func_defaults
...     def wrapper(*args, **kwargs):
...         func.func_defaults = deepcopy(defaults)
...         return func(*args, **kwargs)
...     return wrapper
...
.>>> @copydefaults
... def f(a, x = []):
...     x.append(a)
...     print x
...
.>>>
.>>> f(1)
[1]
.>>> f(2)
[2]
.>>> f(3)
[3]



-tomer


From cvrebert at gmail.com  Sat Jan 27 23:12:34 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Sat, 27 Jan 2007 14:12:34 -0800
Subject: [Python-ideas] my take on mutable default arguments
In-Reply-To: <1d85506f0701271330o1df7628fy2e82cedf97bada68@mail.gmail.com>
References: <1d85506f0701271330o1df7628fy2e82cedf97bada68@mail.gmail.com>
Message-ID: <45BBCE52.5030207@gmail.com>

As pointed out by Calvin Spealman in an earlier email:

Calvin Spealman wrote:
 > Deep copy of the originally created
 > default argument can be expensive and would not work in any useful way
 > with non-literals as defaults, such as function calls or subscript
 > lookups or even simple attributes.

However, your decorator is useful in several common cases.
- Chris Rebert

tomer filiba wrote:
> i thought this code be solved nicely with a decorator... it needs some
> more work, but it would make a good cookbook recipe:
> 
> .>>> from copy import deepcopy.
> .>>>
> .>>> def defaults(**kwdefs):
> ...     def deco(func):
> ...         def wrapper(*args, **kwargs):
> ...             for k,v in kwdefs.iteritems():
> ...                 if k not in kwargs:
> ...                     kwargs[k] = deepcopy(v)
> ...             return func(*args, **kwargs)
> ...         return wrapper
> ...     return deco
> ...
> .>>> @defaults(x = [])
> ... def foo(a, x):
> ...     x.append(a)
> ...     print x
> ...
> .>>> foo(5)
> [5]
> .>>> foo(5)
> [5]
> .>>> foo(5)
> [5]
> 
> maybe it should be done by copying func_defaults... then it could
> be written as
> 
> @copydefaults
> def f(a, b = 5, c = []):
>     ...
> 
> 
> -tomer
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
> 


From tomerfiliba at gmail.com  Sat Jan 27 23:22:32 2007
From: tomerfiliba at gmail.com (tomer filiba)
Date: Sun, 28 Jan 2007 00:22:32 +0200
Subject: [Python-ideas] my take on mutable default arguments
In-Reply-To: <45BBCE52.5030207@gmail.com>
References: <1d85506f0701271330o1df7628fy2e82cedf97bada68@mail.gmail.com>
	<45BBCE52.5030207@gmail.com>
Message-ID: <1d85506f0701271422j5650fcdajefb0ab33215c1de3@mail.gmail.com>

On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:
> As pointed out by Calvin Spealman in an earlier email:
>
> Calvin Spealman wrote:
>  > Deep copy of the originally created
>  > default argument can be expensive and would not work in any useful way
>  > with non-literals as defaults, such as function calls or subscript
>  > lookups or even simple attributes.

first, please refer to my second version (@copydefaults).

second, theoretically all objects are copyable, as long as they
define __copy__ or __deepcopy__. it's up to you.

third, you'll use this decorator only when you want "fresh copies" of the
default values... not "just for fun", and since you'll need these fresh copies
anyhow, creating the value by hand ("if x is None: x = []") will not yield
a dramatic performance gain, as most default objects are "primitive"
anyway (empty lists and stuff, not dicts of 10000 keys)

the main issue is readability and auto-documentation. using
"def f(x = [])" is better documenting than "def f(x = None)"

and fourth, my apologies, but such a decorator already existed (and even
with the same semantics that i used):
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/303440


-tomer


From cvrebert at gmail.com  Sun Jan 28 20:22:44 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Sun, 28 Jan 2007 11:22:44 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
Message-ID: <45BCF804.7040207@gmail.com>

The following is a proto-PEP based on the discussion in the thread
"fixing mutable default argument values". Comments would be greatly
appreciated.
- Chris Rebert

Title: Fixing Non-constant Default Arguments

Abstract

     This PEP proposes new semantics for default arguments to remove 
boilerplate code associated with non-constant default argument values,
allowing them to be expressed more clearly and succinctly.


Motivation

     Currently, to write functions using non-constant default arguments, 
one must use the idiom:

     def foo(non_const=None):
         if non_const is None:
             non_const = some_expr
         #rest of function

or equivalent code. Naive programmers desiring mutable default arguments 
often make the mistake of writing the following:

     def foo(mutable=some_expr_producing_mutable):
         #rest of function

However, this does not work as intended, as 
'some_expr_producing_mutable' is evaluated only *once* at 
definition-time, rather than once per call at call-time.  This results 
in all calls to 'foo' using the same default value, which can result in 
unintended consequences.  This necessitates the previously mentioned 
idiom. This unintuitive behavior is such a frequent stumbling block for 
newbies that it is present in at least 3 lists of Python's problems [0] 
[1] [2].
     There are currently few, if any, known good uses of the current 
behavior of mutable default arguments.  The most common one is to 
preserve function state between calls.  However, as one of the lists [2] 
comments, this purpose is much better served by decorators, classes, or 
(though less preferred) global variables.
     Therefore, since the current semantics aren't useful for 
non-constant default values and an idiom is necessary to work around 
this deficiency, why not change the semantics so that people can write 
what they mean more directly, without the annoying boilerplate?


Rationale

     Originally, it was proposed that all default argument values be 
deep-copied from the original (evaluated at definition-time) at each 
invocation of the function where the default value was required. 
However, this doesn't take into account default values that are not 
literals, e.g. function calls, subscripts, attribute accesses.  Thus, 
the new idea was to re-evaluate the default arguments at each call where 
they were needed. There was some concern over the possible performance 
hit this could cause, and whether there should be new syntax so that 
code could use the existing semantics for performance reasons.  Some of 
the proposed syntaxes were:

     def foo(bar=<baz>):
         #code

     def foo(bar=new baz):
         #code

     def foo(bar=fresh baz):
         #code

     def foo(bar=separate baz):
         #code

     def foo(bar=another baz):
         #code

     def foo(bar=unique baz):
         #code

where the new keyword (or angle brackets) would indicate that the 
parameter's default argument should use the new semantics.  Other 
parameters would continue to use the old semantics. It was generally 
agreed that the angle-bracket syntax was particularly ugly, leading to 
the proposal of the other syntaxes. However, having 2 different sets of 
semantics could be confusing and leaving in the old semantics just for 
performance might be premature optimization. Refactorings to deal with 
the possible performance hit are discussed below.


Specification

     The current semantics for default arguments are replaced by the 
following semantics:
     - Whenever a function is called, and the caller does not provide a
     value for a parameter with a default expression, the parameter's
     default expression shall be evaluated in the function's scope.  The
     resulting value shall be assigned to a local variable in the
     function's scope with the same name as the parameter.
     - The default argument expressions shall be evaluated before the
     body of the function.
     - The evaluation of default argument expressions shall proceed in
     the same order as that of the parameter list in the function's
     definition.
Given these semantics, it makes more sense to refer to default argument
expressions rather than default argument values, as the expression is
re-evaluated at each call, rather than just once at definition-time.
Therefore, we shall do so hereafter.

Demonstrative examples of new semantics:
     #default argument expressions can refer to
     #variables in the enclosing scope...
     CONST = "hi"
     def foo(a=CONST):
         print a

     >>> foo()
     hi
     >>> CONST="bye"
     >>> foo()
     bye

     #...or even other arguments
     def ncopies(container, n=len(container)):
         return [container for i in range(n)]

     >>> ncopies([1, 2], 5)
     [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]]
     >>> ncopies([1, 2, 3])
     [[1, 2, 3], [1, 2, 3], [1, 2, 3]]
     >>> #ncopies grabbed n from [1, 2, 3]'s length (3)

     #default argument expressions are arbitrary expressions
     def my_sum(lst):
         cur_sum = lst[0]
         for i in lst[1:]: cur_sum += i
         return cur_sum

     def bar(b=my_sum((["b"] * (2 * 3))[:4])):
         print b

     >>> bar()
     bbbb

     #default argument expressions are re-evaluated at every call...
     from random import randint
     def baz(c=randint(1,3)):
         print c

     >>> baz()
     2
     >>> baz()
     3

     #...but only when they're required
     def silly():
         print "spam"
         return 42

     def qux(d=silly()):
         pass

     >>> qux()
     spam
     >>> qux(17)
     >>> qux(d=17)
     >>> qux(*[17])
     >>> qux(**{'d':17})
     >>> #no output because silly() never called because d's value was 
specified in the calls

     #Rule 3
     count = 0
     def next():
         global count
         count += 1
         return count - 1

     def frobnicate(g=next(), h=next(), i=next()):
         print g, h, i

     >>> frobnicate()
     0 1 2
     >>> #g, h, and i's default argument expressions are evaluated in 
the same order as the parameter definition


Backwards Compatibility

     This change in semantics breaks all code which uses mutable default 
argument values. Such code can be refactored from:

     def foo(bar=mutable):
         #code

to

     def stateify(state):
         def _wrap(func):
             def _wrapper(*args, **kwds):
                 kwds['bar'] = state
                 return func(*args, **kwds)
             return _wrapper
         return _wrap

     @stateify(mutable)
     def foo(bar):
         #code

or

     state = mutable
     def foo(bar=state):
         #code

or

     class Baz(object):
         def __init__(self):
             self.state = mutable

         def foo(self, bar=self.state):
             #code

The changes in this PEP are backwards-compatible with all code whose 
default argument values are immutable, including code using the idiom 
mentioned in the 'Motivation' section. However, such values will now be 
recomputed for each call for which they are required. This may cause 
performance degradation. If such recomputation is significantly 
expensive, the same refactorings mentioned above can be used.

     In relation to Python 3.0, this PEP's proposal is compatible with 
those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend 
on the acceptance of either of those PEPs.


Reference Implementation

     All code of the form:

         def foo(bar=some_expr, baz=other_expr):
             #body

     Should act as if it had read (in pseudo-Python):

         def foo(bar=_undefined, baz=_undefined):
             if bar is _undefined:
                 bar = some_expr
             if baz is _undefined:
                 baz = other_expr
             #body

where _undefined is the value given to a parameter when the caller 
didn't specify a value for it. This is not intended to be a literal 
translation, but rather a demonstration as to how Python's internal 
argument-handling machinery should be changed.


References

     [0] 10 Python pitfalls
         http://zephyrfalcon.org/labs/python_pitfalls.html

     [1] Python Gotchas
         http://www.ferg.org/projects/python_gotchas.html#contents_item_6

     [2] When Pythons Attack
     http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2

     [3] Keyword-Only Arguments
         http://www.python.org/dev/peps/pep-3102/

     [4] Function Annotations
         http://www.python.org/dev/peps/pep-3107/


From gagsl-py at yahoo.com.ar  Sun Jan 28 01:42:37 2007
From: gagsl-py at yahoo.com.ar (Gabriel Genellina)
Date: Sat, 27 Jan 2007 21:42:37 -0300
Subject: [Python-ideas] my take on mutable default arguments
References: <1d85506f0701271330o1df7628fy2e82cedf97bada68@mail.gmail.com><45BBCE52.5030207@gmail.com>
	<1d85506f0701271422j5650fcdajefb0ab33215c1de3@mail.gmail.com>
Message-ID: <epgri3$q2r$1@sea.gmane.org>

"tomer filiba" <tomerfiliba at gmail.com> 
escribi? en el mensaje 
news:1d85506f0701271422j5650fcdajefb0ab33215c1de3 at mail.gmail.com...

> On 1/28/07, Chris Rebert 
> <cvrebert at gmail.com> wrote:
>> As pointed out by Calvin Spealman in an earlier email:
>>
>> Calvin Spealman wrote:
>>  > Deep copy of the originally created
>>  > default argument can be expensive and would not work in any useful way
>>  > with non-literals as defaults, such as function calls or subscript
>>  > lookups or even simple attributes.
>
> first, please refer to my second version (@copydefaults).
>
> second, theoretically all objects are copyable, as long as they
> define __copy__ or __deepcopy__. it's up to you.

No, I think he's pointing another problem. For function calls, subscripts or 
attributes, it's important *when* you execute it. By example:

def foo1(x, prompt=sys.ps2):
  print prompt,x

def foo2(x, prompt=None):
  if prompt is None: prompt=sys.ps2
  print prompt,x

@copydefaults
def foo3(x, prompt=sys.ps2):
  print prompt,x

If the intent of the author is "use sys.ps2 unless I pass an explicit 
prompt" neither foo1 nor foo3 will work if sys.ps2 is changed.

> second, theoretically all objects are copyable, as long as they
> define __copy__ or __deepcopy__. it's up to you.

Not all objects are copyable, or not as one would expect:

@copydefaults
def foo4(x, outfile=sys.stdout):
  "Use sys.stdout unless an explicit outfile is given"
  outfile.write(x)

foo4 doesn't even work, it raises: ValueError: I/O operation on closed file. 
Even if you manage to special-case files, or "fix" how they are copied, it 
won't work as intended because sys.stdout could have been reassigned after 
the function was defined.
So you have to use the old idiom anyway:

def foo5(x, outfile=None):
  if outfile is None: outfile = sys.stdout
  outfile.write(x)

> third, you'll use this decorator only when you want "fresh copies" of the
> default values... not "just for fun", [...]

Some way to say "I want this default argument to be re-evaluated when I 
don't provide an explicit value" would be useful, but this proposal does not 
help on this.

-- 
Gabriel Genellina 




From rnd at onego.ru  Mon Jan 29 08:38:39 2007
From: rnd at onego.ru (Roman Susi)
Date: Mon, 29 Jan 2007 09:38:39 +0200
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BCF804.7040207@gmail.com>
References: <45BCF804.7040207@gmail.com>
Message-ID: <45BDA47F.4040603@onego.ru>

Hello!

I'd liked to say outright that this bad idea which complicates matters 
more than provides solutions.
Right now it is enough to know that the part from def to ":" is executed 
at definition time. This is what
incremental dynamic semantics is about. So, the suggestion is good only 
as separated feature, but is IMHO wrong
if considered in the language design as a whole.

So things like

def foo(non_const=None):
    non_const = non_const or []

are good becuase explicitely tell you that the mutable object is to be 
created at call-time, not def-time.

And I do not like PEP 3107 neither: its overly complex.

If there is a need for Python type checking, I'd suggested to make a 
special superset which could be used
to write compiled extensions as well (Pyrex comes to mind).


Regards,
Roman

P.S. However, I may be wrong. In that case my syntax suggestion would be 
this:

def foo(non_const or []):
    ...

where [] is executed at runtime BECAUSE at def time non_const is somehow 
True and that is enough to leave [] alone.
I have not checked, but I believe it is backward compatible.
Anyway, could you summarize both contr-argument and this syntax proposal 
in the PEP?



Chris Rebert wrote:

>The following is a proto-PEP based on the discussion in the thread
>"fixing mutable default argument values". Comments would be greatly
>appreciated.
>- Chris Rebert
>
>Title: Fixing Non-constant Default Arguments
>
>Abstract
>
>     This PEP proposes new semantics for default arguments to remove 
>boilerplate code associated with non-constant default argument values,
>allowing them to be expressed more clearly and succinctly.
>
>
>Motivation
>
>     Currently, to write functions using non-constant default arguments, 
>one must use the idiom:
>
>     def foo(non_const=None):
>         if non_const is None:
>             non_const = some_expr
>         #rest of function
>
>or equivalent code. Naive programmers desiring mutable default arguments 
>often make the mistake of writing the following:
>
>     def foo(mutable=some_expr_producing_mutable):
>         #rest of function
>
>However, this does not work as intended, as 
>'some_expr_producing_mutable' is evaluated only *once* at 
>definition-time, rather than once per call at call-time.  This results 
>in all calls to 'foo' using the same default value, which can result in 
>unintended consequences.  This necessitates the previously mentioned 
>idiom. This unintuitive behavior is such a frequent stumbling block for 
>newbies that it is present in at least 3 lists of Python's problems [0] 
>[1] [2].
>     There are currently few, if any, known good uses of the current 
>behavior of mutable default arguments.  The most common one is to 
>preserve function state between calls.  However, as one of the lists [2] 
>comments, this purpose is much better served by decorators, classes, or 
>(though less preferred) global variables.
>     Therefore, since the current semantics aren't useful for 
>non-constant default values and an idiom is necessary to work around 
>this deficiency, why not change the semantics so that people can write 
>what they mean more directly, without the annoying boilerplate?
>
>
>Rationale
>
>     Originally, it was proposed that all default argument values be 
>deep-copied from the original (evaluated at definition-time) at each 
>invocation of the function where the default value was required. 
>However, this doesn't take into account default values that are not 
>literals, e.g. function calls, subscripts, attribute accesses.  Thus, 
>the new idea was to re-evaluate the default arguments at each call where 
>they were needed. There was some concern over the possible performance 
>hit this could cause, and whether there should be new syntax so that 
>code could use the existing semantics for performance reasons.  Some of 
>the proposed syntaxes were:
>
>     def foo(bar=<baz>):
>         #code
>
>     def foo(bar=new baz):
>         #code
>
>     def foo(bar=fresh baz):
>         #code
>
>     def foo(bar=separate baz):
>         #code
>
>     def foo(bar=another baz):
>         #code
>
>     def foo(bar=unique baz):
>         #code
>
>where the new keyword (or angle brackets) would indicate that the 
>parameter's default argument should use the new semantics.  Other 
>parameters would continue to use the old semantics. It was generally 
>agreed that the angle-bracket syntax was particularly ugly, leading to 
>the proposal of the other syntaxes. However, having 2 different sets of 
>semantics could be confusing and leaving in the old semantics just for 
>performance might be premature optimization. Refactorings to deal with 
>the possible performance hit are discussed below.
>
>
>Specification
>
>     The current semantics for default arguments are replaced by the 
>following semantics:
>     - Whenever a function is called, and the caller does not provide a
>     value for a parameter with a default expression, the parameter's
>     default expression shall be evaluated in the function's scope.  The
>     resulting value shall be assigned to a local variable in the
>     function's scope with the same name as the parameter.
>     - The default argument expressions shall be evaluated before the
>     body of the function.
>     - The evaluation of default argument expressions shall proceed in
>     the same order as that of the parameter list in the function's
>     definition.
>Given these semantics, it makes more sense to refer to default argument
>expressions rather than default argument values, as the expression is
>re-evaluated at each call, rather than just once at definition-time.
>Therefore, we shall do so hereafter.
>
>Demonstrative examples of new semantics:
>     #default argument expressions can refer to
>     #variables in the enclosing scope...
>     CONST = "hi"
>     def foo(a=CONST):
>         print a
>
>     >>> foo()
>     hi
>     >>> CONST="bye"
>     >>> foo()
>     bye
>
>     #...or even other arguments
>     def ncopies(container, n=len(container)):
>         return [container for i in range(n)]
>
>     >>> ncopies([1, 2], 5)
>     [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]]
>     >>> ncopies([1, 2, 3])
>     [[1, 2, 3], [1, 2, 3], [1, 2, 3]]
>     >>> #ncopies grabbed n from [1, 2, 3]'s length (3)
>
>     #default argument expressions are arbitrary expressions
>     def my_sum(lst):
>         cur_sum = lst[0]
>         for i in lst[1:]: cur_sum += i
>         return cur_sum
>
>     def bar(b=my_sum((["b"] * (2 * 3))[:4])):
>         print b
>
>     >>> bar()
>     bbbb
>
>     #default argument expressions are re-evaluated at every call...
>     from random import randint
>     def baz(c=randint(1,3)):
>         print c
>
>     >>> baz()
>     2
>     >>> baz()
>     3
>
>     #...but only when they're required
>     def silly():
>         print "spam"
>         return 42
>
>     def qux(d=silly()):
>         pass
>
>     >>> qux()
>     spam
>     >>> qux(17)
>     >>> qux(d=17)
>     >>> qux(*[17])
>     >>> qux(**{'d':17})
>     >>> #no output because silly() never called because d's value was 
>specified in the calls
>
>     #Rule 3
>     count = 0
>     def next():
>         global count
>         count += 1
>         return count - 1
>
>     def frobnicate(g=next(), h=next(), i=next()):
>         print g, h, i
>
>     >>> frobnicate()
>     0 1 2
>     >>> #g, h, and i's default argument expressions are evaluated in 
>the same order as the parameter definition
>
>
>Backwards Compatibility
>
>     This change in semantics breaks all code which uses mutable default 
>argument values. Such code can be refactored from:
>
>     def foo(bar=mutable):
>         #code
>
>to
>
>     def stateify(state):
>         def _wrap(func):
>             def _wrapper(*args, **kwds):
>                 kwds['bar'] = state
>                 return func(*args, **kwds)
>             return _wrapper
>         return _wrap
>
>     @stateify(mutable)
>     def foo(bar):
>         #code
>
>or
>
>     state = mutable
>     def foo(bar=state):
>         #code
>
>or
>
>     class Baz(object):
>         def __init__(self):
>             self.state = mutable
>
>         def foo(self, bar=self.state):
>             #code
>
>The changes in this PEP are backwards-compatible with all code whose 
>default argument values are immutable, including code using the idiom 
>mentioned in the 'Motivation' section. However, such values will now be 
>recomputed for each call for which they are required. This may cause 
>performance degradation. If such recomputation is significantly 
>expensive, the same refactorings mentioned above can be used.
>
>     In relation to Python 3.0, this PEP's proposal is compatible with 
>those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend 
>on the acceptance of either of those PEPs.
>
>
>Reference Implementation
>
>     All code of the form:
>
>         def foo(bar=some_expr, baz=other_expr):
>             #body
>
>     Should act as if it had read (in pseudo-Python):
>
>         def foo(bar=_undefined, baz=_undefined):
>             if bar is _undefined:
>                 bar = some_expr
>             if baz is _undefined:
>                 baz = other_expr
>             #body
>
>where _undefined is the value given to a parameter when the caller 
>didn't specify a value for it. This is not intended to be a literal 
>translation, but rather a demonstration as to how Python's internal 
>argument-handling machinery should be changed.
>
>
>References
>
>     [0] 10 Python pitfalls
>         http://zephyrfalcon.org/labs/python_pitfalls.html
>
>     [1] Python Gotchas
>         http://www.ferg.org/projects/python_gotchas.html#contents_item_6
>
>     [2] When Pythons Attack
>     http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2
>
>     [3] Keyword-Only Arguments
>         http://www.python.org/dev/peps/pep-3102/
>
>     [4] Function Annotations
>         http://www.python.org/dev/peps/pep-3107/
>_______________________________________________
>Python-ideas mailing list
>Python-ideas at python.org
>http://mail.python.org/mailman/listinfo/python-ideas
>
>
>!DSPAM:45bcf8295417644621095!
>
>  
>



From jimjjewett at gmail.com  Mon Jan 29 15:42:40 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Mon, 29 Jan 2007 09:42:40 -0500
Subject: [Python-ideas] simplification pep-like thing
In-Reply-To: <1d85506f0701260713k1fe0d548u8103e86a7dfffdf7@mail.gmail.com>
References: <1d85506f0701260713k1fe0d548u8103e86a7dfffdf7@mail.gmail.com>
Message-ID: <fb6fbf560701290642k7d04430ajd2b97356b33839db@mail.gmail.com>

On 1/26/07, tomer filiba <tomerfiliba at gmail.com> wrote:
> This proposal does not state what "atomic" means -- this is
> open to the decision of the class. The only restriction imposed is,
> collections of any kind must be simplified as tuples.

It is worth stating the minimum that a compliant serializer must be
able to treat atomically.  For instance, is it always sufficient to
reduce state to instances of (builtin, not subclasses of) string,
tuple, float, or int?

> Simplification Protocol
> =======================
> This proposal introduces two new special methods:

>     def __simplify__(self):
>         return type, state

If I understand correctly, this returns the actual type object, rather
than its name, or the source code to rebuild that type.  Are there any
requirements on what sort of type objects the serializer must be able
to support?

> I do not see a need for a convenience routine such as
> simplify(obj) <--> obj.__simplify__(), since this API is not designed
> for everyday usage. This is the case with __reduce__ today.

reduce is nasty for beginners.  In fairness, I think much of the
problem is that reduce and __reduce__ (as well as __reduce_ex__) both
exist, but are unrelated.  So adding simplify probably isn't required,
but reserving the name might be.

> Note that this proposal does not define how atomic objects are to be
> converted to strings, or how a 'recursive simplifier' should work. These
> issues are to be resolved by the implementation of the serializer.

Is there some minimum requirement on is-stability?

For example, would the following relationships be preserved after
deserialization?

    >>> a1=object()
    >>> a2=a1
    >>> b=object()
    >>> a1 is a2
    True
    >>> a1 is b
    False

-jJ


From tomerfiliba at gmail.com  Mon Jan 29 16:54:45 2007
From: tomerfiliba at gmail.com (tomer filiba)
Date: Mon, 29 Jan 2007 17:54:45 +0200
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
Message-ID: <1d85506f0701290754n688a622ft835c4203c749f31f@mail.gmail.com>

there really is no need to have something like that built into the language.
most default arguments are immutable anyhow, and i'd guess most of the
mutable defaults can be addressed with the suggested @copydefaults
decorator.

as for uncopyable or stateful defaults (dev = sys.stdout), which require
reevaluation,
you can just use this modified version of my first suggested decorator:

>>> def reeval(**kwdefs):
...     def deco(func):
...         def wrapper(*args, **kwargs):
...             for k, v in kwdefs.iteritems():
...                 if k not in kwargs:
...                     kwargs[k] = v()               # <--- this is the big
change
...             return func(*args, **kwargs)
...         return wrapper
...     return deco
...

the defaults are now provided as *functions*, which are evaluated at the
time of calling the function.

>>> @reeval(device = lambda: sys.stdout)
... def say(text, device):
...     device.write("%s: %s\n" % (device.name, text))
...     device.flush()

this means you can do things like --

>>> say("hello1")
<stdout>: hello1
>>>
>>> say("hello2", device = sys.stderr)
<stderr>: hello2
>>>
>>> sys.stdout = sys.stderr
>>> say("hello3")
<stderr>: hello3
>>>
>>> sys.stdout = sys.__stdout__
>>> say("hello4")
<stdout>: hello4

decorators are powerful enough for this not-so-common case.

it would be nice to have a collection of useful decorators as part of stdlib

(such as these ones, but including memoize and many others)...
but that's a different issue.

maybe we should really start a list of useful decorators to be included
as an stdlib module.


-tomer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20070129/8f758c37/attachment.html>

From jan.kanis at phil.uu.nl  Mon Jan 29 17:03:52 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Mon, 29 Jan 2007 17:03:52 +0100
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BCF804.7040207@gmail.com>
References: <45BCF804.7040207@gmail.com>
Message-ID: <op.tmxawql4d64u53@aidwe2058w.minlnv.agro.nl>

Well, I (obviously) like the idea, but your pep misses some important  
points (and some not so important).

The important one is about the scoping of variables in default  
expressions. The pep says nothing about them.
If I read the pep correctly, any variables in default expressions are  
handled the same as variables in the body of a function. This means the  
compiler decides if they should be local, lexical or global. If there is  
an assignment to a variable, the compiler makes it a local, else it finds  
the right enclosing scope (lexical or global). In the current python, this  
works fine:

  >>> a = 123
  >>> def foo(b=a):
       	a = 2
       	print a, b

  >>> foo()
  2 123
  >>> a = 42
  >>> foo()
  2 123

In the pep, the a in the default expression would be handled just like any  
other a in the function body, which means it wil become a _local_  
variable. Calling the function would then result in an UnboundLocalError.  
Just like this in current python:

  >>> a = 123
  >>> def foo(b=None):
       	b = a if b==None else b
       	a = 2
       	print a

  >>> foo()

  Traceback (most recent call last):
    File "<pyshell#22>", line 1, in <module>
      foo()
    File "<pyshell#21>", line 2, in foo
      b = a if b==None else b
  UnboundLocalError: local variable 'a' referenced before assignment

The solution, I think, as I wrote in my previous messages, is to have the  
compiler explicitly make variables in default expressions lexical or  
global variables.
This would still break the foo() in my example above, because you can't  
assign to a lexical variable, or to a global that isn't declared global.  
Therefore I think the compiler should distinguish between the a in the  
default expression and the a in the function body, and treat them as two  
different variables. You can think about it as if the compiler silently  
renames one of them (without this rename being visible to python code.   
AFAIK the bytecode is stack based and closure vars are put in some kind of  
anonymous cell, which means they both don't actually have a name anyway.)

see below for more comments regarding this and other things


On Sun, 28 Jan 2007 20:22:44 +0100, Chris Rebert <cvrebert at gmail.com>  
wrote:

> The following is a proto-PEP based on the discussion in the thread
> "fixing mutable default argument values". Comments would be greatly
> appreciated.
> - Chris Rebert
>
> Title: Fixing Non-constant Default Arguments
>
> Abstract
>
>      This PEP proposes new semantics for default arguments to remove
> boilerplate code associated with non-constant default argument values,
> allowing them to be expressed more clearly and succinctly.
>
>
> Motivation
>
>      Currently, to write functions using non-constant default arguments,
> one must use the idiom:
>
>      def foo(non_const=None):
>          if non_const is None:
>              non_const = some_expr
>          #rest of function
>
> or equivalent code. Naive programmers desiring mutable default arguments
> often make the mistake of writing the following:
>
>      def foo(mutable=some_expr_producing_mutable):
>          #rest of function
>
> However, this does not work as intended, as
> 'some_expr_producing_mutable' is evaluated only *once* at
> definition-time, rather than once per call at call-time.  This results
> in all calls to 'foo' using the same default value, which can result in
> unintended consequences.  This necessitates the previously mentioned
> idiom. This unintuitive behavior is such a frequent stumbling block for
> newbies that it is present in at least 3 lists of Python's problems [0]
> [1] [2].

Also, I just found out that python's own documentation refers to this with  
an "Important warning: The default value is evaluated only once. This  
makes a difference when the default is a mutable object such as a list,  
dictionary, or instances of most classes. ..."
(http://docs.python.org/tut/node6.html#SECTION006710000000000000000)
this indicates imo that also the python doc writers don't think of the  
current situation as optimal.

>      There are currently few, if any, known good uses of the current
> behavior of mutable default arguments.  The most common one is to
> preserve function state between calls.  However, as one of the lists [2]
> comments, this purpose is much better served by decorators, classes, or
> (though less preferred) global variables.
>      Therefore, since the current semantics aren't useful for
> non-constant default values and an idiom is necessary to work around
> this deficiency, why not change the semantics so that people can write
> what they mean more directly, without the annoying boilerplate?
>
>
> Rationale
>
>      Originally, it was proposed that all default argument values be
> deep-copied from the original (evaluated at definition-time) at each
> invocation of the function where the default value was required.
> However, this doesn't take into account default values that are not
> literals, e.g. function calls, subscripts, attribute accesses.  Thus,
> the new idea was to re-evaluate the default arguments at each call where
> they were needed. There was some concern over the possible performance
> hit this could cause, and whether there should be new syntax so that
> code could use the existing semantics for performance reasons.  Some of
> the proposed syntaxes were:
>
>      def foo(bar=<baz>):
>          #code
>
>      def foo(bar=new baz):
>          #code
>
>      def foo(bar=fresh baz):
>          #code
>
>      def foo(bar=separate baz):
>          #code
>
>      def foo(bar=another baz):
>          #code
>
>      def foo(bar=unique baz):
>          #code
>
> where the new keyword (or angle brackets) would indicate that the
> parameter's default argument should use the new semantics.  Other
> parameters would continue to use the old semantics. It was generally
> agreed that the angle-bracket syntax was particularly ugly, leading to
> the proposal of the other syntaxes. However, having 2 different sets of
> semantics could be confusing and leaving in the old semantics just for
> performance might be premature optimization. Refactorings to deal with
> the possible performance hit are discussed below.
>
>
> Specification
>
>      The current semantics for default arguments are replaced by the
> following semantics:
>      - Whenever a function is called, and the caller does not provide a
>      value for a parameter with a default expression, the parameter's
>      default expression shall be evaluated in the function's scope.  The
>      resulting value shall be assigned to a local variable in the
>      function's scope with the same name as the parameter.

Include something saying that any variables in a default expression shall  
be lexical variables, with their scope being the first outer scope that  
defines a variable with the same name (they should just use the same rules  
as other lexical/closure variables), and that if the function body defines  
a local variable with the same name as a variable in a default expression,  
those variables shall be handled as two separate variables.

>      - The default argument expressions shall be evaluated before the
>      body of the function.
>      - The evaluation of default argument expressions shall proceed in
>      the same order as that of the parameter list in the function's
>      definition.
> Given these semantics, it makes more sense to refer to default argument
> expressions rather than default argument values, as the expression is
> re-evaluated at each call, rather than just once at definition-time.
> Therefore, we shall do so hereafter.
>
> Demonstrative examples of new semantics:
>      #default argument expressions can refer to
>      #variables in the enclosing scope...
>      CONST = "hi"
>      def foo(a=CONST):
>          print a
>
>      >>> foo()
>      hi
>      >>> CONST="bye"
>      >>> foo()
>      bye
>
>      #...or even other arguments
>      def ncopies(container, n=len(container)):
>          return [container for i in range(n)]
>
>      >>> ncopies([1, 2], 5)
>      [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]]
>      >>> ncopies([1, 2, 3])
>      [[1, 2, 3], [1, 2, 3], [1, 2, 3]]
>      >>> #ncopies grabbed n from [1, 2, 3]'s length (3)

I'm not sure if this can be combined elegantly with what I said about  
variables being lexical variables. The first argument to ncopies,  
'container', is clearly a local variable to ncopies. The 'container' in  
the second arg default expr should, if my comments above are accepted, be  
a lexical variable referring to the 'container' in the global scope.
The best way to combine the two features seems to be to let 'container' be  
a local var if any of the preceding args is named 'container', and let it  
be a lexically scoped variable otherwise. However, I'm not convinced this  
complexity is worth it and the vars in default expressions shouldn't just  
always be lexical vars.

>
>      #default argument expressions are arbitrary expressions
>      def my_sum(lst):
>          cur_sum = lst[0]
>          for i in lst[1:]: cur_sum += i
>          return cur_sum
>
>      def bar(b=my_sum((["b"] * (2 * 3))[:4])):
>          print b
>
>      >>> bar()
>      bbbb
>
>      #default argument expressions are re-evaluated at every call...
>      from random import randint
>      def baz(c=randint(1,3)):
>          print c
>
>      >>> baz()
>      2
>      >>> baz()
>      3
>
>      #...but only when they're required
>      def silly():
>          print "spam"
>          return 42
>
>      def qux(d=silly()):
>          pass
>
>      >>> qux()
>      spam
>      >>> qux(17)
>      >>> qux(d=17)
>      >>> qux(*[17])
>      >>> qux(**{'d':17})
>      >>> #no output because silly() never called because d's value was
> specified in the calls
>
>      #Rule 3
>      count = 0
>      def next():
>          global count
>          count += 1
>          return count - 1
>
>      def frobnicate(g=next(), h=next(), i=next()):
>          print g, h, i
>
>      >>> frobnicate()
>      0 1 2
>      >>> #g, h, and i's default argument expressions are evaluated in
> the same order as the parameter definition
>
>
> Backwards Compatibility
>
>      This change in semantics breaks all code which uses mutable default
> argument values. Such code can be refactored from:

Wow, let's not scare everyone away just yet. This should read:
"This change in semantics breaks code which uses mutable default argument  
expressions and depends on those expressions being evaluated only once, or  
code that assigns new incompatible values in a parent scope to variables  
used in default expressions"

>
>      def foo(bar=mutable):
>          #code

if 'mutable' is just a single variable, this isn't gonna break, unless the  
global scope decides to do something like this:

def foo(bar=mutable):
     #code

mutable = incompatible_mutable
# ...
foo()

>
> to
>
>      def stateify(state):
>          def _wrap(func):
>              def _wrapper(*args, **kwds):
>                  kwds['bar'] = state
>                  return func(*args, **kwds)
>              return _wrapper
>          return _wrap
>
>      @stateify(mutable)
>      def foo(bar):
>          #code
>
> or
>
>      state = mutable
>      def foo(bar=state):
>          #code
>
> or
>
>      class Baz(object):
>          def __init__(self):
>              self.state = mutable
>
>          def foo(self, bar=self.state):
>              #code

Minor point: the stateify decorator looks a bit scary to me as it uses  
three levels of nested functions. (that's inherent to decorators, but  
still.) Suggest you name the class and global var solutions first, and the  
decorator as last, just to prevent people from stopping reading the pep  
and voting '-1' right when they hit the decorator solution.

>
> The changes in this PEP are backwards-compatible with all code whose
> default argument values are immutable

...or don't depend on being evaluated only once, and don't modify in a  
parent scope the variables in default expressions in an incompatible way,

(hmm, the 'or' and 'and' may need some disambiguation parentheses...)

> including code using the idiom
> mentioned in the 'Motivation' section. However, such values will now be
> recomputed for each call for which they are required. This may cause
> performance degradation. If such recomputation is significantly
> expensive, the same refactorings mentioned above can be used.
>
>      In relation to Python 3.0, this PEP's proposal is compatible with
> those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend
> on the acceptance of either of those PEPs.
>
>
> Reference Implementation
>
>      All code of the form:
>
>          def foo(bar=some_expr, baz=other_expr):
>              #body
>
>      Should act as if it had read (in pseudo-Python):
>
>          def foo(bar=_undefined, baz=_undefined):
>              if bar is _undefined:
>                  bar = some_expr
>              if baz is _undefined:
>                  baz = other_expr
>              #body

and, if there are any variables occuring in the function body and in  
some_expr or other_expr, rename those in the function body to something  
that doesn't name-clash.

>
> where _undefined is the value given to a parameter when the caller
> didn't specify a value for it. This is not intended to be a literal
> translation, but rather a demonstration as to how Python's internal
> argument-handling machinery should be changed.
>
>
> References
>
>      [0] 10 Python pitfalls
>          http://zephyrfalcon.org/labs/python_pitfalls.html
>
>      [1] Python Gotchas
>          http://www.ferg.org/projects/python_gotchas.html#contents_item_6
>
>      [2] When Pythons Attack
>      http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2
>
>      [3] Keyword-Only Arguments
>          http://www.python.org/dev/peps/pep-3102/
>
>      [4] Function Annotations
>          http://www.python.org/dev/peps/pep-3107/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas

I'm not quite sure if your decision not to include the  
default-expr-vars-become-lexical part was intentional or not. If it is,  
can you tell me why you'd want that? Else you can incorporate my comments  
in the pep.
Another minor point: I'm personally not too fond of too many 'shall's  
close together. It makes me think of lots of bureaucracy and  
design-by-committee. I think several peps don't have a  shall in them, but  
maybe it is the right language for this one. What's the right language to  
use in peps?

Well, that's about it I think.

- Jan


From jimjjewett at gmail.com  Mon Jan 29 23:51:04 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Mon, 29 Jan 2007 17:51:04 -0500
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BCF804.7040207@gmail.com>
References: <45BCF804.7040207@gmail.com>
Message-ID: <fb6fbf560701291451i2b8442bdof217cf172726cf66@mail.gmail.com>

On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:

> Naive programmers desiring mutable default arguments
> often make the mistake of writing the following:

>      def foo(mutable=some_expr_producing_mutable):
>          #rest of function

Yes, it is an ugly gotcha, but so is the alternative.

If it is changed, then just as many naive programmers (though perhaps
not exactly the same ones) will make the opposite mistake -- and so
will some experienced programmers who are used to current semantics.

In a dynamic language, it makes perfect sense to reevaluate the entire
call signature with each call -- but in almost any language, it makes
perfect sense for the signature to be a constant.  Usually, it doesn't
matter which you assume, which is why this problem doesn't get ironed
out in the first few minutes of writing python.

>      There are currently few, if any, known good uses of the current
> behavior of mutable default arguments.  The most common one is to
> preserve function state between calls.  However, as one of the lists [2]
> comments, this purpose is much better served by decorators, classes, or
> (though less preferred) global variables.

I disagree.  This is particularly wrong for someone coming from a
functional background.

A class plus an instantiation seems far too heavyweight for what ought
to be a simple function.  I'm not talking (only) about the runtime;
the number of methods and lines of code is the real barrier for me.
I'll sometimes do it (or use a global) anyhow, but it feels wrong.  If
I had felt that need more often when I was first learning python (or
before I knew about the  __call__ workaround), I might have just
written the language off as no less bloated than java.

You see the problem with globals, but decorators are in some sense
worse -- a function cannot see its own decorations.  At best, it can
*assume* that repeating its own name (despite Don't Repeat Yourself)
will get another reference to self, but this isn't always true.
Programmers used to creating functions outside of toplevel (or
class-level) will be more aware of this, and see the suggestion as an
indication that python is inherently buggy.

>      def foo(bar=new baz):
>          #code

This would be less bad.

That said, I fear many new programmers would fail to understand when
they needed new and when they didn't, so that in practice, it would be
just optional random noise.

> Demonstrative examples of new semantics:
>      #default argument expressions can refer to
>      #variables in the enclosing scope...
>      CONST = "hi"
>      def foo(a=CONST):
>          print a

This would work if there were any easy way to create a new scope.  In
Lisp, it makes sense.  In python, it would probably be better to just
find a way for functions to refer to their own decorations reliably.

> Backwards Compatibility
>
>      This change in semantics breaks all code which uses mutable default
> argument values. Such code can be refactored from:
>
>      def foo(bar=mutable):
>          #code
>
> to

    [a decorator option uses 3 levels of nested functions, which aren't even
    generic enough for reuse, unless you give up introspection.]

No.  It could be done with a (more complex) decorator, if that
decorator came with the stdlib (probably in functools), but that is
heavy backwards-incompatibility to change a corner case (most defaults
aren't mutable) in a taste-dependent manner.

-jJ


From jimjjewett at gmail.com  Tue Jan 30 00:11:11 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Mon, 29 Jan 2007 18:11:11 -0500
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BDA47F.4040603@onego.ru>
References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru>
Message-ID: <fb6fbf560701291511g3a981b0an2ecfc78635fa9c4e@mail.gmail.com>

On 1/29/07, Roman Susi <rnd at onego.ru> wrote:

> P.S. However, I may be wrong. In that case my syntax suggestion would be
> this:

> def foo(non_const or []):
>     ...
>
> where [] is executed at runtime BECAUSE at def time non_const is somehow
> True and that is enough to leave [] alone.

It would also be possible to treat literals (like "[]") as "do it over
each time", and more general expressions (like "list()") as they are
treated today.

Though like Roman, I think this would still be worse than the status quo.

-jJ


From rhamph at gmail.com  Tue Jan 30 01:07:00 2007
From: rhamph at gmail.com (Adam Olsen)
Date: Mon, 29 Jan 2007 17:07:00 -0700
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <fb6fbf560701291511g3a981b0an2ecfc78635fa9c4e@mail.gmail.com>
References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru>
	<fb6fbf560701291511g3a981b0an2ecfc78635fa9c4e@mail.gmail.com>
Message-ID: <aac2c7cb0701291606w36460832ob70648727aac9519@mail.gmail.com>

On 1/29/07, Jim Jewett <jimjjewett at gmail.com> wrote:
> On 1/29/07, Roman Susi <rnd at onego.ru> wrote:
>
> > P.S. However, I may be wrong. In that case my syntax suggestion would be
> > this:
>
> > def foo(non_const or []):
> >     ...
> >
> > where [] is executed at runtime BECAUSE at def time non_const is somehow
> > True and that is enough to leave [] alone.
>
> It would also be possible to treat literals (like "[]") as "do it over
> each time", and more general expressions (like "list()") as they are
> treated today.
>
> Though like Roman, I think this would still be worse than the status quo.

Another, more bizarre approach would be to require default arguments
be hashable.  This would prevent lists from being used, protecting
people from mistakes.

None of the options given sit right with me though.

-- 
Adam Olsen, aka Rhamphoryncus


From cvrebert at gmail.com  Tue Jan 30 04:17:15 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Mon, 29 Jan 2007 19:17:15 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <fb6fbf560701291511g3a981b0an2ecfc78635fa9c4e@mail.gmail.com>
References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru>
	<fb6fbf560701291511g3a981b0an2ecfc78635fa9c4e@mail.gmail.com>
Message-ID: <45BEB8BB.7010806@gmail.com>

Jim Jewett wrote:
> On 1/29/07, Roman Susi <rnd at onego.ru> wrote:
> 
>> P.S. However, I may be wrong. In that case my syntax suggestion would be
>> this:
> 
>> def foo(non_const or []):
>>     ...
>>
>> where [] is executed at runtime BECAUSE at def time non_const is somehow
>> True and that is enough to leave [] alone.
> 
> It would also be possible to treat literals (like "[]") as "do it over
> each time", and more general expressions (like "list()") as they are
> treated today.
As discussed in the proto-PEP, there are many cases where non-literal 
default values need to be re-evaluated at each call. A solution to this 
problem needs to take this into account. Also, the proto-PEP's proposal 
tries to have the behavior of default arguments be as uniform as 
possible and not "magically" decide the correct behavior, especially by 
using something as arbitrary as what expression is used for the default 
argument.

- Chris Rebert


From gsakkis at rutgers.edu  Tue Jan 30 05:16:50 2007
From: gsakkis at rutgers.edu (George Sakkis)
Date: Mon, 29 Jan 2007 23:16:50 -0500
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BDA47F.4040603@onego.ru>
References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru>
Message-ID: <91ad5bf80701292016m63bf1b28w84e2c9f62fb7ea0e@mail.gmail.com>

> Chris Rebert wrote:
>
> >The following is a proto-PEP based on the discussion in the thread
> >"fixing mutable default argument values". Comments would be greatly
> >appreciated.

As I see it, the main objection to this is the inversal of the current
default semantics: every default argument is treated as if it were
mutable, or re-evaluatable more generally. Although mutable default
arguments are useful some times and would be nice to have, they are
most likely less common than the immutable ones, so the latter should
be the default. I'm sure that the python-devs and the BDFL would have
thought about it quite a bit when the current semantics were decided,
and it's unlikely they'll change their mind now without very good
reasons.

OTOH, a proposal that leaves the current semantics as is and adds a
mechanism to specify default arguments to be evaluated at call-time
rather than definition-time would have more chances. Here's a
proof-of-concept solution that specifies explicitly the re-evaluatable
default expressions as "deferred":


import inspect

class Deferred(object):
    def __init__(self, expr):
        self.expr = expr

def eval_deferreds(func):
    varnames,_,_,defaults = inspect.getargspec(func)
    num_varnames = len(varnames); num_defaults = len(defaults)
    def wrapper(*args, **kwds):
        if len(args) >= num_varnames: # defaults not used here
            return func(*args,**kwds)
        f_locals = dict(zip(varnames,args))
        used_defaults = min(num_defaults, num_varnames-len(args))
        for var,default in zip(varnames[-used_defaults:],
defaults[-used_defaults:]):
            if var in kwds: # passed as keyword argument; don't use the default
                value = kwds[var]
            elif not isinstance(default, Deferred): # non re-evaluatable default
                value = default
            else: # evaluatable default in f_locals
                value = eval(default.expr, func.func_globals, f_locals)
            f_locals[var] = value
        f_locals.update(kwds) # add any extra keyword arguments
        return func(**f_locals)
    return wrapper


#======= example ==============================

W = 1   # some global

@eval_deferreds
def f(x, y=Deferred('x**2+W'), z=Deferred('[]')):
    z.append(x)
    z.append(y)
    return z

from collections import deque
print f(3)                    # [3,10]
W=3; print f(4)           # [4,19]
print f(4,5)                 # [4,5]
print f(-1, z=deque())  # deque([-1,4])


Regards,
George


From collinw at gmail.com  Tue Jan 30 05:36:51 2007
From: collinw at gmail.com (Collin Winter)
Date: Mon, 29 Jan 2007 22:36:51 -0600
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BCF804.7040207@gmail.com>
References: <45BCF804.7040207@gmail.com>
Message-ID: <43aa6ff70701292036t7f0a2a96hc013ecd47957de4d@mail.gmail.com>

On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:
> Rationale
[snip]
> There was some concern over the possible performance
> hit this could cause, and whether there should be new syntax so that
> code could use the existing semantics for performance reasons.  Some of
> the proposed syntaxes were:
>
>      def foo(bar=<baz>):
>          #code
>
>      def foo(bar=new baz):
>          #code
>
>      def foo(bar=fresh baz):
>          #code
>
>      def foo(bar=separate baz):
>          #code
>
>      def foo(bar=another baz):
>          #code
>
>      def foo(bar=unique baz):
>          #code
>
> where the new keyword (or angle brackets) would indicate that the
> parameter's default argument should use the new semantics.

Syntax changes are a huge stick to wield against such a small problem.
I realize the boilerplate is annoying, but Tomer has pointed out a
number of decorator-based solutions [1] that could easily be adapted
to any unusual needs you have.

Also, you haven't talked at all about how all this might be
accomplished. You say

> Given these semantics, it makes more sense to refer to default argument
> expressions rather than default argument values, as the expression is
> re-evaluated at each call, rather than just once at definition-time.

but how do you intend to capture these "default argument expressions"?
Emit additional bytecode for functions making use of these special
default arguments?

Collin Winter


From cvrebert at gmail.com  Tue Jan 30 05:54:19 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Mon, 29 Jan 2007 20:54:19 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <43aa6ff70701292036t7f0a2a96hc013ecd47957de4d@mail.gmail.com>
References: <45BCF804.7040207@gmail.com>
	<43aa6ff70701292036t7f0a2a96hc013ecd47957de4d@mail.gmail.com>
Message-ID: <45BECF7B.6090503@gmail.com>

Collin Winter wrote:
> On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:
> Syntax changes are a huge stick to wield against such a small problem.
> I realize the boilerplate is annoying, but Tomer has pointed out a
> number of decorator-based solutions [1] that could easily be adapted
> to any unusual needs you have.

One of his decorators assumes that the default value merely needs to be 
copied across calls. However, as the PEP says, there are cases where 
this isn't sufficient or just doesn't work. His second decorator 
involves using lambdas (ick!), and doesn't allow you to refer to other 
arguments in determining the default value (think 'self' as a hint for 
how this could be useful). Also, I just think it's plain nicer/prettier 
if decorators don't have to be used for this.

> Also, you haven't talked at all about how all this might be
> accomplished. You say
> 
>> Given these semantics, it makes more sense to refer to default argument
>> expressions rather than default argument values, as the expression is
>> re-evaluated at each call, rather than just once at definition-time.
> 
> but how do you intend to capture these "default argument expressions"?
> Emit additional bytecode for functions making use of these special
> default arguments?

The "Reference Implementation" section of the PEP discusses this. I 
don't personally know Python's internals, but I imagine this proposal 
would just change how default arguments are compiled and might entail 
some changes to the interpreter's argument-processing machinery. As Jan 
Kanis pointed out, the PEP does need to elaborate on exactly what kind 
of variables the default arguments use/are. I'm working on revising this 
in the next draft.

- Chris Rebert


From cvrebert at gmail.com  Tue Jan 30 06:09:29 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Mon, 29 Jan 2007 21:09:29 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <fb6fbf560701291451i2b8442bdof217cf172726cf66@mail.gmail.com>
References: <45BCF804.7040207@gmail.com>
	<fb6fbf560701291451i2b8442bdof217cf172726cf66@mail.gmail.com>
Message-ID: <45BED309.10301@gmail.com>

Jim Jewett wrote:
> On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:
> 
>> Naive programmers desiring mutable default arguments
>> often make the mistake of writing the following:
> 
>>      def foo(mutable=some_expr_producing_mutable):
>>          #rest of function
> 
> Yes, it is an ugly gotcha, but so is the alternative.
> 
> If it is changed, then just as many naive programmers (though perhaps
> not exactly the same ones) will make the opposite mistake -- and so
> will some experienced programmers who are used to current semantics.

Anyone (ab)using the current semantics with the above construct is or 
ought to be aware that such usage is unpythonic. Also, due to the 
hairiness/obscurity of the construct, hardly anyone uses it, so the 
impact of removing it should be relatively minor. Additionally, I see 
making the opposite mistake as usually only maybe causing a performance 
problem, as opposed to causing the program to work incorrectly. As 
discussed in my proposal, this might be premature optimization. If it is 
a serious issue (which hasn't been indicated by the discussion so far), 
we can add syntax for the old/new semantics, which I also mentioned in 
the proposal.

>>      There are currently few, if any, known good uses of the current
>> behavior of mutable default arguments.  The most common one is to
>> preserve function state between calls.  However, as one of the lists [2]
>> comments, this purpose is much better served by decorators, classes, or
>> (though less preferred) global variables.
> 
> I disagree.  This is particularly wrong for someone coming from a
> functional background.

I assume you disagree with the "purpose is much better served by 
decorators, classes, or local variables" part as opposed to the "default 
mutable arguments with current semantics have few good uses" part. 
Please correct me if I'm in error.

> A class plus an instantiation seems far too heavyweight for what ought
> to be a simple function.  I'm not talking (only) about the runtime;
> the number of methods and lines of code is the real barrier for me.
> I'll sometimes do it (or use a global) anyhow, but it feels wrong.  If
> I had felt that need more often when I was first learning python (or
> before I knew about the  __call__ workaround), I might have just
> written the language off as no less bloated than java.

I'm sorry, but when you have a function sharing state between calls, 
that just screams to me that you should make it a method of an object so 
you don't have to store the state in some roundabout way, such as in 
mutable default arguments. If performance is your concern, the decorator 
version might perform better (I don't really know), or in the extreme 
case, you could use a global variable, which is definitely faster. Speed 
and elegance often come in inverse proportions. Also, I've revised the 
refactoring in question so that you no longer need to instanciate the 
class, which at least makes it marginally better.

> You see the problem with globals, but decorators are in some sense
> worse -- a function cannot see its own decorations.  At best, it can
> *assume* that repeating its own name (despite Don't Repeat Yourself)
> will get another reference to self, but this isn't always true.

I really don't see how the decorator in the PEP is any worse than other 
decorators in this regard. The problem you describe currently applies to 
all decorated functions, though functools.wraps might help mitigate this 
situation.

> Programmers used to creating functions outside of toplevel (or
> class-level) will be more aware of this, and see the suggestion as an
> indication that python is inherently buggy.
> 
>>      def foo(bar=new baz):
>>          #code
> 
> This would be less bad.
> 
> That said, I fear many new programmers would fail to understand when
> they needed new and when they didn't, so that in practice, it would be
> just optional random noise.

This is part of the reason I'm trying to avoid adding new syntax. 
However, I assert that at least 'new' is clearer than the' x=None; if x 
is None: x=expr' idiom in that it expresses one's intent more clearly. 
Also, this would at least be a prettier way to spell the idiom even if 
the reason still needed explaining. Alternatively, we could make the new 
semantics the default and have syntax to use the old semantics via 
'once' or some other keyword. This does nicely emphasize that such 
semantics would be purely for optimization reasons. I think I'll add 
this to the PEP.

>> Demonstrative examples of new semantics:
>>      #default argument expressions can refer to
>>      #variables in the enclosing scope...
>>      CONST = "hi"
>>      def foo(a=CONST):
>>          print a
> 
> This would work if there were any easy way to create a new scope.  In
> Lisp, it makes sense.  In python, it would probably be better to just
> find a way for functions to refer to their own decorations reliably.

This is outside of the scope of my PEP. However, the below improvement 
should help and you could always use one of the other refactorings to 
work around this issue.

>> Backwards Compatibility
>>
>>      This change in semantics breaks all code which uses mutable default
>> argument values. Such code can be refactored from:
>>
>>      def foo(bar=mutable):
>>          #code
>>
>> to
> 
>    [a decorator option uses 3 levels of nested functions, which aren't even
>    generic enough for reuse, unless you give up introspection.]
> 
> No.  It could be done with a (more complex) decorator, if that
> decorator came with the stdlib (probably in functools)

Agreed, the decorator could be better. I've just enhanced it using 
functools.wraps, which should help with the introspection issues you 
raise. Other improvements to the refactoring code in the PEP are welcomed.

- Chris Rebert


From collinw at gmail.com  Tue Jan 30 06:22:10 2007
From: collinw at gmail.com (Collin Winter)
Date: Mon, 29 Jan 2007 23:22:10 -0600
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BECF7B.6090503@gmail.com>
References: <45BCF804.7040207@gmail.com>
	<43aa6ff70701292036t7f0a2a96hc013ecd47957de4d@mail.gmail.com>
	<45BECF7B.6090503@gmail.com>
Message-ID: <43aa6ff70701292122o6248a9afi3a139a106fe717f5@mail.gmail.com>

On 1/29/07, Chris Rebert <cvrebert at gmail.com> wrote:
> Collin Winter wrote:
> > On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:
> > Syntax changes are a huge stick to wield against such a small problem.
> > I realize the boilerplate is annoying, but Tomer has pointed out a
> > number of decorator-based solutions [1] that could easily be adapted
> > to any unusual needs you have.
>
> One of his decorators assumes that the default value merely needs to be
> copied across calls. However, as the PEP says, there are cases where
> this isn't sufficient or just doesn't work.

Tomer wasn't proposing that any of those decorators be included in the
standard library. Those cases where one of decorators isn't sufficient
as written? Modify it, use it in your code, post it to the cookbook.
Not every n-line function should be turned into syntax.

> His second decorator
> involves using lambdas (ick!), and doesn't allow you to refer to other
> arguments in determining the default value (think 'self' as a hint for
> how this could be useful).

You can't do that as things are, so that's not a strike against the
decorator-based approach.

> > Also, you haven't talked at all about how all this might be
> > accomplished. You say
> >
> >> Given these semantics, it makes more sense to refer to default argument
> >> expressions rather than default argument values, as the expression is
> >> re-evaluated at each call, rather than just once at definition-time.
> >
> > but how do you intend to capture these "default argument expressions"?
> > Emit additional bytecode for functions making use of these special
> > default arguments?
>
> The "Reference Implementation" section of the PEP discusses this. I
> don't personally know Python's internals, but I imagine this proposal
> would just change how default arguments are compiled and might entail
> some changes to the interpreter's argument-processing machinery.

As you work on this, think about how you'll explain the new semantics
in the documentation. Rewriting http://docs.python.org/ref/calls.html
and http://docs.python.org/ref/function.html -- paying particular
attention to paragraphs 2-5 in the former -- would be a good start.

Collin Winter


From cvrebert at gmail.com  Tue Jan 30 06:23:25 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Mon, 29 Jan 2007 21:23:25 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BDA47F.4040603@onego.ru>
References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru>
Message-ID: <45BED64D.10801@gmail.com>

Roman Susi wrote:
> Hello!
> 
> I'd liked to say outright that this bad idea which complicates matters 
> more than provides solutions.
> Right now it is enough to know that the part from def to ":" is executed 
> at definition time. This is what
> incremental dynamic semantics is about. So, the suggestion is good only 
> as separated feature, but is IMHO wrong
> if considered in the language design as a whole.

You're entitled to your own opinion on the PEP.

> So things like
> 
> def foo(non_const=None):
>    non_const = non_const or []
> 
> are good becuase explicitely tell you that the mutable object is to be 
> created at call-time, not def-time.

The 'new' (or similar) keyword (might) indicate the new semantics, or 
alternatively, 'old' (or a similar) keyword (might) indicate the old 
semantics. If the new semantics become the default (as the PEP 
proposes), then this point is moot anyway as it will be explicit by way 
of the language definition.

> And I do not like PEP 3107 neither: its overly complex.
> 
> If there is a need for Python type checking, I'd suggested to make a 
> special superset which could be used
> to write compiled extensions as well (Pyrex comes to mind).

That's not part of my proposal, nor does my PEP depend on that one. I 
merely mention PEP 3107 when considering compatibility with other 
3100-series PEPs.

> P.S. However, I may be wrong. In that case my syntax suggestion would be 
> this:
> 
> def foo(non_const or []):
>    ...
> 
> where [] is executed at runtime BECAUSE at def time non_const is somehow 
> True and that is enough to leave [] alone.
> I have not checked, but I believe it is backward compatible.
> Anyway, could you summarize both contr-argument and this syntax proposal 
> in the PEP?

I don't quite understand exactly how this would work and would like more 
details on it, but once you've explained it, of course I'd be happy to 
include it in the next draft.

- Chris Rebert


From cvrebert at gmail.com  Tue Jan 30 06:27:53 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Mon, 29 Jan 2007 21:27:53 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <aac2c7cb0701291606w36460832ob70648727aac9519@mail.gmail.com>
References: <45BCF804.7040207@gmail.com>
	<45BDA47F.4040603@onego.ru>	<fb6fbf560701291511g3a981b0an2ecfc78635fa9c4e@mail.gmail.com>
	<aac2c7cb0701291606w36460832ob70648727aac9519@mail.gmail.com>
Message-ID: <45BED759.6060109@gmail.com>

Adam Olsen wrote:
> [snip]
> Another, more bizarre approach would be to require default arguments
> be hashable.  This would prevent lists from being used, protecting
> people from mistakes.

If my PEP ends up being rejected, I would fully support this. It 
wouldn't solve the more general problem, but at least it would prevent a 
common newbie mistake.

- Chris Rebert


From cvrebert at gmail.com  Tue Jan 30 06:40:35 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Mon, 29 Jan 2007 21:40:35 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <91ad5bf80701292016m63bf1b28w84e2c9f62fb7ea0e@mail.gmail.com>
References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru>
	<91ad5bf80701292016m63bf1b28w84e2c9f62fb7ea0e@mail.gmail.com>
Message-ID: <45BEDA53.9010103@gmail.com>

George Sakkis wrote:
> As I see it, the main objection to this is the inversal of the current
> default semantics: every default argument is treated as if it were
> mutable, or re-evaluatable more generally. Although mutable default
> arguments are useful some times and would be nice to have, they are
> most likely less common than the immutable ones, so the latter should
> be the default.

Why? Yes, there _might_ be performance issues (which have yet to be 
demonstrated or deeply speculated upon), but re-evaluating immutable 
default arguments wouldn't affect a program's correct operation.

> I'm sure that the python-devs and the BDFL would have
> thought about it quite a bit when the current semantics were decided,

...which was probably a while ago. They might reconsider the issue now 
that some time has passed and they've seen how their decision has worked 
out. But yes, your analysis is a definite possibility.

> and it's unlikely they'll change their mind now without very good
> reasons.

I hope to provide those reasons in my PEP.

> OTOH, a proposal that leaves the current semantics as is and adds a
> mechanism to specify default arguments to be evaluated at call-time
> rather than definition-time would have more chances.

My PEP does discuss the possibility of adding new syntax for the new 
semantics and leaving the old semantics as the default. The final 
proposal will take into account the community's opinion as to the 
specifics of how the syntax/semantics ought to be changed.

> Here's a
> proof-of-concept solution that specifies explicitly the re-evaluatable
> default expressions as "deferred":
> 
> 
> import inspect
> 
> class Deferred(object):
>    def __init__(self, expr):
>        self.expr = expr
> 
> def eval_deferreds(func):
>    varnames,_,_,defaults = inspect.getargspec(func)
>    num_varnames = len(varnames); num_defaults = len(defaults)
>    def wrapper(*args, **kwds):
>        if len(args) >= num_varnames: # defaults not used here
>            return func(*args,**kwds)
>        f_locals = dict(zip(varnames,args))
>        used_defaults = min(num_defaults, num_varnames-len(args))
>        for var,default in zip(varnames[-used_defaults:],
> defaults[-used_defaults:]):
>            if var in kwds: # passed as keyword argument; don't use the 
> default
>                value = kwds[var]
>            elif not isinstance(default, Deferred): # non re-evaluatable 
> default
>                value = default
>            else: # evaluatable default in f_locals
>                value = eval(default.expr, func.func_globals, f_locals)
>            f_locals[var] = value
>        f_locals.update(kwds) # add any extra keyword arguments
>        return func(**f_locals)
>    return wrapper
> 
> 
> #======= example ==============================
> 
> W = 1   # some global
> 
> @eval_deferreds
> def f(x, y=Deferred('x**2+W'), z=Deferred('[]')):
>    z.append(x)
>    z.append(y)
>    return z
> 
> from collections import deque
> print f(3)                    # [3,10]
> W=3; print f(4)           # [4,19]
> print f(4,5)                 # [4,5]
> print f(-1, z=deque())  # deque([-1,4])
> 

While that is some pretty nifty/fancy coding, the use of strings for the 
default values does seem a bit kludgey. However, if my proposal does not 
end up getting approved, I'll be sure to recommend that some of the 
great decorators mentioned on this thread get added to the standard library.

- Chris Rebert


From jcarlson at uci.edu  Tue Jan 30 06:50:01 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Mon, 29 Jan 2007 21:50:01 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BECF7B.6090503@gmail.com>
References: <43aa6ff70701292036t7f0a2a96hc013ecd47957de4d@mail.gmail.com>
	<45BECF7B.6090503@gmail.com>
Message-ID: <20070129212157.5A2B.JCARLSON@uci.edu>


Chris Rebert <cvrebert at gmail.com> wrote:
> 
> Collin Winter wrote:
> > On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:
> > Syntax changes are a huge stick to wield against such a small problem.
> > I realize the boilerplate is annoying, but Tomer has pointed out a
> > number of decorator-based solutions [1] that could easily be adapted
> > to any unusual needs you have.
> 
> One of his decorators assumes that the default value merely needs to be 
> copied across calls. However, as the PEP says, there are cases where 
> this isn't sufficient or just doesn't work. His second decorator 
> involves using lambdas (ick!),
[snip]

Using the features of a language to attempt to compensate for that same
language's (argued) shortcomings are a valid _and encouraged_ approach. 
Your poo-pooing of Python conditionals, decorators, and lambdas to solve
this particular problem, to me, seems like you want a *particular
solution*.

I don't see a problem with the current default argument semantics.  Why? 
Because in the case where I would want to receive a mutable parameter,
like in the case below that wouldn't work with Python's standard
semantics...

    def append_10(lst=[]):
        lst.append(10)
        return lst

I would presumably change it to...

    def append_10(lst=None):
        if lst is None: lst = []
        lst.append(10)
        return lst

Or some variant thereof.  Now, here's the thing; if I want a mutable
argument, then None is a nonsensical value to pass, generally, as it is
not mutable.  So I don't buy the whole "but then None would no longer be a
valid argument to pass" bull that was offered as a reason why the above
isn't a reasonable translation (I can't remember who offered it).

I'm also not convinced by either of the 3 pages that talk about Python
"gotchas".  You get bitten by it, you learn it, understand it, and move
on.  If they can't figure it out, I'm not sure I want them writing
Python software anyways; I certainly wouldn't want them to work on any
of the Python software I work on and use.


 - Josiah



From collinw at gmail.com  Tue Jan 30 06:52:59 2007
From: collinw at gmail.com (Collin Winter)
Date: Mon, 29 Jan 2007 23:52:59 -0600
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BEDA53.9010103@gmail.com>
References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru>
	<91ad5bf80701292016m63bf1b28w84e2c9f62fb7ea0e@mail.gmail.com>
	<45BEDA53.9010103@gmail.com>
Message-ID: <43aa6ff70701292152q158092d3xf1b37b4ff4ff7438@mail.gmail.com>

On 1/29/07, Chris Rebert <cvrebert at gmail.com> wrote:
> George Sakkis wrote:
> > As I see it, the main objection to this is the inversal of the current
> > default semantics: every default argument is treated as if it were
> > mutable, or re-evaluatable more generally. Although mutable default
> > arguments are useful some times and would be nice to have, they are
> > most likely less common than the immutable ones, so the latter should
> > be the default.
>
> Why? Yes, there _might_ be performance issues (which have yet to be
> demonstrated or deeply speculated upon), but re-evaluating immutable
> default arguments wouldn't affect a program's correct operation.

If the underlying intent of your proposal -- that all default
arguments be re-evaluated with every call -- were to be approved,
there would undoubtedly be a serious performance impact. The
alternative is an ugly, narrow-use syntax that seeks to eliminate two
lines of boilerplate per default argument, boilerplate that can
already be replaced with decorators.

As for whether these effects have "yet to be demonstrated", as you
say, the burden is on you, the PEP author, to investigate and resolve,
mitigate or justify any and all performance changes.

Collin Winter


From cvrebert at gmail.com  Tue Jan 30 08:09:37 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Mon, 29 Jan 2007 23:09:37 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <op.tmxawql4d64u53@aidwe2058w.minlnv.agro.nl>
References: <45BCF804.7040207@gmail.com>
	<op.tmxawql4d64u53@aidwe2058w.minlnv.agro.nl>
Message-ID: <45BEEF31.6070108@gmail.com>

Wow, that's a lot to think about.
Yes, the exact nature of the variables in default arguments does need 
clarification. I'll probably go with something like your proposal, but I 
need to consider a few things, particularly the 'ncopies' situation.
I added a reference to the Python documentation you mentioned. Thanks!

 >>      def foo(bar=mutable):
 >>          #code
 >
 > if 'mutable' is just a single variable, this isn't gonna break, unless
 > the global scope decides to do something like this:
 >
 > def foo(bar=mutable):
 >     #code
 >
 > mutable = incompatible_mutable
 > # ...
 > foo()

Actually, I was considering the case where mutable is a constant (e.g. a 
list), in which case it *will* break since the expr 'mutable' will be 
re-evaluated at every call, so modifying it won't have the same effect 
on future calls that it used to.

I included your rephrasing under "Backwards Compatibility".
I shuffled around the refactorings as you suggested.
My decision not to include the default-expr-vars-become-lexical part was 
completely unintentional, as you can guess from above.
I also reworded the sentences using 'shall'. I don't really get why that 
matters, but what the heck.

Thanks for your useful comments and suggestions.

- Chris Rebert

Jan Kanis wrote:
> Well, I (obviously) like the idea, but your pep misses some important 
> points (and some not so important).
> 
> The important one is about the scoping of variables in default 
> expressions. The pep says nothing about them.
> If I read the pep correctly, any variables in default expressions are 
> handled the same as variables in the body of a function. This means the 
> compiler decides if they should be local, lexical or global. If there is 
> an assignment to a variable, the compiler makes it a local, else it 
> finds the right enclosing scope (lexical or global). In the current 
> python, this works fine:
> 
>  >>> a = 123
>  >>> def foo(b=a):
>           a = 2
>           print a, b
> 
>  >>> foo()
>  2 123
>  >>> a = 42
>  >>> foo()
>  2 123
> 
> In the pep, the a in the default expression would be handled just like 
> any other a in the function body, which means it wil become a _local_ 
> variable. Calling the function would then result in an 
> UnboundLocalError. Just like this in current python:
> 
>  >>> a = 123
>  >>> def foo(b=None):
>           b = a if b==None else b
>           a = 2
>           print a
> 
>  >>> foo()
> 
>  Traceback (most recent call last):
>    File "<pyshell#22>", line 1, in <module>
>      foo()
>    File "<pyshell#21>", line 2, in foo
>      b = a if b==None else b
>  UnboundLocalError: local variable 'a' referenced before assignment
> 
> The solution, I think, as I wrote in my previous messages, is to have 
> the compiler explicitly make variables in default expressions lexical or 
> global variables.
> This would still break the foo() in my example above, because you can't 
> assign to a lexical variable, or to a global that isn't declared global. 
> Therefore I think the compiler should distinguish between the a in the 
> default expression and the a in the function body, and treat them as two 
> different variables. You can think about it as if the compiler silently 
> renames one of them (without this rename being visible to python code.  
> AFAIK the bytecode is stack based and closure vars are put in some kind 
> of anonymous cell, which means they both don't actually have a name 
> anyway.)
> 
> see below for more comments regarding this and other things
> 
> 
> On Sun, 28 Jan 2007 20:22:44 +0100, Chris Rebert <cvrebert at gmail.com> 
> wrote:
> 
>> The following is a proto-PEP based on the discussion in the thread
>> "fixing mutable default argument values". Comments would be greatly
>> appreciated.
>> - Chris Rebert
>>
>> Title: Fixing Non-constant Default Arguments
>>
>> Abstract
>>
>>      This PEP proposes new semantics for default arguments to remove
>> boilerplate code associated with non-constant default argument values,
>> allowing them to be expressed more clearly and succinctly.
>>
>>
>> Motivation
>>
>>      Currently, to write functions using non-constant default arguments,
>> one must use the idiom:
>>
>>      def foo(non_const=None):
>>          if non_const is None:
>>              non_const = some_expr
>>          #rest of function
>>
>> or equivalent code. Naive programmers desiring mutable default arguments
>> often make the mistake of writing the following:
>>
>>      def foo(mutable=some_expr_producing_mutable):
>>          #rest of function
>>
>> However, this does not work as intended, as
>> 'some_expr_producing_mutable' is evaluated only *once* at
>> definition-time, rather than once per call at call-time.  This results
>> in all calls to 'foo' using the same default value, which can result in
>> unintended consequences.  This necessitates the previously mentioned
>> idiom. This unintuitive behavior is such a frequent stumbling block for
>> newbies that it is present in at least 3 lists of Python's problems [0]
>> [1] [2].
> 
> Also, I just found out that python's own documentation refers to this 
> with an "Important warning: The default value is evaluated only once. 
> This makes a difference when the default is a mutable object such as a 
> list, dictionary, or instances of most classes. ..."
> (http://docs.python.org/tut/node6.html#SECTION006710000000000000000)
> this indicates imo that also the python doc writers don't think of the 
> current situation as optimal.
> 
>>      There are currently few, if any, known good uses of the current
>> behavior of mutable default arguments.  The most common one is to
>> preserve function state between calls.  However, as one of the lists [2]
>> comments, this purpose is much better served by decorators, classes, or
>> (though less preferred) global variables.
>>      Therefore, since the current semantics aren't useful for
>> non-constant default values and an idiom is necessary to work around
>> this deficiency, why not change the semantics so that people can write
>> what they mean more directly, without the annoying boilerplate?
>>
>>
>> Rationale
>>
>>      Originally, it was proposed that all default argument values be
>> deep-copied from the original (evaluated at definition-time) at each
>> invocation of the function where the default value was required.
>> However, this doesn't take into account default values that are not
>> literals, e.g. function calls, subscripts, attribute accesses.  Thus,
>> the new idea was to re-evaluate the default arguments at each call where
>> they were needed. There was some concern over the possible performance
>> hit this could cause, and whether there should be new syntax so that
>> code could use the existing semantics for performance reasons.  Some of
>> the proposed syntaxes were:
>>
>>      def foo(bar=<baz>):
>>          #code
>>
>>      def foo(bar=new baz):
>>          #code
>>
>>      def foo(bar=fresh baz):
>>          #code
>>
>>      def foo(bar=separate baz):
>>          #code
>>
>>      def foo(bar=another baz):
>>          #code
>>
>>      def foo(bar=unique baz):
>>          #code
>>
>> where the new keyword (or angle brackets) would indicate that the
>> parameter's default argument should use the new semantics.  Other
>> parameters would continue to use the old semantics. It was generally
>> agreed that the angle-bracket syntax was particularly ugly, leading to
>> the proposal of the other syntaxes. However, having 2 different sets of
>> semantics could be confusing and leaving in the old semantics just for
>> performance might be premature optimization. Refactorings to deal with
>> the possible performance hit are discussed below.
>>
>>
>> Specification
>>
>>      The current semantics for default arguments are replaced by the
>> following semantics:
>>      - Whenever a function is called, and the caller does not provide a
>>      value for a parameter with a default expression, the parameter's
>>      default expression shall be evaluated in the function's scope.  The
>>      resulting value shall be assigned to a local variable in the
>>      function's scope with the same name as the parameter.
> 
> Include something saying that any variables in a default expression 
> shall be lexical variables, with their scope being the first outer scope 
> that defines a variable with the same name (they should just use the 
> same rules as other lexical/closure variables), and that if the function 
> body defines a local variable with the same name as a variable in a 
> default expression, those variables shall be handled as two separate 
> variables.
> 
>>      - The default argument expressions shall be evaluated before the
>>      body of the function.
>>      - The evaluation of default argument expressions shall proceed in
>>      the same order as that of the parameter list in the function's
>>      definition.
>> Given these semantics, it makes more sense to refer to default argument
>> expressions rather than default argument values, as the expression is
>> re-evaluated at each call, rather than just once at definition-time.
>> Therefore, we shall do so hereafter.
>>
>> Demonstrative examples of new semantics:
>>      #default argument expressions can refer to
>>      #variables in the enclosing scope...
>>      CONST = "hi"
>>      def foo(a=CONST):
>>          print a
>>
>>      >>> foo()
>>      hi
>>      >>> CONST="bye"
>>      >>> foo()
>>      bye
>>
>>      #...or even other arguments
>>      def ncopies(container, n=len(container)):
>>          return [container for i in range(n)]
>>
>>      >>> ncopies([1, 2], 5)
>>      [[1, 2], [1, 2], [1, 2], [1, 2], [1, 2]]
>>      >>> ncopies([1, 2, 3])
>>      [[1, 2, 3], [1, 2, 3], [1, 2, 3]]
>>      >>> #ncopies grabbed n from [1, 2, 3]'s length (3)
> 
> I'm not sure if this can be combined elegantly with what I said about 
> variables being lexical variables. The first argument to ncopies, 
> 'container', is clearly a local variable to ncopies. The 'container' in 
> the second arg default expr should, if my comments above are accepted, 
> be a lexical variable referring to the 'container' in the global scope.
> The best way to combine the two features seems to be to let 'container' 
> be a local var if any of the preceding args is named 'container', and 
> let it be a lexically scoped variable otherwise. However, I'm not 
> convinced this complexity is worth it and the vars in default 
> expressions shouldn't just always be lexical vars.
> 
>>
>>      #default argument expressions are arbitrary expressions
>>      def my_sum(lst):
>>          cur_sum = lst[0]
>>          for i in lst[1:]: cur_sum += i
>>          return cur_sum
>>
>>      def bar(b=my_sum((["b"] * (2 * 3))[:4])):
>>          print b
>>
>>      >>> bar()
>>      bbbb
>>
>>      #default argument expressions are re-evaluated at every call...
>>      from random import randint
>>      def baz(c=randint(1,3)):
>>          print c
>>
>>      >>> baz()
>>      2
>>      >>> baz()
>>      3
>>
>>      #...but only when they're required
>>      def silly():
>>          print "spam"
>>          return 42
>>
>>      def qux(d=silly()):
>>          pass
>>
>>      >>> qux()
>>      spam
>>      >>> qux(17)
>>      >>> qux(d=17)
>>      >>> qux(*[17])
>>      >>> qux(**{'d':17})
>>      >>> #no output because silly() never called because d's value was
>> specified in the calls
>>
>>      #Rule 3
>>      count = 0
>>      def next():
>>          global count
>>          count += 1
>>          return count - 1
>>
>>      def frobnicate(g=next(), h=next(), i=next()):
>>          print g, h, i
>>
>>      >>> frobnicate()
>>      0 1 2
>>      >>> #g, h, and i's default argument expressions are evaluated in
>> the same order as the parameter definition
>>
>>
>> Backwards Compatibility
>>
>>      This change in semantics breaks all code which uses mutable default
>> argument values. Such code can be refactored from:
> 
> Wow, let's not scare everyone away just yet. This should read:
> "This change in semantics breaks code which uses mutable default 
> argument expressions and depends on those expressions being evaluated 
> only once, or code that assigns new incompatible values in a parent 
> scope to variables used in default expressions"
> 
>>
>>      def foo(bar=mutable):
>>          #code
> 
> if 'mutable' is just a single variable, this isn't gonna break, unless 
> the global scope decides to do something like this:
> 
> def foo(bar=mutable):
>     #code
> 
> mutable = incompatible_mutable
> # ...
> foo()
> 
>>
>> to
>>
>>      def stateify(state):
>>          def _wrap(func):
>>              def _wrapper(*args, **kwds):
>>                  kwds['bar'] = state
>>                  return func(*args, **kwds)
>>              return _wrapper
>>          return _wrap
>>
>>      @stateify(mutable)
>>      def foo(bar):
>>          #code
>>
>> or
>>
>>      state = mutable
>>      def foo(bar=state):
>>          #code
>>
>> or
>>
>>      class Baz(object):
>>          def __init__(self):
>>              self.state = mutable
>>
>>          def foo(self, bar=self.state):
>>              #code
> 
> Minor point: the stateify decorator looks a bit scary to me as it uses 
> three levels of nested functions. (that's inherent to decorators, but 
> still.) Suggest you name the class and global var solutions first, and 
> the decorator as last, just to prevent people from stopping reading the 
> pep and voting '-1' right when they hit the decorator solution.
> 
>>
>> The changes in this PEP are backwards-compatible with all code whose
>> default argument values are immutable
> 
> ...or don't depend on being evaluated only once, and don't modify in a 
> parent scope the variables in default expressions in an incompatible way,
> 
> (hmm, the 'or' and 'and' may need some disambiguation parentheses...)
> 
>> including code using the idiom
>> mentioned in the 'Motivation' section. However, such values will now be
>> recomputed for each call for which they are required. This may cause
>> performance degradation. If such recomputation is significantly
>> expensive, the same refactorings mentioned above can be used.
>>
>>      In relation to Python 3.0, this PEP's proposal is compatible with
>> those of PEP 3102 [3] and PEP 3107 [4]. Also, this PEP does not depend
>> on the acceptance of either of those PEPs.
>>
>>
>> Reference Implementation
>>
>>      All code of the form:
>>
>>          def foo(bar=some_expr, baz=other_expr):
>>              #body
>>
>>      Should act as if it had read (in pseudo-Python):
>>
>>          def foo(bar=_undefined, baz=_undefined):
>>              if bar is _undefined:
>>                  bar = some_expr
>>              if baz is _undefined:
>>                  baz = other_expr
>>              #body
> 
> and, if there are any variables occuring in the function body and in 
> some_expr or other_expr, rename those in the function body to something 
> that doesn't name-clash.
> 
>>
>> where _undefined is the value given to a parameter when the caller
>> didn't specify a value for it. This is not intended to be a literal
>> translation, but rather a demonstration as to how Python's internal
>> argument-handling machinery should be changed.
>>
>>
>> References
>>
>>      [0] 10 Python pitfalls
>>          http://zephyrfalcon.org/labs/python_pitfalls.html
>>
>>      [1] Python Gotchas
>>          http://www.ferg.org/projects/python_gotchas.html#contents_item_6
>>
>>      [2] When Pythons Attack
>>      
>> http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2
>>
>>      [3] Keyword-Only Arguments
>>          http://www.python.org/dev/peps/pep-3102/
>>
>>      [4] Function Annotations
>>          http://www.python.org/dev/peps/pep-3107/
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
> 
> I'm not quite sure if your decision not to include the 
> default-expr-vars-become-lexical part was intentional or not. If it is, 
> can you tell me why you'd want that? Else you can incorporate my 
> comments in the pep.
> Another minor point: I'm personally not too fond of too many 'shall's 
> close together. It makes me think of lots of bureaucracy and 
> design-by-committee. I think several peps don't have a  shall in them, 
> but maybe it is the right language for this one. What's the right 
> language to use in peps?
> 
> Well, that's about it I think.
> 
> - Jan
> 


From jimjjewett at gmail.com  Tue Jan 30 08:19:11 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Tue, 30 Jan 2007 02:19:11 -0500
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BED309.10301@gmail.com>
References: <45BCF804.7040207@gmail.com>
	<fb6fbf560701291451i2b8442bdof217cf172726cf66@mail.gmail.com>
	<45BED309.10301@gmail.com>
Message-ID: <fb6fbf560701292319l4b42f0b4nc33da1eb1b00ce96@mail.gmail.com>

On 1/30/07, Chris Rebert <cvrebert at gmail.com> wrote:
> Jim Jewett wrote:
> > On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:

> >> Naive programmers desiring mutable default arguments
> >> often make the mistake of writing the following:

> >>      def foo(mutable=some_expr_producing_mutable):
> >>          #rest of function

> > Yes, it is an ugly gotcha, but so is the alternative.

> > If it is changed, then just as many naive programmers (though perhaps
> > not exactly the same ones) will make the opposite mistake -- and so
> > will some experienced programmers who are used to current semantics.

> Anyone (ab)using the current semantics with the above construct is or
> ought to be aware that such usage is unpythonic.

Are you worried about naive programmers or not?

> Also, due to the
> hairiness/obscurity of the construct, hardly anyone uses it, so the
> impact of removing it should be relatively minor.

I just did found 181 such uses in my own installation.  29 were either
3rd-party or test code, but that still leaves over 150 uses in the
standard library.  It is possible that some of them would also work
with your semantics, or may even be bugs today -- but you would have
to go through them one-by-one as part of the PEP process.

> Additionally, I see
> making the opposite mistake as usually only maybe causing a performance
> problem, as opposed to causing the program to work incorrectly.

That is true only when the argument is a cache.

Generally, resetting the storage will make the program work
incorrectly, but the program won't fail on the first data tested --
which means the bug is less likely to be caught.

> >>      There are currently few, if any, known good uses of the current
> >> behavior of mutable default arguments.  The most common one is to
> >> preserve function state between calls.  However, as one of the lists [2]
> >> comments, this purpose is much better served by decorators, classes, or
> >> (though less preferred) global variables.

> > I disagree.  This is particularly wrong for someone coming from a
> > functional background.

> I assume you disagree with the "purpose is much better served by
> decorators, classes, or local variables" part as opposed to the "default
> mutable arguments with current semantics have few good uses" part.
> Please correct me if I'm in error.

I disagree with both.  There are many uses for preserving function
state between calls.  C uses static local variables.   Object Oriented
languages often use objects.

> > A class plus an instantiation seems far too heavyweight for what ought
> > to be a simple function.  I'm not talking (only) about the runtime;
> > the number of methods and lines of code is the real barrier for me.
> > I'll sometimes do it (or use a global) anyhow, but it feels wrong.  If
> > I had felt that need more often when I was first learning python (or
> > before I knew about the  __call__ workaround), I might have just
> > written the language off as no less bloated than java.

> I'm sorry, but when you have a function sharing state between calls,
> that just screams to me that you should make it a method of an object

That's because you drank the OO koolaid.  Python tries not to enforce
any particular programming style.  Its object model happens to be
pretty good, but there are still times when OO isn't the answer.

> so
> you don't have to store the state in some roundabout way, such as in
> mutable default arguments. If performance is your concern, the decorator
> version might perform better (I don't really know), or in the extreme
> case, you could use a global variable, which is definitely faster.

I think there is something wrong with your benchmark.

The default argument is accessible straight from the function object
itself; even a cell variable shouldn't be that fast, let alone global
variables.

That said, speed is *not* my concern -- code size is.  If I have to
add several lines of boilerplate, the code becomes much less readable.
 That is roughly the same justification you have for not liking the
"if arg is None:  arg=foo()" idiom.  The difference is that you're
suggesting adding far more than a line or two.

> > You see the problem with globals, but decorators are in some sense
> > worse -- a function cannot see its own decorations.  At best, it can
> > *assume* that repeating its own name (despite Don't Repeat Yourself)
> > will get another reference to self, but this isn't always true.

> > Programmers used to creating functions outside of toplevel (or
> > class-level) will be more aware of this, and see the suggestion as an
> > indication that python is inherently buggy.

> I really don't see how the decorator in the PEP is any worse than other
> decorators in this regard. The problem you describe currently applies to
> all decorated functions, though functools.wraps might help mitigate this
> situation.

Either you effectively hide the real function so as to create a
lexical closure (plenty of extra work, both mentally and
computationally) or you accept buggy code.  The bug possibility isn't
inherent to decorations; only to functions reading their *own*
decorations.

> Alternatively, we could make the new
> semantics the default and have syntax to use the old semantics via
> 'once' or some other keyword. This does nicely emphasize that such
> semantics would be purely for optimization reasons.

Using a mutable default is almost never for optimization reasons.

-jJ


From cvrebert at gmail.com  Tue Jan 30 08:25:58 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Mon, 29 Jan 2007 23:25:58 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <20070129212157.5A2B.JCARLSON@uci.edu>
References: <43aa6ff70701292036t7f0a2a96hc013ecd47957de4d@mail.gmail.com>
	<45BECF7B.6090503@gmail.com> <20070129212157.5A2B.JCARLSON@uci.edu>
Message-ID: <45BEF306.5030801@gmail.com>

Josiah Carlson wrote:
> Using the features of a language to attempt to compensate for that same
> language's (argued) shortcomings are a valid _and encouraged_ approach. 
> Your poo-pooing of Python conditionals, decorators, and lambdas to solve
> this particular problem, to me, seems like you want a *particular
> solution*.

No, it's just that why use several different decorators when a slight 
change in semantics renders them all obsolete? Why have to use a 
decorator/lambda/conditional and remember when to use which? I'm trying 
to generalize the pattern of "reevaluate/copy default values each time 
they're required" into new behavior for default arguments. Indeed, as I 
stated in another email (no idea which one, there's been too many), I 
wholeheartedly support adding some of those decorators to the standard 
library should my PEP get rejected.

> I don't see a problem with the current default argument semantics.  Why? 
> Because in the case where I would want to receive a mutable parameter,
> like in the case below that wouldn't work with Python's standard
> semantics...
> 
>     def append_10(lst=[]):
>         lst.append(10)
>         return lst
> 
> I would presumably change it to...
> 
>     def append_10(lst=None):
>         if lst is None: lst = []
>         lst.append(10)
>         return lst
> 
> Or some variant thereof.  Now, here's the thing; if I want a mutable
> argument, then None is a nonsensical value to pass, generally, as it is
> not mutable.  So I don't buy the whole "but then None would no longer be a
> valid argument to pass" bull that was offered as a reason why the above
> isn't a reasonable translation (I can't remember who offered it).

Nowhere has it been proposed to forbid passing None as an argument. The 
basic core of the proposal is to have Python compile the code such that 
the boilerplate is either removed entirely (if the new semantics are the 
default) by no longer being necessary, or dramatically shortened via the 
addition of a new keyword (which would indicate the new semantics). Yes, 
you can manually transform your code to use the idiom you mention. But 
why should you have to? Why not have Python do the heavy-lifting for you?
As stated in the "Reference Implementation" section of the PEP, the 
'translation' provided is not intended to be a literal translation of 
the code at compile-time, but rather explain how Python's 
argument-handling machinery should be changed. The 'translation' looks 
similar to the existing idiom because that's what it aims to replace. 
But note that it's *Python* that's doing this 'translation', not the 
programmer! That's the gain! The programmer no longer needs to write the 
boilerplate.

> I'm also not convinced by either of the 3 pages that talk about Python
> "gotchas".  You get bitten by it, you learn it, understand it, and move
> on.  If they can't figure it out, I'm not sure I want them writing
> Python software anyways; I certainly wouldn't want them to work on any
> of the Python software I work on and use.

I'm not here to argue that people shouldn't be made to understand the 
difference between mutable and immutable values. They definitely should 
know the difference. I'm just advocating a language change to make 
certain kinds of functions less verbose. If you're worried about losing 
this 'teaching tool' (which another respondent was worried about), it 
will still exist in the form of:

x = [[0]*4]*4
x[3][1] = 7
x[0][1] == 7 #TRUE!

- Chris Rebert


From jan.kanis at phil.uu.nl  Tue Jan 30 11:36:26 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Tue, 30 Jan 2007 11:36:26 +0100
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BEDA53.9010103@gmail.com>
References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru>
	<91ad5bf80701292016m63bf1b28w84e2c9f62fb7ea0e@mail.gmail.com>
	<45BEDA53.9010103@gmail.com>
Message-ID: <op.tmyqe0xld64u53@aidwe2058w.minlnv.agro.nl>

On Tue, 30 Jan 2007 06:40:35 +0100, Chris Rebert <cvrebert at gmail.com>  
wrote:
> George Sakkis wrote:
>> I'm sure that the python-devs and the BDFL would have
>> thought about it quite a bit when the current semantics were decided,
>
> ...which was probably a while ago. They might reconsider the issue now
> that some time has passed and they've seen how their decision has worked
> out. But yes, your analysis is a definite possibility.

Just looked it up, and python has had lexical variables since version 2.1,  
and default arguments since long before that. (forever?) Without lexical  
variables it's near impossible to implement re-evaluating default  
arguments, because the variables those default expressions refer to may no  
longer be available at function call time. So, The BDFL and other python  
devs didn't really have a choice but to have the default expressions  
evaluate at definition time (or implement lexical scopes, which is what  
has happened by now).

Jan


From jan.kanis at phil.uu.nl  Tue Jan 30 12:21:10 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Tue, 30 Jan 2007 12:21:10 +0100
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BEEF31.6070108@gmail.com>
References: <45BCF804.7040207@gmail.com>
	<op.tmxawql4d64u53@aidwe2058w.minlnv.agro.nl>
	<45BEEF31.6070108@gmail.com>
Message-ID: <op.tmyshkdzd64u53@aidwe2058w.minlnv.agro.nl>

On Tue, 30 Jan 2007 08:09:37 +0100, Chris Rebert <cvrebert at gmail.com>  
wrote:

> I also reworded the sentences using 'shall'. I don't really get why that  
> matters, but what the heck.

Well, it doesn't. Just makes it nicer to read. (As long as the text is  
still clear, that is.)


> Jan Kanis wrote:
>> Well, I (obviously) like the idea, but your pep misses some important  
>> points (and some not so important).
>>  The important one is about the scoping of variables in default  
>> expressions. The pep says nothing about them.
>> If I read the pep correctly, any variables in default expressions are  
>> handled the same as variables in the body of a function. This means the  
>> compiler decides if they should be local, lexical or global. If there  
>> is an assignment to a variable, the compiler makes it a local, else it  
>> finds the right enclosing scope (lexical or global). In the current  
>> python, this works fine:
>>   >>> a = 123
>>  >>> def foo(b=a):
>>           a = 2
>>           print a, b
>>   >>> foo()
>>  2 123
>>  >>> a = 42
>>  >>> foo()
>>  2 123
>>  In the pep, the a in the default expression would be handled just like  
>> any other a in the function body, which means it wil become a _local_  
>> variable. Calling the function would then result in an  
>> UnboundLocalError. Just like this in current python:
>>   >>> a = 123
>>  >>> def foo(b=None):
>>           b = a if b==None else b
>>           a = 2
>>           print a
>>   >>> foo()
>>   Traceback (most recent call last):
>>    File "<pyshell#22>", line 1, in <module>
>>      foo()
>>    File "<pyshell#21>", line 2, in foo
>>      b = a if b==None else b
>>  UnboundLocalError: local variable 'a' referenced before assignment
>>  The solution, I think, as I wrote in my previous messages, is to have  
>> the compiler explicitly make variables in default expressions lexical  
>> or global variables.
>> This would still break the foo() in my example above, because you can't  
>> assign to a lexical variable, or to a global that isn't declared  
>> global. Therefore I think the compiler should distinguish between the a  
>> in the default expression and the a in the function body, and treat  
>> them as two different variables. You can think about it as if the  
>> compiler silently renames one of them (without this rename being  
>> visible to python code.  AFAIK the bytecode is stack based and closure  
>> vars are put in some kind of anonymous cell, which means they both  
>> don't actually have a name anyway.)


Also, I've been thinking about the part that I suggested about having  
python distinguish between lexically scoped variables in default  
expressions and local variables that may have the same name.
To clarify my proposal, a demonstration:

Current python:

a = 4
b = 5
def foo(x = [b]*a):
	x = copy.deepcopy(x)  # one possible workaround to have the same list on  
every call
	b = 'bla'         # no name clash of this b and b in default arg expr
	x += [b]
	print x

foo()
  # prints [5, 5, 5, 5, 'bla']
foo()
  # prints [5, 5, 5, 5, 'bla']

proposed semantics:

a = 4
b = 5
def foo(x = [b]*a):
	 # deepcopy() no longer nescessary
	b = 'bla'   # no name clash of this b and b in default arg expr, the  
compiler distinguishes them
	x += [b]
	print x

foo()
  # prints [5, 5, 5, 5, 'bla']
foo()
  # prints [5, 5, 5, 5, 'bla']


This would work the same as this code in the current python. Imagine it as  
the compiler silently transforming the above code to this code:

a = 4
b = 5
def foo(x = _undefined):
	if x == _undefined:
		x = [b]*a
	_b = 'bla'
	x += [_b]
	print x

foo()
  # prints [5, 5, 5, 5, 'bla']
foo()
  # prints [5, 5, 5, 5, 'bla']

note: the use of deepcopy assumes that whatever is passed to foo is  
deepcopyable. With the new semantics this function can be used without  
these kind of assumptions.

However, I've been thinking about weather it is a good idea to have python  
distinguish between the local and lexical 'b' in the above code, and do  
the what you can think of as implicit renaming. Making the distinction  
makes the whole proposal more backward compatible, however I think that if  
backward compatibility were not an issue, it would be best not to make the  
distinction and just have the programmer choose a different name for the  
local variable. This would improve readability as there are no two  
different variables with the same name in the same piece of code, and  
decrease language complexity as there is no rule that's best explained in  
terms of variable name rewriting.
Not making the distinction does cause the default arguments proposal to be  
less backward compatible, but I don't know if that is really a big  
problem. I assume this pep wil be a py3k pep anyway, so there are going to  
be incompatibilities anyway. Also, the compiler is perfectly able to see  
at compile time if a variable used in a default expression and as a local,  
so it is able to raise a very descriptive error.
All in all, I think this part of my proposal is not such a good idea,  
unless there's going to be some kind of transitional implementation of  
this pep in py 2.x

meta-note: I hope it is clear what I'm trying to say. Basically, I'm  
clarifying, arguing against, and retracting a part of my own proposal I  
made in an earlier mail.


- Jan


From rrr at ronadam.com  Tue Jan 30 12:35:34 2007
From: rrr at ronadam.com (Ron Adam)
Date: Tue, 30 Jan 2007 05:35:34 -0600
Subject: [Python-ideas] Import and '..', '../..' in serach path.
Message-ID: <45BF2D86.2080801@ronadam.com>


In order to resolve a path conflict where I'm working on several copies of the 
same package.  I found it useful to add the following near the top of modules in 
a package or sub package.


Module in package:

     import sys
     sys.path = ['..'] + sys.path

     import package.module          # Imports module in "this!" package.


Note: There could still be conflicts if a module with the same name is in the 
same directory as the package.  But that's much less likely than one in the rest 
of the path.


Module in sub-package:

     import sys
     sys.path = ['../..'] + sys.path

     import package.subpackage.module   # finds "self" (subpackage) reliably.


By explicitly adding the packages parent directory to the *front* of sys.path it 
resolves cases where imports using absolute imports, import modules from another 
package because they are found first in the search path.


Adding this tip to the documentation some where would be nice.  (providing there 
is no major surprising side effects.)  Of course I may have missed some obvious 
way to do this.  If so, it wasn't in an obvious place to be found. I looked.   ;-)


----------------------------------


It might be useful to have a built-in function to do this.  A function could 
also check for __init__ files and raise errors if they are missing.

    set_package_name(dotted.name)  # Replaces import sys & path modification

Where dotted.name is the full package + sub-package name the current module is 
located in.  The function would search upwards to get the root package directory 
and add that to the *front* of sys.path.


Module in package:

     set_package_name('package')    # Add parent directory to front of sys.path

     import packagename.module  # Finds module in "this!" package reliably.


Module in subpackage:

     set_package_name('package.subpackage')

     import package.subpackage.module  # Finds "self" (subpackage) reliably.


----------------------------------


It may also be able to modify the import behavior to allow relative imports to 
work when the module is run as script.

     set_package_name('package')

     from . import module1    # Imports modules from "this" package.
     from . import module2


Currently an exception is raised you try to run a module with relative 
references as a script.

	ValueError: Attempted relative import in non-package


I think it is very handy to be able to run tests as scripts and keep them in a 
sub-package. Especially while I'm writing them.

Another benefit of using relative imports with an absolute specified package 
name, is if you rename a package or relocate a submodule, you only have one line 
to change.


Cheers,

Ron

















From rrr at ronadam.com  Tue Jan 30 13:13:31 2007
From: rrr at ronadam.com (Ron Adam)
Date: Tue, 30 Jan 2007 06:13:31 -0600
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <20070129212157.5A2B.JCARLSON@uci.edu>
References: <43aa6ff70701292036t7f0a2a96hc013ecd47957de4d@mail.gmail.com>	<45BECF7B.6090503@gmail.com>
	<20070129212157.5A2B.JCARLSON@uci.edu>
Message-ID: <45BF366B.9040107@ronadam.com>

Josiah Carlson wrote:
> Chris Rebert <cvrebert at gmail.com> wrote:
>> Collin Winter wrote:
>>> On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:
>>> Syntax changes are a huge stick to wield against such a small problem.
>>> I realize the boilerplate is annoying, but Tomer has pointed out a
>>> number of decorator-based solutions [1] that could easily be adapted
>>> to any unusual needs you have.
>> One of his decorators assumes that the default value merely needs to be 
>> copied across calls. However, as the PEP says, there are cases where 
>> this isn't sufficient or just doesn't work. His second decorator 
>> involves using lambdas (ick!),
> [snip]
> 
> Using the features of a language to attempt to compensate for that same
> language's (argued) shortcomings are a valid _and encouraged_ approach. 
> Your poo-pooing of Python conditionals, decorators, and lambdas to solve
> this particular problem, to me, seems like you want a *particular
> solution*.
> 
> I don't see a problem with the current default argument semantics.  Why? 
> Because in the case where I would want to receive a mutable parameter,
> like in the case below that wouldn't work with Python's standard
> semantics...
> 
>     def append_10(lst=[]):
>         lst.append(10)
>         return lst
> 
> I would presumably change it to...
> 
>     def append_10(lst=None):
>         if lst is None: lst = []
>         lst.append(10)
>         return lst

Strings sometimes work nice.  Then you have a hint as to what should go there.

      def append_10(lst='[]'):
          if lst == '[]': lst = []
          lst.append(10)
          return lst


> Or some variant thereof.  Now, here's the thing; if I want a mutable
> argument, then None is a nonsensical value to pass, generally, as it is
> not mutable.  So I don't buy the whole "but then None would no longer be a
> valid argument to pass" bull that was offered as a reason why the above
> isn't a reasonable translation (I can't remember who offered it).
> 
> I'm also not convinced by either of the 3 pages that talk about Python
> "gotchas".  You get bitten by it, you learn it, understand it, and move
> on.  If they can't figure it out, I'm not sure I want them writing
> Python software anyways; I certainly wouldn't want them to work on any
> of the Python software I work on and use.
> 
> 
>  - Josiah

I agree,  What seems like a gotcha is some situations can be a feature in others.

At the most, documenting things like this should be more visible.  Like maybe
adding a [Gotcha! Mutable arguments...] info box's in the documents.  I like
those little info box's in manuals, especially if they are written with a bit of
humor.

Cheers,

Ron




From jan.kanis at phil.uu.nl  Tue Jan 30 13:24:43 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Tue, 30 Jan 2007 13:24:43 +0100
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <fb6fbf560701292319l4b42f0b4nc33da1eb1b00ce96@mail.gmail.com>
References: <45BCF804.7040207@gmail.com>
	<fb6fbf560701291451i2b8442bdof217cf172726cf66@mail.gmail.com>
	<45BED309.10301@gmail.com>
	<fb6fbf560701292319l4b42f0b4nc33da1eb1b00ce96@mail.gmail.com>
Message-ID: <op.tmyvfhe1d64u53@aidwe2058w.minlnv.agro.nl>

On Tue, 30 Jan 2007 08:19:11 +0100, Jim Jewett <jimjjewett at gmail.com>  
wrote:
> On 1/30/07, Chris Rebert <cvrebert at gmail.com> wrote:
>> Also, due to the
>> hairiness/obscurity of the construct, hardly anyone uses it, so the
>> impact of removing it should be relatively minor.
>
> I just did found 181 such uses in my own installation.  29 were either
> 3rd-party or test code, but that still leaves over 150 uses in the
> standard library.  It is possible that some of them would also work
> with your semantics, or may even be bugs today -- but you would have
> to go through them one-by-one as part of the PEP process.

Well, it seems that numbers are gonna be needed anyway to convince people.  
But that's certainly possible. What exactly did you search for, and what  
did you find?
I did some preliminary grepping through the standardlib myself. regex =  
"def.*\(.*=.*\):", searching through all .py files in /Lib:
2455 matches in total in 474 files out of 998
I manually looked through the first bunch of files, those contained 145  
matches. (the first files sorted alphabetically up until and including  
cookielib.py) By far the most of them had literals as default value. A  
huge amount of them used the =None idiom, I estimate perhaps one third.
10 matches used variables but will not break under the pep's semantics.
I found 1 (one) file that used the current semantics to keep state between  
calls, in two inner functions. This can easily be changed.
So, that leaves 145-12=133 uses of default values that are just constants.

If I have time and figure out the right regexes I'll try and come up with  
some more numbers on the entire stdlib, and the ammount of uses of =None.

- Jan


From eopadoan at altavix.com  Tue Jan 30 13:43:09 2007
From: eopadoan at altavix.com (Eduardo "EdCrypt" O. Padoan)
Date: Tue, 30 Jan 2007 10:43:09 -0200
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <dea92f560701300442w1531fea0u5ba78439d47b5d25@mail.gmail.com>
References: <45BCF804.7040207@gmail.com>
	<fb6fbf560701291451i2b8442bdof217cf172726cf66@mail.gmail.com>
	<45BED309.10301@gmail.com>
	<fb6fbf560701292319l4b42f0b4nc33da1eb1b00ce96@mail.gmail.com>
	<op.tmyvfhe1d64u53@aidwe2058w.minlnv.agro.nl>
	<dea92f560701300442w1531fea0u5ba78439d47b5d25@mail.gmail.com>
Message-ID: <dea92f560701300443r74d8f791l96b22f1dafb961e3@mail.gmail.com>

On 1/30/07, Eduardo EdCrypt O. Padoan <eopadoan at altavix.com> wrote:
> > If I have time and figure out the right regexes I'll try and come up with
> > some more numbers on the entire stdlib, and the ammount of uses of =None.
> >
>
> Some uses of the spam=[] and ham=None in Python projects, including
> Python itself:
>
> http://www.google.com/codesearch?q=def.*%5C(.*%3D%5C%5B%5C%5D.*%5C)%3A%20lang%3APython&hl=en&btnG=Search+Code
> http://www.google.com/codesearch?hl=en&lr=&q=def.*%5C%28.*%3DNone.*%5C%29%3A+lang%3APython&btnG=Search
>
> In this second search, I need a way to search that, in the body of the
> function, we have something like "if foo is not None: foo = []" (and
> foo = {} too)
>
> --
> EduardoOPadoan (eopadoan->altavix::com)
> Bookmarks: http://del.icio.us/edcrypt
> Blog: http://edcrypt.blogspot.com
> Jabber: edcrypt at jabber dot org
> ICQ: 161480283
> GTalk: eduardo dot padoan at gmail dot com
> MSN: eopadoan at altavix dot com
>


-- 
EduardoOPadoan (eopadoan->altavix::com)
Bookmarks: http://del.icio.us/edcrypt
Blog: http://edcrypt.blogspot.com
Jabber: edcrypt at jabber dot org
ICQ: 161480283
GTalk: eduardo dot padoan at gmail dot com
MSN: eopadoan at altavix dot com


From jan.kanis at phil.uu.nl  Tue Jan 30 14:18:35 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Tue, 30 Jan 2007 14:18:35 +0100
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <20070126212657.5A06.JCARLSON@uci.edu>
References: <fb6fbf560701250641k45f24f40qfea54fa07e6b40c4@mail.gmail.com>
	<op.tmsemwqxd64u53@e500> <20070126212657.5A06.JCARLSON@uci.edu>
Message-ID: <op.tmyxw9g2d64u53@aidwe2058w.minlnv.agro.nl>

On Mon, 29 Jan 2007 08:38:39 +0100, Roman Susi <rnd at onego.ru> wrote:
> This is what
> incremental dynamic semantics is about. So, the suggestion is good only
> as separated feature, but is IMHO wrong
> if considered in the language design as a whole.

wtf is incremental dynamic semantics, in this context? I did some googling  
but all I found referred to techniques related to programming  
environments. This proposal is just about a change in the language.


On Sat, 27 Jan 2007 06:30:00 +0100, Josiah Carlson <jcarlson at uci.edu>  
wrote:
> "Jan Kanis" <jan.kanis at phil.uu.nl> wrote:
>> I hope we agree
>> that the problem we're trying to solve is that while
[snip]
> I'm going to have to disagree on the 'non-intuitive and ugly' claim.  We
> are just going to have to agree to disagree.
>

On Mon, 29 Jan 2007 08:38:39 +0100, Roman Susi <rnd at onego.ru> wrote:
> Hello!
>
> I'd liked to say outright that this bad idea which complicates matters
> more than provides solutions.
> Right now it is enough to know that the part from def to ":" is executed
> at definition time.


Well, it's good to be clear on where the disagreements lie. However I'm  
not yet ready to let it rest at that without some more arguments.

As Chris pointed out in his first mail, this 'wart' is mentioned on  
several lists of python misfeatures: [0][1][2]. I'd like to add to this  
that even the python documentation finds this issue severe enough to issue  
an "Important warning"[4].

It seems clear that this behaviour is a gotcha, at least for newbies. This  
could be excused if there is a good reason to spend the additional time  
learning this behaviour, but some of the links state, and my assumption  
is, that there are very few situations where re-evaluating causes a  
problem and which isn't easily fixable.
The semantics which I'd like to have are even easier than the current  
semantics: everything in a function, be it before or after the colon, is  
executed when the function is called.
Of course, as Collin Winters pointed out, the burden of proof of showing  
that these semantics aren't going to be a problem is still on the pep  
proponents.

On the other hand, are there really any good reasons to choose the current  
semantics of evaluation at definition time? What I've heard basically  
boils down to two arguments:
- "let's not change anything", i.e. resist change because it is change,  
which I don't think is a very pythonic argument.
- Arguments based on the assumption that people actually do make lots of  
use of the fact that default arguments are shared between function  
invocations, many of which will result in (much) more code if it has to be  
transformed to using one of the alternative idioms. If this is true, it is  
a valid argument. I guess there's still some stdlib grepping to do to  
decide this.

So, are there any _other_ arguments in favour of the current semantics??


- Jan


[0] 10 Python pitfalls (http://zephyrfalcon.org/labs/python_pitfalls.html)
[1] Python Gotchas
(http://www.ferg.org/projects/python_gotchas.html#contents_item_6)
[2] When Pythons Attack
(http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2)
[4] Python manual - 4. More control flow tools  
(http://docs.python.org/tut/node6.html#SECTION006710000000000000000)


From veloso at verylowsodium.com  Tue Jan 30 16:48:54 2007
From: veloso at verylowsodium.com (Greg Falcon)
Date: Tue, 30 Jan 2007 10:48:54 -0500
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <op.tmyxw9g2d64u53@aidwe2058w.minlnv.agro.nl>
References: <fb6fbf560701250641k45f24f40qfea54fa07e6b40c4@mail.gmail.com>
	<op.tmsemwqxd64u53@e500> <20070126212657.5A06.JCARLSON@uci.edu>
	<op.tmyxw9g2d64u53@aidwe2058w.minlnv.agro.nl>
Message-ID: <3cdcefb80701300748y3c92d544g6414bb5438d28305@mail.gmail.com>

On 1/30/07, Jan Kanis <jan.kanis at phil.uu.nl> wrote:
> On the other hand, are there really any good reasons to choose the current
> semantics of evaluation at definition time?

While I sympathize with the programmer that falls for this common
Python gotcha, and would not have minded if Python's semantics were
different from the start (though the current behavior is cleaner and
more consistent), making such a radical change to such a core part of
the language semantics now is a very bad idea for many reasons.

> What I've heard basically
> boils down to two arguments:
> - "let's not change anything", i.e. resist change because it is change,
> which I don't think is a very pythonic argument.

The argument here is not "let's not change anything because it's
change," but rather "let's not break large amounts of existing code
without a very good reason."  As has been stated here by others,
making obsolete a common two-line idiom is not a compelling enough
reason to do so.

Helping out beginning Python programmers, while well-intentioned,
doesn't feel like enough of a motivation either.  Notice that the main
challenge for the novice programmer is not to learn how default
arguments work -- novices can learn to recognize and write the idiom
easily enough -- but rather to learn how variables and objects work in
general.

>>> a=b=['foo']
>>> c=d=42
>>> a+=['bar']
>>> c+=1
>>> b
['foo', 'bar']
>>> d
42

At some point in his Python career, a novice is going to have to
understand why b "changed" but d didn't.  Fixing the default argument
"wart" doesn't remove the necessity to understand the nature of
mutable objects and variable bindings in Python; it just postpones the
problem.  This is a fact worth keeping in mind when deciding whether
the sweeping change in semantics is worth the costs.

> - Arguments based on the assumption that people actually do make lots of
> use of the fact that default arguments are shared between function
> invocations, many of which will result in (much) more code if it has to be
> transformed to using one of the alternative idioms. If this is true, it is
> a valid argument. I guess there's still some stdlib grepping to do to
> decide this.

Though it's been decried here as unPythonic, I can't be the only
person who uses the idiom
def foo(..., cache={}):
for making a cache when the function in question does not rise to the
level of deserving to be a class object instead.  I don't apologize
for finding it less ugly than using a global variable.

I know I'm not the only user of the idiom because I didn't invent it
-- I learned it from the Python community.  And the fact that people
have already found usages of the current default argument behavior in
the standard library is an argument against the "unPythonic" claim.

I'm reminded of GvR's post on what happened when he made strings
non-iterable in a local build (iterable strings being another "wart"
that people thought needed fixing):
http://mail.python.org/pipermail/python-3000/2006-April/000824.html

> So, are there any _other_ arguments in favour of the current semantics??

Yes.  First, consistency.  What do the three following Python
constructs have in common?

1) lambda x=foo(): None
2) (x for x in foo())
3) def bar(x=foo()):
     pass

Answer: all three evaluate foo() immediately, choosing not to defer
the evaluation to when the resulting object is invoked, even though
they all reasonably could.

It's especially notable that the recently-added feature (generator
expressions) follows existing precedent.  This was not accidental, but
rather a considered design decision.  Two paragraphs from PEP 289
could apply equally well to your proposal:

| Various use cases were proposed for binding all free variables when
| the generator is defined. And some proponents felt that the resulting
| expressions would be easier to understand and debug if bound
| immediately.

| However, Python takes a late binding approach to lambda expressions
| and has no precedent for automatic, early binding. It was felt that
| introducing a new paradigm would unnecessarily introduce complexity.

In fact, the situation here is worse.  PEP 289 is arguing against
early binding of free variables as being complex.  You're not
proposing an early binding, but rather a whole new meaning of the "="
token, "save this expression for conditional evaluation later."  It's
never meant anything like that before.


Second, the a tool can't fix all usages of the old idiom.  When things
break, they can break in subtle or confusing ways.  Consider my module
"greeter":

== begin greeter.py ==
import sys
def say_hi(out = sys.stdout):
 print >> out, "Hi!"
del sys # don't want to leak greeter.sys to the outside world
== end greeter.py ==

Nothing I've done here is strange or unidiomatic, and yet your
proposed change breaks it, and it's unclear how an automated tool
should fix it.  What's worse about the breakage is that it doesn't
break when greeter is imported, or even when greeter.say_hi is called
with an argument.  It might take a while before getting a very
surprising error "global name 'sys' is not defined".


Third, the old idiom is less surprising.

def foo(x=None):
 if x is None:
   x=<some_expr>

<some_expr> may take arbitrarily long to complete.  It may have side
effects.  It may throw an exception.  It is evaluated inside the
function call, but only evaluated when the default value is used (or
the function is passed None).

There is nothing surprising about any of that.  Now:

def foo(x=<some_expr>):
 pass

Everything I said before applies.  The expression can take a long
time, have side effects, throw an exception.  It is conditionally
evaluated inside the function call.

Only now, all of that is terribly confusing and surprising (IMO).


Greg F


From duda.piotr at gmail.com  Tue Jan 30 18:05:40 2007
From: duda.piotr at gmail.com (Piotr Duda)
Date: Tue, 30 Jan 2007 18:05:40 +0100
Subject: [Python-ideas] Mutable default arguments - another approach
Message-ID: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com>

I think that this problem can be solved by the following change of
default argument behavior:
default arguments are evaluated in definition time (like in current
implementation), but right after being evaluated, result object is
checked if it's mutable (for example by checking of presence __copy__
special method or being instance of built in (sub)class
list/dict/set), if object is mutable, argument is marked by
COPY_DEF_ARG flag.
There are two reasons for this check being  done there:
1. performance
2. it can be controlled by "from __future__ import ..." statement in
per-file manner

Then if default argument is needed in function call, first
COPY_DEF_ARG flag is checked if not set, default argument behaves
exactly like in current implementation, if the flag is set, it's
shallow copy is used instead.

Adding following classes/functions to stdlib allow reproduce old
behavior as well as add some new.

class IterateDefaultArg(object):
	def __init__(self, iterator):
		self.__iterator = iterator
	def __copy__(self):
		return self.__iterator.next()

class DefaultArgWrapper(object):
	def __init__(self, generatorfunc):
		self.__generatorfunc = generatorfunc
	def __call__(self, *args, **kwargs):
		return DefaultArgObject(self.__generatorfunc(*args, **kwargs))

@DefaultArgWrapper
def nocopyarg(obj):
	while 1:
		yield obj

With this current definition like:

def foo(cache = {}):
  ...

need to be replaced by:

def foo(cache = nocopyarg({})):
  ...

If one want to use deep copy instead copy, it might be done like this:

@DefaultArgWrapper
def deepcopyarg(obj):
        from copy import deepcopy
        while 1:
                yield deepcopy(obj)

def foo(x = deepcopyarg(<some expression>)):
  ...


P.S. sorry for my bad English

-- 
????????
??????

From jcarlson at uci.edu  Tue Jan 30 18:16:03 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Tue, 30 Jan 2007 09:16:03 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BEF306.5030801@gmail.com>
References: <20070129212157.5A2B.JCARLSON@uci.edu> <45BEF306.5030801@gmail.com>
Message-ID: <20070129233754.5A32.JCARLSON@uci.edu>


Chris Rebert <cvrebert at gmail.com> wrote:
> 
> Josiah Carlson wrote:
> > Using the features of a language to attempt to compensate for that same
> > language's (argued) shortcomings are a valid _and encouraged_ approach. 
> > Your poo-pooing of Python conditionals, decorators, and lambdas to solve
> > this particular problem, to me, seems like you want a *particular
> > solution*.
> 
> No, it's just that why use several different decorators when a slight 
> change in semantics renders them all obsolete? Why have to use a 
> decorator/lambda/conditional and remember when to use which? I'm trying 
> to generalize the pattern of "reevaluate/copy default values each time 
> they're required" into new behavior for default arguments. Indeed, as I 
> stated in another email (no idea which one, there's been too many), I 
> wholeheartedly support adding some of those decorators to the standard 
> library should my PEP get rejected.

The conditional works in all cases.  The lambda works in all cases. The
set of decorators seems to need to be tuned depending on the situation.
Me, I would just use a conditional or even a plain if statement, as it
is already sufficiently universal to handle every case. Discussions
about 'but then I can't mask global names with local names' are going to
fall on deaf ears; it's seen as poor form, and discussing what one can
do using poor Python form is not topical.


> > I don't see a problem with the current default argument semantics.  Why? 
> > Because in the case where I would want to receive a mutable parameter,
> > like in the case below that wouldn't work with Python's standard
> > semantics...
> > 
> >     def append_10(lst=[]):
> >         lst.append(10)
> >         return lst
> > 
> > I would presumably change it to...
> > 
> >     def append_10(lst=None):
> >         if lst is None: lst = []
> >         lst.append(10)
> >         return lst
> > 
> > Or some variant thereof.  Now, here's the thing; if I want a mutable
> > argument, then None is a nonsensical value to pass, generally, as it is
> > not mutable.  So I don't buy the whole "but then None would no longer be a
> > valid argument to pass" bull that was offered as a reason why the above
> > isn't a reasonable translation (I can't remember who offered it).
> 
> Nowhere has it been proposed to forbid passing None as an argument.

And I never claimed as much.  There was a previous post by someone (I
can't remember who offered it), saying that replacing 'lst=[]' with
'lst=None' would make it so that a user couldn't signal *something
special* with a 'None' argument.  I was trying to get across that such
reasoning doesn't make sense in this particular context.


> The 
> basic core of the proposal is to have Python compile the code such that 
> the boilerplate is either removed entirely (if the new semantics are the 
> default) by no longer being necessary, or dramatically shortened via the 
> addition of a new keyword (which would indicate the new semantics). Yes, 
> you can manually transform your code to use the idiom you mention. But 
> why should you have to? Why not have Python do the heavy-lifting for you?

Heavy lifting?  One line change and one line addition is heavy lifting? 
There has got to be an episode of Monty Python embedded in this thread
somewhere, because I can't help laughing at your claim.


> > I'm also not convinced by either of the 3 pages that talk about Python
> > "gotchas".  You get bitten by it, you learn it, understand it, and move
> > on.  If they can't figure it out, I'm not sure I want them writing
> > Python software anyways; I certainly wouldn't want them to work on any
> > of the Python software I work on and use.
> 
> I'm not here to argue that people shouldn't be made to understand the 
> difference between mutable and immutable values. They definitely should 
> know the difference. I'm just advocating a language change to make 
> certain kinds of functions less verbose.

My point is that the change results in *certain* functions being
trivially less verbose.  I've spent more time working around re's call
semantics than I have dealing with (incorrect assumptions about Python's)
default argument semantics.  As I don't see a need to change re, I don't
see a need to change default argument semantics.


 - Josiah



From jimjjewett at gmail.com  Tue Jan 30 18:22:01 2007
From: jimjjewett at gmail.com (Jim Jewett)
Date: Tue, 30 Jan 2007 12:22:01 -0500
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <op.tmyvfhe1d64u53@aidwe2058w.minlnv.agro.nl>
References: <45BCF804.7040207@gmail.com>
	<fb6fbf560701291451i2b8442bdof217cf172726cf66@mail.gmail.com>
	<45BED309.10301@gmail.com>
	<fb6fbf560701292319l4b42f0b4nc33da1eb1b00ce96@mail.gmail.com>
	<op.tmyvfhe1d64u53@aidwe2058w.minlnv.agro.nl>
Message-ID: <fb6fbf560701300922u33fb5f16v234300220b48d6f4@mail.gmail.com>

On 1/30/07, Jan Kanis <jan.kanis at phil.uu.nl> wrote:
> On Tue, 30 Jan 2007 08:19:11 +0100, Jim Jewett <jimjjewett at gmail.com>
> wrote:
> > On 1/30/07, Chris Rebert <cvrebert at gmail.com> wrote:
> >> Also, due to the
> >> hairiness/obscurity of the construct, hardly anyone uses it, so the
> >> impact of removing it should be relatively minor.

> > I just did found 181 such uses in my own installation.  29 were either
> > 3rd-party or test code, but that still leaves over 150 uses in the
> > standard library.  It is possible that some of them would also work
> > with your semantics, or may even be bugs today -- but you would have
> > to go through them one-by-one as part of the PEP process.

> But that's certainly possible. What exactly did you search for, and what
> did you find?

In idle, edit menu, find in files, regex 'def.*=(\\[|\\{)(\\]|\\})'

In other words, it caught "={}" or "=[]" on a def line.

It did not catch other (possibly mutable) default args, nor did it
catch multi-line definitions, nor did it even catch extra whitespace.
I didn't notice any false positives where the "={}" was really in a
comment.

I didn't even look at defaults of None (let alone other singletons) to
see if they were part of this idiom.

They mostly seemed to be passing state, but the __init__ methods in
particular may have been using a class-level variable by accident.
(such as the cnf argument to Tk widgets.)  I'll try to check more
closely later.

-jJ


From george.sakkis at gmail.com  Tue Jan 30 18:22:54 2007
From: george.sakkis at gmail.com (George Sakkis)
Date: Tue, 30 Jan 2007 12:22:54 -0500
Subject: [Python-ideas] Mutable default arguments - another approach
In-Reply-To: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com>
References: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com>
Message-ID: <91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com>

On 1/30/07, Piotr Duda <duda.piotr at gmail.com> wrote:

> I think that this problem can be solved by the following change of
> default argument behavior:
> default arguments are evaluated in definition time (like in current
> implementation), but right after being evaluated, result object is
> checked if it's mutable (for example by checking of presence __copy__
> special method or being instance of built in (sub)class
> list/dict/set)
>
> (snipped)
>

AFAIK there's no reliable way of deciding whether an arbitrary object
is mutable or not, so the rest of the post is irrelevant. Besides,
mutable default arguments is just a specific use case; the general
problem discussed here is call-time vs definition-time semantics for
arbitrary default argument expressions (function calls, attribute
lookups, etc.).

George


From duda.piotr at gmail.com  Tue Jan 30 19:18:46 2007
From: duda.piotr at gmail.com (Piotr Duda)
Date: Tue, 30 Jan 2007 19:18:46 +0100
Subject: [Python-ideas] Mutable default arguments - another approach
In-Reply-To: <91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com>
References: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com>
	<91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com>
Message-ID: <3df8f2650701301018i22fff8deq86829a292dceb8b2@mail.gmail.com>

2007/1/30, George Sakkis <george.sakkis at gmail.com>:
> On 1/30/07, Piotr Duda <duda.piotr at gmail.com> wrote:
>
> > I think that this problem can be solved by the following change of
> > default argument behavior:
> > default arguments are evaluated in definition time (like in current
> > implementation), but right after being evaluated, result object is
> > checked if it's mutable (for example by checking of presence __copy__
> > special method or being instance of built in (sub)class
> > list/dict/set)
> >
> > (snipped)
> >
>
> AFAIK there's no reliable way of deciding whether an arbitrary object
> is mutable or not, so the rest of the post is irrelevant. Besides,
> mutable default arguments is just a specific use case; the general
> problem discussed here is call-time vs definition-time semantics for
> arbitrary default argument expressions (function calls, attribute
> lookups, etc.).

AFAIK there's no reliable way of deciding whether an arbitrary object
has any feature (except it's type at c level), in worst case you can
treat all object as mutable, but that itself doesn't make rest of my
post irrelevant (have you even read it?).
Read my previous post again, and give me examples of what you cannot
do regarding default arguments.


-- 
????????
??????

From rnd at onego.ru  Tue Jan 30 20:29:34 2007
From: rnd at onego.ru (Roman Susi)
Date: Tue, 30 Jan 2007 21:29:34 +0200
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <op.tmyxw9g2d64u53@aidwe2058w.minlnv.agro.nl>
References: <fb6fbf560701250641k45f24f40qfea54fa07e6b40c4@mail.gmail.com>
	<op.tmsemwqxd64u53@e500> <20070126212657.5A06.JCARLSON@uci.edu>
	<op.tmyxw9g2d64u53@aidwe2058w.minlnv.agro.nl>
Message-ID: <45BF9C9E.8010206@onego.ru>

Jan Kanis wrote:
> On Mon, 29 Jan 2007 08:38:39 +0100, Roman Susi <rnd at onego.ru> wrote:
> 
>> This is what
>> incremental dynamic semantics is about. So, the suggestion is good only
>> as separated feature, but is IMHO wrong
>> if considered in the language design as a whole.
> 
> 
> wtf is incremental dynamic semantics, in this context? I did some
> googling  but all I found referred to techniques related to programming 
> environments. This proposal is just about a change in the language.

Oh, I thought this is very fundamental and well-known as it is the
second line of Guido's definition of Python:

"Python is an interpreted, object-oriented, high-level programming
language with dynamic semantics. ..."

As I understand it, it means that statements are read sequencially and
the semantics of names depends on the flow control.

The proposal (as it seems to me) wants to make change to the language
inconsistent with nature dynamic semantics. Its like put implicit:

if RUN_FIRST_TIME:
    do this
else:
    do that

for the same line of code (def statement's first line).


> [snip]


> Well, it's good to be clear on where the disagreements lie. However I'm 
> not yet ready to let it rest at that without some more arguments.
> 
> As Chris pointed out in his first mail, this 'wart' is mentioned on 
> several lists of python misfeatures: [0][1][2]. I'd like to add to this 
> that even the python documentation finds this issue severe enough to
> issue  an "Important warning"[4].
> 
> It seems clear that this behaviour is a gotcha, at least for newbies.
> This  could be excused if there is a good reason to spend the additional
> time  learning this behaviour, but some of the links state, and my

But as somebody already said the alternative is even worse...
Its quite easier to mention 3-5 Python warts up front to newbies than to
introduce subtle exception for semantics and noise words such as "new"
which do not have any other use elsewhere.

> assumption  is, that there are very few situations where re-evaluating
> causes a  problem and which isn't easily fixable.
> The semantics which I'd like to have are even easier than the current 
> semantics: everything in a function, be it before or after the colon,
> is  executed when the function is called.
> Of course, as Collin Winters pointed out, the burden of proof of
> showing  that these semantics aren't going to be a problem is still on
> the pep  proponents.
> 

> So, are there any _other_ arguments in favour of the current semantics??

Repeat the mantra:

    Python's dynamic semantics will not change
    Python's dynamic semantics will not change
    Python's dynamic semantics will not change

;-)


> 
> - Jan


Roman


From brett at python.org  Tue Jan 30 20:32:21 2007
From: brett at python.org (Brett Cannon)
Date: Tue, 30 Jan 2007 11:32:21 -0800
Subject: [Python-ideas] Import and '..', '../..' in serach path.
In-Reply-To: <45BF2D86.2080801@ronadam.com>
References: <45BF2D86.2080801@ronadam.com>
Message-ID: <bbaeab100701301132l5623625chb203d47d55e10f0d@mail.gmail.com>

On 1/30/07, Ron Adam <rrr at ronadam.com> wrote:
>
> In order to resolve a path conflict where I'm working on several copies of the
> same package.  I found it useful to add the following near the top of modules in
> a package or sub package.
>
>
> Module in package:
>
>      import sys
>      sys.path = ['..'] + sys.path
>
>      import package.module          # Imports module in "this!" package.
>
>
> Note: There could still be conflicts if a module with the same name is in the
> same directory as the package.  But that's much less likely than one in the rest
> of the path.
>
>
> Module in sub-package:
>
>      import sys
>      sys.path = ['../..'] + sys.path
>
>      import package.subpackage.module   # finds "self" (subpackage) reliably.
>
>
> By explicitly adding the packages parent directory to the *front* of sys.path it
> resolves cases where imports using absolute imports, import modules from another
> package because they are found first in the search path.

Why aren't you using relative imports (e.g., ``from . import
module``)?  That should be doing exactly what you want.  That uses
__path__ which is set to the path of the package.

-Brett


From rnd at onego.ru  Tue Jan 30 20:48:57 2007
From: rnd at onego.ru (Roman Susi)
Date: Tue, 30 Jan 2007 21:48:57 +0200
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BED64D.10801@gmail.com>
References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru>
	<45BED64D.10801@gmail.com>
Message-ID: <45BFA129.1030805@onego.ru>

Chris Rebert wrote:
> Roman Susi wrote:
> 
>> Hello!
>>
>> I'd liked to say outright that this bad idea which complicates matters

[skip]

> 
>> P.S. However, I may be wrong. In that case my syntax suggestion would
>> be this:
>>
>> def foo(non_const or []):
>>    ...
>>
>> where [] is executed at runtime BECAUSE at def time non_const is
>> somehow True and that is enough to leave [] alone.
>> I have not checked, but I believe it is backward compatible.
>> Anyway, could you summarize both contr-argument and this syntax
>> proposal in the PEP?
> 
> 
> I don't quite understand exactly how this would work and would like more
> details on it, but once you've explained it, of course I'd be happy to
> include it in the next draft.

Simple.


def foo(non_const or []):
    ...

is equivalent to

def foo(non_const=None):
    if non_const is None:
        none_const = []
    ...


And this will be as before:

def foo(non_const=[]):
    ...

Also, I thing that programmers should not use subtle difference between
None and other False values, so something like

def foo(non_const=None):
    non_const = none_const or []

is also valid.

Another approach (if you want to pursue the feature) could be
complication to name binding protocol.

a = []

will be as before, but default value assignment could trigger some extra
method. So, you can explicitly regulate your instance reaction to
default-value assignment:

class MyMutableList:
    ...
    def __default__(self, old_default):
        return old_default.copy()




Regards,
Roman


> - Chris Rebert
> 



From rnd at onego.ru  Tue Jan 30 21:03:25 2007
From: rnd at onego.ru (Roman Susi)
Date: Tue, 30 Jan 2007 22:03:25 +0200
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BED309.10301@gmail.com>
References: <45BCF804.7040207@gmail.com>	<fb6fbf560701291451i2b8442bdof217cf172726cf66@mail.gmail.com>
	<45BED309.10301@gmail.com>
Message-ID: <45BFA48D.2020501@onego.ru>

Chris Rebert wrote:
> Jim Jewett wrote:
> 
>>On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:

/skip/

>>
>>>     def foo(bar=new baz):
>>>         #code
>>
>>This would be less bad.
>>
>>That said, I fear many new programmers would fail to understand when
>>they needed new and when they didn't, so that in practice, it would be
>>just optional random noise.
> 
> 
> This is part of the reason I'm trying to avoid adding new syntax. 
> However, I assert that at least 'new' is clearer than the' x=None; if x 

But if you are concerned with the current None, there could be some
other, new False value serving the same need, like:

    def foo(x, y, z, bar=Missing, qux=Missing):
        if baz is Missing:
            baz = []
        #code

or even:

    def foo(x, y, z, bar=, qux=):
        if baz is Missing:
            baz = []
        #code

at least, it doesn't require decorators, is backward compatible
(hopefully no grammar conflicts in there), reads as English.



> is None: x=expr' idiom in that it expresses one's intent more clearly. 
> Also, this would at least be a prettier way to spell the idiom even if 
> the reason still needed explaining. Alternatively, we could make the new 
> semantics the default and have syntax to use the old semantics via 
> 'once' or some other keyword. This does nicely emphasize that such 
> semantics would be purely for optimization reasons. I think I'll add 
> this to the PEP.
> 
> 
>>>Demonstrative examples of new semantics:
>>>     #default argument expressions can refer to
>>>     #variables in the enclosing scope...
>>>     CONST = "hi"
>>>     def foo(a=CONST):
>>>         print a
>>
>>This would work if there were any easy way to create a new scope.  In
>>Lisp, it makes sense.  In python, it would probably be better to just
>>find a way for functions to refer to their own decorations reliably.
> 
> 
> This is outside of the scope of my PEP. However, the below improvement 
> should help and you could always use one of the other refactorings to 
> work around this issue.
> 
> 
>>>Backwards Compatibility
>>>
>>>     This change in semantics breaks all code which uses mutable default
>>>argument values. Such code can be refactored from:
>>>
>>>     def foo(bar=mutable):
>>>         #code
>>>
>>>to
>>
>>   [a decorator option uses 3 levels of nested functions, which aren't even
>>   generic enough for reuse, unless you give up introspection.]
>>
>>No.  It could be done with a (more complex) decorator, if that
>>decorator came with the stdlib (probably in functools)
> 
> 
> Agreed, the decorator could be better. I've just enhanced it using 
> functools.wraps, which should help with the introspection issues you 
> raise. Other improvements to the refactoring code in the PEP are welcomed.
> 
> - Chris Rebert


From g.brandl at gmx.net  Tue Jan 30 21:17:55 2007
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 30 Jan 2007 21:17:55 +0100
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BFA48D.2020501@onego.ru>
References: <45BCF804.7040207@gmail.com>	<fb6fbf560701291451i2b8442bdof217cf172726cf66@mail.gmail.com>	<45BED309.10301@gmail.com>
	<45BFA48D.2020501@onego.ru>
Message-ID: <epo95k$9up$1@sea.gmane.org>

Roman Susi schrieb:
> Chris Rebert wrote:
>> Jim Jewett wrote:
>> 
>>>On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:
> 
> /skip/
> 
>>>
>>>>     def foo(bar=new baz):
>>>>         #code
>>>
>>>This would be less bad.
>>>
>>>That said, I fear many new programmers would fail to understand when
>>>they needed new and when they didn't, so that in practice, it would be
>>>just optional random noise.
>> 
>> 
>> This is part of the reason I'm trying to avoid adding new syntax. 
>> However, I assert that at least 'new' is clearer than the' x=None; if x 
> 
> But if you are concerned with the current None, there could be some
> other, new False value serving the same need, like:
> 
>     def foo(x, y, z, bar=Missing, qux=Missing):
>         if baz is Missing:
>             baz = []
>         #code
> 
> or even:
> 
>     def foo(x, y, z, bar=, qux=):
>         if baz is Missing:
>             baz = []
>         #code

why not even

def foo(x, y, z, bar=..., qux=...):
    if bar is Ellipsis:
        # foo

;)

Georg



From rnd at onego.ru  Tue Jan 30 21:33:31 2007
From: rnd at onego.ru (Roman Susi)
Date: Tue, 30 Jan 2007 22:33:31 +0200
Subject: [Python-ideas] Mutable default arguments - another approach
In-Reply-To: <91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com>
References: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com>
	<91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com>
Message-ID: <45BFAB9B.4090904@onego.ru>

George Sakkis wrote:
> On 1/30/07, Piotr Duda <duda.piotr at gmail.com> wrote:
> 
> 
>>I think that this problem can be solved by the following change of
>>default argument behavior:
>>default arguments are evaluated in definition time (like in current
>>implementation), but right after being evaluated, result object is
>>checked if it's mutable (for example by checking of presence __copy__
>>special method or being instance of built in (sub)class
>>list/dict/set)
>>
>>(snipped)
>>
> 
> 
> AFAIK there's no reliable way of deciding whether an arbitrary object
> is mutable or not, so the rest of the post is irrelevant. Besides,
> mutable default arguments is just a specific use case; the general
> problem discussed here is call-time vs definition-time semantics for
> arbitrary default argument expressions (function calls, attribute
> lookups, etc.).
> 
> George

Maybe Python interpreter could be better at deciding if the programmer
which wrote the code is novice or experienced.

For example,

from __present__ import _I_AM_PYTHON_EXPERT

at the beginning of each module will do ;-)

Or, alternatively, Python interpreter will read user's directory for
some obscure configuration file where programmer will put True's into
originally:

I_KNOW_WHAT_MUTABLE_MEANS  = False
I_AM_AWARE_OF_MUTABLE_DEFAULT_ARGUMENT = False
...

(And of course, there will be secret

I_AM_GUIDO_VAN_ROSSUM = True

which will override all the previous settings.)

AFAIK, no language has this feature yet (not sure about Perl).


Regards,
Roman



From rrr at ronadam.com  Tue Jan 30 21:38:53 2007
From: rrr at ronadam.com (Ron Adam)
Date: Tue, 30 Jan 2007 14:38:53 -0600
Subject: [Python-ideas] Import and '..', '../..' in serach path.
In-Reply-To: <bbaeab100701301132l5623625chb203d47d55e10f0d@mail.gmail.com>
References: <45BF2D86.2080801@ronadam.com>
	<bbaeab100701301132l5623625chb203d47d55e10f0d@mail.gmail.com>
Message-ID: <45BFACDD.5040602@ronadam.com>

Brett Cannon wrote:
> On 1/30/07, Ron Adam <rrr at ronadam.com> wrote:
>>
>> In order to resolve a path conflict where I'm working on several 
>> copies of the
>> same package.  I found it useful to add the following near the top of 
>> modules in
>> a package or sub package.
>>
>>
>> Module in package:
>>
>>      import sys
>>      sys.path = ['..'] + sys.path
>>
>>      import package.module          # Imports module in "this!" package.
>>
>>
>> Note: There could still be conflicts if a module with the same name is 
>> in the
>> same directory as the package.  But that's much less likely than one 
>> in the rest
>> of the path.
>>
>>
>> Module in sub-package:
>>
>>      import sys
>>      sys.path = ['../..'] + sys.path
>>
>>      import package.subpackage.module   # finds "self" (subpackage) 
>> reliably.
>>
>>
>> By explicitly adding the packages parent directory to the *front* of 
>> sys.path it
>> resolves cases where imports using absolute imports, import modules 
>> from another
>> package because they are found first in the search path.
> 
> Why aren't you using relative imports (e.g., ``from . import
> module``)?  That should be doing exactly what you want.  That uses
> __path__ which is set to the path of the package.
> 
> -Brett


But if you try to run a module in a package, vs from a package, with relative
from . imports, you will get:

      ValueError: Attempted relative import in non-package

And there is no __path__ attribute to look at since no package has been imported
yet.

So you need to import the package first, and then you may not import the correct 
package even then if there is another package or module with the same name.

For example if you are running a test module from test sub-package that has a
"if __name__=='__main__': _test()" at the bottom. Something that is not
uncommon. It may attempt to import modules from a different package. In my case
it was a package with the same name in a different SVN branch. The tests
actually ran, but with errors because it wasn't exactly the same. It wasn't
obvious what was going on either.

Cheers,
Ron



From gsakkis at rutgers.edu  Tue Jan 30 23:11:18 2007
From: gsakkis at rutgers.edu (George Sakkis)
Date: Tue, 30 Jan 2007 17:11:18 -0500
Subject: [Python-ideas] Mutable default arguments - another approach
In-Reply-To: <45BFAB9B.4090904@onego.ru>
References: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com>
	<91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com>
	<45BFAB9B.4090904@onego.ru>
Message-ID: <91ad5bf80701301411l72943aediad759282697630c8@mail.gmail.com>

On 1/30/07, Roman Susi <rnd at onego.ru> wrote:

> George Sakkis wrote:
> > On 1/30/07, Piotr Duda <duda.piotr at gmail.com> wrote:
> >
> >
> >>I think that this problem can be solved by the following change of
> >>default argument behavior:
> >>default arguments are evaluated in definition time (like in current
> >>implementation), but right after being evaluated, result object is
> >>checked if it's mutable (for example by checking of presence __copy__
> >>special method or being instance of built in (sub)class
> >>list/dict/set)
> >>
> >>(snipped)
> >>
> >
> >
> > AFAIK there's no reliable way of deciding whether an arbitrary object
> > is mutable or not, so the rest of the post is irrelevant. Besides,
> > mutable default arguments is just a specific use case; the general
> > problem discussed here is call-time vs definition-time semantics for
> > arbitrary default argument expressions (function calls, attribute
> > lookups, etc.).
> >
> > George
>
> Maybe Python interpreter could be better at deciding if the programmer
> which wrote the code is novice or experienced.
>
> For example,
>
> from __present__ import _I_AM_PYTHON_EXPERT
>
> at the beginning of each module will do ;-)

That would be a good start, but certainly not the end; how about
programmers who *read* other people's code ? Their experience level
may well be quite different, and the code should be understandable by
these poor souls too. The python source files would have to be able to
sense when they're being read or printed, determine the experience
level of the reader and transform themselves to accomodate his level.
Anyone wanna write the PEP for Python-7000 ? ;-)

George


From brett at python.org  Tue Jan 30 23:21:49 2007
From: brett at python.org (Brett Cannon)
Date: Tue, 30 Jan 2007 14:21:49 -0800
Subject: [Python-ideas] Import and '..', '../..' in serach path.
In-Reply-To: <45BFACDD.5040602@ronadam.com>
References: <45BF2D86.2080801@ronadam.com>
	<bbaeab100701301132l5623625chb203d47d55e10f0d@mail.gmail.com>
	<45BFACDD.5040602@ronadam.com>
Message-ID: <bbaeab100701301421y175d2d04ha4d445489d88203@mail.gmail.com>

On 1/30/07, Ron Adam <rrr at ronadam.com> wrote:
> Brett Cannon wrote:
> > On 1/30/07, Ron Adam <rrr at ronadam.com> wrote:
> >>
> >> In order to resolve a path conflict where I'm working on several
> >> copies of the
> >> same package.  I found it useful to add the following near the top of
> >> modules in
> >> a package or sub package.
> >>
> >>
> >> Module in package:
> >>
> >>      import sys
> >>      sys.path = ['..'] + sys.path
> >>
> >>      import package.module          # Imports module in "this!" package.
> >>
> >>
> >> Note: There could still be conflicts if a module with the same name is
> >> in the
> >> same directory as the package.  But that's much less likely than one
> >> in the rest
> >> of the path.
> >>
> >>
> >> Module in sub-package:
> >>
> >>      import sys
> >>      sys.path = ['../..'] + sys.path
> >>
> >>      import package.subpackage.module   # finds "self" (subpackage)
> >> reliably.
> >>
> >>
> >> By explicitly adding the packages parent directory to the *front* of
> >> sys.path it
> >> resolves cases where imports using absolute imports, import modules
> >> from another
> >> package because they are found first in the search path.
> >
> > Why aren't you using relative imports (e.g., ``from . import
> > module``)?  That should be doing exactly what you want.  That uses
> > __path__ which is set to the path of the package.
> >
> > -Brett
>
>
> But if you try to run a module in a package, vs from a package, with relative
> from . imports, you will get:
>
>       ValueError: Attempted relative import in non-package
>
> And there is no __path__ attribute to look at since no package has been imported
> yet.

Yeah, I have been bitten by that.  You need to have that initial
package import.  I got nailed by this when using '-m' and a module
contained within a package for executing tests.

>
> So you need to import the package first, and then you may not import the correct
> package even then if there is another package or module with the same name.
>

Well, just don't do that.  =)  You really shouldn't have multiple
things with the same name in sys.path.  But as you noticed, you can
control it with sys.path if you really need to.

I just don't see this as a common enough problem to warrant
documenting it somewhere (and I have no clue where that "somewhere"
would be).

-Brett


From jcarlson at uci.edu  Tue Jan 30 23:31:36 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Tue, 30 Jan 2007 14:31:36 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BFA48D.2020501@onego.ru>
References: <45BED309.10301@gmail.com> <45BFA48D.2020501@onego.ru>
Message-ID: <20070130142937.5A38.JCARLSON@uci.edu>


Roman Susi <rnd at onego.ru> wrote:
>     def foo(x, y, z, bar=Missing, qux=Missing):
>         if baz is Missing:
>             baz = []
>         #code

With the proper definition of Missing, the above is fine, and is more or
less creating a new 'None' value.

> or even:
> 
>     def foo(x, y, z, bar=, qux=):
>         if baz is Missing:
>             baz = []
>         #code
> 
> at least, it doesn't require decorators, is backward compatible
> (hopefully no grammar conflicts in there), reads as English.

The above with a missing value for a default *is not* backwards
compatible with previous Pythons.  New syntax is, by definition, not
backwards compatible.

 - Josiah



From cvrebert at gmail.com  Wed Jan 31 00:08:53 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Tue, 30 Jan 2007 15:08:53 -0800
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <45BF9C9E.8010206@onego.ru>
References: <fb6fbf560701250641k45f24f40qfea54fa07e6b40c4@mail.gmail.com>
	<op.tmsemwqxd64u53@e500> <20070126212657.5A06.JCARLSON@uci.edu>
	<op.tmyxw9g2d64u53@aidwe2058w.minlnv.agro.nl>
	<45BF9C9E.8010206@onego.ru>
Message-ID: <47c890dc0701301508yeaab646t85efb2cf9f69e4df@mail.gmail.com>

On 1/30/07, Roman Susi <rnd at onego.ru> wrote:
> The proposal (as it seems to me) wants to make change to the language
> inconsistent with nature dynamic semantics. Its like put implicit:
>
> if RUN_FIRST_TIME:
>     do this
> else:
>     do that
>
> for the same line of code (def statement's first line).

No, my proto-PEP has never contained semantics like that anywhere in it.

> > Well, it's good to be clear on where the disagreements lie. However I'm
> > not yet ready to let it rest at that without some more arguments.
> >
> > As Chris pointed out in his first mail, this 'wart' is mentioned on
> > several lists of python misfeatures: [0][1][2]. I'd like to add to this
> > that even the python documentation finds this issue severe enough to
> > issue  an "Important warning"[4].
> >
> > It seems clear that this behaviour is a gotcha, at least for newbies.
> > This  could be excused if there is a good reason to spend the additional
> > time  learning this behaviour, but some of the links state, and my
>
> But as somebody already said the alternative is even worse...
> Its quite easier to mention 3-5 Python warts up front to newbies than to
> introduce subtle exception for semantics and noise words such as "new"
> which do not have any other use elsewhere.

Sidenote: There could instead be a new keyword 'once' to indicate the
old semantics. I think the PEP related to adding a switch statement
proposes the same keyword for a similar use.

- Chris Rebert


From jan.kanis at phil.uu.nl  Wed Jan 31 01:28:31 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Wed, 31 Jan 2007 01:28:31 +0100
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <20070130142937.5A38.JCARLSON@uci.edu>
References: <45BED309.10301@gmail.com> <45BFA48D.2020501@onego.ru>
	<20070130142937.5A38.JCARLSON@uci.edu>
Message-ID: <op.tmzsxtjhd64u53@e500>

On Tue, 30 Jan 2007 23:31:36 +0100, Josiah Carlson <jcarlson at uci.edu>  
wrote:
> Roman Susi <rnd at onego.ru> wrote:
>>
>>     def foo(x, y, z, bar=, qux=):
>>         if baz is Missing:
>>             baz = []
>>         #code
>>
>> at least, it doesn't require decorators, is backward compatible
>> (hopefully no grammar conflicts in there), reads as English.
>
> The above with a missing value for a default *is not* backwards
> compatible with previous Pythons.  New syntax is, by definition, not
> backwards compatible.
>
>  - Josiah

As a matter of fact, backward-compatible syntax changes are certainly  
possible. (ever wondered how C++ got it's syntax?) Any valid current  
python is still going to behave exactly the same if this syntax were to be  
accepted. Talking about backward compatibility, I think it is safe to  
ignore any text files that don't get accepted by the python interpreter.  
This syntax change would certainly not break any existing production  
python code.
(note: the above statements do not entail in any way that I am in favour  
of this syntax change)

- Jan


From jan.kanis at phil.uu.nl  Wed Jan 31 01:33:47 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Wed, 31 Jan 2007 01:33:47 +0100
Subject: [Python-ideas] Mutable default arguments - another approach
In-Reply-To: <91ad5bf80701301411l72943aediad759282697630c8@mail.gmail.com>
References: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com>
	<91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com>
	<45BFAB9B.4090904@onego.ru>
	<91ad5bf80701301411l72943aediad759282697630c8@mail.gmail.com>
Message-ID: <op.tmzs6lb7d64u53@e500>

Nah, this is all old school. If this pep is gonna succeed it needs some  
buzzwords like Web 2.0:
The interpreter should check with google what kinds of programming-related  
search terms the programmer has used and for how long. Using google  
desktop search, it can also include the kind of documents the programmer  
has and when he's read them. With GvR working where he is, I don't think  
implementing this is going to be a big problem.
;-)

On Tue, 30 Jan 2007 23:11:18 +0100, George Sakkis <gsakkis at rutgers.edu>  
wrote:

> On 1/30/07, Roman Susi <rnd at onego.ru> wrote:
>
>> Maybe Python interpreter could be better at deciding if the programmer
>> which wrote the code is novice or experienced.
>>
>> For example,
>>
>> from __present__ import _I_AM_PYTHON_EXPERT
>>
>> at the beginning of each module will do ;-)
>
> That would be a good start, but certainly not the end; how about
> programmers who *read* other people's code ? Their experience level
> may well be quite different, and the code should be understandable by
> these poor souls too. The python source files would have to be able to
> sense when they're being read or printed, determine the experience
> level of the reader and transform themselves to accomodate his level.
> Anyone wanna write the PEP for Python-7000 ? ;-)
>
> George
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas




From jan.kanis at phil.uu.nl  Wed Jan 31 02:16:37 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Wed, 31 Jan 2007 02:16:37 +0100
Subject: [Python-ideas] fixing mutable default argument values
In-Reply-To: <45BF9C9E.8010206@onego.ru>
References: <fb6fbf560701250641k45f24f40qfea54fa07e6b40c4@mail.gmail.com>
	<op.tmsemwqxd64u53@e500> <20070126212657.5A06.JCARLSON@uci.edu>
	<op.tmyxw9g2d64u53@aidwe2058w.minlnv.agro.nl>
	<45BF9C9E.8010206@onego.ru>
Message-ID: <op.tmzu5zd1d64u53@e500>

On Tue, 30 Jan 2007 20:29:34 +0100, Roman Susi <rnd at onego.ru> wrote:

> Jan Kanis wrote:
>> On Mon, 29 Jan 2007 08:38:39 +0100, Roman Susi <rnd at onego.ru> wrote:
>>
>>> This is what
>>> incremental dynamic semantics is about. So, the suggestion is good only
>>> as separated feature, but is IMHO wrong
>>> if considered in the language design as a whole.
>>
>>
>> wtf is incremental dynamic semantics, in this context? I did some
>> googling  but all I found referred to techniques related to programming
>> environments. This proposal is just about a change in the language.
>
> Oh, I thought this is very fundamental and well-known as it is the
> second line of Guido's definition of Python:
>
> "Python is an interpreted, object-oriented, high-level programming
> language with dynamic semantics. ..."
>
> As I understand it, it means that statements are read sequencially and
> the semantics of names depends on the flow control.

I never made a link with this description of python and the term  
'incremental dynamic semantics'. Guess I learned something new today.

> The proposal (as it seems to me) wants to make change to the language
> inconsistent with nature dynamic semantics. Its like put implicit:
>
> if RUN_FIRST_TIME:
>     do this
> else:
>     do that
>
> for the same line of code (def statement's first line).
>
>
>> [snip]
>

I'm certainly no proponent of doing something like this. Do note that the  
discussion is between evaluating default expressions at definition time or  
at call time, and the pep entails evaluating default expressions on  
/every/ function call, just like the body of the function is evaluated on  
every call. There's already other code being evaluated at both def time  
and call time, so the proposal does not introduce anything new to pythons  
evaluation model.
IMO having the default exprs evaluate at call time is in no way against  
pythons language design.

>
>> Well, it's good to be clear on where the disagreements lie. However I'm
>> not yet ready to let it rest at that without some more arguments.
>>
>> As Chris pointed out in his first mail, this 'wart' is mentioned on
>> several lists of python misfeatures: [0][1][2]. I'd like to add to this
>> that even the python documentation finds this issue severe enough to
>> issue  an "Important warning"[4].
>>
>> It seems clear that this behaviour is a gotcha, at least for newbies.
>> This  could be excused if there is a good reason to spend the additional
>> time  learning this behaviour, but some of the links state, and my
>
> But as somebody already said the alternative is even worse...
> Its quite easier to mention 3-5 Python warts up front to newbies than to
> introduce subtle exception for semantics and noise words such as "new"
> which do not have any other use elsewhere.

But it's much better to just eliminate the warts without introducing  
subtle exceptions. The pep proposes evaluating default exprs on every  
function call, just like the function body is. No new exceptions are  
introduced. The fact that newbies often expect default values to be fresh  
on every call seems to entail that they won't be surprised a lot when the  
idiom of using default values as caches won't work, if the pep gets  
accepted. Old time pythoneers who know what's new in 3.0 won't be  
surprised either. I'm not in favour of introducing new noise words or  
other new syntax, I just want to 'fix' pythons current semantics.

>> assumption  is, that there are very few situations where re-evaluating
>> causes a  problem and which isn't easily fixable.
>> The semantics which I'd like to have are even easier than the current
>> semantics: everything in a function, be it before or after the colon,
>> is  executed when the function is called.
>> Of course, as Collin Winters pointed out, the burden of proof of
>> showing  that these semantics aren't going to be a problem is still on
>> the pep  proponents.
>>
>
>> So, are there any _other_ arguments in favour of the current semantics??
>
> Repeat the mantra:
>
>     Python's dynamic semantics will not change
>     Python's dynamic semantics will not change
>     Python's dynamic semantics will not change
>
> ;-)

If anything, the proposal is going to _improve_ pythons dynamicness and  
late-bindingness.

Oh, and it's nice to know repeating mantras now count as 'arguments' <wink>

- Jan


From jcarlson at uci.edu  Wed Jan 31 02:35:08 2007
From: jcarlson at uci.edu (Josiah Carlson)
Date: Tue, 30 Jan 2007 17:35:08 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <op.tmzsxtjhd64u53@e500>
References: <20070130142937.5A38.JCARLSON@uci.edu> <op.tmzsxtjhd64u53@e500>
Message-ID: <20070130173337.5A4D.JCARLSON@uci.edu>


"Jan Kanis" <jan.kanis at phil.uu.nl> wrote:
> 
> On Tue, 30 Jan 2007 23:31:36 +0100, Josiah Carlson <jcarlson at uci.edu>  
> wrote:
> > Roman Susi <rnd at onego.ru> wrote:
> >>
> >>     def foo(x, y, z, bar=, qux=):
> >>         if baz is Missing:
> >>             baz = []
> >>         #code
> >>
> >> at least, it doesn't require decorators, is backward compatible
> >> (hopefully no grammar conflicts in there), reads as English.
> >
> > The above with a missing value for a default *is not* backwards
> > compatible with previous Pythons.  New syntax is, by definition, not
> > backwards compatible.
> >
> >  - Josiah
> 
> As a matter of fact, backward-compatible syntax changes are certainly  
> possible. (ever wondered how C++ got it's syntax?) Any valid current  
> python is still going to behave exactly the same if this syntax were to be  
> accepted. Talking about backward compatibility, I think it is safe to  
> ignore any text files that don't get accepted by the python interpreter.  
> This syntax change would certainly not break any existing production  
> python code.
> (note: the above statements do not entail in any way that I am in favour  
> of this syntax change)

Fowards compatible then.  That is to say, writing for the new proposed
system will break compatibility with older Pythons.  On the other hand,
using currently-available language syntax and semantics is compatible
among the entire range of Pythons available today.


 - Josiah



From jan.kanis at phil.uu.nl  Wed Jan 31 02:31:06 2007
From: jan.kanis at phil.uu.nl (Jan Kanis)
Date: Wed, 31 Jan 2007 02:31:06 +0100
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <20070129233754.5A32.JCARLSON@uci.edu>
References: <20070129212157.5A2B.JCARLSON@uci.edu> <45BEF306.5030801@gmail.com>
	<20070129233754.5A32.JCARLSON@uci.edu>
Message-ID: <op.tmzvt4had64u53@e500>

On Tue, 30 Jan 2007 18:16:03 +0100, Josiah Carlson <jcarlson at uci.edu>  
wrote:
>
> Chris Rebert <cvrebert at gmail.com> wrote:
>>
>> > Because in the case where I would want to receive a mutable parameter,
>> > like in the case below that wouldn't work with Python's standard
>> > semantics...
>> >
>> >     def append_10(lst=[]):
>> >         lst.append(10)
>> >         return lst
>> >
>> > I would presumably change it to...
>> >
>> >     def append_10(lst=None):
>> >         if lst is None: lst = []
>> >         lst.append(10)
>> >         return lst
>> >
>> > Or some variant thereof.  Now, here's the thing; if I want a mutable
>> > argument, then None is a nonsensical value to pass, generally, as it  
>> is
>> > not mutable.  So I don't buy the whole "but then None would no longer  
>> be a
>> > valid argument to pass" bull that was offered as a reason why the  
>> above
>> > isn't a reasonable translation (I can't remember who offered it).
>>
>> Nowhere has it been proposed to forbid passing None as an argument.
>
> And I never claimed as much.  There was a previous post by someone (I
> can't remember who offered it), saying that replacing 'lst=[]' with
> 'lst=None' would make it so that a user couldn't signal *something
> special* with a 'None' argument.  I was trying to get across that such
> reasoning doesn't make sense in this particular context.

The above example code was mine. I was just trying to explain that the  
proposed semantics should not mistake a caller-provided argument value of  
None for a no-argument-provided situation and evaluate the default  
expression. If the proposal were to be implemented naively by literally  
rewriting the python code in the way that I used to explain the proposal  
and compiling the rewritten code, this bug could happen.

- Jan


From cvrebert at gmail.com  Wed Jan 31 02:54:13 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Tue, 30 Jan 2007 17:54:13 -0800
Subject: [Python-ideas] Mutable default arguments - another approach
In-Reply-To: <3df8f2650701301018i22fff8deq86829a292dceb8b2@mail.gmail.com>
References: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com>	<91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com>
	<3df8f2650701301018i22fff8deq86829a292dceb8b2@mail.gmail.com>
Message-ID: <45BFF6C5.8020909@gmail.com>

Piotr Duda wrote:
> Read my previous post again, and give me examples of what you cannot
> do regarding default arguments.

First, I'd like to register my dislike for implicitly determining when a 
default argument will be copied on every call.

Second, as per your request:

def foo(bar=None):
     if bar is None: bar = function_call()
     #code

def foo(bar=None):
     if bar is None: bar = something.method_call()
     #code

#These were just a few. There are others.

These cannot be rewritten using simple copying decorators. Various 
decorators have been presented that account for some of these (and 
other) situations. However, IMHO they're not too elegant/clean. Thus, a 
slightly more sophisticated solution is required.

- Chris Rebert


From cvrebert at gmail.com  Wed Jan 31 02:59:05 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Tue, 30 Jan 2007 17:59:05 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <43aa6ff70701292152q158092d3xf1b37b4ff4ff7438@mail.gmail.com>
References: <45BCF804.7040207@gmail.com> <45BDA47F.4040603@onego.ru>	
	<91ad5bf80701292016m63bf1b28w84e2c9f62fb7ea0e@mail.gmail.com>	
	<45BEDA53.9010103@gmail.com>
	<43aa6ff70701292152q158092d3xf1b37b4ff4ff7438@mail.gmail.com>
Message-ID: <45BFF7E9.1080305@gmail.com>

Collin Winter wrote:
 >> On 1/29/07, Chris Rebert <cvrebert at gmail.com> wrote:
>> Why? Yes, there _might_ be performance issues (which have yet to be
>> demonstrated or deeply speculated upon), but re-evaluating immutable
>> default arguments wouldn't affect a program's correct operation.
> 
> If the underlying intent of your proposal -- that all default
> arguments be re-evaluated with every call -- were to be approved,
> there would undoubtedly be a serious performance impact.
[snip]
> As for whether these effects have "yet to be demonstrated", as you
> say, the burden is on you, the PEP author, to investigate and resolve,
> mitigate or justify any and all performance changes.
> 
> Collin Winter
> 

You do have a point there, it is my responsibility. I therefore shall 
endeavor to find out how much of a performance hit my suggested change 
would have.

- Chris Rebert


From cvrebert at gmail.com  Wed Jan 31 03:05:13 2007
From: cvrebert at gmail.com (Chris Rebert)
Date: Tue, 30 Jan 2007 18:05:13 -0800
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BFA48D.2020501@onego.ru>
References: <45BCF804.7040207@gmail.com>	<fb6fbf560701291451i2b8442bdof217cf172726cf66@mail.gmail.com>
	<45BED309.10301@gmail.com> <45BFA48D.2020501@onego.ru>
Message-ID: <45BFF959.4030906@gmail.com>

Roman Susi wrote:
> Chris Rebert wrote:
>> Jim Jewett wrote:
>>
>>> On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:
> 
> /skip/
> 
>>>>     def foo(bar=new baz):
>>>>         #code
>>> This would be less bad.
>>>
>>> That said, I fear many new programmers would fail to understand when
>>> they needed new and when they didn't, so that in practice, it would be
>>> just optional random noise.
>>
>> This is part of the reason I'm trying to avoid adding new syntax. 
>> However, I assert that at least 'new' is clearer than the' x=None; if x 
> 
> But if you are concerned with the current None, there could be some
> other, new False value serving the same need, like:
> 
>     def foo(x, y, z, bar=Missing, qux=Missing):
>         if baz is Missing:
>             baz = []
>         #code
> 
> or even:
> 
>     def foo(x, y, z, bar=, qux=):
>         if baz is Missing:
>             baz = []
>         #code
> 
> at least, it doesn't require decorators, is backward compatible
> (hopefully no grammar conflicts in there), reads as English.

Those examples are only slightly better than using None. The convention 
of using None to indicate use of a mutable/non-constant default value is 
about as clear as your examples.

The point wasn't that None wasn't descriptive/specific enough, but 
rather why have to write the 'bar=None' and then the 'if bar is None' 
when 'bar=[]' could be sufficient? Also, this indicates exactly what the 
default value is, as opposed to None/Missing, which is just an opaque 
indicator for 'some mutable/non-constant default value'.

Thus, I see no need for 'Missing' or a similar constant.

- Chris Rebert


From rrr at ronadam.com  Wed Jan 31 05:17:34 2007
From: rrr at ronadam.com (Ron Adam)
Date: Tue, 30 Jan 2007 22:17:34 -0600
Subject: [Python-ideas] Import and '..', '../..' in serach path.
In-Reply-To: <bbaeab100701301421y175d2d04ha4d445489d88203@mail.gmail.com>
References: <45BF2D86.2080801@ronadam.com>	
	<bbaeab100701301132l5623625chb203d47d55e10f0d@mail.gmail.com>	
	<45BFACDD.5040602@ronadam.com>
	<bbaeab100701301421y175d2d04ha4d445489d88203@mail.gmail.com>
Message-ID: <45C0185E.2090408@ronadam.com>

Brett Cannon wrote:
> On 1/30/07, Ron Adam <rrr at ronadam.com> wrote:
>> Brett Cannon wrote:
>> > On 1/30/07, Ron Adam <rrr at ronadam.com> wrote:
>> >>
>> >> In order to resolve a path conflict where I'm working on several
>> >> copies of the
>> >> same package.  I found it useful to add the following near the top of
>> >> modules in
>> >> a package or sub package.
>> >>
>> >>
>> >> Module in package:
>> >>
>> >>      import sys
>> >>      sys.path = ['..'] + sys.path
>> >>
>> >>      import package.module          # Imports module in "this!" 
>> package.
>> >>
>> >>
>> >> Note: There could still be conflicts if a module with the same name is
>> >> in the
>> >> same directory as the package.  But that's much less likely than one
>> >> in the rest
>> >> of the path.
>> >>
>> >>
>> >> Module in sub-package:
>> >>
>> >>      import sys
>> >>      sys.path = ['../..'] + sys.path
>> >>
>> >>      import package.subpackage.module   # finds "self" (subpackage)
>> >> reliably.
>> >>
>> >>
>> >> By explicitly adding the packages parent directory to the *front* of
>> >> sys.path it
>> >> resolves cases where imports using absolute imports, import modules
>> >> from another
>> >> package because they are found first in the search path.
>> >
>> > Why aren't you using relative imports (e.g., ``from . import
>> > module``)?  That should be doing exactly what you want.  That uses
>> > __path__ which is set to the path of the package.
>> >
>> > -Brett
>>
>>
>> But if you try to run a module in a package, vs from a package, with 
>> relative
>> from . imports, you will get:
>>
>>       ValueError: Attempted relative import in non-package
>>
>> And there is no __path__ attribute to look at since no package has 
>> been imported
>> yet.
> 
> Yeah, I have been bitten by that.  You need to have that initial
> package import.  I got nailed by this when using '-m' and a module
> contained within a package for executing tests.

And you can't set __path__ manually, that only changes the Exception to an 
Import Error.


>> So you need to import the package first, and then you may not import 
>> the correct
>> package even then if there is another package or module with the same 
>> name.
>>
> 
> Well, just don't do that.  =)  You really shouldn't have multiple
> things with the same name in sys.path.  But as you noticed, you can
> control it with sys.path if you really need to.
> 
> I just don't see this as a common enough problem to warrant
> documenting it somewhere (and I have no clue where that "somewhere"
> would be).


I expected that someone would say that.

How about adding a note at the bottom of PEP 328. That was one of the places I 
looked for a way to resolve this.  Or possibly posting a message on 
python-ideas.  Consider it done  ;-)

A note on PEP 328 might be a good idea still.  I have no idea what the best 
wording would be.  The import tutorial is another place where a note clarifying 
relative imports can't be used for stand alone scripts is needed.


The nice thing about adjusting sys.path directly is you can then move back and 
forth from different same named versions in development without doing renames. 
If you do a rename, you also have to remember to rename all the submodule 
absolute references in the package as well.  Another alternative is to hide the 
ones you don't want by renaming them.  Then you have to do renames to switch 
back to them.  That makes it more difficult to do side by side comparisons as well.

Altering PYTHONPATH is another way to do it, but you also need to switch it back 
if you switch versions.  But if you forget to edit it back, and use a version 
not on the path, it will import modules from the version on the path.

In this case the package I'm working on may replace pydoc.py module in pythons 
library. (no promises though) The name matters so that site.py can locate the 
help function.  It's convenient for me to be able to easily and dependably 
compare different working versions.


A few things to consider...

-----
According to PEP 328:

You may use relative imports freely. In Python 2.6, any import statement that 
results in an intra-package import will raise DeprecationWarning (this also 
applies to from <> import that fails to use the relative import syntax). In 
Python 2.7, import will always be an absolute import (and the __future__ 
directive will no longer be needed).
-----

So It will probably come up more often as more people start using absolute 
imports and the '.' style relative imports. Or as you put it, more people get 
"bitten".

Take a look at how much __name__ == '__main__' is used in the standard library. 
  Those modules won't be able to use relative imports.  They will need to be 
updated and can use only absolute imports.

Some people will probably want a way to have relative imports work with modules 
meant to be run as scripts (in packages).

An optional function (or property) may be a good way to resolve both of these 
package self reference situations without any changes to the default behavior.


Cheers,
   Ron




From rnd at onego.ru  Wed Jan 31 09:04:33 2007
From: rnd at onego.ru (Roman Susi)
Date: Wed, 31 Jan 2007 10:04:33 +0200
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <20070130173337.5A4D.JCARLSON@uci.edu>
References: <20070130142937.5A38.JCARLSON@uci.edu> <op.tmzsxtjhd64u53@e500>
	<20070130173337.5A4D.JCARLSON@uci.edu>
Message-ID: <45C04D91.8030906@onego.ru>

Josiah Carlson wrote:
> "Jan Kanis" <jan.kanis at phil.uu.nl> wrote:
> 
>>On Tue, 30 Jan 2007 23:31:36 +0100, Josiah Carlson <jcarlson at uci.edu>  
>>wrote:
>>
>>>Roman Susi <rnd at onego.ru> wrote:
>>>
>>>>    def foo(x, y, z, bar=, qux=):
>>>>        if baz is Missing:
>>>>            baz = []
>>>>        #code
>>>>
>>>>at least, it doesn't require decorators, is backward compatible
>>>>(hopefully no grammar conflicts in there), reads as English.
>>>
>>>The above with a missing value for a default *is not* backwards
>>>compatible with previous Pythons.  New syntax is, by definition, not
>>>backwards compatible.
>>>
>>> - Josiah
>>
>>As a matter of fact, backward-compatible syntax changes are certainly  
>>possible. (ever wondered how C++ got it's syntax?) Any valid current  
>>python is still going to behave exactly the same if this syntax were to be  
>>accepted. Talking about backward compatibility, I think it is safe to  
>>ignore any text files that don't get accepted by the python interpreter.  
>>This syntax change would certainly not break any existing production  
>>python code.
>>(note: the above statements do not entail in any way that I am in favour  
>>of this syntax change)
> 
> 
> Fowards compatible then.  That is to say, writing for the new proposed
> system will break compatibility with older Pythons.  On the other hand,
> using currently-available language syntax and semantics is compatible
> among the entire range of Pythons available today.

>From wikipedia:


"... a product is said to be backward compatible (or downward
compatible) when it is able to take the place of an older product, by
interoperating with other products that were designed for the older
product."

"Forward compatibility (sometimes confused with extensibility) is the
ability of a system to accept input intended for later versions of itself."


So, the right term is that Python 3000 will be in the syntactic aspect
we discuss backward compatible because it will accept old syntax too.


Regards,
Roman



>  - Josiah
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
> 
> 
> !DSPAM:45bff1845411635716952!
> 



From rnd at onego.ru  Wed Jan 31 09:24:04 2007
From: rnd at onego.ru (Roman Susi)
Date: Wed, 31 Jan 2007 10:24:04 +0200
Subject: [Python-ideas] proto-PEP: Fixing Non-constant Default Arguments
In-Reply-To: <45BFF959.4030906@gmail.com>
References: <45BCF804.7040207@gmail.com>	<fb6fbf560701291451i2b8442bdof217cf172726cf66@mail.gmail.com>
	<45BED309.10301@gmail.com> <45BFA48D.2020501@onego.ru>
	<45BFF959.4030906@gmail.com>
Message-ID: <45C05224.6090407@onego.ru>

Chris Rebert wrote:
> Roman Susi wrote:
> 
>> Chris Rebert wrote:
>>
>>> Jim Jewett wrote:
>>>
>>>> On 1/28/07, Chris Rebert <cvrebert at gmail.com> wrote:
>>
>>
>> /skip/
>>
>>>>>     def foo(bar=new baz):
>>>>>         #code
>>>>
>>>> This would be less bad.
>>>>
>>>> That said, I fear many new programmers would fail to understand when
>>>> they needed new and when they didn't, so that in practice, it would be
>>>> just optional random noise.
>>>
>>>
>>> This is part of the reason I'm trying to avoid adding new syntax.
>>> However, I assert that at least 'new' is clearer than the' x=None; if x 
>>
>>
>> But if you are concerned with the current None, there could be some
>> other, new False value serving the same need, like:
>>
>>     def foo(x, y, z, bar=Missing, qux=Missing):
>>         if baz is Missing:
>>             baz = []
>>         #code
>>
>> or even:
>>
>>     def foo(x, y, z, bar=, qux=):
>>         if baz is Missing:
>>             baz = []
>>         #code
>>
>> at least, it doesn't require decorators, is backward compatible
>> (hopefully no grammar conflicts in there), reads as English.
> 
> 
> Those examples are only slightly better than using None. The convention
> of using None to indicate use of a mutable/non-constant default value is
> about as clear as your examples.
> 
> The point wasn't that None wasn't descriptive/specific enough, but
> rather why have to write the 'bar=None' and then the 'if bar is None'
> when 'bar=[]' could be sufficient? Also, this indicates exactly what the
> default value is, as opposed to None/Missing, which is just an opaque
> indicator for 'some mutable/non-constant default value'.
> 
> Thus, I see no need for 'Missing' or a similar constant.

And I think it is overkill to add new syntax there. For example, if the
function or method is near the external (human or network) interface
border it is good to check ALL arguments anyway. So the extra

     if baz is Missing:
         baz = []

doesn't spoil the impression:

     def foo(x, y, z, baz=, qux=):
         ...
         if baz is Missing:
             baz = []

         ...
         if type(baz) != type([]) or len(baz) > 1000 or min(baz) < 0 or ...:
            raise ...

         #code


What I'd wanted to say is that =None or =Missing is not just
"indicator for 'some mutable/non-constant default value'." but
may also mean that the argument is subject to further constraints and
thus funtion code has checks for that.

So, instead of having clear indicator of missing argument we have some
fresh instance of a list or dict.

Also, in many situation another approach is used nowadays:

     def foo(x, y, z, **args):
        ...
        baz = args.get("baz", [])
        qux = args.get("qux", {})


Which has the effect proto-PEP author would like to achieve.
Of course, no automatic tools are going to make type checks with that
helpful function signature.

Also, while its offtopic in this post, I'd really liked Python 3000 to
have more unification for dict vs attribute access because there are use
cases (Storage in web.py, Namespace in mxTools - I am sure many other
projects have similar things) which single purpose is to have both dict
and attribute syntax in one data object interchangeably:

     def foo(**args):
        ...
        baz = args.baz
        qux = args["qux"]


Regards,
Roman

> 
> - Chris Rebert


From duda.piotr at gmail.com  Wed Jan 31 14:19:06 2007
From: duda.piotr at gmail.com (Piotr Duda)
Date: Wed, 31 Jan 2007 14:19:06 +0100
Subject: [Python-ideas] Mutable default arguments - another approach
In-Reply-To: <45BFF6C5.8020909@gmail.com>
References: <3df8f2650701300905p75302fb8x23296da519d90b7a@mail.gmail.com>
	<91ad5bf80701300922t360560d3y4e83a6c578f50338@mail.gmail.com>
	<3df8f2650701301018i22fff8deq86829a292dceb8b2@mail.gmail.com>
	<45BFF6C5.8020909@gmail.com>
Message-ID: <3df8f2650701310519g28f09105t6e367cdd5a59981a@mail.gmail.com>

2007/1/31, Chris Rebert <cvrebert at gmail.com>:
> Piotr Duda wrote:
> > Read my previous post again, and give me examples of what you cannot
> > do regarding default arguments.
>
> First, I'd like to register my dislike for implicitly determining when a
> default argument will be copied on every call.
In my solution you can always declare this explicitly.
Only problem here is to determine which objects are copied, simplest
rule here is to copy all object except instances of built-in immutable
types.

>
> Second, as per your request:
>
> def foo(bar=None):
>      if bar is None: bar = function_call()
>      #code
>
> def foo(bar=None):
>      if bar is None: bar = something.method_call()
>      #code

In my solution you can do both.
Assuming there are all classes/methods defined in my first post, use:

@DefaultArgWrapper
def calldefarg(callable): #this function should be defined in stdlib
    while 1:
        yield callable()

first example

def foo(bar = calldefarg(function_call)):
    #code

here i assume that function_call is available at definition call, if
it isn't then use this

def foo(bar = calldefarg(lambda: function_call())):
    #code

and the second example

def foo(bar = calldefarg(something.method_call)):
    #code

or if something isn't available at definition time

def foo(bar = calldefarg(lambda: something.method_call())):
    #code

there's one restriction, that 'something' cannot depend on other
function argument(s).

> #These were just a few. There are others.
>
> These cannot be rewritten using simple copying decorators. Various
> decorators have been presented that account for some of these (and
> other) situations. However, IMHO they're not too elegant/clean. Thus, a
> slightly more sophisticated solution is required.

Are you referring here to my solution?

-- 
????????
??????

From brett at python.org  Wed Jan 31 20:48:53 2007
From: brett at python.org (Brett Cannon)
Date: Wed, 31 Jan 2007 11:48:53 -0800
Subject: [Python-ideas] Import and '..', '../..' in serach path.
In-Reply-To: <45C0185E.2090408@ronadam.com>
References: <45BF2D86.2080801@ronadam.com>
	<bbaeab100701301132l5623625chb203d47d55e10f0d@mail.gmail.com>
	<45BFACDD.5040602@ronadam.com>
	<bbaeab100701301421y175d2d04ha4d445489d88203@mail.gmail.com>
	<45C0185E.2090408@ronadam.com>
Message-ID: <bbaeab100701311148l2c1d8b41qc1bc3f35b405d857@mail.gmail.com>

On 1/30/07, Ron Adam <rrr at ronadam.com> wrote:
> Brett Cannon wrote:
[SNIP]
> > I just don't see this as a common enough problem to warrant
> > documenting it somewhere (and I have no clue where that "somewhere"
> > would be).
>
>
> I expected that someone would say that.
>
> How about adding a note at the bottom of PEP 328. That was one of the places I
> looked for a way to resolve this.  Or possibly posting a message on
> python-ideas.  Consider it done  ;-)
>
> A note on PEP 328 might be a good idea still.  I have no idea what the best
> wording would be.  The import tutorial is another place where a note clarifying
> relative imports can't be used for stand alone scripts is needed.
>

Sure, but that is not my PEP.  That is something to bug Thomas Wouters about.

>
> The nice thing about adjusting sys.path directly is you can then move back and
> forth from different same named versions in development without doing renames.
> If you do a rename, you also have to remember to rename all the submodule
> absolute references in the package as well.  Another alternative is to hide the
> ones you don't want by renaming them.  Then you have to do renames to switch
> back to them.  That makes it more difficult to do side by side comparisons as well.
>

Right but the point of relative imports is to help prevent renaming.
It's a catch-22 situation of what matters more to you.

> Altering PYTHONPATH is another way to do it, but you also need to switch it back
> if you switch versions.  But if you forget to edit it back, and use a version
> not on the path, it will import modules from the version on the path.
>
> In this case the package I'm working on may replace pydoc.py module in pythons
> library. (no promises though) The name matters so that site.py can locate the
> help function.  It's convenient for me to be able to easily and dependably
> compare different working versions.
>
>
> A few things to consider...
>
> -----
> According to PEP 328:
>
> You may use relative imports freely. In Python 2.6, any import statement that
> results in an intra-package import will raise DeprecationWarning (this also
> applies to from <> import that fails to use the relative import syntax). In
> Python 2.7, import will always be an absolute import (and the __future__
> directive will no longer be needed).
> -----
>
> So It will probably come up more often as more people start using absolute
> imports and the '.' style relative imports. Or as you put it, more people get
> "bitten".
>

Probably.

> Take a look at how much __name__ == '__main__' is used in the standard library.
>   Those modules won't be able to use relative imports.  They will need to be
> updated and can use only absolute imports.
>

It's possible.  Maybe this will finally push forward the idea of
having a proper 'main' function or something.

> Some people will probably want a way to have relative imports work with modules
> meant to be run as scripts (in packages).
>
> An optional function (or property) may be a good way to resolve both of these
> package self reference situations without any changes to the default behavior.
>

Yeah, some built-in or something might do it that lets you decorate
what function is to be the 'main' function and maybe do some magic to
make it resolve properly.

-Brett


From rrr at ronadam.com  Wed Jan 31 23:45:45 2007
From: rrr at ronadam.com (Ron Adam)
Date: Wed, 31 Jan 2007 16:45:45 -0600
Subject: [Python-ideas] Import and '..', '../..' in serach path.
In-Reply-To: <bbaeab100701311148l2c1d8b41qc1bc3f35b405d857@mail.gmail.com>
References: <45BF2D86.2080801@ronadam.com>	
	<bbaeab100701301132l5623625chb203d47d55e10f0d@mail.gmail.com>	
	<45BFACDD.5040602@ronadam.com>	
	<bbaeab100701301421y175d2d04ha4d445489d88203@mail.gmail.com>	
	<45C0185E.2090408@ronadam.com>
	<bbaeab100701311148l2c1d8b41qc1bc3f35b405d857@mail.gmail.com>
Message-ID: <45C11C19.5030703@ronadam.com>

Brett Cannon wrote:
> On 1/30/07, Ron Adam <rrr at ronadam.com> wrote:
>> Brett Cannon wrote:
> [SNIP]
>> > I just don't see this as a common enough problem to warrant
>> > documenting it somewhere (and I have no clue where that "somewhere"
>> > would be).
>>
>>
>> I expected that someone would say that.
>>
>> How about adding a note at the bottom of PEP 328. That was one of the 
>> places I
>> looked for a way to resolve this.  Or possibly posting a message on
>> python-ideas.  Consider it done  ;-)
>>
>> A note on PEP 328 might be a good idea still.  I have no idea what the 
>> best
>> wording would be.  The import tutorial is another place where a note 
>> clarifying
>> relative imports can't be used for stand alone scripts is needed.
>>
> 
> Sure, but that is not my PEP.  That is something to bug Thomas Wouters 
> about.

Ok, Maybe later.  I looked into it further and I think it needs more thought in 
any case.


>> The nice thing about adjusting sys.path directly is you can then move 
>> back and
>> forth from different same named versions in development without doing 
>> renames.
>> If you do a rename, you also have to remember to rename all the submodule
>> absolute references in the package as well.  Another alternative is to 
>> hide the
>> ones you don't want by renaming them.  Then you have to do renames to 
>> switch
>> back to them.  That makes it more difficult to do side by side 
>> comparisons as well.

> Right but the point of relative imports is to help prevent renaming.
> It's a catch-22 situation of what matters more to you.

Right, thats what I tried first.  It would have been nice if that worked.


>> Take a look at how much __name__ == '__main__' is used in the standard 
>> library.
>>   Those modules won't be able to use relative imports.  They will need 
>> to be
>> updated and can use only absolute imports.

> It's possible.  Maybe this will finally push forward the idea of
> having a proper 'main' function or something.

<clip>

> Yeah, some built-in or something might do it that lets you decorate
> what function is to be the 'main' function and maybe do some magic to
> make it resolve properly.

It's not the main function that is the problem, it's the imports at the top of 
the file before the main function.  If they import ok, then any main function 
will work also.


I did some experimenting and just adding '../..' in front of the path has some 
problems of its own.  It's relative to the directory you run it from instead of 
the directory the script is in.  So it may or may not allow the script to run.

What is needed is to append the actual root package location to the front os 
sys.path. If absolute imports are used for intra-package imports, then there is 
not chance of shadowing the library by doing this.


This is what I came up with, but there may be an easier way. (and shorter)

def pkg_dir(file):
     import os
     file = os.path.abspath(file)
     path = os.path.dirname(file)
     # Next line should check for (.pyo, .pyc) files too.
     while os.path.exists(os.path.join(path, '__init__.py')):
         path = os.path.abspath(os.path.join(path, os.pardir))
     return path


# Library imports
import sys
sys.path = [pkg_dir(__file__)] + sys.path

# Absolute package imports here
...

tests():
     ...

if __name__ == '__main__':
    tests()


This does what you would expect it to do. It always finds it's own packages 
modules and it's not sensitive to what location you run it from.  But it's bit 
bulky to be putting everywhere.


The real problem isn't that you might get errors, it is that you *may not*.  For 
example, if you decide to modify an existing package, and so copy it to a new 
location for editing.  Then you do your edits, and after everything looks good 
you run the tests in the tests sub module. It passes, Yay!  So then you copy it 
back to it's original location.

Unfortunately the tests may have imported the unedited modules in the live 
package. So the edited package you replaced it with may have errors.  How 
serious that is depends on what the package is used for.  Those mars satellites 
can get pretty expensive after you loose 2 or 3 of them.  ;-)

I admit that's pretty unlikely because the package would probably go though some 
pretty rigorous testing in that case.  But still, there is the chance of 
something like that happening in more casual circumstances.  I don't think this 
is that far fetched.

And the other benefit is it just makes it easier to run modules in different 
circumstances without problems.  On the path, directly, indirectly... while 
editing.  That is really what I want at the moment.  It should *just work*.


Cheers,
    Ron