From cmjohnson.mailinglist at gmail.com  Wed Jun  1 06:52:05 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Tue, 31 May 2011 18:52:05 -1000
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
Message-ID: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>

We all know that the following code won't work because of UnboundLocalError
and that to get around it, one needs to use nonlocal:

>>> def accum():
...     x = 0
...     def inner():
...         x += 1
...         return x
...     return inner
...
>>> inc = accum()
>>> inc()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in inner
UnboundLocalError: local variable 'x' referenced before assignment

But why does this happen? Let's think about this a little more closely: +=
is not the same as =. A += can only happen if the left-hand term was already
defined. So, why does the compiler treat this as though there were an
assignment inside the function? Compare:

>>> def accum():
...     x = []
...     def inner():
...         x.append(1)
...         return x
...     return inner
...
>>> inc = accum()
>>> inc()
[1]
>>> inc()
[1, 1]
>>> inc()
[1, 1, 1]

So, if I changed += to .append, the code suddenly works fine. Heck, I could
also change it to x.__iadd__ if x happens to have that attribute.

As we all know, adding an = anywhere to the function bound will cause x to
be considered a local. So, for example, we can make the .append example fail
by adding some unreachable code:

>>> def accum():
...     x = []
...     def inner():
...         x.append(1)
...         return x
...         x = 0 #Won't ever be reached, but will cause x to be considered
a local
...     return inner
...
>>> inc = accum()
>>> inc()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in inner
UnboundLocalError: local variable 'x' referenced before assignment

So, my proposal is that += by itself should not cause x to be considered a
local variable. There should need to be a normal = assignment for the
compiler to count x as a local. If the objection to my proposal is that I'm
being "implicit and not explicit" because it would be like there's an
implicit "nonlocal," my rebuttal is that we already have "implicit"
nonlocals in the case of .append.

-- Carl Johnson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110531/5c3ac55b/attachment.html>

From g.brandl at gmx.net  Wed Jun  1 07:48:19 2011
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 01 Jun 2011 07:48:19 +0200
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
Message-ID: <is4jqs$or8$1@dough.gmane.org>

On 01.06.2011 06:52, Carl M. Johnson wrote:
> We all know that the following code won't work because of UnboundLocalError and
> that to get around it, one needs to use nonlocal:
> 
>>>> def accum():
> ...     x = 0
> ...     def inner():
> ...         x += 1
> ...         return x
> ...     return inner
> ... 
>>>> inc = accum()
>>>> inc()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "<stdin>", line 4, in inner
> UnboundLocalError: local variable 'x' referenced before assignment
> 
> But why does this happen? Let's think about this a little more closely: += is
> not the same as =. A += can only happen if the left-hand term was already
> defined. So, why does the compiler treat this as though there were an assignment
> inside the function?

Because x += y is equivalent to

x = x.__iadd__(y)

and therefore an assignment is going on here.  Therefore, it's only logical to
treat it as such when determining scopes.

Georg



From cmjohnson.mailinglist at gmail.com  Wed Jun  1 08:48:55 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Tue, 31 May 2011 20:48:55 -1000
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <is4jqs$or8$1@dough.gmane.org>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
	<is4jqs$or8$1@dough.gmane.org>
Message-ID: <BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>

On Tue, May 31, 2011 at 7:48 PM, Georg Brandl <g.brandl at gmx.net> wrote:

> Because x += y is equivalent to
>
> x = x.__iadd__(y)
>
> and therefore an assignment is going on here.  Therefore, it's only logical
> to
> treat it as such when determining scopes.
>
>
But the difference is that you can only use += if the LHS name already
exists and is defined. So, it couldn't possibly be referring to a local name
if it's the only assignment-like statement within a function body. How could
it refer to a local if it has to refer to something that already exists?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110531/518b247c/attachment.html>

From g.brandl at gmx.net  Wed Jun  1 09:05:58 2011
From: g.brandl at gmx.net (Georg Brandl)
Date: Wed, 01 Jun 2011 09:05:58 +0200
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
	<is4jqs$or8$1@dough.gmane.org>
	<BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>
Message-ID: <is4ocf$fv8$1@dough.gmane.org>

On 01.06.2011 08:48, Carl M. Johnson wrote:
> 
> 
> On Tue, May 31, 2011 at 7:48 PM, Georg Brandl
> <g.brandl at gmx.net
> <mailto:g.brandl at gmx.net>> wrote:
> 
>     Because x += y is equivalent to
> 
>     x = x.__iadd__(y)
> 
>     and therefore an assignment is going on here.  Therefore, it's only logical to
>     treat it as such when determining scopes.
> 
> 
> But the difference is that you can only use += if the LHS name already exists
> and is defined. So, it couldn't possibly be referring to a local name if it's
> the only assignment-like statement within a function body. How could it refer to
> a local if it has to refer to something that already exists?

Sure, this can only work if the local is assigned somewhere before the augmented
assign statement.  But this is just like accessing a local before its
assignment: in the case of

x = 1
def f():
    print x
    x = 2

we also don't treat the first "x" reference as a nonlocal.

And the fact remains that augassign *is* an assignment, and the rule is that
assignments to out-of-scope names are only allowed when declared using "global"
or "nonlocal".

Georg



From cmjohnson.mailinglist at gmail.com  Wed Jun  1 10:26:02 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Tue, 31 May 2011 22:26:02 -1000
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <is4ocf$fv8$1@dough.gmane.org>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
	<is4jqs$or8$1@dough.gmane.org>
	<BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>
	<is4ocf$fv8$1@dough.gmane.org>
Message-ID: <BANLkTik6Zy_T0n_p-WeoL5RZYKvFYtyfsA@mail.gmail.com>

On Tue, May 31, 2011 at 9:05 PM, Georg Brandl <g.brandl at gmx.net> wrote:

> Sure, this can only work if the local is assigned somewhere before the
> augmented
> assign statement.  But this is just like accessing a local before its
> assignment: in the case of
>
> x = 1
> def f():
>    print x
>    x = 2
>
> we also don't treat the first "x" reference as a nonlocal.
>

I don't think that's a counterexample to the point I'm trying to make. We
all agree that if there's an x= somewhere in the function body, then we have
to treat the variable as a local. The only possible way around that would be
to solve the halting problem in order to figure out if a particular line of
code will be reached or not. Agreed, sure, we have to treat the LHS of = as
a local. But += is fundamentally different. You cannot have a += statement
unless somewhere out there there is a matching = statement. It cannot exist
independently. It never works on its own. So, if there is a += statement in
the function body and there isn't an = statement in the function body it
cannot work. Ever. All function bodies that have a += but no corresponding =
or nonlocal are, as of today, broken code. So, if we were to change Python
to make += not cause a variable to become a local, it wouldn't change how
any (working) Python code today functions (it might causes some tests to
change if they were counting on the error). This would be a completely
backwards compatible change.

Or am I missing something? Is there any scenario where you can get away with
using += without = or nonlocal? I guess you could do something with
locals().update or the stackframe, but my understanding is that those hacks
don't count for language purposes.

-- Carl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110531/3e26b148/attachment.html>

From p.f.moore at gmail.com  Wed Jun  1 10:51:33 2011
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 1 Jun 2011 09:51:33 +0100
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <BANLkTik6Zy_T0n_p-WeoL5RZYKvFYtyfsA@mail.gmail.com>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
	<is4jqs$or8$1@dough.gmane.org>
	<BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>
	<is4ocf$fv8$1@dough.gmane.org>
	<BANLkTik6Zy_T0n_p-WeoL5RZYKvFYtyfsA@mail.gmail.com>
Message-ID: <BANLkTinUEh-ivR9Gd8Ft_JXYKywR+fZ9OQ@mail.gmail.com>

On 1 June 2011 09:26, Carl M. Johnson <cmjohnson.mailinglist at gmail.com> wrote:
> I don't think that's a counterexample to the point I'm trying to make. We
> all agree that if there's an x= somewhere in the function body, then we have
> to treat the variable as a local. The only possible way around that would be
> to solve the halting problem in order to figure out if a particular line of
> code will be reached or not. Agreed, sure, we have to treat the LHS of = as
> a local. But += is fundamentally different. You cannot have a += statement
> unless somewhere out there there is a matching = statement. It cannot exist
> independently. It never works on its own. So, if there is a += statement in
> the function body and there isn't an = statement in the function body it
> cannot work. Ever. All function bodies that have a += but no corresponding =
> or nonlocal are, as of today, broken code. So, if we were to change Python
> to make += not cause a variable to become a local, it wouldn't change how
> any (working) Python code today functions (it might causes some tests to
> change if they were counting on the error). This would be a completely
> backwards compatible change.
> Or am I missing something? Is there any scenario where you can get away with
> using += without = or nonlocal? I guess you could do something with
> locals().update or the stackframe, but my understanding is that those hacks
> don't count for language purposes.

The place to start here is section 4.1 of the language reference
(Naming and Binding). Specifically, "A scope defines the visibility of
a name within a block. If a local variable is defined in a block, its
scope includes that block." Your modification of augmented assignment
implies that a block can contain 2 different scopes - consider

x = 1
def f():
    # The next statement uses the global x
    x += 1
    x = 2
    # From here, you have a local x

That fundamentally changes the language semantics.

If you want to push this change, I'd suggest you start by proposing a
change to the language reference section I mentioned above to define
your proposed new scoping rules. In my view, that would be
sufficiently hard that it'd kill this proposal, but if you can manage
to do it, then you may have a chance to get your change accepted.

Paul.


From jh at improva.dk  Wed Jun  1 11:09:06 2011
From: jh at improva.dk (Jacob Holm)
Date: Wed, 01 Jun 2011 11:09:06 +0200
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <BANLkTinUEh-ivR9Gd8Ft_JXYKywR+fZ9OQ@mail.gmail.com>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>	<is4jqs$or8$1@dough.gmane.org>	<BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>	<is4ocf$fv8$1@dough.gmane.org>	<BANLkTik6Zy_T0n_p-WeoL5RZYKvFYtyfsA@mail.gmail.com>
	<BANLkTinUEh-ivR9Gd8Ft_JXYKywR+fZ9OQ@mail.gmail.com>
Message-ID: <4DE601B2.20708@improva.dk>

I think you missed this statement, even though you quoted it.

On 2011-06-01 10:51, Paul Moore wrote:
> On 1 June 2011 09:26, Carl M. Johnson <cmjohnson.mailinglist at gmail.com> wrote:
>> We
>> all agree that if there's an x= somewhere in the function body, then we have
>> to treat the variable as a local. 
> 

This means that your example:

> x = 1
> def f():
>     # The next statement uses the global x
>     x += 1
>     x = 2
>     # From here, you have a local x
> 

Would behave exactly as it does today under the proposed new semantics.
 Specifically, the "x = 2" statement (and the lack of a nonlocal
statement) forces x to be local throughout the function, and the "x +=
1" statement then tries to read the local "x" and fails.

> That fundamentally changes the language semantics.

I don't think it does.  It only makes a difference for functions that
contains an augmented assignment to a name without also containing a
regular assignment to that name.  This case will change from being an
error to doing something well-defined and useful.

FWIW, I'm +1 on the idea.


Best regards

 - Jacob


From cmjohnson.mailinglist at gmail.com  Wed Jun  1 11:41:06 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Tue, 31 May 2011 23:41:06 -1000
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <4DE601B2.20708@improva.dk>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
	<is4jqs$or8$1@dough.gmane.org>
	<BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>
	<is4ocf$fv8$1@dough.gmane.org>
	<BANLkTik6Zy_T0n_p-WeoL5RZYKvFYtyfsA@mail.gmail.com>
	<BANLkTinUEh-ivR9Gd8Ft_JXYKywR+fZ9OQ@mail.gmail.com>
	<4DE601B2.20708@improva.dk>
Message-ID: <BANLkTikCXV9JBBPAb_GxmGR04v9zJgBQYA@mail.gmail.com>

On Tue, May 31, 2011 at 11:09 PM, Jacob Holm <jh at improva.dk> wrote:

>

> x = 1
> > def f():
> >     # The next statement uses the global x
> >     x += 1
> >     x = 2
> >     # From here, you have a local x
> >
>
>
>  Specifically, the "x = 2" statement (and the lack of a nonlocal
> statement) forces x to be local throughout the function, and the "x +=
> 1" statement then tries to read the local "x" and fails.
>


Yes, Jacob has got exactly what I was proposing. x += 1; x = 2 should
continue to fail, since there would be a = statement in the function body in
that case.

-- Carl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110531/339a5b43/attachment.html>

From rob.cliffe at btinternet.com  Wed Jun  1 12:43:14 2011
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Wed, 01 Jun 2011 11:43:14 +0100
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <BANLkTikCXV9JBBPAb_GxmGR04v9zJgBQYA@mail.gmail.com>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>	<is4jqs$or8$1@dough.gmane.org>	<BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>	<is4ocf$fv8$1@dough.gmane.org>	<BANLkTik6Zy_T0n_p-WeoL5RZYKvFYtyfsA@mail.gmail.com>	<BANLkTinUEh-ivR9Gd8Ft_JXYKywR+fZ9OQ@mail.gmail.com>	<4DE601B2.20708@improva.dk>
	<BANLkTikCXV9JBBPAb_GxmGR04v9zJgBQYA@mail.gmail.com>
Message-ID: <4DE617C2.2020602@btinternet.com>


> Yes, Jacob has got exactly what I was proposing. x += 1; x = 2 should 
> continue to fail, since there would be a = statement in the function 
> body in that case.
>
> -- Carl

My first reaction was: +1 on the proposed change.  It seemed logical.

Then I had a reservation: it would widen the semantic difference between
         x += 1
and
         x = x + 1
which could trip someone innocently making a "trivial" code change from 
the former to the latter (x unintentionally becomes a local).

So how about going further and say that x is only interpreted as local 
if there is at least one NON-augmented assignment in which x appears as 
a target on the LHS but x does NOT appear on the RHS?
I.e.
     x = x + 1
(like "x += 1") does not (by itself) make x local.

Or is this getting too hard to explain?

Best wishes
Rob Cliffe


From andreengels at gmail.com  Wed Jun  1 12:51:49 2011
From: andreengels at gmail.com (Andre Engels)
Date: Wed, 1 Jun 2011 12:51:49 +0200
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <4DE617C2.2020602@btinternet.com>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
	<is4jqs$or8$1@dough.gmane.org>
	<BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>
	<is4ocf$fv8$1@dough.gmane.org>
	<BANLkTik6Zy_T0n_p-WeoL5RZYKvFYtyfsA@mail.gmail.com>
	<BANLkTinUEh-ivR9Gd8Ft_JXYKywR+fZ9OQ@mail.gmail.com>
	<4DE601B2.20708@improva.dk>
	<BANLkTikCXV9JBBPAb_GxmGR04v9zJgBQYA@mail.gmail.com>
	<4DE617C2.2020602@btinternet.com>
Message-ID: <BANLkTimHdWjYP1khp9VKt_FnoQ6tNCoVRQ@mail.gmail.com>

On Wed, Jun 1, 2011 at 12:43 PM, Rob Cliffe <rob.cliffe at btinternet.com> wrote:

> My first reaction was: +1 on the proposed change. ?It seemed logical.
>
> Then I had a reservation: it would widen the semantic difference between
> ? ? ? ?x += 1
> and
> ? ? ? ?x = x + 1
> which could trip someone innocently making a "trivial" code change from the
> former to the latter (x unintentionally becomes a local).
>
> So how about going further and say that x is only interpreted as local if
> there is at least one NON-augmented assignment in which x appears as a
> target on the LHS but x does NOT appear on the RHS?
> I.e.
> ? ?x = x + 1
> (like "x += 1") does not (by itself) make x local.
>
> Or is this getting too hard to explain?

I think so; it also has the same disadvantage you mention of getting a
semantic change from seemingly neutral changes, but for other changes.
For example

x = 1 if x == 0 else x-1

would keep x global, but changing it to:

if x == 0:
    x = 1
else:
    x = x-1

would not do so.

-- 
Andr? Engels, andreengels at gmail.com


From ethan at stoneleaf.us  Wed Jun  1 13:24:30 2011
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 01 Jun 2011 04:24:30 -0700
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>	<is4jqs$or8$1@dough.gmane.org>
	<BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>
Message-ID: <4DE6216E.1000204@stoneleaf.us>

Carl M. Johnson wrote:
> On Tue, May 31, 2011 at 7:48 PM, Georg Brandl wrote:
>> 
>> Because x += y is equivalent to
>> 
>> x = x.__iadd__(y)
>> 
>> and therefore an assignment is going on here.  Therefore, it's only
>> logical to treat it as such when determining scopes.
> 
> But the difference is that you can only use += if the LHS name already 
> exists and is defined. So, it couldn't possibly be referring to a local 
> name if it's the only assignment-like statement within a function body. 
> How could it refer to a local if it has to refer to something that 
> already exists?

Two problems.  Firstly, what error should be raised here?

--> def accum():
...     x = 0
...     def inner():
...         x1 += 1
...         return x
...     return inner

Secondly, the += operator may or may not be a mutating operator 
depending on the object it's used on:  if the object does not have a 
__iadd__ method, it's not mutating; even if it does have an __iadd__ 
method, it may not be mutating -- it's up to the object to decide.

--> class ex_int(int):
...   def __iadd__(self, other):
...     return self + other
...
--> x = ex_int(7)
--> x.__iadd__(3)
10
--> x
7
--> x = [1, 2, 3]
--> x.__iadd__([4])
[1, 2, 3, 4]
--> x
[1, 2, 3, 4]

-1 on changing the semantics.

~Ethan~


From ncoghlan at gmail.com  Wed Jun  1 13:50:20 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 1 Jun 2011 21:50:20 +1000
Subject: [Python-ideas] Adding 'bytes' as alias for 'latin_1' codec.
In-Reply-To: <79306.1306858606@parc.com>
References: <BANLkTin1P5BVUisF-SfW=-TUeQbumDdBvA@mail.gmail.com>
	<irkc54$gih$1@dough.gmane.org> <irl5n7$l1q$1@dough.gmane.org>
	<BANLkTikQXPJ3Y0SBfG5UKhu1S-CVDo5sng@mail.gmail.com>
	<DF3948D2-C0AC-4B6E-A722-25004C48C0E2@masklinn.net>
	<irmk2f$ess$1@dough.gmane.org>
	<BANLkTikTnwrGA2N2oaxwNsqYDVeWO6CXVQ@mail.gmail.com>
	<BANLkTinsqOGZzU7DXBShde9N1kudoSFP8Q@mail.gmail.com>
	<4DE0481E.7010005@canterbury.ac.nz>
	<BANLkTimZRE15gPqVPZR27pQtWYgntycz9w@mail.gmail.com>
	<79306.1306858606@parc.com>
Message-ID: <BANLkTi=pNpXnqiXKbtGL2cvt6Kb+HBu4XQ@mail.gmail.com>

On Wed, Jun 1, 2011 at 2:16 AM, Bill Janssen <janssen at parc.com> wrote:
> I like the deprecations you suggest, but I'd prefer to see a more
> general solution: the 'str' type extended so that it had two possible
> representations for strings, the current format and an "encoded" format,
> which would be kept as an array of bytes plus an encoding. ?It would
> transcode only as necessary -- for example, the 're' module might
> require the current Unicode encoding. ?An explicit method would be added
> to allow the user to force transcoding.
>
> This would complicate life at the C level, to be sure. ?Though, perhaps
> not so much, given the proper macrology.

See PEP 393 - it is basically this idea (although the encodings are
fixed for the various sizes rather than allowing arbitrary encodings
in the 8-bit internal format).

Cheers,
Nick.


-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From andrew at acooke.org  Wed Jun  1 14:29:10 2011
From: andrew at acooke.org (andrew cooke)
Date: Wed, 1 Jun 2011 08:29:10 -0400
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <is4jqs$or8$1@dough.gmane.org>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
	<is4jqs$or8$1@dough.gmane.org>
Message-ID: <20110601122910.GA19508@acooke.org>

On Wed, Jun 01, 2011 at 07:48:19AM +0200, Georg Brandl wrote:
> Because x += y is equivalent to
> 
> x = x.__iadd__(y)
> 
> and therefore an assignment is going on here.  Therefore, it's only logical to
> treat it as such when determining scopes.
> 
> Georg


And it's like this so that immutable classes work correctly, as far as I can
see.

So one way to answer the original idea is: because of immutable classes, +=
does not have the same semantics as .append()

Andrew


From ncoghlan at gmail.com  Wed Jun  1 14:43:07 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 1 Jun 2011 22:43:07 +1000
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
Message-ID: <BANLkTi=tujW7maFT1jNPFkxK65+DfrGVCw@mail.gmail.com>

On Wed, Jun 1, 2011 at 2:52 PM, Carl M. Johnson
<cmjohnson.mailinglist at gmail.com> wrote:
> We all know that the following code won't work because of UnboundLocalError
> and that to get around it, one needs to use nonlocal:

There's no fundamental reason this couldn't change, but actually
changing it simply isn't worth the hassle, so the status quo wins the
stalemate.

I elaborated further on this point when the topic came up last year:
http://mail.python.org/pipermail/python-ideas/2010-June/007448.html

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From paul-python at svensson.org  Wed Jun  1 15:03:40 2011
From: paul-python at svensson.org (Paul Svensson)
Date: Wed, 1 Jun 2011 09:03:40 -0400 (EDT)
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <is4jqs$or8$1@dough.gmane.org>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
	<is4jqs$or8$1@dough.gmane.org>
Message-ID: <20110601081137.B68837@familjen.svensson.org>

On Wed, 1 Jun 2011, Georg Brandl wrote:

> On 01.06.2011 06:52, Carl M. Johnson wrote:
>> We all know that the following code won't work because of UnboundLocalError and
>> that to get around it, one needs to use nonlocal:
>>
>>>>> def accum():
>> ...     x = 0
>> ...     def inner():
>> ...         x += 1
>> ...         return x
>> ...     return inner
>> ...
>>>>> inc = accum()
>>>>> inc()
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>>   File "<stdin>", line 4, in inner
>> UnboundLocalError: local variable 'x' referenced before assignment
>>
>> But why does this happen? Let's think about this a little more closely: += is
>> not the same as =. A += can only happen if the left-hand term was already
>> defined. So, why does the compiler treat this as though there were an assignment
>> inside the function?
>
> Because x += y is equivalent to
>
> x = x.__iadd__(y)
>
> and therefore an assignment is going on here.  Therefore, it's only logical to
> treat it as such when determining scopes.

Off on a bit of a tangent here - this behaviour always bugged me:

 	--> x = ([],)
 	--> x[0] += ['a']
 	Traceback (most recent call last):
 	  File "<stdin>", line 1, in <module>
 	TypeError: 'tuple' object does not support item assignment
 	--> x
 	(['a'],)
 	-->

I understand by the definition why this happens as it does, but intuitively,
I'd expect an operation to either fail and raise, or succeed and not.

I see two possible ways to make this behave: we can look before we leap,
and raise the exception before calling __iadd__, if the assigment would fail;
or, we can change the definition to only perform the assignment if __iadd__
returns something other than self.

Both these are, to some extent, incompatible language changes.  Both change
how I think about the original proposal: with the first option, it softens
the argument about __iadd__ being called before the assignment, so strengthens
the case for the status quo; with the second option, the definition of
__iadd__ gets more complicated, making me less inclined to dive into this
definition to explain the locality of the assigned variable, preferring it
to be defined separately, and simply.

Back from the tangent, I think Carl's proposal would make Python more
difficult to understand rather than less, so -1 from me.

 	/Paul


From janssen at parc.com  Wed Jun  1 18:34:03 2011
From: janssen at parc.com (Bill Janssen)
Date: Wed, 1 Jun 2011 09:34:03 PDT
Subject: [Python-ideas] Adding 'bytes' as alias for 'latin_1' codec.
In-Reply-To: <BANLkTi=pNpXnqiXKbtGL2cvt6Kb+HBu4XQ@mail.gmail.com>
References: <BANLkTin1P5BVUisF-SfW=-TUeQbumDdBvA@mail.gmail.com>
	<irkc54$gih$1@dough.gmane.org> <irl5n7$l1q$1@dough.gmane.org>
	<BANLkTikQXPJ3Y0SBfG5UKhu1S-CVDo5sng@mail.gmail.com>
	<DF3948D2-C0AC-4B6E-A722-25004C48C0E2@masklinn.net>
	<irmk2f$ess$1@dough.gmane.org>
	<BANLkTikTnwrGA2N2oaxwNsqYDVeWO6CXVQ@mail.gmail.com>
	<BANLkTinsqOGZzU7DXBShde9N1kudoSFP8Q@mail.gmail.com>
	<4DE0481E.7010005@canterbury.ac.nz>
	<BANLkTimZRE15gPqVPZR27pQtWYgntycz9w@mail.gmail.com>
	<79306.1306858606@parc.com>
	<BANLkTi=pNpXnqiXKbtGL2cvt6Kb+HBu4XQ@mail.gmail.com>
Message-ID: <98069.1306946043@parc.com>

Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Wed, Jun 1, 2011 at 2:16 AM, Bill Janssen <janssen at parc.com> wrote:
> > I like the deprecations you suggest, but I'd prefer to see a more
> > general solution: the 'str' type extended so that it had two possible
> > representations for strings, the current format and an "encoded" format,
> > which would be kept as an array of bytes plus an encoding. ?It would
> > transcode only as necessary -- for example, the 're' module might
> > require the current Unicode encoding. ?An explicit method would be added
> > to allow the user to force transcoding.
> >
> > This would complicate life at the C level, to be sure. ?Though, perhaps
> > not so much, given the proper macrology.
> 
> See PEP 393 - it is basically this idea

Should have realized Martin would have thought of this :-).  I'm not
sure how I missed it back in January -- high drama at work distracted
me, I guess.

I might do it a bit differently, with just one pointer, say, "data", and
a field which carries the encoding (possibly as a pointer to the
appropriate codec).  "data" would point to a buffer of the correct type.
New strings would by default still be created as UCS-2 or UCS-4 Unicode,
just as per today.

I'd also allow any encoding which we have a codec for, so that if you
are reading from a file containing encoded text, you can carry the exact
bytes around unless you need to do something which isn't supported for
that encoding -- in which case things get Unicodified behind the scenes.
We'd smarten the various string methods over time so that most of them
would work so long as the operands matched.  str.index, for instance,
wouldn't require decoding unless the two strings were of different
encodings.  Yes, there'd be some "magic" going on, but it wouldn't be
worse than the automatic coercions Python does now -- that's just what a
HLL does for you.

> (although the encodings are
> fixed for the various sizes rather than allowing arbitrary encodings
> in the 8-bit internal format).

IMO, the thing that bit us on the fundament with the 2.x str/unicode
divide, and continues to bite us with the 3.x str/bytes divide is that
we don't carry the encoding as part of the 2.x 'str' value (or as part
of the 3.x 'bytes' value).  The key here is to store the encoding
internally in the string object, so that it's available to do automatic
coercion when necessary, rather than *requiring* all coercions to be
done manually by some program code.

Bill


From ethan at stoneleaf.us  Wed Jun  1 18:56:47 2011
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 01 Jun 2011 09:56:47 -0700
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <BANLkTi=tujW7maFT1jNPFkxK65+DfrGVCw@mail.gmail.com>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
	<BANLkTi=tujW7maFT1jNPFkxK65+DfrGVCw@mail.gmail.com>
Message-ID: <4DE66F4F.5000706@stoneleaf.us>

Nick Coghlan wrote:
> On Wed, Jun 1, 2011 at 2:52 PM, Carl M. Johnson
> <cmjohnson.mailinglist at gmail.com> wrote:
>> We all know that the following code won't work because of UnboundLocalError
>> and that to get around it, one needs to use nonlocal:
> 
> There's no fundamental reason this couldn't change, but actually
> changing it simply isn't worth the hassle, so the status quo wins the
> stalemate.
> 
> I elaborated further on this point when the topic came up last year:
> http://mail.python.org/pipermail/python-ideas/2010-June/007448.html

Maybe I get to learn something new about Python today.

Several times in that thread it was stated that

--> a += 1

is a shortcut for

--> a.__iadd__(1)

It seems to me that this is an implementation detail, and that the 
actual "longcut" is

--> a = a + 1

Likewise, the shortcut of

--> some_list[func_with_side_effects()] += some_value

is the same as

--> index = func_with_side_effects()
--> some_list[index] = some_list[index] + some_value

Is my understanding correct?

~Ethan~


From tjreedy at udel.edu  Wed Jun  1 19:18:50 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 01 Jun 2011 13:18:50 -0400
Subject: [Python-ideas] Adding 'bytes' as alias for 'latin_1' codec.
In-Reply-To: <98069.1306946043@parc.com>
References: <BANLkTin1P5BVUisF-SfW=-TUeQbumDdBvA@mail.gmail.com>	<irkc54$gih$1@dough.gmane.org>
	<irl5n7$l1q$1@dough.gmane.org>	<BANLkTikQXPJ3Y0SBfG5UKhu1S-CVDo5sng@mail.gmail.com>	<DF3948D2-C0AC-4B6E-A722-25004C48C0E2@masklinn.net>	<irmk2f$ess$1@dough.gmane.org>	<BANLkTikTnwrGA2N2oaxwNsqYDVeWO6CXVQ@mail.gmail.com>	<BANLkTinsqOGZzU7DXBShde9N1kudoSFP8Q@mail.gmail.com>	<4DE0481E.7010005@canterbury.ac.nz>	<BANLkTimZRE15gPqVPZR27pQtWYgntycz9w@mail.gmail.com>	<79306.1306858606@parc.com>	<BANLkTi=pNpXnqiXKbtGL2cvt6Kb+HBu4XQ@mail.gmail.com>
	<98069.1306946043@parc.com>
Message-ID: <is5s9r$nnh$1@dough.gmane.org>

On 6/1/2011 12:34 PM, Bill Janssen wrote:

> IMO, the thing that bit us on the fundament with the 2.x str/unicode
> divide, and continues to bite us with the 3.x str/bytes divide is that
> we don't carry the encoding as part of the 2.x 'str' value (or as part
> of the 3.x 'bytes' value).  The key here is to store the encoding
> internally in the string object, so that it's available to do automatic
> coercion when necessary, rather than *requiring* all coercions to be
> done manually by some program code.

Some time ago, I posted here a proposal to do just that -- add an 
encoding field to byte strings (or, I believe, add a new class). It was 
horribly shot down. Something like 'conceptually wrong, some bytes have 
0 or multiple encodings, can just use an attribute or tuple, don't need it'.

-- 
Terry Jan Reedy



From bruce at leapyear.org  Wed Jun  1 19:32:49 2011
From: bruce at leapyear.org (Bruce Leban)
Date: Wed, 1 Jun 2011 10:32:49 -0700
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <4DE66F4F.5000706@stoneleaf.us>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
	<BANLkTi=tujW7maFT1jNPFkxK65+DfrGVCw@mail.gmail.com>
	<4DE66F4F.5000706@stoneleaf.us>
Message-ID: <BANLkTikWeWRdCdDpHKvsR4NsCga1bkM8hw@mail.gmail.com>

On Wed, Jun 1, 2011 at 9:56 AM, Ethan Furman <ethan at stoneleaf.us> wrote:

>
> Several times in that thread it was stated that
>
> --> a += 1
>
> is a shortcut for
>
> --> a.__iadd__(1)
>
> It seems to me that this is an implementation detail, and that the actual
> "longcut" is
>
> --> a = a + 1
>
> ...
>
> ~Ethan~
>

a += 1 is not a shortcut for a.__iadd__(1). It's a shortcut for a =
a.__iadd(1). Otherwise this wouldn't work:

>>> x = (1,)
>>> x += (2,)
>>> x
(1, 2)

Note the difference between these two is one opcode:

>>> def f(x,y):
x += y
 >>> dis.dis(f)
  2           0 LOAD_FAST                0 (x)
              3 LOAD_FAST                1 (y)
              6 *INPLACE_ADD*
              7 STORE_FAST               0 (x)
             10 LOAD_CONST               0 (None)
             13 RETURN_VALUE
>>> def g(x,y):
 x = x + y

>>> dis.dis(g)
  2           0 LOAD_FAST                0 (x)
              3 LOAD_FAST                1 (y)
              6 *BINARY_ADD*
              7 STORE_FAST               0 (x)
             10 LOAD_CONST               0 (None)
             13 RETURN_VALUE

--- Bruce
Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
Latest tweet: SO disappointed end of the world didn't happen AGAIN!
#y2k<http://twitter.com/#!/search?q=%23y2k>
 #rapture <http://twitter.com/#!/search?q=%23rapture> Now waiting for 2038!
#unixrapture <http://twitter.com/#!/search?q=%23unixrapture>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110601/878e89ae/attachment.html>

From ethan at stoneleaf.us  Wed Jun  1 19:51:30 2011
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 01 Jun 2011 10:51:30 -0700
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <BANLkTikWeWRdCdDpHKvsR4NsCga1bkM8hw@mail.gmail.com>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
	<BANLkTi=tujW7maFT1jNPFkxK65+DfrGVCw@mail.gmail.com>
	<4DE66F4F.5000706@stoneleaf.us>
	<BANLkTikWeWRdCdDpHKvsR4NsCga1bkM8hw@mail.gmail.com>
Message-ID: <4DE67C22.4000502@stoneleaf.us>

Bruce Leban wrote:
> 
> 
> On Wed, Jun 1, 2011 at 9:56 AM, Ethan Furman wrote:
> 
> 
>     Several times in that thread it was stated that
> 
>     --> a += 1
> 
>     is a shortcut for
> 
>     --> a.__iadd__(1)
> 
>     It seems to me that this is an implementation detail, and that the
>     actual "longcut" is
> 
>     --> a = a + 1
> 
>     ...
> 
>     ~Ethan~
> 
>  
> a += 1 is not a shortcut for a.__iadd__(1). It's a shortcut for a = 
> a.__iadd(1). Otherwise this wouldn't work:

Right -- typo on my part, sorry.


> Note the difference between these two is one opcode:
> 
>  >>> def f(x,y):
> x += y
>  >>> dis.dis(f)
>   2           0 LOAD_FAST                0 (x)
>               3 LOAD_FAST                1 (y)
>               6 *INPLACE_ADD*         
>               7 STORE_FAST               0 (x)
>              10 LOAD_CONST               0 (None)
>              13 RETURN_VALUE        
>  >>> def g(x,y):
> x = x + y
> 
>  >>> dis.dis(g)
>   2           0 LOAD_FAST                0 (x)
>               3 LOAD_FAST                1 (y)
>               6 *BINARY_ADD*          
>               7 STORE_FAST               0 (x)
>              10 LOAD_CONST               0 (None)
>              13 RETURN_VALUE        

Note also that INPLACE_ADD will call the the BINARY_ADD method if no 
__iadd__ method exists.

~Ethan~


From tjreedy at udel.edu  Wed Jun  1 19:41:26 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 01 Jun 2011 13:41:26 -0400
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
Message-ID: <is5tk7$ir$1@dough.gmane.org>

On 6/1/2011 12:52 AM, Carl M. Johnson wrote:

> So, my proposal is that += by itself should not cause x to be considered
> a local variable.

Right now, 'augmented assigment' is uniformly what it says: an 
assignment with augmented behavior. 'expr1 op= expr2' is *defined* as 
being the same as 'expr1 = expr1 op expr2' except that expr1 is 
evauluated just once*, and if expr1 evaluates to a mutable, the op can 
be done in place. Some consider the second exception to be a confusing 
complication and a mistake. Your proposal would require a rewrite of the 
definition and would add additional complication. Some would then want 
another exception for when expr1 evaluates to a mutable within an 
immutable (see Paul Svensson's post).

While I do understand your point, I also value uniformity.
-1

* It is actually more complicate than than. Expr1 is partially evaluated 
just once to an internal reference rather than to an object. That 
reference is then used once to fetch the existing object and once again 
to rebind to the new or mutated object. Still, it is one behavior for 
all occurrences.

-- 
Terry Jan Reedy



From ethan at stoneleaf.us  Wed Jun  1 19:58:12 2011
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 01 Jun 2011 10:58:12 -0700
Subject: [Python-ideas] Adding 'bytes' as alias for 'latin_1' codec.
In-Reply-To: <is5s9r$nnh$1@dough.gmane.org>
References: <BANLkTin1P5BVUisF-SfW=-TUeQbumDdBvA@mail.gmail.com>	<irkc54$gih$1@dough.gmane.org>	<irl5n7$l1q$1@dough.gmane.org>	<BANLkTikQXPJ3Y0SBfG5UKhu1S-CVDo5sng@mail.gmail.com>	<DF3948D2-C0AC-4B6E-A722-25004C48C0E2@masklinn.net>	<irmk2f$ess$1@dough.gmane.org>	<BANLkTikTnwrGA2N2oaxwNsqYDVeWO6CXVQ@mail.gmail.com>	<BANLkTinsqOGZzU7DXBShde9N1kudoSFP8Q@mail.gmail.com>	<4DE0481E.7010005@canterbury.ac.nz>	<BANLkTimZRE15gPqVPZR27pQtWYgntycz9w@mail.gmail.com>	<79306.1306858606@parc.com>	<BANLkTi=pNpXnqiXKbtGL2cvt6Kb+HBu4XQ@mail.gmail.com>	<98069.1306946043@parc.com>
	<is5s9r$nnh$1@dough.gmane.org>
Message-ID: <4DE67DB4.3070304@stoneleaf.us>

Terry Reedy wrote:
> On 6/1/2011 12:34 PM, Bill Janssen wrote:
> 
>> IMO, the thing that bit us on the fundament with the 2.x str/unicode
>> divide, and continues to bite us with the 3.x str/bytes divide is that
>> we don't carry the encoding as part of the 2.x 'str' value (or as part
>> of the 3.x 'bytes' value).  The key here is to store the encoding
>> internally in the string object, so that it's available to do automatic
>> coercion when necessary, rather than *requiring* all coercions to be
>> done manually by some program code.
> 
> Some time ago, I posted here a proposal to do just that -- add an 
> encoding field to byte strings (or, I believe, add a new class). It was 
> horribly shot down. Something like 'conceptually wrong, some bytes have 
> 0 or multiple encodings, can just use an attribute or tuple, don't need 
> it'.
> 
A byte stream with multiple encodings?  Now *that* seems wrong!

It could also be handled by having the encoding field set to some 
special value indicating Unknown.

~Ethan~


From tjreedy at udel.edu  Thu Jun  2 00:15:47 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 01 Jun 2011 18:15:47 -0400
Subject: [Python-ideas] Adding 'bytes' as alias for 'latin_1' codec.
In-Reply-To: <4DE67DB4.3070304@stoneleaf.us>
References: <BANLkTin1P5BVUisF-SfW=-TUeQbumDdBvA@mail.gmail.com>	<irkc54$gih$1@dough.gmane.org>	<irl5n7$l1q$1@dough.gmane.org>	<BANLkTikQXPJ3Y0SBfG5UKhu1S-CVDo5sng@mail.gmail.com>	<DF3948D2-C0AC-4B6E-A722-25004C48C0E2@masklinn.net>	<irmk2f$ess$1@dough.gmane.org>	<BANLkTikTnwrGA2N2oaxwNsqYDVeWO6CXVQ@mail.gmail.com>	<BANLkTinsqOGZzU7DXBShde9N1kudoSFP8Q@mail.gmail.com>	<4DE0481E.7010005@canterbury.ac.nz>	<BANLkTimZRE15gPqVPZR27pQtWYgntycz9w@mail.gmail.com>	<79306.1306858606@parc.com>	<BANLkTi=pNpXnqiXKbtGL2cvt6Kb+HBu4XQ@mail.gmail.com>	<98069.1306946043@parc.com>	<is5s9r$nnh$1@dough.gmane.org>
	<4DE67DB4.3070304@stoneleaf.us>
Message-ID: <is6dmk$53d$1@dough.gmane.org>

On 6/1/2011 1:58 PM, Ethan Furman wrote:

> A byte stream with multiple encodings? Now *that* seems wrong!

No, it is standard in many protocols. Ascii coded characters and numbers 
are mixed with binary coded numbers and binary blobs with their own 
codings. Bytes are not text, so don't think in terms of just text encodings.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Thu Jun  2 00:42:03 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 01 Jun 2011 18:42:03 -0400
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <BANLkTikWeWRdCdDpHKvsR4NsCga1bkM8hw@mail.gmail.com>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>	<BANLkTi=tujW7maFT1jNPFkxK65+DfrGVCw@mail.gmail.com>	<4DE66F4F.5000706@stoneleaf.us>
	<BANLkTikWeWRdCdDpHKvsR4NsCga1bkM8hw@mail.gmail.com>
Message-ID: <is6f7t$dc2$1@dough.gmane.org>

On 6/1/2011 1:32 PM, Bruce Leban wrote:

>  >>> def f(x,y):
> x += y
>  >>> dis.dis(f)
>    2           0 LOAD_FAST                0 (x)
>                3 LOAD_FAST                1 (y)
>                6 *INPLACE_ADD*
>                7 STORE_FAST               0 (x)
>               10 LOAD_CONST               0 (None)
>               13 RETURN_VALUE
>  >>> def g(x,y):
> x = x + y
>
>  >>> dis.dis(g)
>    2           0 LOAD_FAST                0 (x)
>                3 LOAD_FAST                1 (y)
>                6 *BINARY_ADD*
>                7 STORE_FAST               0 (x)
>               10 LOAD_CONST               0 (None)
>               13 RETURN_VALUE

(In 3.2, one no longer needs to wrap code in a function to dis it.
see below.)

To see the 'calculate the source/target just once instead of twice' 
part, you need a source/target that actually requires calculation.

 >>> from dis import dis
 >>> dis('x[i] = x[i] + 1')
   1           0 LOAD_NAME                0 (x)
               3 LOAD_NAME                1 (i)
               6 BINARY_SUBSCR
               7 LOAD_CONST               0 (1)
              10 BINARY_ADD
              11 LOAD_NAME                0 (x)
              14 LOAD_NAME                1 (i)
              17 STORE_SUBSCR
              18 LOAD_CONST               1 (None)
              21 RETURN_VALUE

 >>> dis('x[i] += 1')
   1           0 LOAD_NAME                0 (x)
               3 LOAD_NAME                1 (i)
               6 DUP_TOP_TWO
               7 BINARY_SUBSCR
               8 LOAD_CONST               0 (1)
              11 INPLACE_ADD
              12 ROT_THREE
              13 STORE_SUBSCR
              14 LOAD_CONST               1 (None)
              17 RETURN_VALUE

Even this does not show much difference as the dup and rotate substitute 
for two loads but do not actually save any calculation. However,

 >>> dis('a.b[c+d] = a.b[c+d] + 1')
   1           0 LOAD_NAME                0 (a)
               3 LOAD_ATTR                1 (b)
               6 LOAD_NAME                2 (c)
               9 LOAD_NAME                3 (d)
              12 BINARY_ADD
              13 BINARY_SUBSCR
              14 LOAD_CONST               0 (1)
              17 BINARY_ADD
              18 LOAD_NAME                0 (a)
              21 LOAD_ATTR                1 (b)
              24 LOAD_NAME                2 (c)
              27 LOAD_NAME                3 (d)
              30 BINARY_ADD
              31 STORE_SUBSCR
              32 LOAD_CONST               1 (None)
              35 RETURN_VALUE

 >>> dis('a.b[c+d] += 1')
   1           0 LOAD_NAME                0 (a)
               3 LOAD_ATTR                1 (b)
               6 LOAD_NAME                2 (c)
               9 LOAD_NAME                3 (d)
              12 BINARY_ADD
              13 DUP_TOP_TWO
              14 BINARY_SUBSCR
              15 LOAD_CONST               0 (1)
              18 INPLACE_ADD
              19 ROT_THREE
              20 STORE_SUBSCR
              21 LOAD_CONST               1 (None)
              24 RETURN_VALUE

The latter has the same dup-rotate in place of a bit more calculation. 
The same would be true of, for instance, f(a).b.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Thu Jun  2 00:56:52 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 01 Jun 2011 18:56:52 -0400
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <is5tk7$ir$1@dough.gmane.org>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
	<is5tk7$ir$1@dough.gmane.org>
Message-ID: <is6g3l$hit$1@dough.gmane.org>

On 6/1/2011 1:41 PM, Terry Reedy wrote:
> On 6/1/2011 12:52 AM, Carl M. Johnson wrote:
>
>> So, my proposal is that += by itself should not cause x to be considered
>> a local variable.

> While I do understand your point, I also value uniformity.
> -1

There is another problem I had not thought of before. Right now, Python 
has (always had) a simple rule: code in a function CANNOT rebind names 
in outer scopes unless the function has a global or nonlocal 
declaration. This simple, uniform rule benefits not only the interpreter 
but human readers. It should not be broken.

def f():
   'doc for f'
   def g():
     'docstring of g'
     <body of g>
   <body of f>

If g is the only nested function and the body of g does not have a 
nonlocal declaration (which OUGHT to be at the top if present), then a 
reader or maintainer of f knows (without reading g in detail) that 
nothing other that <body of f> can rebind f's locals.

-- 
Terry Jan Reedy



From steve at pearwood.info  Thu Jun  2 01:21:12 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 02 Jun 2011 09:21:12 +1000
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <BANLkTik6Zy_T0n_p-WeoL5RZYKvFYtyfsA@mail.gmail.com>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>	<is4jqs$or8$1@dough.gmane.org>	<BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>	<is4ocf$fv8$1@dough.gmane.org>
	<BANLkTik6Zy_T0n_p-WeoL5RZYKvFYtyfsA@mail.gmail.com>
Message-ID: <4DE6C968.3000301@pearwood.info>

Carl M. Johnson wrote:

> Agreed, sure, we have to treat the LHS of = as
> a local. But += is fundamentally different. 


No it's not. It is fundamentally the same. Augmented assignment in 
Python *is* assignment, equivalent to x = x.__iadd__(other). That alone 
should be enough to kill this proposal stone dead.

+= is not, except by accident, an in-place addition operator. It is 
always a re-binding. (Mutable objects are free to mutate in place, if 
they choose, but the re-binding still takes place.)



> You cannot have a += statement
> unless somewhere out there there is a matching = statement. It cannot exist
> independently. It never works on its own. 


Neither does *any* attempt to access an unbound local. Python doesn't, 
and shouldn't, try to guess what you actually intended so as to make it 
work. If you want x to refer to a nonlocal, or a global, declare it as such.

print x; x = 1 will fail unless there is an earlier x = something.

x = x+1 will fail unless there is an earlier x = something.

x += 1 will fail unless there is an earlier x = something.


Why single out x += 1 for changed semantics to the rule that any 
assignment makes x a local? What if you don't have a non-local x, should 
Python guess that you wanted a global?

Currently, the rule is simple: any assignment tells the compiler to 
treat x as local. If you want nonlocal or global, you have to declare it 
as such. Nice and simple. What actual real-world problem are you trying 
to solve that you want to change this behaviour?


-1 on this change.



-- 
Steven


From ethan at stoneleaf.us  Thu Jun  2 01:47:19 2011
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 01 Jun 2011 16:47:19 -0700
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <4DE6C968.3000301@pearwood.info>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>	<is4jqs$or8$1@dough.gmane.org>	<BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>	<is4ocf$fv8$1@dough.gmane.org>	<BANLkTik6Zy_T0n_p-WeoL5RZYKvFYtyfsA@mail.gmail.com>
	<4DE6C968.3000301@pearwood.info>
Message-ID: <4DE6CF87.3080604@stoneleaf.us>

Steven D'Aprano wrote:
> Carl M. Johnson wrote:
> 
>> Agreed, sure, we have to treat the LHS of = as
>> a local. But += is fundamentally different. 
> 
> 
> No it's not. It is fundamentally the same. Augmented assignment in 
> Python *is* assignment, equivalent to x = x.__iadd__(other).

Or x = x.__add__(other) if no __iadd__ exists for the object.

~Ethan~


From g.brandl at gmx.net  Thu Jun  2 07:17:19 2011
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 02 Jun 2011 07:17:19 +0200
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <BANLkTik6Zy_T0n_p-WeoL5RZYKvFYtyfsA@mail.gmail.com>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
	<is4jqs$or8$1@dough.gmane.org>
	<BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>
	<is4ocf$fv8$1@dough.gmane.org>
	<BANLkTik6Zy_T0n_p-WeoL5RZYKvFYtyfsA@mail.gmail.com>
Message-ID: <is76cn$kmp$1@dough.gmane.org>

On 01.06.2011 10:26, Carl M. Johnson wrote:
> 
> 
> On Tue, May 31, 2011 at 9:05 PM, Georg Brandl <g.brandl at gmx.net
> <mailto:g.brandl at gmx.net>> wrote:
> 
>     Sure, this can only work if the local is assigned somewhere before the augmented
>     assign statement.  But this is just like accessing a local before its
>     assignment: in the case of
> 
>     x = 1
>     def f():
>        print x
>        x = 2
> 
>     we also don't treat the first "x" reference as a nonlocal.
> 
>  
> I don't think that's a counterexample to the point I'm trying to make. We all
> agree that if there's an x= somewhere in the function body, then we have to
> treat the variable as a local. The only possible way around that would be to
> solve the halting problem in order to figure out if a particular line of code
> will be reached or not. Agreed, sure, we have to treat the LHS of = as a local.
> But += is fundamentally different.

You keep saying that, but I just can't see how += is fundamentally different
from =, given its definition as x = x.__iadd__(y).

This is a situation that comes up from time to time, where it seems logical to
make a change that satisfies "DWIM" feelings, but makes the languge more
inconsistent by introducing special cases.  This doesn't feel right to me (and
the Zen agrees ;)

Georg



From cmjohnson.mailinglist at gmail.com  Thu Jun  2 07:17:58 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Wed, 1 Jun 2011 19:17:58 -1000
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <4DE6C968.3000301@pearwood.info>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
	<is4jqs$or8$1@dough.gmane.org>
	<BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>
	<is4ocf$fv8$1@dough.gmane.org>
	<BANLkTik6Zy_T0n_p-WeoL5RZYKvFYtyfsA@mail.gmail.com>
	<4DE6C968.3000301@pearwood.info>
Message-ID: <BANLkTik1niTVHVG10N0boZartFDx2dNr6A@mail.gmail.com>

On Wed, Jun 1, 2011 at 1:21 PM, Steven D'Aprano <steve at pearwood.info> wrote:

> Currently, the rule is simple: any assignment tells the compiler to treat x
> as local. If you want nonlocal or global, you have to declare it as such.
> Nice and simple. What actual real-world problem are you trying to solve that
> you want to change this behaviour?


The best counter-arguments I've heard so far are Nick's (it would be a pain
to go into the guts and change this, and you also need to think about PyPy,
Jython, IronPy, etc., etc.) and this one.

In terms of "real world problems" this solves, it makes the solution to the
Paul Graham language challenge problem (build a function that returns an
accumulator) one line shorter. Which is a bit silly, but so far as I can
tell, nonlocal was created just to say we have an answer to the Paul Graham
question. ;-)

I think the benefit of saving that one line is probably outweighed by the
brittleness that this would create (ie. changing x +=  1 to x = x + 1 could
break code), so I withdraw the proposal, at least for now.

One additional problem that I ran into is this:

>>> def f():
...  nonlocal count
...  return count
...
SyntaxError: no binding for nonlocal 'count' found

Nonlocal fails at the compilation stage if the variable isn't found. On the
other hand, attribute lookup is delayed until runtime, so if by accident you
did

def f():
    count = 0
    def g():
        cont += 1 #oops typo.
        return cont
    return g

it's not clear when the function should fail: compile time or runtime.

-- Carl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110601/56aeb92e/attachment.html>

From ncoghlan at gmail.com  Thu Jun  2 07:37:47 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 2 Jun 2011 15:37:47 +1000
Subject: [Python-ideas] Adding 'bytes' as alias for 'latin_1' codec.
In-Reply-To: <4DE67DB4.3070304@stoneleaf.us>
References: <BANLkTin1P5BVUisF-SfW=-TUeQbumDdBvA@mail.gmail.com>
	<irkc54$gih$1@dough.gmane.org> <irl5n7$l1q$1@dough.gmane.org>
	<BANLkTikQXPJ3Y0SBfG5UKhu1S-CVDo5sng@mail.gmail.com>
	<DF3948D2-C0AC-4B6E-A722-25004C48C0E2@masklinn.net>
	<irmk2f$ess$1@dough.gmane.org>
	<BANLkTikTnwrGA2N2oaxwNsqYDVeWO6CXVQ@mail.gmail.com>
	<BANLkTinsqOGZzU7DXBShde9N1kudoSFP8Q@mail.gmail.com>
	<4DE0481E.7010005@canterbury.ac.nz>
	<BANLkTimZRE15gPqVPZR27pQtWYgntycz9w@mail.gmail.com>
	<79306.1306858606@parc.com>
	<BANLkTi=pNpXnqiXKbtGL2cvt6Kb+HBu4XQ@mail.gmail.com>
	<98069.1306946043@parc.com> <is5s9r$nnh$1@dough.gmane.org>
	<4DE67DB4.3070304@stoneleaf.us>
Message-ID: <BANLkTikLfwLVD2VgUtsL+dF2prii+hKy9Q@mail.gmail.com>

On Thu, Jun 2, 2011 at 3:58 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> A byte stream with multiple encodings? ?Now *that* seems wrong!

Unicode encodings are just one serialisation format specific to text
data. bytes objects may contain *any* serialisation format (e.g. zip
archives, Python pickles, Python marshal files, packed binary data,
innumerable wire protocols both standard and proprietary).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Thu Jun  2 07:49:58 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 2 Jun 2011 15:49:58 +1000
Subject: [Python-ideas] Why does += trigger UnboundLocalError?
In-Reply-To: <BANLkTik1niTVHVG10N0boZartFDx2dNr6A@mail.gmail.com>
References: <BANLkTimfZkoedSmNAW0BhunF2HkMP0S__w@mail.gmail.com>
	<is4jqs$or8$1@dough.gmane.org>
	<BANLkTin+1n3H6YgVd27H3u0wJeBvy6Ldcg@mail.gmail.com>
	<is4ocf$fv8$1@dough.gmane.org>
	<BANLkTik6Zy_T0n_p-WeoL5RZYKvFYtyfsA@mail.gmail.com>
	<4DE6C968.3000301@pearwood.info>
	<BANLkTik1niTVHVG10N0boZartFDx2dNr6A@mail.gmail.com>
Message-ID: <BANLkTinDac9EMquXcMZmAo8X5kgUfFJt7Q@mail.gmail.com>

On Thu, Jun 2, 2011 at 3:17 PM, Carl M. Johnson
<cmjohnson.mailinglist at gmail.com> wrote:
>
>
> On Wed, Jun 1, 2011 at 1:21 PM, Steven D'Aprano <steve at pearwood.info> wrote:
>>
>> Currently, the rule is simple: any assignment tells the compiler to treat
>> x as local. If you want nonlocal or global, you have to declare it as such.
>> Nice and simple. What actual real-world problem are you trying to solve that
>> you want to change this behaviour?
>
> The best counter-arguments I've heard so far are Nick's (it would be a pain
> to go into the guts and change this, and you also need to think about PyPy,
> Jython, IronPy, etc., etc.) and this one.
> In terms of "real world problems" this solves, it makes the solution to the
> Paul Graham language challenge problem (build a function that returns an
> accumulator) one line shorter. Which is a bit silly, but so far as I can
> tell, nonlocal was created just to say we have an answer to the Paul Graham
> question. ;-)

Nah, nonlocal was added because the introduction of decorators
increased the use of closures, and boxing and unboxing variables
manually is a PITA.

Note that the "translation" of 'x += y' to 'x = x + y' is and always
has been a gross oversimplification (albeit a useful one).

Reality is complicated by possible provision of __iadd__ by the
assignment target, as well as the need to pair up
__getitem__/__setitem__ and __getattr__/__setattr__ appropriately when
the target is a subscript operation or attribute access.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From tjreedy at udel.edu  Thu Jun  2 08:30:28 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 02 Jun 2011 02:30:28 -0400
Subject: [Python-ideas] Adding 'bytes' as alias for 'latin_1' codec.
In-Reply-To: <BANLkTikLfwLVD2VgUtsL+dF2prii+hKy9Q@mail.gmail.com>
References: <BANLkTin1P5BVUisF-SfW=-TUeQbumDdBvA@mail.gmail.com>	<irkc54$gih$1@dough.gmane.org>
	<irl5n7$l1q$1@dough.gmane.org>	<BANLkTikQXPJ3Y0SBfG5UKhu1S-CVDo5sng@mail.gmail.com>	<DF3948D2-C0AC-4B6E-A722-25004C48C0E2@masklinn.net>	<irmk2f$ess$1@dough.gmane.org>	<BANLkTikTnwrGA2N2oaxwNsqYDVeWO6CXVQ@mail.gmail.com>	<BANLkTinsqOGZzU7DXBShde9N1kudoSFP8Q@mail.gmail.com>	<4DE0481E.7010005@canterbury.ac.nz>	<BANLkTimZRE15gPqVPZR27pQtWYgntycz9w@mail.gmail.com>	<79306.1306858606@parc.com>	<BANLkTi=pNpXnqiXKbtGL2cvt6Kb+HBu4XQ@mail.gmail.com>	<98069.1306946043@parc.com>
	<is5s9r$nnh$1@dough.gmane.org>	<4DE67DB4.3070304@stoneleaf.us>
	<BANLkTikLfwLVD2VgUtsL+dF2prii+hKy9Q@mail.gmail.com>
Message-ID: <is7am5$8f8$1@dough.gmane.org>

On 6/2/2011 1:37 AM, Nick Coghlan wrote:
> On Thu, Jun 2, 2011 at 3:58 AM, Ethan Furman<ethan at stoneleaf.us>  wrote:
>> A byte stream with multiple encodings?  Now *that* seems wrong!
>
> Unicode encodings are just one serialisation format specific to text
> data. bytes objects may contain *any* serialisation format (e.g. zip
> archives, Python pickles, Python marshal files, packed binary data,
> innumerable wire protocols both standard and proprietary).

One result of this thread is that I see much better the value of 
separating the ancient human level concepts of character and text from 
the (3) decades old computer concept of byte. Numbers, lists, and dicts 
are other old human concepts. As Nick implies above, bytes (or bits 
within them) are used to encode all data for computer processing.

The confusion of character with byte in the original design of Python 
both privileged and burdened text processing.

-- 
Terry Jan Reedy



From guido at python.org  Thu Jun  2 19:58:55 2011
From: guido at python.org (Guido van Rossum)
Date: Thu, 2 Jun 2011 10:58:55 -0700
Subject: [Python-ideas] Adding 'bytes' as alias for 'latin_1' codec.
In-Reply-To: <is7am5$8f8$1@dough.gmane.org>
References: <BANLkTin1P5BVUisF-SfW=-TUeQbumDdBvA@mail.gmail.com>
	<irkc54$gih$1@dough.gmane.org> <irl5n7$l1q$1@dough.gmane.org>
	<BANLkTikQXPJ3Y0SBfG5UKhu1S-CVDo5sng@mail.gmail.com>
	<DF3948D2-C0AC-4B6E-A722-25004C48C0E2@masklinn.net>
	<irmk2f$ess$1@dough.gmane.org>
	<BANLkTikTnwrGA2N2oaxwNsqYDVeWO6CXVQ@mail.gmail.com>
	<BANLkTinsqOGZzU7DXBShde9N1kudoSFP8Q@mail.gmail.com>
	<4DE0481E.7010005@canterbury.ac.nz>
	<BANLkTimZRE15gPqVPZR27pQtWYgntycz9w@mail.gmail.com>
	<79306.1306858606@parc.com>
	<BANLkTi=pNpXnqiXKbtGL2cvt6Kb+HBu4XQ@mail.gmail.com>
	<98069.1306946043@parc.com>
	<is5s9r$nnh$1@dough.gmane.org> <4DE67DB4.3070304@stoneleaf.us>
	<BANLkTikLfwLVD2VgUtsL+dF2prii+hKy9Q@mail.gmail.com>
	<is7am5$8f8$1@dough.gmane.org>
Message-ID: <BANLkTik75wOp+XXj=xs_L+au379JFJz7_A@mail.gmail.com>

On Wed, Jun 1, 2011 at 11:30 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> The confusion of character with byte in the original design of Python both
> privileged and burdened text processing.

Right. And it wasn't only Python: most languages created around or
before that time had the same issues (perhaps starting with C's use of
"char" meaning byte). Even most IP protocols developed in the 1990s
confuse character set and encoding (witness HTTP's "Content-type:
text/plain; charset=utf-8").

I'm glad in Python 3 we undertook to improve the distinction.

-- 
--Guido van Rossum (python.org/~guido)


From guido at python.org  Thu Jun  2 20:11:09 2011
From: guido at python.org (Guido van Rossum)
Date: Thu, 2 Jun 2011 11:11:09 -0700
Subject: [Python-ideas] Minor tweak to PEP 8?
In-Reply-To: <BANLkTinrUQKO0trKgWyUZLWyhBxO10Vu=w@mail.gmail.com>
References: <20110510104754.4689cc5e@bhuda.mired.org>
	<BANLkTi=txKo3DPWOBj5ZD09V5jBPcT6Srw@mail.gmail.com>
	<87y62ejl2j.fsf@benfinney.id.au>
	<BANLkTikV5PppTUkOVn-Sxd6HWdG3=H0Z2g@mail.gmail.com>
	<BANLkTi=p7prG6R+dVhzeBANWbK_jL2bPOg@mail.gmail.com>
	<BANLkTi=0QgcC5NC0ECLKjJcG_Nmt-aSA-g@mail.gmail.com>
	<BANLkTinrUQKO0trKgWyUZLWyhBxO10Vu=w@mail.gmail.com>
Message-ID: <BANLkTin+Ubc3zVY51c=Q0NckBj4kvMuS+A@mail.gmail.com>

FYI, I've submitted this change to PEP 8, with the help of a draft
patch by Steven Klass.

--Guido

On Wed, May 11, 2011 at 7:23 AM, Guido van Rossum <guido at python.org> wrote:
> At Google we use the following rule (from
> http://google-styleguide.googlecode.com/svn/trunk/pyguide.html#Indentation):
>
> Yes:? # Aligned with opening delimiter
> ? ? ? foo = long_function_name(var_one, var_two,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?var_three, var_four)
>
> ? ? ? # 4-space hanging indent; nothing on first line
> ? ? ? foo = long_function_name(
> ? ? ? ? ? var_one, var_two, var_three,
> ? ? ? ? ? var_four)
>
> No: ? # Stuff on first line forbidden
> ? ? ? foo = long_function_name(var_one, var_two,
> ? ? ? ? ? var_three, var_four)
>
> ? ? ? # 2-space hanging indent forbidden
> ? ? ? foo = long_function_name(
> ? ? ? ? var_one, var_two, var_three,
> ? ? ? ? var_four)
>
> I propose we somehow incorporate these two allowed alternatives into PEP 8.
> They both serve a purpose.
>
> --
> --Guido van Rossum (python.org/~guido)
>
>



-- 
--Guido van Rossum (python.org/~guido)


From barry at python.org  Thu Jun  2 21:00:31 2011
From: barry at python.org (Barry Warsaw)
Date: Thu, 2 Jun 2011 15:00:31 -0400
Subject: [Python-ideas] Minor tweak to PEP 8?
References: <20110510104754.4689cc5e@bhuda.mired.org>
	<BANLkTi=txKo3DPWOBj5ZD09V5jBPcT6Srw@mail.gmail.com>
	<87y62ejl2j.fsf@benfinney.id.au>
	<BANLkTikV5PppTUkOVn-Sxd6HWdG3=H0Z2g@mail.gmail.com>
	<BANLkTi=p7prG6R+dVhzeBANWbK_jL2bPOg@mail.gmail.com>
	<BANLkTi=0QgcC5NC0ECLKjJcG_Nmt-aSA-g@mail.gmail.com>
	<BANLkTinrUQKO0trKgWyUZLWyhBxO10Vu=w@mail.gmail.com>
	<BANLkTin+Ubc3zVY51c=Q0NckBj4kvMuS+A@mail.gmail.com>
Message-ID: <20110602150031.2b98524f@neurotica.wooz.org>

On Jun 02, 2011, at 11:11 AM, Guido van Rossum wrote:

>FYI, I've submitted this change to PEP 8, with the help of a draft
>patch by Steven Klass.

Thanks Guido.  This is probably the right mailing list to follow up to (ignore
my python-dev followup to the -checkins message).

I agree with the change, except for the recommendation to double-indent.  Yes,
double-indent does look better with Google's 2-space indentation level rule,
but it looks excessive (to my eyes anyway) with a 4-space rule.  One
indentation level looks fine.  I posted some examples to python-dev.

Is it worth softening the PEP 8 recommendation on double-indents?

Cheers,
-Barry

>On Wed, May 11, 2011 at 7:23 AM, Guido van Rossum <guido-+ZN9ApsXKcEdnm+yROfE0A at public.gmane.org> wrote:
>> At Google we use the following rule (from
>> http://google-styleguide.googlecode.com/svn/trunk/pyguide.html#Indentation):
>>
>> Yes:? # Aligned with opening delimiter
>> ? ? ? foo = long_function_name(var_one, var_two,
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?var_three, var_four)
>>
>> ? ? ? # 4-space hanging indent; nothing on first line
>> ? ? ? foo = long_function_name(
>> ? ? ? ? ? var_one, var_two, var_three,
>> ? ? ? ? ? var_four)
>>
>> No: ? # Stuff on first line forbidden
>> ? ? ? foo = long_function_name(var_one, var_two,
>> ? ? ? ? ? var_three, var_four)
>>
>> ? ? ? # 2-space hanging indent forbidden
>> ? ? ? foo = long_function_name(
>> ? ? ? ? var_one, var_two, var_three,
>> ? ? ? ? var_four)
>>
>> I propose we somehow incorporate these two allowed alternatives into PEP 8.
>> They both serve a purpose.
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>>
>
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110602/02ea76b3/attachment.pgp>

From tjreedy at udel.edu  Thu Jun  2 22:14:30 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 02 Jun 2011 16:14:30 -0400
Subject: [Python-ideas] Adding 'bytes' as alias for 'latin_1' codec.
In-Reply-To: <BANLkTik75wOp+XXj=xs_L+au379JFJz7_A@mail.gmail.com>
References: <BANLkTin1P5BVUisF-SfW=-TUeQbumDdBvA@mail.gmail.com>	<irkc54$gih$1@dough.gmane.org>
	<irl5n7$l1q$1@dough.gmane.org>	<BANLkTikQXPJ3Y0SBfG5UKhu1S-CVDo5sng@mail.gmail.com>	<DF3948D2-C0AC-4B6E-A722-25004C48C0E2@masklinn.net>	<irmk2f$ess$1@dough.gmane.org>	<BANLkTikTnwrGA2N2oaxwNsqYDVeWO6CXVQ@mail.gmail.com>	<BANLkTinsqOGZzU7DXBShde9N1kudoSFP8Q@mail.gmail.com>	<4DE0481E.7010005@canterbury.ac.nz>	<BANLkTimZRE15gPqVPZR27pQtWYgntycz9w@mail.gmail.com>	<79306.1306858606@parc.com>	<BANLkTi=pNpXnqiXKbtGL2cvt6Kb+HBu4XQ@mail.gmail.com>	<98069.1306946043@parc.com>	<is5s9r$nnh$1@dough.gmane.org>
	<4DE67DB4.3070304@stoneleaf.us>	<BANLkTikLfwLVD2VgUtsL+dF2prii+hKy9Q@mail.gmail.com>	<is7am5$8f8$1@dough.gmane.org>
	<BANLkTik75wOp+XXj=xs_L+au379JFJz7_A@mail.gmail.com>
Message-ID: <is8qv8$jan$1@dough.gmane.org>

On 6/2/2011 1:58 PM, Guido van Rossum wrote:
> On Wed, Jun 1, 2011 at 11:30 PM, Terry Reedy<tjreedy at udel.edu>  wrote:
>> The confusion of character with byte in the original design of Python both
>> privileged and burdened text processing.
>
> Right. And it wasn't only Python: most languages created around or
> before that time had the same issues (perhaps starting with C's use of
> "char" meaning byte). Even most IP protocols developed in the 1990s
> confuse character set and encoding (witness HTTP's "Content-type:
> text/plain; charset=utf-8").

I hold Python to a higher standard. But yes, that is badly confused.

> I'm glad in Python 3 we undertook to improve the distinction.

I am a bit embarassed that I did not see sooner that characters are for 
people and bytes for computers. Thus Python produces both character and 
byte serializations for objects.

On the coding front: when I first did statistics on computers (1970s), 
all data were coded with numbers. For instance, Sex: male = 1; female = 
2; unknown = 9. In the 1980s, we could use letters (which became ascii 
codes): male = 'm'; female = 'f'; unknown = ' '. For a US-only project, 
this seemed like an advance. So I though then.

For a global project, it would have been the opposite. For a Spanish 
speaker, 'm' might seem to mean 'mujer' (woman). For many others around 
the world, euro-indic digits are more familiar and easier to read than 
latin letters. I am less ethnocentric now.

I'm glad Python has become more of a global language, even if English based.

-- 
Terry Jan Reedy



From python-dev at realityexists.net  Fri Jun  3 06:07:39 2011
From: python-dev at realityexists.net (Evan Martin)
Date: Fri, 03 Jun 2011 14:07:39 +1000
Subject: [Python-ideas] date.datetime() method to convert from date to
	datetime
Message-ID: <4DE85E0B.5090906@realityexists.net>

There is a datetime.date() method for converting from a datetime to a 
date, but no stdlib method to do the opposite conversion. Could a 
date.datetime() method be added that returns a datetime with the time 
component set to zero? Alternatively, if the object is already a 
datetime, it could simply return itself.

This isn't exactly hard to do in user code, but the obvious way of doing 
it is a bit too verbose for such a simple operation. Less verbose ways 
are not obvious or are error-prone, as this StackOverflow question 
shows: 
http://stackoverflow.com/questions/1937622/convert-date-to-datetime-in-python

--
Evan Martin


From ncoghlan at gmail.com  Fri Jun  3 06:40:36 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 3 Jun 2011 14:40:36 +1000
Subject: [Python-ideas] Adding 'bytes' as alias for 'latin_1' codec.
In-Reply-To: <is8qv8$jan$1@dough.gmane.org>
References: <BANLkTin1P5BVUisF-SfW=-TUeQbumDdBvA@mail.gmail.com>
	<irkc54$gih$1@dough.gmane.org> <irl5n7$l1q$1@dough.gmane.org>
	<BANLkTikQXPJ3Y0SBfG5UKhu1S-CVDo5sng@mail.gmail.com>
	<DF3948D2-C0AC-4B6E-A722-25004C48C0E2@masklinn.net>
	<irmk2f$ess$1@dough.gmane.org>
	<BANLkTikTnwrGA2N2oaxwNsqYDVeWO6CXVQ@mail.gmail.com>
	<BANLkTinsqOGZzU7DXBShde9N1kudoSFP8Q@mail.gmail.com>
	<4DE0481E.7010005@canterbury.ac.nz>
	<BANLkTimZRE15gPqVPZR27pQtWYgntycz9w@mail.gmail.com>
	<79306.1306858606@parc.com>
	<BANLkTi=pNpXnqiXKbtGL2cvt6Kb+HBu4XQ@mail.gmail.com>
	<98069.1306946043@parc.com> <is5s9r$nnh$1@dough.gmane.org>
	<4DE67DB4.3070304@stoneleaf.us>
	<BANLkTikLfwLVD2VgUtsL+dF2prii+hKy9Q@mail.gmail.com>
	<is7am5$8f8$1@dough.gmane.org>
	<BANLkTik75wOp+XXj=xs_L+au379JFJz7_A@mail.gmail.com>
	<is8qv8$jan$1@dough.gmane.org>
Message-ID: <BANLkTi=gA_36eZZC+k=mGr+_JWxGQ2G4-g@mail.gmail.com>

On Fri, Jun 3, 2011 at 6:14 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> I am a bit embarassed that I did not see sooner that characters are for
> people and bytes for computers. Thus Python produces both character and byte
> serializations for objects.

FWIW, even after being involved in the assorted bytes/str design
discussions for Py3k, I didn't really "get it" myself until I made the
changes to urllib.parse in Python 3.2 to get most of the APIs to
accept both str objects and byte sequences.

The contrast between my first attempt (which tried to provide a common
code path that handled both strings and byte sequences without
trashing the encoding of the latter) and my second (which just decodes
and reencodes byte sequences using strict ASCII and punts on malformed
URLs containing non-ASCII values) was amazing. My original plan was to
benchmark them before choosing, but the latter approach was so much
simpler and cleaner than the former that it wasn't even a contest.

Focusing efforts on things like PEP 393, and perhaps even a memoryview
based "strview" is likely to be a more fruitful way forward than
trying to shoehorn text-specific concerns into the general binary
storage types (and, as noted, the long release cycle means the
standard library is the wrong place for that kind of experimentation).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Fri Jun  3 06:45:25 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 3 Jun 2011 14:45:25 +1000
Subject: [Python-ideas] date.datetime() method to convert from date to
	datetime
In-Reply-To: <4DE85E0B.5090906@realityexists.net>
References: <4DE85E0B.5090906@realityexists.net>
Message-ID: <BANLkTik3ARejgx_cWNavJHkyo6ADGNM0vw@mail.gmail.com>

On Fri, Jun 3, 2011 at 2:07 PM, Evan Martin
<python-dev at realityexists.net> wrote:
> There is a datetime.date() method for converting from a datetime to a date,
> but no stdlib method to do the opposite conversion. Could a date.datetime()
> method be added that returns a datetime with the time component set to zero?
> Alternatively, if the object is already a datetime, it could simply return
> itself.
>
> This isn't exactly hard to do in user code, but the obvious way of doing it
> is a bit too verbose for such a simple operation. Less verbose ways are not
> obvious or are error-prone, as this StackOverflow question shows:
> http://stackoverflow.com/questions/1937622/convert-date-to-datetime-in-python

Alternatively, the second argument to datetime.combine() could be made
optional (defaulting to midnight).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ben+python at benfinney.id.au  Fri Jun  3 08:03:29 2011
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 03 Jun 2011 16:03:29 +1000
Subject: [Python-ideas] date.datetime() method to convert from date to
	datetime
References: <4DE85E0B.5090906@realityexists.net>
Message-ID: <87wrh34fke.fsf@benfinney.id.au>

Evan Martin
<python-dev at realityexists.net> writes:

> There is a datetime.date() method for converting from a datetime to a
> date, but no stdlib method to do the opposite conversion. Could a
> date.datetime() method be added that returns a datetime with the time
> component set to zero?

What is ?zero? for a time-of-day? Do you mean ?midnight on that day?? In
what timezone?

-- 
 \       ?Faith, n. Belief without evidence in what is told by one who |
  `\   speaks without knowledge, of things without parallel.? ?Ambrose |
_o__)                           Bierce, _The Devil's Dictionary_, 1906 |
Ben Finney



From python-dev at realityexists.net  Fri Jun  3 14:46:35 2011
From: python-dev at realityexists.net (Evan Martin)
Date: Fri, 03 Jun 2011 22:46:35 +1000
Subject: [Python-ideas] date.datetime() method to convert from date to
 datetime
In-Reply-To: <BANLkTik3ARejgx_cWNavJHkyo6ADGNM0vw@mail.gmail.com>
References: <4DE85E0B.5090906@realityexists.net>
	<BANLkTik3ARejgx_cWNavJHkyo6ADGNM0vw@mail.gmail.com>
Message-ID: <4DE8D7AB.3040902@realityexists.net>

Yes, zero means the midnight of the date. No timezone - it's a naive 
datetime, same as the original date. The method would return exactly the 
same result as datetime.combine(date, time())

--
Evan Martin


From alexander.belopolsky at gmail.com  Fri Jun  3 15:11:32 2011
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 3 Jun 2011 09:11:32 -0400
Subject: [Python-ideas] date.datetime() method to convert from date to
	datetime
In-Reply-To: <4DE85E0B.5090906@realityexists.net>
References: <4DE85E0B.5090906@realityexists.net>
Message-ID: <BANLkTi=y7oBhVoRfQ4O2NWTCK-tORNYcXA@mail.gmail.com>

On Fri, Jun 3, 2011 at 12:07 AM, Evan Martin
<python-dev at realityexists.net> wrote:
> There is a datetime.date() method for converting from a datetime to a date,
> but no stdlib method to do the opposite conversion. Could a date.datetime()
> method be added that returns a datetime with the time component set to zero?
> Alternatively, if the object is already a datetime, it could simply return
> itself.

My preferred alternative to this idea would be to allow datetime
constructor to take date (or datetime).  This would make date/datetime
behave similar to int/float.  If this is done, I would also like the
single-argument constructor to accept str in ISO format.


From ben+python at benfinney.id.au  Sat Jun  4 02:08:32 2011
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 04 Jun 2011 10:08:32 +1000
Subject: [Python-ideas] date.datetime() method to convert from date to
	datetime
References: <4DE85E0B.5090906@realityexists.net>
	<BANLkTi=y7oBhVoRfQ4O2NWTCK-tORNYcXA@mail.gmail.com>
Message-ID: <87oc2e4fwf.fsf@benfinney.id.au>

Alexander Belopolsky
<alexander.belopolsky at gmail.com> writes:

> My preferred alternative to this idea would be to allow datetime
> constructor to take date (or datetime). This would make date/datetime
> behave similar to int/float.

+1. They are conceptually very similar types, so it makes sense for the
constructor to accept each of them.

> If this is done, I would also like the single-argument constructor to
> accept str in ISO format.

?1, please don't overload a type's default constructor to the point of
accepting all sorts of unrelated types.

I think ?datetime.fromstring?, if implemented, should be a separate
alternative constructor (by whatever spelling).

-- 
 \        ?I knew it was a shocking thing to say, but ? no-one has the |
  `\        right to spend their life without being offended.? ?Philip |
_o__)                                              Pullman, 2010-03-28 |
Ben Finney



From python-dev at realityexists.net  Sat Jun  4 03:10:38 2011
From: python-dev at realityexists.net (Evan Martin)
Date: Sat, 04 Jun 2011 11:10:38 +1000
Subject: [Python-ideas] date.datetime() method to convert from date to
 datetime
In-Reply-To: <87oc2e4fwf.fsf@benfinney.id.au>
References: <4DE85E0B.5090906@realityexists.net>	<BANLkTi=y7oBhVoRfQ4O2NWTCK-tORNYcXA@mail.gmail.com>
	<87oc2e4fwf.fsf@benfinney.id.au>
Message-ID: <4DE9860E.1070003@realityexists.net>

On 4/06/2011 10:08 AM, Ben Finney wrote:
> Alexander Belopolsky
> <alexander.belopolsky at gmail.com>  writes:
>
>> My preferred alternative to this idea would be to allow datetime
>> constructor to take date (or datetime). This would make date/datetime
>> behave similar to int/float.
> +1. They are conceptually very similar types, so it makes sense for the
> constructor to accept each of them.

I think that would work well if we had overloaded constructors, but 
without them it might complicate things too much. What would the 
signature look like?

Also, that would be inconsistent with how a datetime is converted to a 
date - as a user I would then expect the date constructor to take a 
datetime, too. If datetime.date() converts a datetime to a date then I 
naturally think to call date.datetime() to go the other way.

--
Evan Martin


From ericsnowcurrently at gmail.com  Sat Jun  4 08:11:31 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Sat, 4 Jun 2011 00:11:31 -0600
Subject: [Python-ideas] objects aware of being bound to a name
Message-ID: <BANLkTi=-oA4bEaBB=g29Rv68j-r7o81SRQ@mail.gmail.com>

When a name is bound to an object, the object doesn't hear about it,
usually.  The exceptions I could
think of are class and function definitions, and imports.  Also, you
can sneak around it using
setattr/descriptors if you control the class...  However, objects are
otherwise (and generally) blind to
their names.

This is relevant when you want an object to be aware of the contexts
in which it exists.  It also relates
to DRY issues that people bring up sometimes with regards to
descriptors and namedtuple (though
I'm not sure its that big a deal).

Here are three approaches that satisfy this situation:

1. have assignment automatically pass the name to the __init__ when
binding a new instance.

2. bind the name to __name__ on the object before calling __init__.

3. call __bound__(self, name, obj) on the object before __init__ is
called (or maybe after).

The first is the approach import/class/def take.  You can't really
generalize that, though, since most
classes don't have a name parameter.  Both the first and second
approach only work when the object
to be bound is instantiated.  The second seems to work better than the
first, but __name__ shouldn't
be re-bound on an object every time it is involved in an assignment.
It might be a good special-case,
though.

I like the third option because it could be tried for any name
binding, from assignment to function
arguments.  However, that may be its downfall too.  I would guess that
name binding happens more
than just once or twice during the course of execution <wink>.  I
would also guess that it would kill
performance.  However, I don't know the ins and outs of the
compiler/runtime so I could be pleasantly
wrong.

In addition to __bound__, an __unbound__ could be leveraged (wait for
it) to let an object know when
it has been unbound from a name (during del or when another object is
bound to the name).  Of course
you get double the performance hit from just __bound__.

Like most things, this can already be done, just not cleanly.  Here's
an example of more or less
equivalent code:

    class Something:
        def __init__(self):
            self._names = set()
        def __bound__(self, name, obj):
            if (name, obj) in set:
                return
            self._names.add((name, obj))
        def __unbound__(self, name, obj):
            self._names.remove((name, obj))

    obj = __import__(__file__.rsplit(".py", 1)[0])

    something = Something()
    something.__bound__("something", obj)

    something.__unbound__("something", obj)
    something = Something()
    something.__bound__("something", obj)

    something.__unbound__("something", obj)
    del something


So you can do it already, explicitly, but it's a mess.  I wouldn't be
surprised if there was a way to
be smarter about when to call __bound__/__unbound__ to alleviate the
performance hit, but I don't
see it.  I also wouldn't be surprised if there was a trivial way to do
this, or if no one's brought it up
because it's such an obviously bad idea! :)  Maybe I just need to get
go some sleep.

Regardless, this idea hit me suddenly while I was working on something
else.  I don't remember
what prompted the idea, but I at least wanted to float it out there.
Even if it's a terrible idea, I think
the concept of letting the bound object know how it's bound is an
interesting one.  It's an angle I
had not considered before.

Thanks,

-eric


From guido at python.org  Sat Jun  4 18:52:03 2011
From: guido at python.org (Guido van Rossum)
Date: Sat, 4 Jun 2011 09:52:03 -0700
Subject: [Python-ideas] Fwd:  Minor tweak to PEP 8?
In-Reply-To: <BANLkTinxg7HGiNn2VvDb8pzphifXbpmWLg@mail.gmail.com>
References: <20110510104754.4689cc5e@bhuda.mired.org>
	<BANLkTi=txKo3DPWOBj5ZD09V5jBPcT6Srw@mail.gmail.com>
	<87y62ejl2j.fsf@benfinney.id.au>
	<BANLkTikV5PppTUkOVn-Sxd6HWdG3=H0Z2g@mail.gmail.com>
	<BANLkTi=p7prG6R+dVhzeBANWbK_jL2bPOg@mail.gmail.com>
	<BANLkTi=0QgcC5NC0ECLKjJcG_Nmt-aSA-g@mail.gmail.com>
	<BANLkTinrUQKO0trKgWyUZLWyhBxO10Vu=w@mail.gmail.com>
	<BANLkTin+Ubc3zVY51c=Q0NckBj4kvMuS+A@mail.gmail.com>
	<20110602150031.2b98524f@neurotica.wooz.org>
	<BANLkTik4MrWhKj0P93_j1AY2DcjC5=Goyw@mail.gmail.com>
	<20110602181113.5be70efc@neurotica.wooz.org>
	<BANLkTinyRoJmM7PdNB22ucN5D3HdWkibMA@mail.gmail.com>
	<20110603101906.25dbf2f7@neurotica.wooz.org>
	<BANLkTi=4pt2qVesPzwFYE7QD02gmSXMv_Q@mail.gmail.com>
	<20110603140923.0c8fcb90@neurotica.wooz.org>
	<BANLkTi=Ta8vxKrHJXhp86vaR7uSeNNF64g@mail.gmail.com>
	<BANLkTinxg7HGiNn2VvDb8pzphifXbpmWLg@mail.gmail.com>
Message-ID: <BANLkTikCBVh6PUuyQOaZU7ngr2=WRfXvgw@mail.gmail.com>

[Correct list address]

Yeah, please prepare a patch. Maybe you can send it to Barry so he can
check it in. I would use 4-space indent for the "Opt:" exanple at the
end though.

On Sat, Jun 4, 2011 at 9:03 AM, Steven Klass <sklass at pointcircle.com> wrote:
> Hey Barry,
> You are correct - it was an artifact from the email tool..
> Guido - it appears we are reaching some consensus. Thoughts? Did you want me
> to update the repo?
> ?? ?Continuation lines should align wrapped elements either vertically
> using
> ?? ?Python's implicit line joining inside parentheses, brackets and braces,
> or
> ?? ?or using a hanging indent. ?When using a hanging indent the following
> ?? ?considerations should be applied; there should be no arguments on the
> ?? ?first line and further indentation should be used to clearly distinguish
> ?? ?the itself as a continuation line.
>
> ?? ?Yes: ?# Aligned with opening delimiter
> ?? ? ? ? ?foo = long_function_name(var_one, var_two,
> ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? var_three, var_four)
> ?? ? ? ? ?# More indentation included to distiguish this from the rest.
> ?? ? ? ? ?def long_function_name(
> ?? ? ? ? ? ? ? ? ?var_one, var_two, var_three,
> ?? ? ? ? ? ? ? ? ?var_four):
> ?? ? ? ? ? ? ?print(var_one)
> ?? ?No: ? # Stuff on first line forbidden when not using vertical alignment
> ?? ? ? ? ?foo = long_function_name(var_one, var_two,
> ?? ? ? ? ? ? ?var_three, var_four)
> ?? ? ? ? ?# Further indentation required as indentation is not
> distiguishable
> ?? ? ? ? ?def long_function_name(
> ?? ? ? ? ? ? ?var_one, var_two, var_three,
> ?? ? ? ? ? ? ?var_four):
> ?? ? ? ? ? ? ?print(var_one)
>
> ?? ?Opt: ?# Extra indentation not necessary.
> ?? ? ? ? ?foo = long_function_name(
> ?? ? ? ? ? ?var_one, var_two,
> ?? ? ? ? ? ?var_three, var_four)
>
>
>
> On Fri, Jun 3, 2011 at 11:09 AM, Barry Warsaw <barry at python.org> wrote:
>>
>> Hi Steven,
>>
>> On Jun 03, 2011, at 08:57 AM, Steven Klass wrote:
>>
>> > ? ?Continuation lines should align wrapped elements either vertically
>> > using
>> > ? ?Python's implicit line joining inside parentheses, brackets and
>> > braces,
>> > ? ?or or using a hanging indent. ?When using a hanging indent the
>> > following
>> > ? ?considerations should be applied; there should be no arguments on the
>> > ? ?first line and further indentation should be used to clearly
>> > distinguish
>> > ? ?the itself as a continuation line.
>> >
>> > ? ?Yes: ?# Aligned with opening delimiter
>> > ? ? ? ? ?foo = long_function_name(var_one, var_two,
>> > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?var_three, var_four)
>> -------------------------------------^
>> I'm sure that's just an email alignment bug, right?
>>
>> >
>> > ? ? ? ? ?# More indentation included to distiguish this from the rest.
>> > ? ? ? ? ?def long_function_name(
>> > ? ? ? ? ? ? ? ? ?var_one, var_two, var_three,
>> > ? ? ? ? ? ? ? ? ?var_four):
>> > ? ? ? ? ? ? ?print(var_one)
>> >
>> >
>> > ? ?No: ? # Stuff on first line forbidden when not using vertical
>> > alignment
>> > ? ? ? ? ?foo = long_function_name(var_one, var_two,
>> > ? ? ? ? ? ? ?var_three, var_four)
>> >
>> > ? ? ? ? ?# Further indentation required as indentation is not
>> > distiguishable
>> > ? ? ? ? ?def long_function_name(
>> > ? ? ? ? ? ? ?var_one, var_two, var_three,
>> > ? ? ? ? ? ? ?var_four):
>> > ? ? ? ? ? ? ?print(var_one)
>> >
>> >
>> >Thoughts?
>>
>> That looks great to me, thanks. ?I would add one more example to the 'Yes'
>> section to cover the case we've been talking about:
>>
>> ? ? ? ?# Extra indentation not necessary.
>> ? ? ? ?foo = long_function_name(
>> ? ? ? ? ? ?var_one, var_two,
>> ? ? ? ? ? ?var_three, var_four)
>>
>> This doesn't say that extra indentation isn't allowed, just that it's not
>> necessary, so I think it strikes the right balance.
>>
>> Cheers,
>> -Barry
>
>
>
> --
> ---
>
> Steven M. Klass
>
> ? 1 (480) 225-1112
> ? sklass at pointcircle.com
>


-- 
--Guido van Rossum (python.org/~guido)


From tjreedy at udel.edu  Sun Jun  5 03:54:42 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 04 Jun 2011 21:54:42 -0400
Subject: [Python-ideas] objects aware of being bound to a name
In-Reply-To: <BANLkTi=-oA4bEaBB=g29Rv68j-r7o81SRQ@mail.gmail.com>
References: <BANLkTi=-oA4bEaBB=g29Rv68j-r7o81SRQ@mail.gmail.com>
Message-ID: <isenl3$3gd$1@dough.gmane.org>

On 6/4/2011 2:11 AM, Eric Snow wrote:
> When a name is bound to an object, the object doesn't hear about it,

Objects are created. Then they may *optionally* be bound to names 
(possibly in a different namespace) and collection slots. Or they may be 
used in an expression and be discarded without every being bound to 
anything.

 > usually.

I believe never.

 > The exceptions I could
> think of are class and function definitions, and imports.

Functions, classes, and modules have __name__ attributes (that cannot be 
deleted, even if they can be replaced). This attribute is set before 
they are optionally bound, so they also do not hear about the subsequent 
bindings. This attribute is used for their string representations for 
humans to read. I cannot think of any other use by the interpreter itself.

 > Also, you can sneak around it using
> setattr/descriptors if you control the class...  However, objects are
> otherwise (and generally) blind to their names.

Object (or definition or intrinsic or attribute) names (exactly 1 for 
certain instances of certain classes) and namespace binding names 
(0-many for every object) are different concepts. Ojects names live on 
the object. The binding names live in one of the many namespaces. Object 
names are not unique among objects. Binding names are unique within each 
namespace.

Objects names do not have to match any of the binding names of the 
object. A common type of example of this is

 >>> import itertools as it
 >>> it.__name__
'itertools'

> This is relevant when you want an object to be aware of the contexts
> in which it exists.

Python objects exist within an anonymous object space with no structure. 
They can be used within any namespace context from which they can be 
accessed via names and slots. Adding a __name__ attribute to instances 
of a class will not say anything about use context.

Modules carry source information in __filename__ for convenience in 
error reporting. Classes and functions have the *name* of their creation 
context in __module__ for the same reason. I do not believe that 
__filename or __module__ have any operational meaning after the object 
is created. For functions (but not classes) __globals__ *is* needed to 
function. Instances have __class__, which is used for both attribute 
lookup and information display.

 > It also relates to DRY issues that people bring up sometimes
 > with regards to descriptors and namedtuple (though
> I'm not sure its that big a deal).

If one wants an object name and initial binding name to be the same, I 
agree that giving the name once is a convenience. Def, class, and import 
*statements* allow this for functions, classes, and modules. Each has a 
syntax peculiar to the class. Lambda expressions, type() calls, 
__import__ calls (and others in inspect create objects with __name__s 
but without an automatic binding operation.

Ideas about not repeating duplicate object/binding names was discussed 
here in a thread with several posts within the last year. I believe one 
idea was a new 'augmented assignment': something like 'x .= y' meaning 
"y.__name__ = 'x'; x = y". But there was some problem with every proposal.

-- 
Terry Jan Reedy



From ncoghlan at gmail.com  Sun Jun  5 07:50:30 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 5 Jun 2011 15:50:30 +1000
Subject: [Python-ideas] objects aware of being bound to a name
In-Reply-To: <isenl3$3gd$1@dough.gmane.org>
References: <BANLkTi=-oA4bEaBB=g29Rv68j-r7o81SRQ@mail.gmail.com>
	<isenl3$3gd$1@dough.gmane.org>
Message-ID: <BANLkTikf8e2soj=E44R3MkNi3o4-3RUTnQ@mail.gmail.com>

On Sun, Jun 5, 2011 at 11:54 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> Functions, classes, and modules have __name__ attributes (that cannot be
> deleted, even if they can be replaced). This attribute is set before they
> are optionally bound, so they also do not hear about the subsequent
> bindings. This attribute is used for their string representations for humans
> to read. I cannot think of any other use by the interpreter itself.

__name__ attributes are also relevant for serialisation (esp. pickling).

However, due to immutable objects, there's no realistic general
purpose solution in this space.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From cmjohnson.mailinglist at gmail.com  Sun Jun  5 10:00:55 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Sat, 4 Jun 2011 22:00:55 -1000
Subject: [Python-ideas] objects aware of being bound to a name
In-Reply-To: <BANLkTikf8e2soj=E44R3MkNi3o4-3RUTnQ@mail.gmail.com>
References: <BANLkTi=-oA4bEaBB=g29Rv68j-r7o81SRQ@mail.gmail.com>
	<isenl3$3gd$1@dough.gmane.org>
	<BANLkTikf8e2soj=E44R3MkNi3o4-3RUTnQ@mail.gmail.com>
Message-ID: <BANLkTi=PK+zO+Xw_+-UbaGDAUYuToeL6OA@mail.gmail.com>

This is in principle doable with metaclasses (like everything else :-D)...

Here's what I whipped up in the console:

>>> class MagicDict(dict):
...     def __setitem__(self, key, value):
...         print("Binding key: {} to value: {}".format(key, value))
...         if hasattr(value, "__bind__"):
...             value.__bind__(key)
...         super().__setitem__(key, value)
...
>>> class Bindable:
...     def __bind__(self, key):
...         print("{} was bound to {}".format(self, key))
...
>>> class MetaBinding(type):
...     @classmethod
...     def __prepare__(metacls, name, bases):
...        return MagicDict()
...
>>> class BindingReports(metaclass=MetaBinding):
...     a = 1
...     b = Bindable()
...     c = "blah"
...
Binding key: __module__ to value: __main__
Binding key: a to value: 1
Binding key: b to value: <__main__.Bindable object at 0x100603b50>
<__main__.Bindable object at 0x100603b50> was bound to b
Binding key: c to value: blah

Not sure how useful this is. I don't like using the term "class" for things
where you're not really trying to bundle together methods or create a type,
just change the way values get bound.

-- Carl Johnson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110604/40db7ff3/attachment.html>

From ericsnowcurrently at gmail.com  Mon Jun  6 17:48:07 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Mon, 6 Jun 2011 09:48:07 -0600
Subject: [Python-ideas] objects aware of being bound to a name
In-Reply-To: <BANLkTikf8e2soj=E44R3MkNi3o4-3RUTnQ@mail.gmail.com>
References: <BANLkTi=-oA4bEaBB=g29Rv68j-r7o81SRQ@mail.gmail.com>
	<isenl3$3gd$1@dough.gmane.org>
	<BANLkTikf8e2soj=E44R3MkNi3o4-3RUTnQ@mail.gmail.com>
Message-ID: <BANLkTi=oE6rbK3LvSSK4SEL7NJwn-YTfxw@mail.gmail.com>

On Sat, Jun 4, 2011 at 11:50 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Sun, Jun 5, 2011 at 11:54 AM, Terry Reedy <tjreedy at udel.edu> wrote:
>> Functions, classes, and modules have __name__ attributes (that cannot be
>> deleted, even if they can be replaced). This attribute is set before they
>> are optionally bound, so they also do not hear about the subsequent
>> bindings. This attribute is used for their string representations for humans
>> to read. I cannot think of any other use by the interpreter itself.
>
> __name__ attributes are also relevant for serialisation (esp. pickling).
>
> However, due to immutable objects, there's no realistic general
> purpose solution in this space.
>

Yeah, I think I was hasty on writing this up.  It's interesting, but
not a great fit, nor very practical.  The immutable objects problem is
definitely a show-stopper.  Thanks for the feedback though.

-eric

> Cheers,
> Nick.
>
> --
> Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From andrew at acooke.org  Mon Jun  6 18:11:23 2011
From: andrew at acooke.org (andrew cooke)
Date: Mon, 6 Jun 2011 12:11:23 -0400
Subject: [Python-ideas] objects aware of being bound to a name
In-Reply-To: <BANLkTi=oE6rbK3LvSSK4SEL7NJwn-YTfxw@mail.gmail.com>
References: <BANLkTi=-oA4bEaBB=g29Rv68j-r7o81SRQ@mail.gmail.com>
	<isenl3$3gd$1@dough.gmane.org>
	<BANLkTikf8e2soj=E44R3MkNi3o4-3RUTnQ@mail.gmail.com>
	<BANLkTi=oE6rbK3LvSSK4SEL7NJwn-YTfxw@mail.gmail.com>
Message-ID: <20110606161123.GE11101@acooke.org>


I'm not sure what you're trying to do (ie if there's any practical problem
that motivates this), but in Lepl (a parser) I use a little trick that lets me
examine variables defined within a "with" scope.  That lets me add "debugging"
at the application level.

There's an example here:
http://www.acooke.org/lepl/debugging.html#variable-traces - everything defined
inside the "with TraceVariables" is found by inspection of some Python
internals doo-hicky, and then modified to produce the debug output (note that
the output incldues the variable *names* which is the kind of thing you are
trying to do here).

Contact me if you want more info.

Andrew



On Mon, Jun 06, 2011 at 09:48:07AM -0600, Eric Snow wrote:
> On Sat, Jun 4, 2011 at 11:50 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > On Sun, Jun 5, 2011 at 11:54 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> >> Functions, classes, and modules have __name__ attributes (that cannot be
> >> deleted, even if they can be replaced). This attribute is set before they
> >> are optionally bound, so they also do not hear about the subsequent
> >> bindings. This attribute is used for their string representations for humans
> >> to read. I cannot think of any other use by the interpreter itself.
> >
> > __name__ attributes are also relevant for serialisation (esp. pickling).
> >
> > However, due to immutable objects, there's no realistic general
> > purpose solution in this space.
> >
> 
> Yeah, I think I was hasty on writing this up.  It's interesting, but
> not a great fit, nor very practical.  The immutable objects problem is
> definitely a show-stopper.  Thanks for the feedback though.
> 
> -eric
> 
> > Cheers,
> > Nick.
> >
> > --
> > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > http://mail.python.org/mailman/listinfo/python-ideas
> >
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
> 


From zuo at chopin.edu.pl  Wed Jun  8 13:01:08 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Wed, 8 Jun 2011 13:01:08 +0200
Subject: [Python-ideas] Iteritems() function?
Message-ID: <20110608110108.GA2703@chopin.edu.pl>

Case
====

Quite typical: iterate and do something with some_items -- a collection
of 2-element items.

    for first, second in some_items:
        ...

But for dicts it must use another form:

    for first, second in some_items.items():
        ...

We must know it'll be a mapping, and even then quite usual bug is to
forget to add that `items()'.

But sometimes it may be a dict {first: second, ...} OR a seq [(first,
second), ...] and in fact we are not interested in it -- we simply want
to iterate over its items... But we are forced to do type/interface
check, e.g.:

    if isinstance(coll, collections.Mapping):
        for first, second in some_items.items(): ...
    else:
        for first, second in some_items: ...

Idea
====

A new function:
    builtins.iteritems()
or
    builtins.iterpairs()
or
    itertools.items()
or
    itertools.pairs()
(don't know what name would be the best)

-- equivalent to:

    def <one of the above names>(coll):
        iterable = (coll.items()
                    if isinstance(coll, collections.Mapping)
                    else coll)
        return iter(iterable)

or maybe something like:

    def <one of the above names>(coll):
        try:
            iterable = coll.items()
        except AttributeError:
            iterable = coll
        return iter(iterable)

Usage
=====

Then, in our example case, we'd do simply:

    for first, second in iteritems(some_items):
        ...

And we don't need to think about some_items type, whether it's a mapping
or 2-tuple sequence. All we need to know is that it's a collection of
2-element items.

Regards,
*j



From masklinn at masklinn.net  Wed Jun  8 13:25:42 2011
From: masklinn at masklinn.net (Masklinn)
Date: Wed, 8 Jun 2011 13:25:42 +0200
Subject: [Python-ideas] Iteritems() function?
In-Reply-To: <20110608110108.GA2703@chopin.edu.pl>
References: <20110608110108.GA2703@chopin.edu.pl>
Message-ID: <518A4DD6-AA95-4D68-9783-5FE993E07B7A@masklinn.net>

On 2011-06-08, at 13:01 , Jan Kaliszewski wrote:
> But sometimes it may be a dict {first: second, ...} OR a seq [(first,
> second), ...] and in fact we are not interested in it -- we simply want
> to iterate over its items... But we are forced to do type/interface
> check, e.g.:
> 
>    if isinstance(coll, collections.Mapping):
>        for first, second in some_items.items(): ...
>    else:
>        for first, second in some_items: ?
You could just convert everything to a dict:

    for first, second in dict(some_items).iteritems():
        # etc?



From phd at phdru.name  Wed Jun  8 13:54:49 2011
From: phd at phdru.name (Oleg Broytman)
Date: Wed, 8 Jun 2011 15:54:49 +0400
Subject: [Python-ideas] Iteritems() function?
In-Reply-To: <20110608110108.GA2703@chopin.edu.pl>
References: <20110608110108.GA2703@chopin.edu.pl>
Message-ID: <20110608115449.GC21059@iskra.aviel.ru>

On Wed, Jun 08, 2011 at 01:01:08PM +0200, Jan Kaliszewski wrote:
> Quite typical: iterate and do something with some_items -- a collection
> of 2-element items.
> 
>     for first, second in some_items:
>         ...
> 
> But for dicts it must use another form:
> 
>     for first, second in some_items.items():
>         ...
> 
> We must know it'll be a mapping, and even then quite usual bug is to
> forget to add that `items()'.

   You don't need a special buitin for that. Just call .items():

if hasattr(some_items, 'items'):
    some_items = some_items.items()
for first, second in some_items:
    ...

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.


From grosser.meister.morti at gmx.net  Wed Jun  8 17:41:28 2011
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Wed, 08 Jun 2011 17:41:28 +0200
Subject: [Python-ideas] Iteritems() function?
In-Reply-To: <20110608115449.GC21059@iskra.aviel.ru>
References: <20110608110108.GA2703@chopin.edu.pl>
	<20110608115449.GC21059@iskra.aviel.ru>
Message-ID: <4DEF9828.9070009@gmx.net>

On 06/08/2011 01:54 PM, Oleg Broytman wrote:
> On Wed, Jun 08, 2011 at 01:01:08PM +0200, Jan Kaliszewski wrote:
>> Quite typical: iterate and do something with some_items -- a collection
>> of 2-element items.
>>
>>      for first, second in some_items:
>>          ...
>>
>> But for dicts it must use another form:
>>
>>      for first, second in some_items.items():
>>          ...
>>
>> We must know it'll be a mapping, and even then quite usual bug is to
>> forget to add that `items()'.
>
>     You don't need a special buitin for that. Just call .items():
>
> if hasattr(some_items, 'items'):
>      some_items = some_items.items()
> for first, second in some_items:
>      ...
>
> Oleg.

So basically it would be:

def items(sequence):
	if hasattr(sequence, 'items'):
		return sequence.items()
	else:
		return sequence

I don't think that is enough complexity to justify an inclusion in builtins or itertools. Anyway, I 
would have expected such a function to do this (so it's not even obvious):

def items(sequence):
	if hasattr(sequence, 'items'):
		return sequence.items()
	else:
		return enumerate(sequence)


	-panzi


From ncoghlan at gmail.com  Wed Jun  8 18:24:30 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 9 Jun 2011 02:24:30 +1000
Subject: [Python-ideas] Iteritems() function?
In-Reply-To: <4DEF9828.9070009@gmx.net>
References: <20110608110108.GA2703@chopin.edu.pl>
	<20110608115449.GC21059@iskra.aviel.ru> <4DEF9828.9070009@gmx.net>
Message-ID: <BANLkTimC0DMKHHuU4ngEP_kO2HKaiDmfBw@mail.gmail.com>

On Thu, Jun 9, 2011 at 1:41 AM, Mathias Panzenb?ck
<grosser.meister.morti at gmx.net> wrote:
> I don't think that is enough complexity to justify an inclusion in builtins
> or itertools. Anyway, I would have expected such a function to do this (so
> it's not even obvious):
>
> def items(sequence):
> ? ? ? ?if hasattr(sequence, 'items'):
> ? ? ? ? ? ? ? ?return sequence.items()
> ? ? ? ?else:
> ? ? ? ? ? ? ? ?return enumerate(sequence)

I expect the use case here is to implement APIs like the dict
constructor and update() method - you can either pass them a mapping,
or else an iterable of key-value pairs. As Oleg noted though, the
traditional way of handling that is to ducktype on the "items" method,
although an isinstance check against Mapping would indeed be an
acceptable substitute these days. Either way, calling items() gets you
an iterable of key-value pairs, so you write your algorithm to work on
that and do a coercion via items() to handle the mapping case.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From matt at vazor.com  Wed Jun  8 19:57:28 2011
From: matt at vazor.com (Matt Billenstein)
Date: Wed, 08 Jun 2011 18:57:28 +0100
Subject: [Python-ideas] Iteritems() function?
Message-ID: <4tiv7fp5lkhs.sts845i@elasticemail.net>

On Wed, Jun 08, 2011 at 01:25:42PM +0200, Masklinn wrote:
> You could just convert everything to a dict:
> 
>     for first, second in dict(some_items).iteritems():
>         # etc?

That option won't necessarily preserve the order of the original sequence where
perhaps it matters...

m

-- 
Matt Billenstein
matt at vazor.com
http://www.vazor.com/

From masklinn at masklinn.net  Wed Jun  8 21:32:37 2011
From: masklinn at masklinn.net (Masklinn)
Date: Wed, 8 Jun 2011 21:32:37 +0200
Subject: [Python-ideas] Iteritems() function?
In-Reply-To: <4tiv7fp5lkhs.sts845i@elasticemail.net>
References: <4tiv7fp5lkhs.sts845i@elasticemail.net>
Message-ID: <F0B2308B-9BC1-4208-B688-1728E2E46F95@masklinn.net>

On 2011-06-08, at 19:57 , Matt Billenstein wrote:
> On Wed, Jun 08, 2011 at 01:25:42PM +0200, Masklinn wrote:
>> You could just convert everything to a dict:
>> 
>>    for first, second in dict(some_items).iteritems():
>>        # etc?
> 
> That option won't necessarily preserve the order of the original sequence where
> perhaps it matters?
> 
That is what collections.OrderedDict is for.

From steve at pearwood.info  Thu Jun  9 02:38:23 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 09 Jun 2011 10:38:23 +1000
Subject: [Python-ideas] Iteritems() function?
In-Reply-To: <F0B2308B-9BC1-4208-B688-1728E2E46F95@masklinn.net>
References: <4tiv7fp5lkhs.sts845i@elasticemail.net>
	<F0B2308B-9BC1-4208-B688-1728E2E46F95@masklinn.net>
Message-ID: <4DF015FF.9050109@pearwood.info>

Masklinn wrote:
> On 2011-06-08, at 19:57 , Matt Billenstein wrote:
>> On Wed, Jun 08, 2011 at 01:25:42PM +0200, Masklinn wrote:
>>> You could just convert everything to a dict:
>>>
>>>    for first, second in dict(some_items).iteritems():
>>>        # etc?
>> That option won't necessarily preserve the order of the original sequence where
>> perhaps it matters?
>>
> That is what collections.OrderedDict is for.

But calling *dict* on the items (as shown above), not OrderedDict, 
doesn't preserve the order.

And frankly, I think it's silly to take an arbitrarily big iterable of 
ordered items, convert it to a dict (ordered or not), only to 
immediately extract an ordered iterable of items again:

OrderedDict(some_items).items()

You already have some_items in the right format for iteration, why 
iterate over it twice instead of once?

Better to use a simple helper function:

def coerce_to_items(obj):
     if has_attr(obj, 'items'):
         return obj.items()
     return obj

which accepts either a mapping (dict or OrderedDict) or an iterable of 
items, and returns an iterable of items.

(I use items() rather than iteritems() because any proposed new 
functionality must be aimed at Python 3, not 2.)

That's simple enough to use in-line, if you use small variable names:

for a,b in (pairs.items() if hasattr(pairs, 'items') else pairs):
     ...


but I think the utility function is better. I don't think it needs to be 
a built-in.


-- 
Steven



From ericsnowcurrently at gmail.com  Fri Jun 10 01:54:14 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 9 Jun 2011 17:54:14 -0600
Subject: [Python-ideas] inheriting docstrings and mutable docstings for
	classes
Message-ID: <BANLkTik=nQ=DwKk-UPfzuuo4SoD7koa1ZA@mail.gmail.com>

I noticed that __doc__ for classes is immutable:

>>> class X:
...     "some doc"
...
>>> X.__doc__
'some doc'
>>> X.__doc__ = "another doc"
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
AttributeError: attribute '__doc__' of 'type' objects is not writable

That is on 3.3, but apparently it's the case all the way back to 2.2.
I mentioned this on python-list and several people indicated that it
should be an unnecessary restriction [1].  Someone else pointed out
that docstrings also behave this way for method objects [2].

I want too see if it would be okay to make __doc__ writable for
classes.  I am not sure about for method objects, since I've never
thought to do that, but it is analogous to class instances, where
__doc__ is mutable and distinct from the class docstring.  I just
don't have any use cases that would dictate changing the docstring of
the method object, the wrapped function, or neither when changing
__doc__.

Someone else on the thread indicated that perhaps docstrings should be
inherited and got some support [3][4].  It makes sense to me for many,
but not necessarily all, cases.  However, if you don't want to inherit
you can simply set an empty docstring.  Docstrings impact help(),
doctests, and some DSLs.  I'm +1 on having __doc__ be inherited.

Thanks,

-eric


[1] http://mail.python.org/pipermail/python-list/2011-June/1274079.html
[2] http://mail.python.org/pipermail/python-list/2011-June/1274080.html
[3] http://mail.python.org/pipermail/python-list/2011-June/1274099.html
[4] http://mail.python.org/pipermail/python-list/2011-June/1274105.html


From ncoghlan at gmail.com  Fri Jun 10 03:05:08 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 10 Jun 2011 11:05:08 +1000
Subject: [Python-ideas] inheriting docstrings and mutable docstings for
	classes
In-Reply-To: <BANLkTik=nQ=DwKk-UPfzuuo4SoD7koa1ZA@mail.gmail.com>
References: <BANLkTik=nQ=DwKk-UPfzuuo4SoD7koa1ZA@mail.gmail.com>
Message-ID: <BANLkTimGVxe0m=txJtEr6=JMpkBOo7ahEQ@mail.gmail.com>

On Fri, Jun 10, 2011 at 9:54 AM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> I'm +1 on having __doc__ be inherited.

-1. Subclasses are not the same thing as the original class so
docstring inheritance should be requested explicitly.

Agreed that docstrings should be writeable after the fact, though
(e.g. functions already work that way - functools.wraps wouldn't work
otherwise).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ericsnowcurrently at gmail.com  Fri Jun 10 03:45:07 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 9 Jun 2011 19:45:07 -0600
Subject: [Python-ideas] inheriting docstrings and mutable docstings for
	classes
In-Reply-To: <BANLkTimGVxe0m=txJtEr6=JMpkBOo7ahEQ@mail.gmail.com>
References: <BANLkTik=nQ=DwKk-UPfzuuo4SoD7koa1ZA@mail.gmail.com>
	<BANLkTimGVxe0m=txJtEr6=JMpkBOo7ahEQ@mail.gmail.com>
Message-ID: <BANLkTinhbrrUkGB7TA-Pxo7NRLumQiCUYw@mail.gmail.com>

On Thu, Jun 9, 2011 at 7:05 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Fri, Jun 10, 2011 at 9:54 AM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> I'm +1 on having __doc__ be inherited.
>
> -1. Subclasses are not the same thing as the original class so
> docstring inheritance should be requested explicitly.
>

Yeah, this one was mostly auxiliary to my main concern, __doc__
mutability for classes.  Other than doctests and documentation/help(),
I haven't used docstrings for much so the idea of it did not seem like
a big deal.  I certainly find myself inheriting docstrings from my
abstract base classes explicitly all the time so that help() will show
the info that is still applicable.

> Agreed that docstrings should be writeable after the fact, though
> (e.g. functions already work that way - functools.wraps wouldn't work
> otherwise).
>

Would this be a very controversial change?  I ask because it's been
this way since 2.2 and no one's changed it.  Thanks.

-eric

> Cheers,
> Nick.
>
> --
> Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia
>


From greg.ewing at canterbury.ac.nz  Fri Jun 10 09:04:32 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 10 Jun 2011 19:04:32 +1200
Subject: [Python-ideas] inheriting docstrings and mutable docstings for
 classes
In-Reply-To: <BANLkTimGVxe0m=txJtEr6=JMpkBOo7ahEQ@mail.gmail.com>
References: <BANLkTik=nQ=DwKk-UPfzuuo4SoD7koa1ZA@mail.gmail.com>
	<BANLkTimGVxe0m=txJtEr6=JMpkBOo7ahEQ@mail.gmail.com>
Message-ID: <4DF1C200.9090405@canterbury.ac.nz>

Nick Coghlan wrote:
> -1. Subclasses are not the same thing as the original class so
> docstring inheritance should be requested explicitly.

The docstring of the class itself probably shouldn't be
inherited automatically. But if you override a method
without changing the API or user-visible behaviour, the
inherited docstring still applies.

Maybe the best thing would be for the inherited docstring
to get put into a different property, such as __basedoc__.
Then tools that examine docstrings can decide for themselves
whether using inherited docstrings makes sense.

-- 
Greg




From ncoghlan at gmail.com  Fri Jun 10 09:07:47 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 10 Jun 2011 17:07:47 +1000
Subject: [Python-ideas] inheriting docstrings and mutable docstings for
	classes
In-Reply-To: <4DF1C200.9090405@canterbury.ac.nz>
References: <BANLkTik=nQ=DwKk-UPfzuuo4SoD7koa1ZA@mail.gmail.com>
	<BANLkTimGVxe0m=txJtEr6=JMpkBOo7ahEQ@mail.gmail.com>
	<4DF1C200.9090405@canterbury.ac.nz>
Message-ID: <BANLkTi=PLiw5HFZgi1Fqw=-8wu72sjDykQ@mail.gmail.com>

On Fri, Jun 10, 2011 at 5:04 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>>
>> -1. Subclasses are not the same thing as the original class so
>> docstring inheritance should be requested explicitly.
>
> The docstring of the class itself probably shouldn't be
> inherited automatically. But if you override a method
> without changing the API or user-visible behaviour, the
> inherited docstring still applies.

I believe Eric created a class decorator recipe on the cookbook site
that does exactly that for methods without their own docstrings.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Fri Jun 10 09:09:41 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 10 Jun 2011 17:09:41 +1000
Subject: [Python-ideas] inheriting docstrings and mutable docstings for
	classes
In-Reply-To: <BANLkTinhbrrUkGB7TA-Pxo7NRLumQiCUYw@mail.gmail.com>
References: <BANLkTik=nQ=DwKk-UPfzuuo4SoD7koa1ZA@mail.gmail.com>
	<BANLkTimGVxe0m=txJtEr6=JMpkBOo7ahEQ@mail.gmail.com>
	<BANLkTinhbrrUkGB7TA-Pxo7NRLumQiCUYw@mail.gmail.com>
Message-ID: <BANLkTi=QQDoVS6yxFQB0GK8eLWeQV9F8GA@mail.gmail.com>

On Fri, Jun 10, 2011 at 11:45 AM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> Agreed that docstrings should be writeable after the fact, though
>> (e.g. functions already work that way - functools.wraps wouldn't work
>> otherwise).
>>
>
> Would this be a very controversial change? ?I ask because it's been
> this way since 2.2 and no one's changed it. ?Thanks.

More likely a matter of nobody needing the functionality enough to
question the behaviour (there are a variety of arbitrary restrictions
still floating around that don't really have a *reason*, it's just
that it was easier to do it that way initially and nobody has cared
enough to change it).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From arnodel at gmail.com  Fri Jun 10 09:28:13 2011
From: arnodel at gmail.com (Arnaud Delobelle)
Date: Fri, 10 Jun 2011 08:28:13 +0100
Subject: [Python-ideas] inheriting docstrings and mutable docstings for
	classes
In-Reply-To: <4DF1C200.9090405@canterbury.ac.nz>
References: <BANLkTik=nQ=DwKk-UPfzuuo4SoD7koa1ZA@mail.gmail.com>
	<BANLkTimGVxe0m=txJtEr6=JMpkBOo7ahEQ@mail.gmail.com>
	<4DF1C200.9090405@canterbury.ac.nz>
Message-ID: <BANLkTikFyP=630E5KA-2fH71ifgwYA0AgA@mail.gmail.com>

On 10 June 2011 08:04, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
[...]
> Maybe the best thing would be for the inherited docstring
> to get put into a different property, such as __basedoc__.
> Then tools that examine docstrings can decide for themselves
> whether using inherited docstrings makes sense.

Given that Python supports multiple inheritance, which parent class's
__doc__ would the __basedoc__ contain?  Also, what would the
__basedoc__ contain if the parent's __doc__ is empty but not its
__basedoc__?

-- 
Arnaud


From dstanek at dstanek.com  Fri Jun 10 12:37:23 2011
From: dstanek at dstanek.com (David Stanek)
Date: Fri, 10 Jun 2011 06:37:23 -0400
Subject: [Python-ideas] inheriting docstrings and mutable docstings for
	classes
In-Reply-To: <4DF1C200.9090405@canterbury.ac.nz>
References: <BANLkTik=nQ=DwKk-UPfzuuo4SoD7koa1ZA@mail.gmail.com>
	<BANLkTimGVxe0m=txJtEr6=JMpkBOo7ahEQ@mail.gmail.com>
	<4DF1C200.9090405@canterbury.ac.nz>
Message-ID: <BANLkTinQf9mAfgxBuVYL8TmNoC571_V-7Q@mail.gmail.com>

On Fri, Jun 10, 2011 at 3:04 AM, Greg Ewing <greg.ewing at canterbury.ac.nz>wrote:

>
> Maybe the best thing would be for the inherited docstring
> to get put into a different property, such as __basedoc__.
> Then tools that examine docstrings can decide for themselves
> whether using inherited docstrings makes sense.
>
>
How would a tool know if the behavior of a method changed without analyzing
the code? I think this could very easily lead to a situation where a
project's generated documentation is incorrect.

-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek
www: http://dstanek.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110610/b58ae078/attachment.html>

From steve at pearwood.info  Fri Jun 10 13:22:15 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 10 Jun 2011 21:22:15 +1000
Subject: [Python-ideas] inheriting docstrings and mutable docstings for
 classes
In-Reply-To: <BANLkTinQf9mAfgxBuVYL8TmNoC571_V-7Q@mail.gmail.com>
References: <BANLkTik=nQ=DwKk-UPfzuuo4SoD7koa1ZA@mail.gmail.com>	<BANLkTimGVxe0m=txJtEr6=JMpkBOo7ahEQ@mail.gmail.com>	<4DF1C200.9090405@canterbury.ac.nz>
	<BANLkTinQf9mAfgxBuVYL8TmNoC571_V-7Q@mail.gmail.com>
Message-ID: <4DF1FE67.5060005@pearwood.info>

David Stanek wrote:
> On Fri, Jun 10, 2011 at 3:04 AM, Greg Ewing <greg.ewing at canterbury.ac.nz>wrote:
> 
>> Maybe the best thing would be for the inherited docstring
>> to get put into a different property, such as __basedoc__.
>> Then tools that examine docstrings can decide for themselves
>> whether using inherited docstrings makes sense.
>>
>>
> How would a tool know if the behavior of a method changed without analyzing
> the code? I think this could very easily lead to a situation where a
> project's generated documentation is incorrect.


That's the developer's problem, and no different from any other case 
where you inherit data without ensuring it is the correct data.


class Parrot:
     colour = 'green'
     def speak(self):
         return "Polly wants a cracker."

class NorwegianBlue(Parrot):
     def speak(self):
         return "I'm pining for the fjords."

assert NorwegianBlue().colour == 'blue'  # oops!


-- 
Steven


From dstanek at dstanek.com  Fri Jun 10 13:40:48 2011
From: dstanek at dstanek.com (David Stanek)
Date: Fri, 10 Jun 2011 07:40:48 -0400
Subject: [Python-ideas] inheriting docstrings and mutable docstings for
	classes
In-Reply-To: <4DF1FE67.5060005@pearwood.info>
References: <BANLkTik=nQ=DwKk-UPfzuuo4SoD7koa1ZA@mail.gmail.com>
	<BANLkTimGVxe0m=txJtEr6=JMpkBOo7ahEQ@mail.gmail.com>
	<4DF1C200.9090405@canterbury.ac.nz>
	<BANLkTinQf9mAfgxBuVYL8TmNoC571_V-7Q@mail.gmail.com>
	<4DF1FE67.5060005@pearwood.info>
Message-ID: <BANLkTi=7rK4iCDj3jkzP_Ma+RpR-i8Ljzg@mail.gmail.com>

On Fri, Jun 10, 2011 at 7:22 AM, Steven D'Aprano <steve at pearwood.info>wrote:

> David Stanek wrote:
>>
>> How would a tool know if the behavior of a method changed without
>> analyzing
>> the code? I think this could very easily lead to a situation where a
>> project's generated documentation is incorrect.
>>
>
>
> That's the developer's problem, and no different from any other case where
> you inherit data without ensuring it is the correct data.
>

This means that a significant amount of existing code may/will have problems
with this type of change. If Sphinx did this automatically I'm sure there
would be lots of incorrect documentation. I think the developer should
explicitly carry over and modify the documentation if necessary.


-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek
www: http://dstanek.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110610/45baa4a2/attachment.html>

From steve at pearwood.info  Fri Jun 10 14:17:25 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 10 Jun 2011 22:17:25 +1000
Subject: [Python-ideas] inheriting docstrings and mutable docstings for
 classes
In-Reply-To: <BANLkTi=7rK4iCDj3jkzP_Ma+RpR-i8Ljzg@mail.gmail.com>
References: <BANLkTik=nQ=DwKk-UPfzuuo4SoD7koa1ZA@mail.gmail.com>	<BANLkTimGVxe0m=txJtEr6=JMpkBOo7ahEQ@mail.gmail.com>	<4DF1C200.9090405@canterbury.ac.nz>	<BANLkTinQf9mAfgxBuVYL8TmNoC571_V-7Q@mail.gmail.com>	<4DF1FE67.5060005@pearwood.info>
	<BANLkTi=7rK4iCDj3jkzP_Ma+RpR-i8Ljzg@mail.gmail.com>
Message-ID: <4DF20B55.2080504@pearwood.info>

David Stanek wrote:
> On Fri, Jun 10, 2011 at 7:22 AM, Steven D'Aprano <steve at pearwood.info>wrote:
> 
>> David Stanek wrote:
>>> How would a tool know if the behavior of a method changed without
>>> analyzing
>>> the code? I think this could very easily lead to a situation where a
>>> project's generated documentation is incorrect.
>>>
>>
>> That's the developer's problem, and no different from any other case where
>> you inherit data without ensuring it is the correct data.
>>
> 
> This means that a significant amount of existing code may/will have problems
> with this type of change. If Sphinx did this automatically I'm sure there
> would be lots of incorrect documentation. I think the developer should
> explicitly carry over and modify the documentation if necessary.


Oh, I'm sorry if I gave the impression that we should inherit docstrings 
by default. That wasn't my intention. I was merely suggesting that we 
shouldn't let the risk of misuse discourage us from a potential change.


-- 
Steven



From digitalxero at gmail.com  Fri Jun 10 15:29:18 2011
From: digitalxero at gmail.com (Dj Gilcrease)
Date: Fri, 10 Jun 2011 09:29:18 -0400
Subject: [Python-ideas] inheriting docstrings and mutable docstings for
	classes
In-Reply-To: <BANLkTikFyP=630E5KA-2fH71ifgwYA0AgA@mail.gmail.com>
References: <BANLkTik=nQ=DwKk-UPfzuuo4SoD7koa1ZA@mail.gmail.com>
	<BANLkTimGVxe0m=txJtEr6=JMpkBOo7ahEQ@mail.gmail.com>
	<4DF1C200.9090405@canterbury.ac.nz>
	<BANLkTikFyP=630E5KA-2fH71ifgwYA0AgA@mail.gmail.com>
Message-ID: <BANLkTinN4+-y9rrnHhydc9EqM_w32BKgFg@mail.gmail.com>

On Fri, Jun 10, 2011 at 3:28 AM, Arnaud Delobelle <arnodel at gmail.com> wrote:
> Given that Python supports multiple inheritance, which parent class's
> __doc__ would the __basedoc__ contain? ?Also, what would the
> __basedoc__ contain if the parent's __doc__ is empty but not its
> __basedoc__?

You would not need a __basedoc__ magic attribute, you can just do

class C(object): pass

docs = [c.__doc__ for c in C.__mro__]

and you get the docs for all bases in the proper mro order

Though I guess __basedocs__ mapping to [c.__doc__ for c in C.__mro__
if c != C] could be handy


From ericsnowcurrently at gmail.com  Fri Jun 10 20:40:40 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 10 Jun 2011 12:40:40 -0600
Subject: [Python-ideas] inheriting docstrings
Message-ID: <BANLkTim7MY6c1i92ad0e-y+QNYwp8q4KNg@mail.gmail.com>

On Fri, Jun 10, 2011 at 7:29 AM, Dj Gilcrease <digitalxero at gmail.com> wrote:
> Though I guess __basedocs__ mapping to [c.__doc__ for c in C.__mro__
> if c != C] could be handy

Or: next(c.__doc__ for c in C.__mro__[1:] if c.__doc__) or None

Right now you could do something like this:

    def get_basedoc(mro):
        return next(c.__doc__ for c in mro[1:] if c.__doc__) or None

    class Meta(type):
        __basedoc__ = property(lambda cls: get_basedoc(cls.__mro__))

But then instances don't get __basedoc__, since the metaclass is not
in the MRO.  To get it on instances you could do this:

    class C:
        __basedoc__ = property(lambda self: get_basedoc(self.__class__.__mro__))

But then getting that attribute on the class will give you the
property object and not the docstring.  I'm not sure of a way to
resolve that.

However, inheriting the docstring between classes is only part of the
problem.  You'll probably want to have the functions on a class also
"inherit" their docstring from the first matching attribute of the
bases on the MRO.

In that case the function will not have enough information (the class
isn't available to the function) to do the lookup.  Instead, the class
would have to do it.  Here's an example:

    class BasedocMethod:
        def __init__(self, f, cls):
            self.f = f
            self.cls = cls
        def __getattribute__(self, name):
            if name == "__basedoc__":
                return object.__getattribute__(self, "__basedoc__")
            return getattr(f, name)
        @property
        def __basedoc__(self):
            for base in self.cls.__mro__:
                basefunc = base.__dict__.get(self.f.__name__)
                if not basefunc:
                    continue
                return getattr(basefunc, "__doc__", None)
            return None

    class Meta(type):
        def __init__(cls, name, bases, namespace):
            for attrname, obj in namespace.items():
                if isinstance(obj, FunctionType):
                    setattr(cls, attrname, BasedocMethod(obj, cls))

Obviously not a perfect example, but hopefully demonstrates the idea.
The whole point is that it would be nice to have a mechanism built in
that makes it easy to inherit docstrings to functions, without
necessarily implicitly inheriting them.  The__basedoc__ idea works for
that.

My main motivation for this docstring inheritance is primarily for the
case of abstract base classes.  My implementations of those rarely
have docstrings unique from that of the base class, nor do the
methods.  I can get around this now with metaclasses and class
decorators, but it would be nice to have a little more help from the
language.  It would always be nice.  :)

-eric

> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From ericsnowcurrently at gmail.com  Fri Jun 10 21:08:08 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 10 Jun 2011 13:08:08 -0600
Subject: [Python-ideas] inheriting docstrings
In-Reply-To: <BANLkTim7MY6c1i92ad0e-y+QNYwp8q4KNg@mail.gmail.com>
References: <BANLkTim7MY6c1i92ad0e-y+QNYwp8q4KNg@mail.gmail.com>
Message-ID: <BANLkTin7MOwLQxwg5x25tY=ycfZZMPmgqQ@mail.gmail.com>

On Fri, Jun 10, 2011 at 12:40 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> Right now you could do something like this:
>
> ? ?def get_basedoc(mro):
> ? ? ? ?return next(c.__doc__ for c in mro[1:] if c.__doc__) or None
>
> ? ?class Meta(type):
> ? ? ? ?__basedoc__ = property(lambda cls: get_basedoc(cls.__mro__))
>
> But then instances don't get __basedoc__, since the metaclass is not
> in the MRO. ?To get it on instances you could do this:
>
> ? ?class C:
> ? ? ? ?__basedoc__ = property(lambda self: get_basedoc(self.__class__.__mro__))
>
> But then getting that attribute on the class will give you the
> property object and not the docstring.
>

> I'm not sure of a way to resolve that.

Duh, someone just pointed out that you use your own descriptor instead
of a property:

class Basedoc:
    def __get__(self, obj, cls):
 ? ? ? ?return next(c.__doc__ for c in cls.__mro__[1:] if c.__doc__) or None

class C:
    __basedoc__ = Basedoc()

Inherit from the class or add it onto the class with a class decorator
or metaclass.

-eric


From ericsnowcurrently at gmail.com  Fri Jun 10 23:39:12 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 10 Jun 2011 15:39:12 -0600
Subject: [Python-ideas] inheriting docstrings
In-Reply-To: <BANLkTin7MOwLQxwg5x25tY=ycfZZMPmgqQ@mail.gmail.com>
References: <BANLkTim7MY6c1i92ad0e-y+QNYwp8q4KNg@mail.gmail.com>
	<BANLkTin7MOwLQxwg5x25tY=ycfZZMPmgqQ@mail.gmail.com>
Message-ID: <BANLkTikwtECthzTh+SWfdJhvQsagjcEb=g@mail.gmail.com>

On Fri, Jun 10, 2011 at 1:04 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Maybe the best thing would be for the inherited docstring
> to get put into a different property, such as __basedoc__.
> Then tools that examine docstrings can decide for themselves
> whether using inherited docstrings makes sense.

Another idea that I like, that someone suggested on python-list [1],
is using the empty string to indicate that you want a docstring to be
inherited.  Here's an approximate implementation using a metaclass:

    class DocDescriptor:
        # as a non-data descriptor
        # but how to make it a data descriptor for the class?
        def __init__(self, docstring):
            self.docstring = docstring
        def __get__(self, obj, cls):
            if self.docstring != '':
                return self.docstring
            return next(c.__doc__ for c in cls.__mro__[1:] if c.__doc__) or ''

    class DocMethod:
        def __init__(self, f, cls):
            self.f = f
            self.cls = cls
        def __getattribute__(self, name):
            if name == '__doc__':
                return object.__getattribute__(self, '__doc__')
            f = object.__getattribute__(self, 'f')
            return getattr(f, name)
        @property
        def __doc__(self):
            f = object.__getattribute__(self, 'f')
            cls = object.__getattribute__(self, 'cls')
            if f.__doc__ != '':
                return f.__doc__
            for base in cls.__mro__:
                basefunc = base.__dict__.get(self.f.__name__)
                if not basefunc:
                    continue
                docstring = getattr(basefunc, '__doc__', None)
                if not docstring:
                    continue
                return docstring
            return ''
        @__doc__.setter
        def __doc__(self, value):
            object.__getattribute__(self, 'f').__doc__ = value

    class Meta(type):
        def __init__(cls, name, bases, namespace):
            docstring = namespace.get('__doc__')
            cls.__doc__ = DocDescriptor(docstring)
            for attrname, obj in namespace.items():
                if isinstance(obj, FunctionType):
                    setattr(cls, attrname, DocMethod(obj, cls))


-eric

[1] http://mail.python.org/pipermail/python-list/2011-June/1274123.html


From zuo at chopin.edu.pl  Sat Jun 11 13:02:02 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Sat, 11 Jun 2011 13:02:02 +0200
Subject: [Python-ideas] inheriting docstrings
In-Reply-To: <BANLkTim7MY6c1i92ad0e-y+QNYwp8q4KNg@mail.gmail.com>
References: <BANLkTim7MY6c1i92ad0e-y+QNYwp8q4KNg@mail.gmail.com>
Message-ID: <20110611110202.GA2395@chopin.edu.pl>

+1 from me for writable (not mutable of course) class __doc__

-1 from me for all that more or less implicit doc inheritance.

Adding some decorator(s) to functools would be much better IMHO, e.g.:

    class MyDict(dict):
    
        @functools.basedoc(dict)
        def __setitem__(self, key, value):
            super(dict, self).__setitem__(key, value)
            ...

or:

    @functools.superdocs  # for methods without docstrings
    class MyDict(dict):
    
        def __setitem__(self, key, value):
            super(dict, self).__setitem__(key, value)
            ...

Cheers.
*j



From zuo at chopin.edu.pl  Sat Jun 11 13:59:23 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Sat, 11 Jun 2011 13:59:23 +0200
Subject: [Python-ideas] inheriting docstrings
In-Reply-To: <20110611110202.GA2395@chopin.edu.pl>
References: <BANLkTim7MY6c1i92ad0e-y+QNYwp8q4KNg@mail.gmail.com>
	<20110611110202.GA2395@chopin.edu.pl>
Message-ID: <20110611115923.GB2395@chopin.edu.pl>

Jan Kaliszewski dixit (2011-06-11, 13:02):

> +1 from me for writable (not mutable of course) class __doc__
> 
> -1 from me for all that more or less implicit doc inheritance.
> 
> Adding some decorator(s) to functools would be much better IMHO, e.g.:
> 
>     class MyDict(dict):
>     
>         @functools.basedoc(dict)
>         def __setitem__(self, key, value):
>             super(dict, self).__setitem__(key, value)
>             ...
> 
> or:
> 
>     @functools.superdocs  # for methods without docstrings
>     class MyDict(dict):
>     
>         def __setitem__(self, key, value):
>             super(dict, self).__setitem__(key, value)
>             ...

Sorry,
  s/ super(dict/ super(MyDict/
of course.


From zuo at chopin.edu.pl  Sat Jun 11 14:09:07 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Sat, 11 Jun 2011 14:09:07 +0200
Subject: [Python-ideas] inheriting docstrings
In-Reply-To: <20110611110202.GA2395@chopin.edu.pl>
References: <BANLkTim7MY6c1i92ad0e-y+QNYwp8q4KNg@mail.gmail.com>
	<20110611110202.GA2395@chopin.edu.pl>
Message-ID: <20110611120907.GC2395@chopin.edu.pl>

Probably better names...

>     class MyDict(dict):
>     
>         @functools.basedoc(dict)
>         def __setitem__(self, key, value):
>             super(dict, self).__setitem__(key, value)
>             ...

'docfrom' instead of 'basedoc'
or maybe:
'inheritingdoc' or 'derivdoc'?

>     @functools.superdocs  # for methods without docstrings
>     class MyDict(dict):
>     
>         def __setitem__(self, key, value):
>             super(dict, self).__setitem__(key, value)
>             ...

'docfromsuper' instead of 'superdocs'

Cheers.
*j



From zuo at chopin.edu.pl  Sat Jun 11 15:30:29 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Sat, 11 Jun 2011 15:30:29 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
Message-ID: <20110611133028.GD2395@chopin.edu.pl>

== Use cases ==

A quite common practice is 'injecting' objects into a function as its
locals, at def-time, using function arguments with default values...

Sometimes to keep state using a mutable container:

    def do_and_remember(val, verbose=False, mem=collections.Counter()):
        result = do_something(val)
        mem[val] += 1
        if verbose:
            print('Done {} times for {!r}'.format(mem[val], val))

Sometimes, when creating functions dynamically (making use of nested
scopes), e.g. to keep some individual function features (usable within
that functions):

    def make_my_callbacks(callback_params):
        my_callbacks = []
        for params in callback_params:
            def fun1(*args, _params=params, **kwargs):
                "...do something with args and params..."
            def fun2(*args, _params=params, **kwargs):
                "...do something with args and params..."
            def fun3(*args, _fun1=fun1, _fun2=fun2, **kwargs):
                """...do something with args and with functions fun1, fun2,
                for example pass them as callbacks to other functions..."
            my_callbacks.append((fun1, fun2, fun3))
        return my_callbacks

Sometimes simply to make critical parts of code optimised...

    def do_it_quickly(fields, _len=len, _split=str.split,
                      _sth=something):
        return [_len(f), _split(f), _sth(f) for f in fields]

...or even for readability -- keeping function-specific constants within
the function definition:

    def check_value(val,
                    VAL_REGEX=re.compile('^...$'),
                    VAL_MAX_LEN=38):
        return len(val) <= VAL_MAX_LEN and VAL_RE.search(val) is not None

In all that cases (and probably some other too) that technique appears
to be quite useful.


== The problem ==

...is that it is not very elegant. We add arguments which:
a) mess up function signatures (both in the code and in auto-generated docs);
b) can be incidentally overriden (especially when a function has an "open"
   signature with **kwargs).


== Proposed solutions ==

I see three possibilities:

1.
To add a new keyword, e.g. `inject':
    def do_and_remember(val, verbose=False):
        inject mem = collections.Counter()
        ...
or maybe:
    def do_and_remember(val, verbose=False):
        inject collections.Counter() as mem
        ...

2. (which personally I would prefer)
To add `dummy' (or `hidden') keyword arguments, defined after **kwargs
(and after bare ** if kwargs are not needed; we have already have
keyword-only arguments after *args or bare *):

    def do_and_remember(val, verbose=False, **, mem=collections.Counter()):
        ...

do_and_remember(val, False, mem='something') would raise TypeError and
`mem' shoudn not appear in help() etc. as a function argument.

3.
To provide a special decorator, e.g. functools.within:
    @functools.within(mem=collections.Counter())
    def do_and_remember(val, verbose=False):
        ...


Regards.
*j



From g.rodola at gmail.com  Sat Jun 11 20:51:56 2011
From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=)
Date: Sat, 11 Jun 2011 20:51:56 +0200
Subject: [Python-ideas] Adding shutil.disk_usage()
Message-ID: <BANLkTimBgYw=XmNQFDbFOH2y5P_qS248yQ@mail.gmail.com>

Hi all,
I've just implemented this functionality in psutil for both POSIX and
Windows and thought it might be nice to have it in shutil module as
well since it's useful when doing system monitoring:
http://code.google.com/p/psutil/issues/detail?id=172

The posix implementation is nothing but a wrapper around os.statvfs():

def disk_usage(path):
    """Return disk usage associated with path."""
    st = os.statvfs(path)
    free = (st.f_bavail * st.f_frsize)
    total = (st.f_blocks * st.f_frsize)
    used = (st.f_blocks - st.f_bfree) * st.f_frsize
    percent = (float(used) / total) * 100
    # NB: the percentage is -5% than what shown by df due to
    # reserved blocks that we are currently not considering:
    # http://goo.gl/sWGbH
    return ntuple_diskinfo(total, used, free, round(percent, 1))

...and reflects what returned by "df /somepath".
The Windows implementation requires GetDiskFreeSpaceEx() which is not
exposed in python stdlib but can be added as a privade module
(Modules/_winutil.c maybe?) or retrieved via ctypes.

Thoughts?


--- Giampaolo
http://code.google.com/p/pyftpdlib
http://code.google.com/p/psutil


From tjreedy at udel.edu  Sat Jun 11 22:09:17 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 11 Jun 2011 16:09:17 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110611133028.GD2395@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>
Message-ID: <it0i1c$2es$1@dough.gmane.org>

On 6/11/2011 9:30 AM, Jan Kaliszewski wrote:
> == Use cases ==
>
> A quite common practice is 'injecting' objects into a function as its
> locals, at def-time, using function arguments with default values...
>
> Sometimes to keep state using a mutable container:
>
>      def do_and_remember(val, verbose=False, mem=collections.Counter()):
>          result = do_something(val)
>          mem[val] += 1
>          if verbose:
>              print('Done {} times for {!r}'.format(mem[val], val))
>
> Sometimes, when creating functions dynamically (making use of nested
> scopes), e.g. to keep some individual function features (usable within
> that functions):
>
>      def make_my_callbacks(callback_params):
>          my_callbacks = []
>          for params in callback_params:
>              def fun1(*args, _params=params, **kwargs):
>                  "...do something with args and params..."
>              def fun2(*args, _params=params, **kwargs):
>                  "...do something with args and params..."
>              def fun3(*args, _fun1=fun1, _fun2=fun2, **kwargs):
>                  """...do something with args and with functions fun1, fun2,
>                  for example pass them as callbacks to other functions..."
>              my_callbacks.append((fun1, fun2, fun3))
>          return my_callbacks
>
> Sometimes simply to make critical parts of code optimised...
>
>      def do_it_quickly(fields, _len=len, _split=str.split,
>                        _sth=something):
>          return [_len(f), _split(f), _sth(f) for f in fields]
>
> ...or even for readability -- keeping function-specific constants within
> the function definition:
>
>      def check_value(val,
>                      VAL_REGEX=re.compile('^...$'),
>                      VAL_MAX_LEN=38):
>          return len(val)<= VAL_MAX_LEN and VAL_RE.search(val) is not None
>
> In all that cases (and probably some other too) that technique appears
> to be quite useful.
>
>
> == The problem ==
>
> ...is that it is not very elegant. We add arguments which:
> a) mess up function signatures (both in the code and in auto-generated docs);
> b) can be incidentally overriden (especially when a function has an "open"
>     signature with **kwargs).

One problem with trying to 'fix' this is that there can be defaulted args
which are not intended to be overwritten by users but which are intended to
be replaced in recursive calls.

> == Proposed solutions ==
>
> I see three possibilities:
>
> 1.
> To add a new keyword, e.g. `inject':
>      def do_and_remember(val, verbose=False):
>          inject mem = collections.Counter()
>          ...
> or maybe:
>      def do_and_remember(val, verbose=False):
>          inject collections.Counter() as mem

The body should all be runtime. Deftime expression should be in the header.

> 2. (which personally I would prefer)
> To add `dummy' (or `hidden') keyword arguments, defined after **kwargs
> (and after bare ** if kwargs are not needed; we have already have
> keyword-only arguments after *args or bare *):
>
>      def do_and_remember(val, verbose=False, **, mem=collections.Counter()):
>          ...

I thought of this while reading 'the problem'. It is at least plausible 
to me.

> do_and_remember(val, False, mem='something') would raise TypeError and
> `mem' shoudn not appear in help() etc. as a function argument.
>
> 3.
> To provide a special decorator, e.g. functools.within:
>      @functools.within(mem=collections.Counter())
>      def do_and_remember(val, verbose=False):
>          ...

The decorator would have to modify the code object as well as the 
function objects, probably in ways not currently allowed.

-- 
Terry Jan Reedy



From arnodel at gmail.com  Sat Jun 11 22:47:59 2011
From: arnodel at gmail.com (Arnaud Delobelle)
Date: Sat, 11 Jun 2011 21:47:59 +0100
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110611133028.GD2395@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>
Message-ID: <4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>


On 11 Jun 2011, at 14:30, Jan Kaliszewski wrote:

> == Use cases ==
> 
> A quite common practice is 'injecting' objects into a function as its
> locals, at def-time, using function arguments with default values...
> 
> Sometimes to keep state using a mutable container:
> 
>    def do_and_remember(val, verbose=False, mem=collections.Counter()):
>        result = do_something(val)
>        mem[val] += 1
>        if verbose:
>            print('Done {} times for {!r}'.format(mem[val], val))
> 
> Sometimes, when creating functions dynamically (making use of nested
> scopes), e.g. to keep some individual function features (usable within
> that functions):
> 
>    def make_my_callbacks(callback_params):
>        my_callbacks = []
>        for params in callback_params:
>            def fun1(*args, _params=params, **kwargs):
>                "...do something with args and params..."
>            def fun2(*args, _params=params, **kwargs):
>                "...do something with args and params..."
>            def fun3(*args, _fun1=fun1, _fun2=fun2, **kwargs):
>                """...do something with args and with functions fun1, fun2,
>                for example pass them as callbacks to other functions..."
>            my_callbacks.append((fun1, fun2, fun3))
>        return my_callbacks
> 
> Sometimes simply to make critical parts of code optimised...
> 
>    def do_it_quickly(fields, _len=len, _split=str.split,
>                      _sth=something):
>        return [_len(f), _split(f), _sth(f) for f in fields]
> 
> ...or even for readability -- keeping function-specific constants within
> the function definition:
> 
>    def check_value(val,
>                    VAL_REGEX=re.compile('^...$'),
>                    VAL_MAX_LEN=38):
>        return len(val) <= VAL_MAX_LEN and VAL_RE.search(val) is not None
> 
> In all that cases (and probably some other too) that technique appears
> to be quite useful.
> 
> 
> == The problem ==
> 
> ...is that it is not very elegant. We add arguments which:
> a) mess up function signatures (both in the code and in auto-generated docs);
> b) can be incidentally overriden (especially when a function has an "open"
>   signature with **kwargs).
> 
> 
> == Proposed solutions ==
> 
> I see three possibilities:
> 
> 1.
> To add a new keyword, e.g. `inject':
>    def do_and_remember(val, verbose=False):
>        inject mem = collections.Counter()
>        ...
> or maybe:
>    def do_and_remember(val, verbose=False):
>        inject collections.Counter() as mem
>        ...
> 
> 2. (which personally I would prefer)
> To add `dummy' (or `hidden') keyword arguments, defined after **kwargs
> (and after bare ** if kwargs are not needed; we have already have
> keyword-only arguments after *args or bare *):
> 
>    def do_and_remember(val, verbose=False, **, mem=collections.Counter()):
>        ...
> 
> do_and_remember(val, False, mem='something') would raise TypeError and
> `mem' shoudn not appear in help() etc. as a function argument.
> 
> 3.
> To provide a special decorator, e.g. functools.within:
>    @functools.within(mem=collections.Counter())
>    def do_and_remember(val, verbose=False):
>        ...

That's hard to do as (assuming the function is defined at the global scope), mem will be compiled as a global, meaning that you will have to modify the bytecode.  Oh but this makes me think about something I wrote a while ago (see below).


4. Use closures.

def factory(mem):
     def do_and_remember(val, verbose=False)
         result = do_something(val)
         mem[val] += 1
         if verbose:
             print('Done {} times for {!r}'.format(mem[val], val))         ....
     return do_and_remember
do_and_remember = factory(mem=collections.Counter())

Added bonus: you can create many instances of do_and_remember.


----------

Related to this, here's a "localize" decorator that I wrote some time ago for fun (I think it was from a discussion on this list).  It was for python 2.x (could easily be modified for 3.x I think, it's a matter of adapting the attribute names of the function object).  It "freezes" all non local variables in the function.  It's a hack! It may be possible to adapt it.

def new_closure(vals):
    args = ','.join('x%i' % i for i in range(len(vals)))
    f = eval("lambda %s:lambda:(%s)" % (args, args))
    return f(*vals).func_closure

def localize(f):
    f_globals = dict((n, f.func_globals[n]) for n in f.func_code.co_names)
    f_closure = ( f.func_closure and
                  new_closure([c.cell_contents for c in f.func_closure]) )
    return type(f)(f.func_code, f_globals, f.func_name,
                   f.func_defaults, f_closure)

# Examples of how localize works:

x, y = 1, 2
@localize
def f():
    return x + y

def test():
    acc = []
    for i in range(10):
        @localize
        def pr(): print i
        acc.append(pr)
    return acc

def lambdatest():
    return [localize(lambda: i) for i in range(10)]

# These examples will behave as follows:
>>> f()
3
>>> x = 3
>>> f()
3
>>> pr = test()
>>> pr[0]()
0
>>> pr[5]()
5
>>> l = lambdatest()
>>> l[2]()
2
>>> l[7]()
7
>>> 


-- 
Arnaud



From greg at krypto.org  Sun Jun 12 02:15:08 2011
From: greg at krypto.org (Gregory P. Smith)
Date: Sat, 11 Jun 2011 17:15:08 -0700
Subject: [Python-ideas] Adding shutil.disk_usage()
In-Reply-To: <BANLkTimBgYw=XmNQFDbFOH2y5P_qS248yQ@mail.gmail.com>
References: <BANLkTimBgYw=XmNQFDbFOH2y5P_qS248yQ@mail.gmail.com>
Message-ID: <BANLkTinFMcNRtEPVqP291kxanB82U9fSew@mail.gmail.com>

On Sat, Jun 11, 2011 at 11:51 AM, Giampaolo Rodol? <g.rodola at gmail.com>wrote:

> Hi all,
> I've just implemented this functionality in psutil for both POSIX and
> Windows and thought it might be nice to have it in shutil module as
> well since it's useful when doing system monitoring:
> http://code.google.com/p/psutil/issues/detail?id=172
>
> The posix implementation is nothing but a wrapper around os.statvfs():
>
> def disk_usage(path):
>    """Return disk usage associated with path."""
>    st = os.statvfs(path)
>    free = (st.f_bavail * st.f_frsize)
>    total = (st.f_blocks * st.f_frsize)
>    used = (st.f_blocks - st.f_bfree) * st.f_frsize
>    percent = (float(used) / total) * 100
>    # NB: the percentage is -5% than what shown by df due to
>    # reserved blocks that we are currently not considering:
>    # http://goo.gl/sWGbH
>    return ntuple_diskinfo(total, used, free, round(percent, 1))
>
> ...and reflects what returned by "df /somepath".
> The Windows implementation requires GetDiskFreeSpaceEx() which is not
> exposed in python stdlib but can be added as a privade module
> (Modules/_winutil.c maybe?) or retrieved via ctypes.
>
> Thoughts?
>

Makes sense to me.  Though I would personally leave the percent calculation
up to the caller or at least leave the rounding to the caller.

I'll leave opinion on which implementation to use on windows up to someone
more familiar with that platform.

Attach your patch(es) implementing it to a feature request issue on
bugs.python.org.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110611/434b8e8b/attachment.html>

From brian.curtin at gmail.com  Sun Jun 12 03:40:38 2011
From: brian.curtin at gmail.com (Brian Curtin)
Date: Sat, 11 Jun 2011 20:40:38 -0500
Subject: [Python-ideas] Adding shutil.disk_usage()
In-Reply-To: <BANLkTimBgYw=XmNQFDbFOH2y5P_qS248yQ@mail.gmail.com>
References: <BANLkTimBgYw=XmNQFDbFOH2y5P_qS248yQ@mail.gmail.com>
Message-ID: <BANLkTi=gcmFqjP40oFhDkKy==hxjjAV8vg@mail.gmail.com>

On Sat, Jun 11, 2011 at 13:51, Giampaolo Rodol? <g.rodola at gmail.com> wrote:

> Hi all,
> I've just implemented this functionality in psutil for both POSIX and
> Windows and thought it might be nice to have it in shutil module as
> well since it's useful when doing system monitoring:
> http://code.google.com/p/psutil/issues/detail?id=172
>
> The posix implementation is nothing but a wrapper around os.statvfs():
>
> def disk_usage(path):
>    """Return disk usage associated with path."""
>    st = os.statvfs(path)
>    free = (st.f_bavail * st.f_frsize)
>    total = (st.f_blocks * st.f_frsize)
>    used = (st.f_blocks - st.f_bfree) * st.f_frsize
>    percent = (float(used) / total) * 100
>    # NB: the percentage is -5% than what shown by df due to
>    # reserved blocks that we are currently not considering:
>    # http://goo.gl/sWGbH
>    return ntuple_diskinfo(total, used, free, round(percent, 1))
>
> ...and reflects what returned by "df /somepath".
> The Windows implementation requires GetDiskFreeSpaceEx() which is not
> exposed in python stdlib but can be added as a privade module
> (Modules/_winutil.c maybe?) or retrieved via ctypes.


The GetDiskFreeSpaceEx call should just be exposed within
Modules/posixmodule.c. See the posix__getfinalpathname function for an
example.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110611/afdd0cc9/attachment.html>

From efotinis at yahoo.com  Sun Jun 12 08:23:41 2011
From: efotinis at yahoo.com (Elias Fotinis)
Date: Sun, 12 Jun 2011 09:23:41 +0300
Subject: [Python-ideas] Adding shutil.disk_usage()
References: <BANLkTimBgYw=XmNQFDbFOH2y5P_qS248yQ@mail.gmail.com>
Message-ID: <op.vwx9drqcd4rfj9@efcore>

On Sat, 11 Jun 2011 21:51:56 +0300, Giampaolo Rodol? <g.rodola-Re5JQEeQqe8AvxtiuMwx3w at public.gmane.org> wrote:

> The posix implementation is nothing but a wrapper around os.statvfs():
[...]
> The Windows implementation requires GetDiskFreeSpaceEx() [...]

I'd suggest mentioning in the documentation that symbolic links are resolved for the supplied path. I know that's fact for GetDiskFreeSpaceEx() and a quick look suggests it's also the case for statvfs(). It would be nice to be aware of this and the fact that it works the same on both platforms.



From ncoghlan at gmail.com  Sun Jun 12 17:44:31 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 13 Jun 2011 01:44:31 +1000
Subject: [Python-ideas] PEP 3150 (statement local namespaces) updated based
	on April discussion
Message-ID: <BANLkTinVg_i0y0DTubNLG5fTBNAt+DVahg@mail.gmail.com>

The rationale and proposed semantics sections of PEP 3150 have been
heavily modified based on the various discussions on the topic back in
April.

http://www.python.org/dev/peps/pep-3150/

(Note that the PEP is still officially Deferred - this entire idea
still intrigues me, but I'm also still not convinced it's worth the
additional language complexity).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From zuo at chopin.edu.pl  Sun Jun 12 18:26:29 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Sun, 12 Jun 2011 18:26:29 +0200
Subject: [Python-ideas] PEP 3150 (statement local namespaces) updated
 based on April discussion
In-Reply-To: <BANLkTinVg_i0y0DTubNLG5fTBNAt+DVahg@mail.gmail.com>
References: <BANLkTinVg_i0y0DTubNLG5fTBNAt+DVahg@mail.gmail.com>
Message-ID: <20110612162629.GB4263@chopin.edu.pl>

Nick Coghlan dixit (2011-06-13, 01:44):

> The rationale and proposed semantics sections of PEP 3150 have been
> heavily modified based on the various discussions on the topic back in
> April.
> 
> http://www.python.org/dev/peps/pep-3150/
> 
> (Note that the PEP is still officially Deferred - this entire idea
> still intrigues me, but I'm also still not convinced it's worth the
> additional language complexity).

Still +3 from me for PEP 3150 :)

*j


From tjreedy at udel.edu  Sun Jun 12 23:20:03 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 12 Jun 2011 17:20:03 -0400
Subject: [Python-ideas] PEP 3150 (statement local namespaces) updated
 based on April discussion
In-Reply-To: <BANLkTinVg_i0y0DTubNLG5fTBNAt+DVahg@mail.gmail.com>
References: <BANLkTinVg_i0y0DTubNLG5fTBNAt+DVahg@mail.gmail.com>
Message-ID: <it3ai3$rvv$1@dough.gmane.org>

On 6/12/2011 11:44 AM, Nick Coghlan wrote:
> The rationale and proposed semantics sections of PEP 3150 have been
> heavily modified based on the various discussions on the topic back in
> April.
>
> http://www.python.org/dev/peps/pep-3150/
>
> (Note that the PEP is still officially Deferred - this entire idea
> still intrigues me, but I'm also still not convinced it's worth the
> additional language complexity).

I tried to read this with an open mind, but I still strongly dislike it. 
I think it would make Python harder to learn and read and would lead to 
more confusion and questions on python-list.

I think having a different rule for function compilation makes the 
proposal even worse, as people would come to expect early binding 
(outside the given context) even more than now. If one does not like 
default args, one can either use class instances or closures. The latter 
were explicitly introduced as an alternative to using default args. Do 
we really need a fourth solution to the same problem?

I do not see comprehensions, which are *expressions*, as a precedent for 
out-of-order *statements*. Nested expressions are not left-to-right 
either, nor are assignments: "a[3] = b*f(c+d)".

'Given' or 'where' constructs are sometimes used in mathematical 
writings, especially formula exposition, but they typically, if not 
always, are isolated (like the examples in the PEP), and not part of a 
code sequence. So this is notreally a precedent to me. Example:

   fv = p * (1 + i/12)**t... , where
     fv = present value
     p  = principle
     i  = nominal annual interest rate
     t  = time in months

-(1+whatever)

-- 
Terry Jan Reedy



From zuo at chopin.edu.pl  Mon Jun 13 00:22:36 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Mon, 13 Jun 2011 00:22:36 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
Message-ID: <20110612222236.GA2540@chopin.edu.pl>

Terry Reedy dixit (2011-06-11, 16:09):

> On 6/11/2011 9:30 AM, Jan Kaliszewski wrote:
[...]
> >In all that cases (and probably some other too) that technique appears
> >to be quite useful.
> >
> >
> >== The problem ==
> >
> >...is that it is not very elegant. We add arguments which:
> >a) mess up function signatures (both in the code and in auto-generated docs);
> >b) can be incidentally overriden (especially when a function has an "open"
> >    signature with **kwargs).
> 
> One problem with trying to 'fix' this is that there can be defaulted args
> which are not intended to be overwritten by users but which are intended to
> be replaced in recursive calls.

I think this is another case... Although I can imagine that such 'private'
arguments could be specified when calling -- after **{...}/bare **, e.g.:

    fun(1, b=3, **{'c':3}, my_secret_hidden_arg='xyz')
    fun(1, b=3, **, my_secret_hidden_arg='xyz')

Though at the first sight I don't like this (`after-** args in calls') idea
so much (contrary to `after-** args in definitions' idea).

[...]
> >2. (which personally I would prefer)
> >To add `dummy' (or `hidden') keyword arguments, defined after **kwargs
> >(and after bare ** if kwargs are not needed; we have already have
> >keyword-only arguments after *args or bare *):
> >
> >     def do_and_remember(val, verbose=False, **, mem=collections.Counter()):
> >         ...
> 
> I thought of this while reading 'the problem'. It is at least
> plausible to me.
[...]

Arnaud Delobelle dixit (2011-06-11, 21:47):

> On 11 Jun 2011, at 14:30, Jan Kaliszewski wrote:
[...]
> > 3.
> > To provide a special decorator, e.g. functools.within:
> >    @functools.within(mem=collections.Counter())
> >    def do_and_remember(val, verbose=False):
> >        ...
> 
> That's hard to do as (assuming the function is defined at the global
> scope), mem will be compiled as a global, meaning that you will have

Here mem is a keyword argument, not a variable. Though I understand that
making it local/closure would need some code/closures hacking... Unless
built in to the interpreter.

> to modify the bytecode.  Oh but this makes me think about something I
> wrote a while ago (see below).
> 
> 
> 4. Use closures.
> 
> def factory(mem):
>      def do_and_remember(val, verbose=False)
>          result = do_something(val)
>          mem[val] += 1
>          if verbose:
>              print('Done {} times for {!r}'.format(mem[val], val))         ....
>      return do_and_remember
> do_and_remember = factory(mem=collections.Counter())
> 
> Added bonus: you can create many instances of do_and_remember.

Yes, but this method makes code longer and more complex.
And simple is better :)

Consider my multi-factory example:

    def make_my_callbacks(callback_params):
        my_callbacks = []
        for params in callback_params:
            def fun1(*args, **kwargs, params=params):
                "...do something with args and params..."
            def fun2(*args, **kwargs, params=params):
                "...do something with args and params..."
            def fun3(*args, **kwargs, fun1=fun1, fun2=fun2):
                """...do something with args and with functions fun1, fun2,
                for example pass them as callbacks to other functions..."
            my_callbacks.append((fun1, fun2, fun3))
        return my_callbacks

...compared to:

    def make_fun1(params):
        def fun1(*args, **kwargs):
            "...do something with args and params..."
        return fun1

    def make_fun2(params):
        def fun2(*args, **kwargs):
            "...do something with args and params..."
        return fun2

    def make_fun3(fun1, fun2):
        def fun3(*args, **kwargs):
            """...do something with args and with functions fun1, fun2,
            for example pass them as callbacks to other functions..."
        return fun3

    def make_my_callbacks(callback_params):
        my_callbacks = []
        for params in callback_params:
            fun1 = make_fun1(params)
            fun2 = make_fun2(params)
            fun3 = make_fun3(fun1, fun2)
            my_callbacks.append((fun1, fun2, fun3))
        return my_callbacks

Though, maybe it'a a matter of individual taste...

> Related to this, here's a "localize" decorator that I wrote some time
> ago for fun (I think it was from a discussion on this list).  It was
> for python 2.x (could easily be modified for 3.x I think, it's a
> matter of adapting the attribute names of the function object).  It
> "freezes" all non local variables in the function.  It's a hack! It
> may be possible to adapt it.
> 
> def new_closure(vals):
>     args = ','.join('x%i' % i for i in range(len(vals)))
>     f = eval("lambda %s:lambda:(%s)" % (args, args))
>     return f(*vals).func_closure
> 
> def localize(f):
>     f_globals = dict((n, f.func_globals[n]) for n in f.func_code.co_names)
>     f_closure = ( f.func_closure and
>                   new_closure([c.cell_contents for c in f.func_closure]) )
>     return type(f)(f.func_code, f_globals, f.func_name,
>                    f.func_defaults, f_closure)

Nice :) (and, as far as I understand, it could be used to implement the
decorator I ment).

Best regards.
*j



From greg.ewing at canterbury.ac.nz  Mon Jun 13 00:30:09 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 13 Jun 2011 10:30:09 +1200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110612222236.GA2540@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
Message-ID: <4DF53DF1.9060804@canterbury.ac.nz>

I'm -1 on any proposal that somehow tries to make the
default-argument hack more acceptable.

The main reason people still feel the need to use it
is that the for-loop is broken, insofar as it doesn't
create a new binding for each iteration.

The right way to address that is to fix the for-loop,
IMO.

-- 
Greg



From zuo at chopin.edu.pl  Mon Jun 13 02:03:32 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Mon, 13 Jun 2011 02:03:32 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF53DF1.9060804@canterbury.ac.nz>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz>
Message-ID: <20110613000332.GA5821@chopin.edu.pl>

Greg Ewing dixit (2011-06-13, 10:30):

> I'm -1 on any proposal that somehow tries to make the
> default-argument hack more acceptable.

My propositions don't make that hack less acceptable -- proposing an
alternative.

> The main reason people still feel the need to use it
> is that the for-loop is broken, insofar as it doesn't
> create a new binding for each iteration.
> 
> The right way to address that is to fix the for-loop,
> IMO.

Do you mean that each iteration should create separate local scope?
Then:

    j = 0
    my_lambdas = []
    for i in range(10):
        print(j)  # would raise UnboundLocalError
        j = i
        my_lambdas.append(lambda: i)

Or that the loop variable should be treated specially?
Then:

    i_lambdas, j_lambdas = [], []
    for i in range(10):
        j = i
        i_lambdas.append(lambda: i)
        j_lambdas.append(lambda: j)
    print(i_lambdas[2]())  # would print 2
    print(j_lambdas[2]())  # would print 9

Cheers.
*j



From zuo at chopin.edu.pl  Mon Jun 13 02:21:36 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Mon, 13 Jun 2011 02:21:36 +0200
Subject: [Python-ideas] PEP 3150 (statement local namespaces) updated
 based on April discussion
In-Reply-To: <BANLkTinVg_i0y0DTubNLG5fTBNAt+DVahg@mail.gmail.com>
References: <BANLkTinVg_i0y0DTubNLG5fTBNAt+DVahg@mail.gmail.com>
Message-ID: <20110613002136.GB5821@chopin.edu.pl>

There are some bugs in the code example in the PEP's section "Detailed
Semantics #1: Early Binding of Variable References":

There is:
        assert seq == ...
...and should be:
        assert map(lambda x: x(), seq) == ...
(3 times)

There is:
        def f(_i=i):
          return i
...and should be:
        def f(_i=i):
          return _i

Cheers.
*j



From zuo at chopin.edu.pl  Mon Jun 13 02:22:49 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Mon, 13 Jun 2011 02:22:49 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110613000332.GA5821@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz>
	<20110613000332.GA5821@chopin.edu.pl>
Message-ID: <20110613002249.GA6020@chopin.edu.pl>

Jan Kaliszewski dixit (2011-06-13, 02:03):

> My propositions don't make that hack less acceptable -- proposing an
> alternative.

Sorry, should be:

My propositions don't make that hack more acceptable -- in fact they
make it less acceptable, proposing an alternative.

*j



From ncoghlan at gmail.com  Mon Jun 13 06:52:45 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 13 Jun 2011 14:52:45 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF53DF1.9060804@canterbury.ac.nz>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz>
Message-ID: <BANLkTimaZW1gJZ6R9tEi-BFc31EcsR3CRw@mail.gmail.com>

On Mon, Jun 13, 2011 at 8:30 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> I'm -1 on any proposal that somehow tries to make the
> default-argument hack more acceptable.
>
> The main reason people still feel the need to use it
> is that the for-loop is broken, insofar as it doesn't
> create a new binding for each iteration.
>
> The right way to address that is to fix the for-loop,
> IMO.

Yikes, now *there's* a radical proposal.

-lots on any idea that would make:

def f():
  i = 0
  def g1():
    return i
  i = 1
  def g2():
    return i
  return [g1, g2]

differ in external behaviour from:

def f():
  result = []
  for i in range(2):
    def g():
      return i
    result.append(g)
  return result

or:

def f():
  return [lambda: i for i in range(2)]

or:

def _inner():
  for i in range(2):
    def g():
      return i
    yield g

def f():
  return list(_inner())

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Mon Jun 13 07:07:54 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 13 Jun 2011 15:07:54 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110611133028.GD2395@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>
Message-ID: <BANLkTi=Ofe0hkTQuRtoAyvot7gd1d+ktcA@mail.gmail.com>

On Sat, Jun 11, 2011 at 11:30 PM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
> 1.
> To add a new keyword, e.g. `inject':
> ? ?def do_and_remember(val, verbose=False):
> ? ? ? ?inject mem = collections.Counter()
> ? ? ? ?...
> or maybe:
> ? ?def do_and_remember(val, verbose=False):
> ? ? ? ?inject collections.Counter() as mem
> ? ? ? ?...

This particular alternative to the default argument hack has come up
before, as has the "hidden parameters after '**'" approach. (I thought
there was a PEP on this, but I can't find anything other than the
reference in the description of Option 4 in PEP 3103 - however, there
is a thread on the topic that starts as part of the PEP 3103
discussion at http://mail.python.org/pipermail/python-dev/2006-June/066603.html)

Institutional-memory'ly yours,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From tjreedy at udel.edu  Mon Jun 13 08:27:53 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 13 Jun 2011 02:27:53 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF53DF1.9060804@canterbury.ac.nz>
References: <20110611133028.GD2395@chopin.edu.pl>	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz>
Message-ID: <it4al9$3c8$1@dough.gmane.org>

On 6/12/2011 6:30 PM, Greg Ewing wrote:
> I'm -1 on any proposal that somehow tries to make the
> default-argument hack more acceptable.
>
> The main reason people still feel the need to use it
> is that the for-loop is broken, insofar as it doesn't
> create a new binding for each iteration.
>
> The right way to address that is to fix the for-loop,

Or use closures, which were partly designed to replace default arg use.

This case is quite different from the multiple capture in for-loop case. 
The OP is simply trying to localize names for speed instead of using 
module constants, which would otherwise do quite fine and are routinely 
used in the stdlib.

-- 
Terry Jan Reedy



From steve at pearwood.info  Mon Jun 13 08:23:38 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 13 Jun 2011 16:23:38 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <it0i1c$2es$1@dough.gmane.org>
References: <20110611133028.GD2395@chopin.edu.pl>
	<it0i1c$2es$1@dough.gmane.org>
Message-ID: <4DF5ACEA.7040603@pearwood.info>

Terry Reedy wrote:
> On 6/11/2011 9:30 AM, Jan Kaliszewski wrote:
>> == Use cases ==
>>
>> A quite common practice is 'injecting' objects into a function as its
>> locals, at def-time, using function arguments with default values...
[...]
> One problem with trying to 'fix' this is that there can be defaulted args
> which are not intended to be overwritten by users but which are intended to
> be replaced in recursive calls.

I think any solution to this would have to be backward compatible. A big 
NO to anything which changes the behaviour of existing code.


>> == Proposed solutions ==
>>
>> I see three possibilities:
>>
>> 1.
>> To add a new keyword, e.g. `inject':
>>      def do_and_remember(val, verbose=False):
>>          inject mem = collections.Counter()
>>          ...


> The body should all be runtime. Deftime expression should be in the header.


That's not even the case now. The global and nonlocal keywords are in 
the body, and they apply at compile-time.

I don't like the name inject as shown, but I like the idea of injecting 
locals into a function from the outside. (Or rather, into a *copy* of 
the function.) This suggests generalising the idea: take any function, 
and make a copy of it with the specified names/values defined as locals. 
The obvious API is a decorator (presumably living in functools).

Assume we can write such a decorator, and postpone discussion of any 
implementation for now.

Firstly, this provides a way of setting locals at function definition 
time without polluting the parameter list and exposing local variables 
to the caller. Function arguments should be used for arguments, not 
internal implementation details.

@inject(mem=collections.Counter())
def do_and_remember(val, verbose=False):
     # like do_and_remember(val, verbose=False, mem=...)


But more importantly, it has wider applications, like testing, 
introspection, or adding logging to functions:

def my_function(alist):
     return random.choice(alist) + 1

You might not be able to modify my_function, it may be part of a library 
you don't control. As written, if you want to test it, you need to 
monkey-patch the random module, which is a dangerous anti-pattern.

Better to do this:

class randomchoice_mock:
     def choice(self, arg):
         return 0

mock = randomchoice_mock()

test_func = inject(random=mock)(my_function)

Because test_func is a copy of my_function, you can be sure that you 
won't break anything.

Adding logging is just as easy.

This strikes me as the best solution: the decorator is at the head of 
the function, so it looks like a declaration, and it has its effect at 
function definition time. But as Terry points out, such a decorator 
might not be currently possible without language support, or at least 
messy byte-code hacking.




-- 
Steven



From steve at pearwood.info  Mon Jun 13 09:11:56 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 13 Jun 2011 17:11:56 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <it4al9$3c8$1@dough.gmane.org>
References: <20110611133028.GD2395@chopin.edu.pl>	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>	<20110612222236.GA2540@chopin.edu.pl>	<4DF53DF1.9060804@canterbury.ac.nz>
	<it4al9$3c8$1@dough.gmane.org>
Message-ID: <4DF5B83C.9070206@pearwood.info>

Terry Reedy wrote:
> On 6/12/2011 6:30 PM, Greg Ewing wrote:
>> I'm -1 on any proposal that somehow tries to make the
>> default-argument hack more acceptable.
>>
>> The main reason people still feel the need to use it
>> is that the for-loop is broken, insofar as it doesn't
>> create a new binding for each iteration.
>>
>> The right way to address that is to fix the for-loop,
> 
> Or use closures, which were partly designed to replace default arg use.

Default args are specifically used in at least one use-case where 
closures give the wrong result.

 >>> funcs = [lambda x: x+i for i in range(10)]
 >>> funcs[0].__closure__  # may be different in Python 2.x
(<cell at 0xb7bd8a1c: int object at 0x81b20c0>,)
 >>> funcs[0](42)  # should return 42+0
51

The usual solution is to *not* use a closure:

 >>> funcs = [lambda x, i=i: x+i for i in range(10)]
 >>> funcs[0].__closure__ is None
True
 >>> funcs[0](42)
42
 >>> funcs[9](42)
51



> This case is quite different from the multiple capture in for-loop case. 
> The OP is simply trying to localize names for speed instead of using 
> module constants, which would otherwise do quite fine and are routinely 
> used in the stdlib.
> 
That's just one use-case. Jan gave two others. Optimizations might be 
common in the stdlib, but it's a hack, and an ugly one.

Function parameters should be kept for actual arguments, not for 
optimizing name look-ups.




-- 
Steven


From ncoghlan at gmail.com  Mon Jun 13 10:05:31 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 13 Jun 2011 18:05:31 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF5B83C.9070206@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz> <it4al9$3c8$1@dough.gmane.org>
	<4DF5B83C.9070206@pearwood.info>
Message-ID: <BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>

On Mon, Jun 13, 2011 at 5:11 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> Function parameters should be kept for actual arguments, not for optimizing
> name look-ups.

Still, the post-** shared state (Jan's option 2) is likely the most
obvious way to get early binding for *any* purpose without polluting
the externally visible parameter list. Several questions that are
unclear in the general case of definition-time code are resolved in
obvious ways by that approach:

Q. When is the code executed?
A. At definition time, just like default argument values

Q. Where are the results of the calculation stored?
A. On the function object, just like default argument values

Q. How does the compiler know to generate local variable lookups for
those attributes?
A. The names are specified in the function header, just like public parameters

Q. What is the advantage over custom classes with __call__ methods?
A. Aside from the obvious speed disadvantage, moving from a function
with state that is preserved between calls to a stateful class that
happens to be callable is a surprisingly large mental shift that may
not fit well with the conceptual structure of a piece of code. While
*technically* they're the same thing (just expressed in different
ways), in reality the difference in relative emphasis of algorithm vs
shared state can make one mode of expression far more natural than the
other in a given context.

    class DoAndRemember():
        def __init__(self):
            self.mem = collections.Counter()

       def __call__(self, val, verbose=False):
           result = do_something(val)
           self.mem[val] += 1
           if verbose:
               print('Done {} times for {!r}'.format(self.mem[val], val))

    do_and_remember = DoAndRemember()

Custom classes also suffer grievously when it comes to supporting
introspection (e.g. try help() or inspect.getargspec() on the above)
and lack natural support for other features of functions (such as easy
decorator compatibility, descriptor protocol support, standard
annotations, appropriate __name__ assignment).


Q. What is the advantage over using an additional level of closure?
A. This is actually the most viable alternative, since the conceptual
model is quite a close match and it doesn't break introspection the
way a custom class does. The problems with this approach are largely
syntactic:

   def _make_do_and_remember():
       mem=collections.Counter()
       def do_and_remember(val, verbose=False):
           result = do_something(val)
           mem[val] += 1
           if verbose:
               print('Done {} times for {!r}'.format(mem[val], val))
       return do_and_remember
  do_and_remember = _make_do_and_remember()

1. The function signature is buried inside "_make_do_and_remember"
(the class approach and even PEP 3150 have the same problem)
2. The name of the function in the current namespace and its __name__
attribute have been decoupled, require explicit repetition to keep
them the same
3. This is basically an unreadable mess

I'd actually be far happier with the default argument hack equivalent:

       def do_and_remember(val, verbose=False, *, _mem=collections.Counter()):
           result = do_something(val)
           _mem[val] += 1
           if verbose:
               print('Done {} times for {!r}'.format(_mem[val], val))

All a "persistent state" proposal would do is create an alternative to
the default argument hack that doesn't suffer from the same problems:

       def do_and_remember(val, verbose=False, **, mem=collections.Counter()):
           result = do_something(val)
           mem[val] += 1
           if verbose:
               print('Done {} times for {!r}'.format(_mem[val], val))

It seems like the path of least resistance to me - the prevalence of
the default argument hack means there's an existing, widespread
practice that solves real programming issues, but is flawed in some
ways (specifically, messing with the function's signature). Allowing
declarations of shared state after the keyword-only arguments seems
like a fairly obvious answer.

The one potential trap is the classic one with immutable nonlocal
variables that haven't been declared as such (this trap also applies
to any existing use of the default argument hack): reassignment will
*not* modify the shared state, only the name binding in the current
invocation.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Mon Jun 13 13:57:06 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 13 Jun 2011 21:57:06 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz> <it4al9$3c8$1@dough.gmane.org>
	<4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
Message-ID: <BANLkTin74200ZuF704Xs27Niu+8weSZenA@mail.gmail.com>

On Mon, Jun 13, 2011 at 6:05 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> All a "persistent state" proposal would do is create an alternative to
> the default argument hack that doesn't suffer from the same problems:
>
> ? ? ? def do_and_remember(val, verbose=False, **, mem=collections.Counter()):
> ? ? ? ? ? result = do_something(val)
> ? ? ? ? ? mem[val] += 1
> ? ? ? ? ? if verbose:
> ? ? ? ? ? ? ? print('Done {} times for {!r}'.format(_mem[val], val))

As yet another shade for this particular bikeshed, this one just occurred to me:

    def do_and_remember(val, verbose=False):
          @def mem=collections.Counter()
          result = do_something(val)
          mem[val] += 1
          if verbose:
              print('Done {} times for {!r}'.format(_mem[val], val))

The @def ("at def") statement is just a new flavour of the same
proposal that has been made many times before: a way to indicate that
a simple assignment statement should be executed once at function
definition time rather than repeatedly on every call to the function.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From arnodel at gmail.com  Mon Jun 13 14:22:07 2011
From: arnodel at gmail.com (Arnaud Delobelle)
Date: Mon, 13 Jun 2011 13:22:07 +0100
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTin74200ZuF704Xs27Niu+8weSZenA@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz> <it4al9$3c8$1@dough.gmane.org>
	<4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
	<BANLkTin74200ZuF704Xs27Niu+8weSZenA@mail.gmail.com>
Message-ID: <BANLkTik+LHqWN_bTzBhu_pL=R_=wgdsW7w@mail.gmail.com>

On 13 June 2011 12:57, Nick Coghlan <ncoghlan at gmail.com> wrote:

> As yet another shade for this particular bikeshed, this one just occurred to me:
>
> ? ?def do_and_remember(val, verbose=False):
> ? ? ? ? ?@def mem=collections.Counter()
> ? ? ? ? ?result = do_something(val)
> ? ? ? ? ?mem[val] += 1
> ? ? ? ? ?if verbose:
> ? ? ? ? ? ? ?print('Done {} times for {!r}'.format(_mem[val], val))

Or to link this to PEP 3150:

    given:
          mem = collections.Counter()
    def do_and_remember(val, verbose=False):
          result = do_something(val)
          mem[val] += 1
          if verbose:
              print('Done {} times for {!r}'.format(_mem[val], val))

(Or the other way around)

-- 
Arnaud


From steve at pearwood.info  Mon Jun 13 16:33:50 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 14 Jun 2011 00:33:50 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>	<20110612222236.GA2540@chopin.edu.pl>	<4DF53DF1.9060804@canterbury.ac.nz>	<it4al9$3c8$1@dough.gmane.org>	<4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
Message-ID: <4DF61FCE.4050206@pearwood.info>

Nick Coghlan wrote:
> On Mon, Jun 13, 2011 at 5:11 PM, Steven D'Aprano <steve at pearwood.info> wrote:
>> Function parameters should be kept for actual arguments, not for optimizing
>> name look-ups.
> 
> Still, the post-** shared state (Jan's option 2) is likely the most
> obvious way to get early binding for *any* purpose without polluting
> the externally visible parameter list. 

I wouldn't call adding even more complexity to function signatures 
"obvious", although I grant that it depends on whether you're Dutch :)

Another disadvantage is that it uses a symbol instead of a word. Too 
many symbols, and your code looks like Perl (or APL). It's hard to 
google for ** to find out what it means. It's harder to talk about a 
symbol than a word. (In written text you can just write ** but in speech 
you have to use circumlocutions or made-up names like double-splat.)

[...]
> It seems like the path of least resistance to me - the prevalence of
> the default argument hack means there's an existing, widespread
> practice that solves real programming issues, but is flawed in some
> ways (specifically, messing with the function's signature). Allowing
> declarations of shared state after the keyword-only arguments seems
> like a fairly obvious answer.

The problem with injecting locals in the parameter list is that it can 
only happen at write-time. That's useful, but there's a major 
opportunity being missed: to be able to inject at runtime. You could add 
test mocks, optimized functions, logging, turn global variables into 
local constants, and probably things I've never thought of.

Here's one use-case to give a flavour of what I have in mind: if you're 
writing Unix-like scripts, one piece of useful functionality is "verbose 
mode". Here's one way of doing so:


def do_work(args, verbose=False):
     if verbose:
          pr = print
     else:
          pr = lambda *args: None
     pr("doing spam")
     spam()
     pr("doing ham")
     ham()
     # and so on

if __name__ == '__main__':
     verbose = '--verbose' in sys.argv
     do_work(my_arguments, verbose)

But why does do_work take a verbose flag? That isn't part of the API for 
the do_work function itself, which might be usefully called by other 
bits of code. The verbose argument is only there to satisfy the needs of 
the user interface. Using a ** hidden argument would solve that problem, 
but you then have to specify the value of verbose at write-time, 
defeating the purpose.

Here's an injection solution. First, the body of the function needs a 
generic hook, with a global do-nothing default:


def hook(*args):
     pass

def do_work(args):
     hook("doing spam")
     spam()
     hook("doing ham")
     ham()
     # and so on

if __name__ == '__main__':
     if '--verbose' in sys.argv:
         wrap = inject(hook=print)
     else:
         wrap = lambda func: func  # do nothing
         # or `inject(hook=hook)` to micro-optimize
     wrap(do_work)(my_arguments)


If you want to add logging, its easy: just add an elif clause with
wrap = inject(hook=logger).

Because you aren't monkey-patching the hook function (or, heaven help 
us, monkey-patching builtins.print!) you don't need to fear 
side-effects. No globals are patched, hence no mysterious 
action-at-a-distance bugs. And because the injected function is a copy 
of the original, other parts of the code that use do_work are unaffected.


But for this to work, you have to be able to inject at run-time, not 
just at write-time.



-- 
Steven


From debatem1 at gmail.com  Mon Jun 13 18:59:55 2011
From: debatem1 at gmail.com (geremy condra)
Date: Mon, 13 Jun 2011 09:59:55 -0700
Subject: [Python-ideas] PEP 3150 (statement local namespaces) updated
 based on April discussion
In-Reply-To: <it3ai3$rvv$1@dough.gmane.org>
References: <BANLkTinVg_i0y0DTubNLG5fTBNAt+DVahg@mail.gmail.com>
	<it3ai3$rvv$1@dough.gmane.org>
Message-ID: <BANLkTi=sZ3UCTWbB=7uTKZJjaS69PPJJww@mail.gmail.com>

On Sun, Jun 12, 2011 at 2:20 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 6/12/2011 11:44 AM, Nick Coghlan wrote:
>>
>> The rationale and proposed semantics sections of PEP 3150 have been
>> heavily modified based on the various discussions on the topic back in
>> April.
>>
>> http://www.python.org/dev/peps/pep-3150/
>>
>> (Note that the PEP is still officially Deferred - this entire idea
>> still intrigues me, but I'm also still not convinced it's worth the
>> additional language complexity).
>
> I tried to read this with an open mind, but I still strongly dislike it. I
> think it would make Python harder to learn and read and would lead to more
> confusion and questions on python-list.
>
> I think having a different rule for function compilation makes the proposal
> even worse, as people would come to expect early binding (outside the given
> context) even more than now. If one does not like default args, one can
> either use class instances or closures. The latter were explicitly
> introduced as an alternative to using default args. Do we really need a
> fourth solution to the same problem?
>
> I do not see comprehensions, which are *expressions*, as a precedent for
> out-of-order *statements*. Nested expressions are not left-to-right either,
> nor are assignments: "a[3] = b*f(c+d)".
>
> 'Given' or 'where' constructs are sometimes used in mathematical writings,
> especially formula exposition, but they typically, if not always, are
> isolated (like the examples in the PEP), and not part of a code sequence. So
> this is notreally a precedent to me. Example:
>
> ?fv = p * (1 + i/12)**t... , where
> ? ?fv = present value
> ? ?p ?= principle
> ? ?i ?= nominal annual interest rate
> ? ?t ?= time in months
>
> -(1+whatever)

I've historically been in favor of this kind of proposal specifically
because I'd like to be able to write the above code, but I have to
admit that many of the examples given in the PEP terrify me. The point
of exposition like this is to enhance readability by making the flow
of information (defined by the equation on the first line) extremely
clear; tucking functions and classes with their own flow into the
out-of-order block just makes it really hard to figure out what's
happening where and when.

Geremy Condra


From jimjjewett at gmail.com  Mon Jun 13 20:11:52 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Mon, 13 Jun 2011 14:11:52 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF61FCE.4050206@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz>
	<it4al9$3c8$1@dough.gmane.org> <4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
	<4DF61FCE.4050206@pearwood.info>
Message-ID: <BANLkTimJW9dJb3z_FssWB1S-efkJUbN=Ew@mail.gmail.com>

On Mon, Jun 13, 2011 at 10:33 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> Nick Coghlan wrote:
>>
>> On Mon, Jun 13, 2011 at 5:11 PM, Steven D'Aprano <steve at pearwood.info>
>> wrote:

>>> Function parameters should be kept for actual arguments, not for
>>> optimizing name look-ups.

Even the bind-it-now behavior isn't always for optimization; it can
also be used as a way of forcing stability in case the global name
gets rebound.  That is often an anti-pattern in practice, but ... not
always.

>> Still, the post-** shared state (Jan's option 2) is likely the most
>> obvious way to get early binding for *any* purpose without polluting
>> the externally visible parameter list.

I would say the most obvious place is in a decorator, using the
function object (or a copy) as the namespace.  Doing this properly
would require some variant of PEP 3130, which was rejected largely for
insufficient use.

> The problem with injecting locals in the parameter list is that it can only
> happen at write-time. That's useful, but there's a major opportunity being
> missed: to be able to inject at runtime. You could add test mocks, optimized
> functions, logging, turn global variables into local constants, and probably
> things I've never thought of.

Using the function object as a namespace (largely) gets around that,
because you can use a with statement to change the settings
temporarily.

> Here's one use-case to give a flavour of what I have in mind: if you're
> writing Unix-like scripts, one piece of useful functionality is "verbose
> mode". Here's one way of doing so:

[A verbose mode -- full example below, but the new spelling here at the top]

Just replace:

> def do_work(args):
> ? ?hook("doing spam")
> ? ?spam()
> ? ?hook("doing ham")
> ? ?ham()

with:

    def do_work(args):
 ? ?    __function__.hook("doing spam")
 ? ?    spam()
 ? ?    __function__.hook("doing ham")
 ? ?    ham()

If you want to change the bindings, just rebind do_work.hook to the
correct function.  If you are doing this as part of a test, do so
within a with statement that sets it back at the end.

(The reason this requires a variant of 3130 is that the name do_work
may itself be rebound, so do_work.hook isn't a reliable pointer.)

-jJ

[only quotes below here]

> def do_work(args, verbose=False):
> ? ?if verbose:
> ? ? ? ? pr = print
> ? ?else:
> ? ? ? ? pr = lambda *args: None
> ? ?pr("doing spam")
> ? ?spam()
> ? ?pr("doing ham")
> ? ?ham()
> ? ?# and so on
>
> if __name__ == '__main__':
> ? ?verbose = '--verbose' in sys.argv
> ? ?do_work(my_arguments, verbose)
>
> But why does do_work take a verbose flag? That isn't part of the API for the
> do_work function itself, which might be usefully called by other bits of
> code. The verbose argument is only there to satisfy the needs of the user
> interface. Using a ** hidden argument would solve that problem, but you then
> have to specify the value of verbose at write-time, defeating the purpose.
>
> Here's an injection solution. First, the body of the function needs a
> generic hook, with a global do-nothing default:
>
>
> def hook(*args):
> ? ?pass
>
> def do_work(args):
> ? ?hook("doing spam")
> ? ?spam()
> ? ?hook("doing ham")
> ? ?ham()
> ? ?# and so on
>
> if __name__ == '__main__':
> ? ?if '--verbose' in sys.argv:
> ? ? ? ?wrap = inject(hook=print)
> ? ?else:
> ? ? ? ?wrap = lambda func: func ?# do nothing
> ? ? ? ?# or `inject(hook=hook)` to micro-optimize
> ? ?wrap(do_work)(my_arguments)
>
>
> If you want to add logging, its easy: just add an elif clause with
> wrap = inject(hook=logger).
>
> Because you aren't monkey-patching the hook function (or, heaven help us,
> monkey-patching builtins.print!) you don't need to fear side-effects. No
> globals are patched, hence no mysterious action-at-a-distance bugs. And
> because the injected function is a copy of the original, other parts of the
> code that use do_work are unaffected.
>
>
> But for this to work, you have to be able to inject at run-time, not just at
> write-time.


From paul at colomiets.name  Mon Jun 13 21:14:31 2011
From: paul at colomiets.name (Paul Colomiets)
Date: Mon, 13 Jun 2011 22:14:31 +0300
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF61FCE.4050206@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz>
	<it4al9$3c8$1@dough.gmane.org> <4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
	<4DF61FCE.4050206@pearwood.info>
Message-ID: <BANLkTikzsAhtcs=evorayQarb+ATEN8Azw@mail.gmail.com>

On Mon, Jun 13, 2011 at 5:33 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> def hook(*args):
> ? ?pass
>
> def do_work(args):
> ? ?hook("doing spam")
> ? ?spam()
> ? ?hook("doing ham")
> ? ?ham()
> ? ?# and so on
>
> if __name__ == '__main__':
> ? ?if '--verbose' in sys.argv:
> ? ? ? ?wrap = inject(hook=print)
> ? ?else:
> ? ? ? ?wrap = lambda func: func ?# do nothing
> ? ? ? ?# or `inject(hook=hook)` to micro-optimize
> ? ?wrap(do_work)(my_arguments)
>
>
> If you want to add logging, its easy: just add an elif clause with
> wrap = inject(hook=logger).

It's quite promising idea.

Currenlty there are notion of cell for closures. What if globals would also
use a cell? So that cell cound be either bound to a value or to a name
in globals or builtin dictionary. With this in mind it could be
possible to either
change binding from name to value or vice versa, our to make a copy of the
function with another cells.

I think this adheres to Python philosophy of having anything modifyable.
It will add at most two words of memory for each cell (name and global dict),
and probably will not make interpreter slower. Also will probably
allow to remove
__globals__ attribute from functions in the long term.

Then it even be possible to make some modules faster by either
    from __future__ import fast_bindings
or it could be done by some external library like:
    __super_freezer_allow__ = True
    ...
    import sys, super_freezer
    super_freezer.apply(sys.modules)

Probably about 80% modules do not need to rebind globals, so they
can run faster. And if you need to monkeypatch them, just either
not freeze globals in this module or change the bindings in all its
functions.

Thoughts?

-- 
Paul


From tjreedy at udel.edu  Mon Jun 13 21:33:24 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 13 Jun 2011 15:33:24 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF5B83C.9070206@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>	<20110612222236.GA2540@chopin.edu.pl>	<4DF53DF1.9060804@canterbury.ac.nz>	<it4al9$3c8$1@dough.gmane.org>
	<4DF5B83C.9070206@pearwood.info>
Message-ID: <it5om4$g44$1@dough.gmane.org>

On 6/13/2011 3:11 AM, Steven D'Aprano wrote:
> Terry Reedy wrote:

>> Or use closures, which were partly designed to replace default arg use.
>
> Default args are specifically used in at least one use-case where
> closures give the wrong result.

I meant an explicit user-defined closure with a separate cell for each 
function ...

>  >>> funcs = [lambda x: x+i for i in range(10)]
>  >>> funcs[0].__closure__ # may be different in Python 2.x
> (<cell at 0xb7bd8a1c: int object at 0x81b20c0>,)
>  >>> funcs[0](42) # should return 42+0
> 51

not this implicit one where each function uses the *same* cell referring 
to the same int object.

 >>> funcs[0].__closure__
(<cell at 0x00FE0330: int object at 0x1E2139A0>,)
 >>> funcs[9].__closure__
(<cell at 0x00FE0330: int object at 0x1E2139A0>,)

The fundamental problem with this code for funcs is that "lambda x: x+i" 
is a *constant* equivalent to "def _(x): return x+i". Executing either 
10 times creates 10 duplicate functions. The hypnotic effect of 'lambda' 
is that some do not immediately see the equivalence.

> The usual solution is to *not* use a closure:
>
>  >>> funcs = [lambda x, i=i: x+i for i in range(10)]
>  >>> funcs[0].__closure__ is None
> True
>  >>> funcs[0](42)
> 42
>  >>> funcs[9](42)
> 51

The explicit closure solution intended to replace "lambda x,i=i:x+i" is

 >>> def makef(j):
	return lambda x: x+j

 >>> funcs = [makef(i) for i in range(10)]
 >>> list(funcs[_](42) for _ in range(10))
[42, 43, 44, 45, 46, 47, 48, 49, 50, 51]
 >>> funcs[0].__closure__
(<cell at 0x00FCB4D0: int object at 0x1E213910>,)
 >>> funcs[9].__closure__
(<cell at 0x00FE0750: int object at 0x1E2139A0>,)

We now have difference cells containing different ints. To get different 
functions from multiple compilations of one body we need either 
different defaults for pseudo-parameters or different closure cells. The 
rationale for adding the latter was partly to be an alternative to the 
former.

Once closure cells were made writable with 'nonlocal', they gained 
additional uses, or rather, replaced the awkward hack of using mutable 
1-element lists as closure contents, with the one elements being the 
true desired content.

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Mon Jun 13 21:57:31 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 13 Jun 2011 15:57:31 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF61FCE.4050206@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>	<20110612222236.GA2540@chopin.edu.pl>	<4DF53DF1.9060804@canterbury.ac.nz>	<it4al9$3c8$1@dough.gmane.org>	<4DF5B83C.9070206@pearwood.info>	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
	<4DF61FCE.4050206@pearwood.info>
Message-ID: <it5q3a$otu$1@dough.gmane.org>

On 6/13/2011 10:33 AM, Steven D'Aprano wrote:

> def hook(*args):
> pass
>
> def do_work(args):
> hook("doing spam")
> spam()
> hook("doing ham")
> ham()

Given the expense of function calls, I would write the above as

hook = None

def do(args):
   if hook: hook("doing spam")
   ...

if __name__ == '__main__':
     if '--verbose' in sys.argv:
         wrap = inject(hook=print)

I do not see the point of all this complication. If you are not trying 
to optimize the function (and adding such hooks is obviously not),
         hook = print
works just fine (in 3.x ;-).

-- 
Terry Jan Reedy



From tjreedy at udel.edu  Mon Jun 13 22:09:18 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 13 Jun 2011 16:09:18 -0400
Subject: [Python-ideas] PEP 3150 (statement local namespaces) updated
 based on April discussion
In-Reply-To: <BANLkTi=sZ3UCTWbB=7uTKZJjaS69PPJJww@mail.gmail.com>
References: <BANLkTinVg_i0y0DTubNLG5fTBNAt+DVahg@mail.gmail.com>	<it3ai3$rvv$1@dough.gmane.org>
	<BANLkTi=sZ3UCTWbB=7uTKZJjaS69PPJJww@mail.gmail.com>
Message-ID: <it5qpe$tdb$1@dough.gmane.org>

On 6/13/2011 12:59 PM, geremy condra wrote:
> On Sun, Jun 12, 2011 at 2:20 PM, Terry Reedy<tjreedy at udel.edu>  wrote:

>> 'Given' or 'where' constructs are sometimes used in mathematical writings,
>> especially formula exposition, but they typically, if not always, are
>> isolated (like the examples in the PEP), and not part of a code sequence. So
>> this is notreally a precedent to me. Example:
>>
>>   fv = p * (1 + i/12)**t... , where
>>     fv = present value
>>     p  = principle
>>     i  = nominal annual interest rate
>>     t  = time in months
>>
>> -(1+whatever)
>
> I've historically been in favor of this kind of proposal specifically
> because I'd like to be able to write the above code,

Part of my point is that the above is not *code* (and hence not an 
argument for formatting code that way). It is a text definition. Someone 
who wrote something like the above might very well then write a *code* 
version the equivalent of

p = input("principle: ")
i = input("nominal annual interest rate: ")
t = input("time in months: ")
print("Future value of ${} at {}% after {} months is ${}"
       .format(p, i, t, p*(1+i/12)**t

 > but I have to
> admit that many of the examples given in the PEP terrify me. The point
> of exposition like this is to enhance readability by making the flow
> of information (defined by the equation on the first line) extremely
> clear; tucking functions and classes with their own flow into the
> out-of-order block just makes it really hard to figure out what's
> happening where and when.

I obviously agree with all this.

-- 
Terry Jan Reedy



From debatem1 at gmail.com  Mon Jun 13 22:37:03 2011
From: debatem1 at gmail.com (geremy condra)
Date: Mon, 13 Jun 2011 13:37:03 -0700
Subject: [Python-ideas] PEP 3150 (statement local namespaces) updated
 based on April discussion
In-Reply-To: <it5qpe$tdb$1@dough.gmane.org>
References: <BANLkTinVg_i0y0DTubNLG5fTBNAt+DVahg@mail.gmail.com>
	<it3ai3$rvv$1@dough.gmane.org>
	<BANLkTi=sZ3UCTWbB=7uTKZJjaS69PPJJww@mail.gmail.com>
	<it5qpe$tdb$1@dough.gmane.org>
Message-ID: <BANLkTinFkgvnFRbf_wOyvuzkCk05Usop8w@mail.gmail.com>

On Mon, Jun 13, 2011 at 1:09 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 6/13/2011 12:59 PM, geremy condra wrote:
>>
>> On Sun, Jun 12, 2011 at 2:20 PM, Terry Reedy<tjreedy at udel.edu> ?wrote:
>
>>> 'Given' or 'where' constructs are sometimes used in mathematical
>>> writings,
>>> especially formula exposition, but they typically, if not always, are
>>> isolated (like the examples in the PEP), and not part of a code sequence.
>>> So
>>> this is notreally a precedent to me. Example:
>>>
>>> ?fv = p * (1 + i/12)**t... , where
>>> ? ?fv = present value
>>> ? ?p ?= principle
>>> ? ?i ?= nominal annual interest rate
>>> ? ?t ?= time in months
>>>
>>> -(1+whatever)
>>
>> I've historically been in favor of this kind of proposal specifically
>> because I'd like to be able to write the above code,
>
> Part of my point is that the above is not *code* (and hence not an argument
> for formatting code that way). It is a text definition. Someone who wrote
> something like the above might very well then write a *code* version the
> equivalent of
>
> p = input("principle: ")
> i = input("nominal annual interest rate: ")
> t = input("time in months: ")
> print("Future value of ${} at {}% after {} months is ${}"
> ? ? ?.format(p, i, t, p*(1+i/12)**t

Meh, semantics. I could, hypothetically, write the following:

y = sqrt(z**2 + x**2) given:
    z = get_adjacent_side_length()
    x = get_opposite_side_length()

and in terms of exposition style it would be basically the same as
what you put up earlier, and is (to me) more readable than:

z = get_adjacent_side_length()
x = get_opposite_side_length()
y = sqrt(z**2 + x**2)

for the same reasons. The question is whether the (admittedly
arguable) gain in readability offered in situations like this is worth
the extra complexity of implementation and the risk of seeing code
like the examples from the PEP in practice.

Also, just to make sure I'm being clear- Nick, I'm not trying to bash
your code. I just think that right now this lends itself to some
really hard-to-understand constructions.

Geremy Condra

Geremy Condra


From zuo at chopin.edu.pl  Tue Jun 14 00:30:27 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Tue, 14 Jun 2011 00:30:27 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTin74200ZuF704Xs27Niu+8weSZenA@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz> <it4al9$3c8$1@dough.gmane.org>
	<4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
	<BANLkTin74200ZuF704Xs27Niu+8weSZenA@mail.gmail.com>
Message-ID: <20110613223027.GB3483@chopin.edu.pl>

Nick Coghlan dixit (2011-06-13, 21:57):

>     def do_and_remember(val, verbose=False):
>           @def mem=collections.Counter()
>           result = do_something(val)
>           mem[val] += 1
>           if verbose:
>               print('Done {} times for {!r}'.format(_mem[val], val))
> 
> The @def ("at def") statement is just a new flavour of the same
> proposal that has been made many times before: a way to indicate that
> a simple assignment statement should be executed once at function
> definition time rather than repeatedly on every call to the function.

If using '@' character, I'd rather prefer:

    @in(mem=collections.Counter())
    def do_and_remember(val, verbose=False):
        result = do_something(val)
        mem[val] += 1
        if verbose:
            print('Done {} times for {!r}'.format(_mem[val], val))

@in (or @with, or @within, or @withlocal, or...) could be a language
syntax construct, not a real decorator, though using -- already well
settled -- decorator-like syntax. Important advantage of this variant is
IMHO that then it is obvious for everybody that the binding(s) is (are)
being done *early*.

Regards.
*j



From greg.ewing at canterbury.ac.nz  Tue Jun 14 00:37:20 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 14 Jun 2011 10:37:20 +1200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTimaZW1gJZ6R9tEi-BFc31EcsR3CRw@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz>
	<BANLkTimaZW1gJZ6R9tEi-BFc31EcsR3CRw@mail.gmail.com>
Message-ID: <4DF69120.6090403@canterbury.ac.nz>

Nick Coghlan wrote:

> -lots on any idea that would make:
> 
> def f():
>   i = 0
>   def g1():
>     return i
>   i = 1
>   def g2():
>     return i
>   return [g1, g2]
> 
> differ in external behaviour from:
> 
> def f():
>   result = []
>   for i in range(2):
>     def g():
>       return i
>     result.append(g)
>   return result

One possible variation of my idea wouldn't change the existing
behaviour of the for-loop at all, but would require you to
explicitly request new-binding behaviour, using something like

    for new i in range(2):
      def g():
        return i

-- 
Greg


From greg.ewing at canterbury.ac.nz  Tue Jun 14 00:19:01 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 14 Jun 2011 10:19:01 +1200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110613000332.GA5821@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz>
	<20110613000332.GA5821@chopin.edu.pl>
Message-ID: <4DF68CD5.4070402@canterbury.ac.nz>

Jan Kaliszewski wrote:

> My propositions don't make that hack less acceptable -- proposing an
> alternative.

You seem to be proposing yet another feature whose main
purpose is to patch over a mismatch between existing
features. That's not the path to elegant language design.

> Do you mean that each iteration should create separate local scope?

No...

> Or that the loop variable should be treated specially?

Yes, but in a way that you're probably not expecting. :-)

My proposal is that, if the loop variable is referenced by
an inner function (and is therefore in a cell), a new cell
is created on each iteration instead of replacing the
contents of the existing cell.

This would mean that:

* If the loop variable is *not* referenced by an inner function
(the vast majority of cases), there would be no change from
current semantics and no impact on performance.

* In any case, the loop variable can still be referenced
after the loop has finished with the expected results.

One objection that's been raised is that, as described,
it's somewhat CPython-specific, and it's uncertain how other
Pythons would get on trying to implement it.

>     i_lambdas, j_lambdas = [], []
>     for i in range(10):
>         j = i
>         i_lambdas.append(lambda: i)
>         j_lambdas.append(lambda: j)
>     print(i_lambdas[2]())  # would print 2
>     print(j_lambdas[2]())  # would print 9
> 

Yes, that's true. An extension to the idea would be to provide
a way of specifying cell-replacement behaviour for any assignment,
maybe something like

   j = new i

Then your example would print 2 both times, and the values of
both i and j after the loop would be 9.

One slightly curly aspect would be that if you *changed* the
value of i or j after the loop, the change would be seen by
the *last* lambdas created, and not any of the others. :-)
But I find it hard to imagine anyone doing this -- if you're
capturing variables in a loop, you don't normally expect to
have access to the loop variable at all after the loop
finishes.

-- 
Greg


From steve at pearwood.info  Tue Jun 14 01:25:52 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 14 Jun 2011 09:25:52 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <it5q3a$otu$1@dough.gmane.org>
References: <20110611133028.GD2395@chopin.edu.pl>	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>	<20110612222236.GA2540@chopin.edu.pl>	<4DF53DF1.9060804@canterbury.ac.nz>	<it4al9$3c8$1@dough.gmane.org>	<4DF5B83C.9070206@pearwood.info>	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>	<4DF61FCE.4050206@pearwood.info>
	<it5q3a$otu$1@dough.gmane.org>
Message-ID: <4DF69C80.9000305@pearwood.info>

Terry Reedy wrote:
[...]
> if __name__ == '__main__':
>     if '--verbose' in sys.argv:
>         wrap = inject(hook=print)
> 
> I do not see the point of all this complication. If you are not trying 
> to optimize the function (and adding such hooks is obviously not),
>         hook = print
> works just fine (in 3.x ;-).


You're modifying a global variable. Now any other function that calls 
do_work() for its own purposes suddenly finds it mysteriously printing. 
A classic action-at-a-distance bug. For a simple stand-alone script, 
there's no problem, but once you have more complexity in your app, or a 
library, things become very different.

My apologies, I've been doing a lot of reading about the pros and cons 
(mostly cons *wink*) of monkey-patching in the Ruby world, the 
open/closed principle, and various forms of bugs caused by the use of 
globals. I assumed that the problems would be blindingly obvious. I 
suppose they were only obvious to me because I'd just immersed myself in 
them for the last day or so!


-- 
Steven



From greg.ewing at canterbury.ac.nz  Tue Jun 14 01:29:40 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 14 Jun 2011 11:29:40 +1200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF61FCE.4050206@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz>
	<it4al9$3c8$1@dough.gmane.org> <4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
	<4DF61FCE.4050206@pearwood.info>
Message-ID: <4DF69D64.7030308@canterbury.ac.nz>

Steven D'Aprano wrote:

> Because you aren't monkey-patching the hook function (or, heaven help 
> us, monkey-patching builtins.print!) you don't need to fear 
> side-effects.

It's still rather non-obvious what's going on, though. Copious
commenting would be needed to make this style of coding
understandable.

Also, it doesn't seem to generalise. What if the function in
question calls other functions, which call other functions,
which themselves need a verbose option? It seems you would
need to explicitly wrap all the sub-function calls to pass
the hook on to them. And what if there is more than one
option to be hooked? You'd rapidly end up with a nightmarish
mess.

Here's another way to approach the problem:

   class HookableWorker(object):

     def hook(self, arg):
       pass

     def do_work(self):
       self.hook("Starting work")
       ...
       self.hook("Stopping work")

   def be_verbose(arg):
     print arg

   def main():
     worker = HookableWorker()
     if "--verbose" in sys.argv:
       worker.hook = be_verbose
     worker.do_work()

Now you can expand the HookableWorker class by adding more methods
that all share the same hook, still without anything being global.

-- 
Greg


From steve at pearwood.info  Tue Jun 14 01:55:26 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 14 Jun 2011 09:55:26 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTimJW9dJb3z_FssWB1S-efkJUbN=Ew@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz>
	<it4al9$3c8$1@dough.gmane.org> <4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
	<4DF61FCE.4050206@pearwood.info>
	<BANLkTimJW9dJb3z_FssWB1S-efkJUbN=Ew@mail.gmail.com>
Message-ID: <4DF6A36E.8090006@pearwood.info>

Jim Jewett wrote:
> On Mon, Jun 13, 2011 at 10:33 AM, Steven D'Aprano <steve at pearwood.info> wrote:
>> Nick Coghlan wrote:
>>> On Mon, Jun 13, 2011 at 5:11 PM, Steven D'Aprano <steve at pearwood.info>
>>> wrote:
> 
>>>> Function parameters should be kept for actual arguments, not for
>>>> optimizing name look-ups.
> 
> Even the bind-it-now behavior isn't always for optimization; it can
> also be used as a way of forcing stability in case the global name
> gets rebound.  That is often an anti-pattern in practice, but ... not
> always.

Acknowledged. But whatever the purpose, my comment still stands: 
function arguments should be used for arguments, not for their 
side-effect of injecting a local variable into the function namespace.



>> The problem with injecting locals in the parameter list is that it can only
>> happen at write-time. That's useful, but there's a major opportunity being
>> missed: to be able to inject at runtime. You could add test mocks, optimized
>> functions, logging, turn global variables into local constants, and probably
>> things I've never thought of.
> 
> Using the function object as a namespace (largely) gets around that,
> because you can use a with statement to change the settings
> temporarily.

You mean something like this?

with make_logging_len() as len:
     x = some_function_that_calls_len()


That's fine for some purposes, but you're still modifying global state. 
If some_function_that_calls_len() calls spam(), and spam() also contains 
a call to len, you've unexpectedly changed the behaviour of spam. If 
that's the behaviour that you want, fine, but it probably isn't.

There are all sorts of opportunities for breaking things when patching 
globals, which makes it somewhat of an anti-pattern. Better to make the 
patched version a local.



>> Here's one use-case to give a flavour of what I have in mind: if you're
>> writing Unix-like scripts, one piece of useful functionality is "verbose
>> mode". Here's one way of doing so:
> 
> [A verbose mode -- full example below, but the new spelling here at the top]
> 
> Just replace:
> 
>> def do_work(args):
>>    hook("doing spam")
>>    spam()
>>    hook("doing ham")
>>    ham()
> 
> with:
> 
>     def do_work(args):
>         __function__.hook("doing spam")
>         spam()
>         __function__.hook("doing ham")
>         ham()
[...]
> (The reason this requires a variant of 3130 is that the name do_work
> may itself be rebound, so do_work.hook isn't a reliable pointer.)

Ah, that's why it doesn't work for me! :)

Even if it did work, you're still messing with global state. If two 
functions are using do_work, and one wants a print hook, and the other 
wants a logging hook (or whatever), only one can be satisfied.

Also this trick can't work for optimizations. A call to do_work.hook 
requires a global lookup followed by a second lookup in the function 
object namespace, which is not as fast as using a local.



-- 
Steven


From steve at pearwood.info  Tue Jun 14 02:21:24 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 14 Jun 2011 10:21:24 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF69D64.7030308@canterbury.ac.nz>
References: <20110611133028.GD2395@chopin.edu.pl>	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>	<20110612222236.GA2540@chopin.edu.pl>	<4DF53DF1.9060804@canterbury.ac.nz>	<it4al9$3c8$1@dough.gmane.org>
	<4DF5B83C.9070206@pearwood.info>	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>	<4DF61FCE.4050206@pearwood.info>
	<4DF69D64.7030308@canterbury.ac.nz>
Message-ID: <4DF6A984.6090304@pearwood.info>

Greg Ewing wrote:
> Steven D'Aprano wrote:
> 
>> Because you aren't monkey-patching the hook function (or, heaven help 
>> us, monkey-patching builtins.print!) you don't need to fear side-effects.
> 
> It's still rather non-obvious what's going on, though. Copious
> commenting would be needed to make this style of coding
> understandable.

I don't think so. The injection happens right at the top of the 
function. True, you need to know what "inject" does, but that's no 
different from any other function. Provided you know that "inject" adds 
a local binding to the function namespace, instead of using a global, 
it's easy to understand what this does:

x = 42

@inject(x=23)
def spam():
     print(x)

Not terribly mysterious. The only tricky thing is that some programmers 
aren't comfortable with the idea that functions are first class objects, 
and so:

@inject(len=my_len)
def spam(arg):
     return len(arg)+1

will discombobulate them. ("What do you mean, len isn't the built-in 
len?") But then again, they're likely to be equally put off by global 
patches too:

len=my_len

def spam(arg):
     return len(arg)+1


Doesn't stop us using that technique when appropriate.


> Also, it doesn't seem to generalise. What if the function in
> question calls other functions, which call other functions,
> which themselves need a verbose option? It seems you would
> need to explicitly wrap all the sub-function calls to pass
> the hook on to them. And what if there is more than one
> option to be hooked? You'd rapidly end up with a nightmarish
> mess.

That's a feature, not a bug!

Patches are *local* to the function, not global. If you want to change 
global state, you can already do it, by monkey-patching the module. We 
don't need a new magic inject function to do that. This is not meant to 
be used for making wholesale changes to multiple functions at once, but 
for localized changes to one function at a time. A scalpel, not a chainsaw.



> Here's another way to approach the problem:
> 
>   class HookableWorker(object):
[...]
> Now you can expand the HookableWorker class by adding more methods
> that all share the same hook, still without anything being global.

Absolutely. And that will still be a viable approach for many things.

But...

* You can only patch things that are already written as a class. If you 
want to add a test mock or logging to a function, this strategy doesn't 
help you because there's nothing to subclass.

* There's a performance and (arguably) readability cost to using 
callable classes instead of functions.

* Nor does it clean up the func(arg, len=len) hack.



-- 
Steven


From ncoghlan at gmail.com  Tue Jun 14 02:43:47 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 14 Jun 2011 10:43:47 +1000
Subject: [Python-ideas] PEP 3150 (statement local namespaces) updated
 based on April discussion
In-Reply-To: <BANLkTinFkgvnFRbf_wOyvuzkCk05Usop8w@mail.gmail.com>
References: <BANLkTinVg_i0y0DTubNLG5fTBNAt+DVahg@mail.gmail.com>
	<it3ai3$rvv$1@dough.gmane.org>
	<BANLkTi=sZ3UCTWbB=7uTKZJjaS69PPJJww@mail.gmail.com>
	<it5qpe$tdb$1@dough.gmane.org>
	<BANLkTinFkgvnFRbf_wOyvuzkCk05Usop8w@mail.gmail.com>
Message-ID: <BANLkTi=vsC5qxGZxRc4SnZd_E32GFDQ9ag@mail.gmail.com>

On Tue, Jun 14, 2011 at 6:37 AM, geremy condra <debatem1 at gmail.com> wrote:
> Also, just to make sure I'm being clear- Nick, I'm not trying to bash
> your code. I just think that right now this lends itself to some
> really hard-to-understand constructions.

There's a reason PEP 3150 has two sections ("PEP Deferral" and "Key
Concern") devoted to explaining why it's current state is Deferred
rather than Draft. When *I'm* not convinced it's a good idea, I'm not
at all inclined to take offense when people see serious problems with
the concept :)

With PEP 343, we managed to find a sweet spot that allowed a great
deal of flexibility without unduly encouraging obfuscated code.

I've yet to find a similar sweet spot with PEP 3150 or related ideas.
I feel there's a seed of a useful concept in there somewhere, but
mostly it just creates two ways to do too many things.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From greg.ewing at canterbury.ac.nz  Tue Jun 14 02:47:10 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 14 Jun 2011 12:47:10 +1200
Subject: [Python-ideas] PEP 3150 (statement local namespaces) updated
 based on April discussion
In-Reply-To: <it5qpe$tdb$1@dough.gmane.org>
References: <BANLkTinVg_i0y0DTubNLG5fTBNAt+DVahg@mail.gmail.com>
	<it3ai3$rvv$1@dough.gmane.org>
	<BANLkTi=sZ3UCTWbB=7uTKZJjaS69PPJJww@mail.gmail.com>
	<it5qpe$tdb$1@dough.gmane.org>
Message-ID: <4DF6AF8E.3060106@canterbury.ac.nz>

Terry Reedy wrote:

> Part of my point is that the above is not *code* (and hence not an 
> argument for formatting code that way).

A couple of things about that:

1) Mathematicians often write things informally that, in a program,
would need to be spelled out formally in code. They get away with
it because they're writing for humans, not computers.

2) Very often they *do* write out the subsequently-defined terms
formally. I came across an example just the other day. From
"Elementary Numerical Analysis, an Algorithmic Approach" by
Samuel D. Conte and Carl de Boor, 3rd edition, page 364:

<quote>
    Runge-Kutta method of order 4:

       y[n+1] = y[n] + (1/6) * (k1 + 2*k2 + 2*k3 + k4)

    where

       k1 = h*f(x[n], y[n])
       k2 = h*f(x[n + h/2], y[n] + k1 / 2)
       k3 = h*f(x[n + h/2], y[n] + k2 / 2)
       k4 = h*f(x[n], y[n] + k3)
</quote>

-- 
Greg


From greg.ewing at canterbury.ac.nz  Tue Jun 14 02:52:28 2011
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 14 Jun 2011 12:52:28 +1200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF69C80.9000305@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz>
	<it4al9$3c8$1@dough.gmane.org> <4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
	<4DF61FCE.4050206@pearwood.info> <it5q3a$otu$1@dough.gmane.org>
	<4DF69C80.9000305@pearwood.info>
Message-ID: <4DF6B0CC.4030002@canterbury.ac.nz>

Steven D'Aprano wrote:

> My apologies, I've been doing a lot of reading about the pros and cons 
> (mostly cons *wink*) of monkey-patching in the Ruby world, the 
> open/closed principle, and various forms of bugs caused by the use of 
> globals.

I think part of the problem is that for the particular example
you chose -- a "verbose" option to a command-line script --
you usually *do* want it to apply to the entire program, so
using a global is perfectly adequate in that case.

-- 
Greg


From ncoghlan at gmail.com  Tue Jun 14 03:12:05 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 14 Jun 2011 11:12:05 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF61FCE.4050206@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz> <it4al9$3c8$1@dough.gmane.org>
	<4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
	<4DF61FCE.4050206@pearwood.info>
Message-ID: <BANLkTik+PFwmKMD9gj1JnDXUQeQgdTfEkA@mail.gmail.com>

On Tue, Jun 14, 2011 at 12:33 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> The problem with injecting locals in the parameter list is that it can only
> happen at write-time. That's useful, but there's a major opportunity being
> missed: to be able to inject at runtime. You could add test mocks, optimized
> functions, logging, turn global variables into local constants, and probably
> things I've never thought of.
>
> Here's one use-case to give a flavour of what I have in mind: if you're
> writing Unix-like scripts, one piece of useful functionality is "verbose
> mode". Here's one way of doing so:
>
>
> def do_work(args, verbose=False):
> ? ?if verbose:
> ? ? ? ? pr = print
> ? ?else:
> ? ? ? ? pr = lambda *args: None
> ? ?pr("doing spam")
> ? ?spam()
> ? ?pr("doing ham")
> ? ?ham()
> ? ?# and so on

> But why does do_work take a verbose flag? That isn't part of the API for the
> do_work function itself, which might be usefully called by other bits of
> code. The verbose argument is only there to satisfy the needs of the user
> interface. Using a ** hidden argument would solve that problem, but you then
> have to specify the value of verbose at write-time, defeating the purpose.
>
> Here's an injection solution. First, the body of the function needs a
> generic hook, with a global do-nothing default:
>
>
> def hook(*args):
> ? ?pass
>
> def do_work(args):
> ? ?hook("doing spam")
> ? ?spam()
> ? ?hook("doing ham")
> ? ?ham()
> ? ?# and so on
>
> if __name__ == '__main__':
> ? ?if '--verbose' in sys.argv:
> ? ? ? ?wrap = inject(hook=print)
> ? ?else:
> ? ? ? ?wrap = lambda func: func ?# do nothing
> ? ? ? ?# or `inject(hook=hook)` to micro-optimize
> ? ?wrap(do_work)(my_arguments)
>
>
> If you want to add logging, its easy: just add an elif clause with
> wrap = inject(hook=logger).
>
> Because you aren't monkey-patching the hook function (or, heaven help us,
> monkey-patching builtins.print!) you don't need to fear side-effects. No
> globals are patched, hence no mysterious action-at-a-distance bugs. And
> because the injected function is a copy of the original, other parts of the
> code that use do_work are unaffected.
>
>
> But for this to work, you have to be able to inject at run-time, not just at
> write-time.

This is getting deep into major structural changes to the way name
lookups work, though. Pre-seeding locals with values that are
calculated at run-time is a much simpler concept.

A more explicit way to do the same thing might work along the following lines:

1. Add a writeable f_initlocals dict attribute to function objects
(None by default)
2. When a function is called, if f_initlocals is not None, use it to
initialise the locals() namespace
3. Add a new "local" statement to tell the compiler to treat names as
local. Using this statement will create an f_initlocals dict mapping
those names to None.
4. Add a new decorator to functools that works like the following:
  def initlocals(**kwargs):
    def inner(f):
      new_names = kwargs.keys() - f.f_initlocals.keys()
      if new_names:
          raise ValueError("{} are not local variables of
{!r}".format(new_names, f))
      f.f_initlocals.update(kwargs)
      return f
    return inner

   @functools.initlocals(mem=collections.Counter())
   def do_and_remember(val, verbose=False):
       local mem
       result = do_something(val)
       mem[val] += 1
       if verbose:
           print('Done {} times for {!r}'.format(mem[val], val))

You could still inject changes at runtime with that concept, but would
need to be careful with thread-safety issues if you only wanted the
change to apply to some invocations and not others.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From zuo at chopin.edu.pl  Tue Jun 14 03:12:14 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Tue, 14 Jun 2011 03:12:14 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110613223027.GB3483@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz> <it4al9$3c8$1@dough.gmane.org>
	<4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
	<BANLkTin74200ZuF704Xs27Niu+8weSZenA@mail.gmail.com>
	<20110613223027.GB3483@chopin.edu.pl>
Message-ID: <20110614011214.GA5291@chopin.edu.pl>

Jan Kaliszewski dixit (2011-06-14, 00:30):

>     @in(mem=collections.Counter())
>     def do_and_remember(val, verbose=False):
>         result = do_something(val)
>         mem[val] += 1
>         if verbose:
>             print('Done {} times for {!r}'.format(_mem[val], val))
> 
> @in (or @with, or @within, or @withlocal, or...) could be a language
> syntax construct

On second thought: no. I mean: no -- for a separate syntax construct
with limited usage possibilities (see: cases mentioned by Steven);
yes -- for language improvements that would make possible one of the
solutions:

1. A real decorator:
  a) quasi-argument-locals-based
     (names could be used to read injected value and later could be
      rebound, like arguments);
or
  b) another-level-closure-based
     (names could not be used to read injected values if rebound later:
      it's *either* a free variable *or* a local variable).

or

2. `after-** hidden pseudo-arguments' (see previous posts...).

Now I don't know which of them I'd prefer...

And probably any of them would need some core-language modifications...
(at least the '2' and '1a' variants) 

Regards.
*j



From bruce at leapyear.org  Tue Jun 14 03:45:05 2011
From: bruce at leapyear.org (Bruce Leban)
Date: Mon, 13 Jun 2011 18:45:05 -0700
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110611133028.GD2395@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>
Message-ID: <BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>

It seems to me this discussion is mixing some different issues.

(1) having secret parameters that don't show in help().

(2) injecting a value in some way

I think these should be thought about separately.

*For secret parameters:* help() could have a convention that it doesn't
display variables with that start with _ or something like that, but that
might have compatibility issues. Alternatively, there could be some new
syntax or a decorator like

  >>> *@help.hide*("x,y")
  def foo(a, b, x=[], y={}): pass

  >>> help(foo)
  Help on function foo in module __main__:

  f(a, b)

*For injecting values:* There are several different cases. I'll use the
arbitrary keyword *special* in the snippets below (but note that the keyword
means something slightly different in each case):

  def foo1():
    x = *special *[]
    ...
    x.append(t)

Sets x every time the function is called to the same static list. What's
special here is that I have one list created that gets reused, not a new
list every time. This is how default arguments work and is useful for an
accumulating list.

  def foo2():
    *special *y = 0
    ...
    y += t

Initializes y once to 0 (at some time before the next line of code is
reached). This is how static works in C++. This is what I want if my
accumulating variable is a counter since numbers are immutable. This case
easily handles the first case. If you never rebind x, you don't need to do
anything special, otherwise something like this:

  def foo1a():
    *special *_x = []
    x = _x
    ... # might rebind x
    x.append(t)

It's a bit clumsy to use the first case to handle the second case:

  def foo2a():
    y = *special *[0]
    ...
    y[0] += t

In addition, there are other use cases being discussed. This creates a new
scope for i every time through the loop:

  def foo3():
    result = []
    for *special *i in range(10):
      def z(): return i
      result.append(z)

And this injects a mock to replace a library function:

  def foo4():
    return random.random()
  w = *special*(random.random=lambda: 0.1) foo4()

Just because we might use similar hacks to do these now, doesn't mean that
they are necessarily the same and I think the discussion has been going in
several different directions simultaneously. I think all these cases have
merits but I don't know which are more important. The last case seems to be
handled reasonably well by various mock libraries using with, so I'm not
particularly worried about it. I would like support for case 1 or 2. I don't
like the idea of using a different function argument hack instead of the
current one.

--- Bruce
Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110613/9766a825/attachment.html>

From stephen at xemacs.org  Tue Jun 14 07:02:13 2011
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 14 Jun 2011 14:02:13 +0900
Subject: [Python-ideas] PEP 3150 (statement local namespaces) updated
 based on April discussion
In-Reply-To: <4DF6AF8E.3060106@canterbury.ac.nz>
References: <BANLkTinVg_i0y0DTubNLG5fTBNAt+DVahg@mail.gmail.com>
	<it3ai3$rvv$1@dough.gmane.org>
	<BANLkTi=sZ3UCTWbB=7uTKZJjaS69PPJJww@mail.gmail.com>
	<it5qpe$tdb$1@dough.gmane.org> <4DF6AF8E.3060106@canterbury.ac.nz>
Message-ID: <87pqmhknu2.fsf@uwakimon.sk.tsukuba.ac.jp>

Greg Ewing writes:

 > 2) Very often [mathematicians] *do* write out the
 >    subsequently-defined terms formally.

It strikes me that this is just the tension between the declarative
style of coding, and the imperative style of doing the same thing.
*The same code ends up being written* (modulo a bit of syntactic sugar
like "where:" or "given:"), just the statement order is permuted.

I have nothing against the declarative style.  But at least for these
simple examples including this syntax in Python violates TOOWTDI, and
my experience in Lisp is that indeed the extra flexibility hinders
readability because whichever style one personally favors, lots of
folks use the other one.

Python already allows you to (verbosely) use this style, anyway.

    def behavior_of_experimental_subject (subject):

        def behavior(now, later):
            return now + later

        now = subject.randomness()
        later = subject.perversity()
        return behavior(now, later)

My feeling here is that if the declarative order is not sufficiently
important to justify the local function and the extra return
statement, using the imperative order won't be that costly in
readability terms.


From cmjohnson.mailinglist at gmail.com  Tue Jun 14 08:06:49 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Mon, 13 Jun 2011 20:06:49 -1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
Message-ID: <BANLkTinuPXRf9qc09mqjTUT6dJjepQ_vMg@mail.gmail.com>

Feels like we're just repeating October 2008:

http://mail.python.org/pipermail/python-ideas/2008-October/thread.html

Are there any considerations we missed that time around?


From zuo at chopin.edu.pl  Tue Jun 14 11:49:58 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Tue, 14 Jun 2011 11:49:58 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTinuPXRf9qc09mqjTUT6dJjepQ_vMg@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<BANLkTinuPXRf9qc09mqjTUT6dJjepQ_vMg@mail.gmail.com>
Message-ID: <20110614094958.GA3754@chopin.edu.pl>

Carl M. Johnson dixit (2011-06-13, 20:06):

> Feels like we're just repeating October 2008:
> 
> http://mail.python.org/pipermail/python-ideas/2008-October/thread.html

Not exactly. That discussion was only about closure-based solutions and
cases. Please note, that e.g. after-**-idea is a bit different.

Cheers.
*j



From zuo at chopin.edu.pl  Tue Jun 14 12:17:55 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Tue, 14 Jun 2011 12:17:55 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
Message-ID: <20110614101755.GB3754@chopin.edu.pl>

Bruce Leban dixit (2011-06-13, 18:45):

> It seems to me this discussion is mixing some different issues.
> 
> (1) having secret parameters that don't show in help().
[...]
> I think these should be thought about separately.
> 
> *For secret parameters:* help() could have a convention that it doesn't
> display variables with that start with _ or something like that, but that

No, the idea is that after-**-constans are not only hidden-in-help-
-arguments but that they also cannot be specified/overriden in a
function call. So their usage would not be a hack that causes risk of
incidental override by caller or that makes function signatures
obfuscated (they would have to be defined separately, after **|**kwargs,
in the righmost signature part).

    def compute(num1, num2, **, MAX_CACHE_LEN=100, cache=dict()):
        try:
            return cache[(num1, num2)]
        except KeyError:
            if len(cache) >= MAX_CACHE_LEN:
                cache.popitem()
            cache[(num1, num2)] = result = _compute(num1, num2)
            return result

    help(compute)                   # -> "... compute(num1, num2)"
    compute(1, 2)                   # OK
    compute(1, 2, MAX_CACHE_LEN=3)  # would raise TypeError
    compute(1, 2, cache={})         # would raise TypeError

----

Open question:

It's obvious that such a repetition must be prohibited (SyntaxError, at
compile time):

    def sth(my_var, **, my_var): "do something"

But in case of:

    def sth(*args, **kwargs, my_var='foo'): "do something"

-- should 'my_var' in kwargs be allowed? (it's a runtime question)
There is no real conflict here, so at the first sight I'd say: yes. 

Regards.
*j



From ncoghlan at gmail.com  Tue Jun 14 12:31:41 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 14 Jun 2011 20:31:41 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF61FCE.4050206@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz> <it4al9$3c8$1@dough.gmane.org>
	<4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
	<4DF61FCE.4050206@pearwood.info>
Message-ID: <BANLkTikLpG8ReRzKxt3gR=ptn5rt4GT_Gw@mail.gmail.com>

On Tue, Jun 14, 2011 at 12:33 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> Nick Coghlan wrote:
>>
>> On Mon, Jun 13, 2011 at 5:11 PM, Steven D'Aprano <steve at pearwood.info>
>> wrote:
>>>
>>> Function parameters should be kept for actual arguments, not for
>>> optimizing
>>> name look-ups.
>>
>> Still, the post-** shared state (Jan's option 2) is likely the most
>> obvious way to get early binding for *any* purpose without polluting
>> the externally visible parameter list.
>
> I wouldn't call adding even more complexity to function signatures
> "obvious", although I grant that it depends on whether you're Dutch :)
>
> Another disadvantage is that it uses a symbol instead of a word. Too many
> symbols, and your code looks like Perl (or APL). It's hard to google for **
> to find out what it means. It's harder to talk about a symbol than a word.
> (In written text you can just write ** but in speech you have to use
> circumlocutions or made-up names like double-splat.)

As with *, ** and @, you don't search for them directly, you search
for "def" (although redirects from the multiplication and power docs
to the def statement docs may not be the worst idea ever).

If we hadn't already added keyword-only arguments in Python 3, I'd
consider this significantly more obscure. Having the function
signature progress from "positional-or-keyword arguments" to
"keyword-only arguments" to "implicit arguments", on the other hand,
seems a lot cleaner than the status quo without being significantly
more complicated. After all, there's no new symbols involved - merely
a modification to allow a bare "**" to delimit the start of the
implicit arguments when arbitrary keyword arguments are not accepted.

Who knows, maybe explicitly teaching that behaviour would make people
less likely to fall into the default argument trap.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From scialexlight at gmail.com  Tue Jun 14 19:44:26 2011
From: scialexlight at gmail.com (Alex Light)
Date: Tue, 14 Jun 2011 13:44:26 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110614011214.GA5291@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>
	<4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz>
	<it4al9$3c8$1@dough.gmane.org> <4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
	<BANLkTin74200ZuF704Xs27Niu+8weSZenA@mail.gmail.com>
	<20110613223027.GB3483@chopin.edu.pl>
	<20110614011214.GA5291@chopin.edu.pl>
Message-ID: <BANLkTinwUQx9pNqbck91uC-yJVacYHXvhA@mail.gmail.com>

On Mon, Jun 13, 2011 at 9:12 PM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
>
> On second thought: no. I mean: no -- for a separate syntax construct
> with limited usage possibilities (see: cases mentioned by Steven);
> yes -- for language improvements that would make possible one of the
> solutions:
>
> 1. A real decorator:
>  a) quasi-argument-locals-based
>     (names could be used to read injected value and later could be
>      rebound, like arguments);
>

Forgive me if im wrong but i believe that this is possible without any
language changes
using pure python.

this is my attempt at it:

>>>from inspect import getfullargspec as argspec
>>>
>>>def makeLocal(**localArgs):
>>>    def decorator(func):
>>>        if (_anyConflicts(func, localArgs)):
>>>            raise Exception("abiguity in names")
>>>
>>>        def inner(*args, **kwargs):
>>>            if (any(Larg in kwargs.keys() for Larg in localArgs.keys())):
>>>                raise NameError("no resetting locals")
>>>
>>>            ## used to restore __globals__ i think this is alright since
>>>            ## iirc __globals__ only holds references anyway
>>>            frmglobals = func.__globals__.copy()
>>>            func.__globals__.update(localArgs)
>>>            ret= func(*args, **kwargs)
>>>            func.__globals__.clear()
>>>            func.__globals__.update(frmglobals)
>>>            return ret
>>>
>>>        inner.__doc__=func.__doc__
>>>        inner.__name__=func.__name__
>>>        return inner
>>>    return decorator
>>>
>>>def _anyConflicts(func, localArgs):
>>>    fa = argspec(func)
>>>    for Larg in localArgs:
>>>        if (Larg in fa.args or
>>>            Larg in fa.kwonlyargs or
>>>            Larg in fa):
>>>            return True
>>>    return False

this uses a closure to hold the values of the injected values and hides them

all test pass exactly as if the values were defined with makeLocals were
globals within the function but all act as if they
are locals outside of it.

this means that
if we define this this
>>>@makeLocal(aList=list())
>>>def add_and_print(arg):
>>>    aList.append(arg)
>>>    print(aList)

and run
>>>add_and_print(1) #it prints
[1]
>>>add_and_print(33) #prints
[1,33]

if we try this
>>>print(aList)
we get a NameError

and if we try this

>>>aList=['this','is','a','different','list']
>>>add_and_print(66)#prints
[1,33,66]
>>>print(aList)
['this','is','a','different','list']

all problems with global values still apply with this however.
for example just as
>>>anInt=33
>>>def increment():
>>>    anInt+=1
>>>foo(22)
throws a UnboundLocalError so does
>>>@makeLocal(anInt=33)
>>>def increment():
>>>    anInt+=1

so what do you think?

--Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110614/e07b8028/attachment.html>

From zuo at chopin.edu.pl  Tue Jun 14 23:27:39 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Tue, 14 Jun 2011 23:27:39 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTinwUQx9pNqbck91uC-yJVacYHXvhA@mail.gmail.com>
References: <4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz> <it4al9$3c8$1@dough.gmane.org>
	<4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
	<BANLkTin74200ZuF704Xs27Niu+8weSZenA@mail.gmail.com>
	<20110613223027.GB3483@chopin.edu.pl>
	<20110614011214.GA5291@chopin.edu.pl>
	<BANLkTinwUQx9pNqbck91uC-yJVacYHXvhA@mail.gmail.com>
Message-ID: <20110614212739.GB2549@chopin.edu.pl>

Alex Light dixit (2011-06-14, 13:44):

> >>>            frmglobals = func.__globals__.copy()
> >>>            func.__globals__.update(localArgs)
> >>>            ret= func(*args, **kwargs)
> >>>            func.__globals__.clear()
> >>>            func.__globals__.update(frmglobals)

Changing global state on each call seems to be both
concurrency-and-recurrency-unsafe and inefficient.

Though that 1a (closure-based) variant should be possible using
techniques like that:
http://mail.python.org/pipermail/python-ideas/2008-October/002227.html

Retards.
*j



From zuo at chopin.edu.pl  Wed Jun 15 02:28:25 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Wed, 15 Jun 2011 02:28:25 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110614101755.GB3754@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
Message-ID: <20110615002825.GA2217@chopin.edu.pl>

Jan Kaliszewski dixit (2011-06-14, 12:17):

> It's obvious that such a repetition must be prohibited (SyntaxError, at
> compile time):
> 
>     def sth(my_var, **, my_var): "do something"

Of course, I ment:
      def sth(my_var, **, my_var='foo'): "do something"

> But in case of:
> 
>     def sth(*args, **kwargs, my_var='foo'): "do something"
> 
> -- should 'my_var' in kwargs be allowed? (it's a runtime question)
> There is no real conflict here, so at the first sight I'd say: yes. 

On second thought: no, such repetitions also should *not* be allowed.
If a programmer, by mistake, would try to specify the argument value in
a call, an explicit TypeError should be raised -- otherwise it'd become
a trap (especially for beginners and absent-minded programmers).

Regards.
*j



From jh at improva.dk  Wed Jun 15 08:21:47 2011
From: jh at improva.dk (Jacob Holm)
Date: Wed, 15 Jun 2011 08:21:47 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110615002825.GA2217@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl>
Message-ID: <4DF84F7B.3010700@improva.dk>

On 2011-06-15 02:28, Jan Kaliszewski wrote:
> Jan Kaliszewski dixit (2011-06-14, 12:17):
>> But in case of:
>>
>>     def sth(*args, **kwargs, my_var='foo'): "do something"
>>
>> -- should 'my_var' in kwargs be allowed? (it's a runtime question)
>> There is no real conflict here, so at the first sight I'd say: yes. 
> 
> On second thought: no, such repetitions also should *not* be allowed.
> If a programmer, by mistake, would try to specify the argument value in
> a call, an explicit TypeError should be raised -- otherwise it'd become
> a trap (especially for beginners and absent-minded programmers).

I disagree.  One of the main selling points of this feature for me is
that adding a few "hidden parameters" to a function does not change the
signature of the function.  If you raise a TypeError when the name of a
hidden parameter is in kwargs this is a change in signature.

- Jacob


From steve at pearwood.info  Wed Jun 15 13:35:31 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 15 Jun 2011 21:35:31 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF84F7B.3010700@improva.dk>
References: <20110611133028.GD2395@chopin.edu.pl>	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>	<20110614101755.GB3754@chopin.edu.pl>	<20110615002825.GA2217@chopin.edu.pl>
	<4DF84F7B.3010700@improva.dk>
Message-ID: <4DF89903.6030202@pearwood.info>

Jacob Holm wrote:
> On 2011-06-15 02:28, Jan Kaliszewski wrote:

>>> -- should 'my_var' in kwargs be allowed? (it's a runtime question)
>>> There is no real conflict here, so at the first sight I'd say: yes. 
>> On second thought: no, such repetitions also should *not* be allowed.
>> If a programmer, by mistake, would try to specify the argument value in
>> a call, an explicit TypeError should be raised -- otherwise it'd become
>> a trap (especially for beginners and absent-minded programmers).
> 
> I disagree.  One of the main selling points of this feature for me is
> that adding a few "hidden parameters" to a function does not change the
> signature of the function.  If you raise a TypeError when the name of a
> hidden parameter is in kwargs this is a change in signature.


This is another reason why function parameters should not be used for 
something that is not a function parameter!

+1 on the ability to inject locals into a function namespace.

-1 on having the syntax for that masquerade as function arguments.




-- 
Steven



From scialexlight at gmail.com  Wed Jun 15 17:48:55 2011
From: scialexlight at gmail.com (Alex Light)
Date: Wed, 15 Jun 2011 11:48:55 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110614212739.GB2549@chopin.edu.pl>
References: <4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz>
	<it4al9$3c8$1@dough.gmane.org> <4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
	<BANLkTin74200ZuF704Xs27Niu+8weSZenA@mail.gmail.com>
	<20110613223027.GB3483@chopin.edu.pl>
	<20110614011214.GA5291@chopin.edu.pl>
	<BANLkTinwUQx9pNqbck91uC-yJVacYHXvhA@mail.gmail.com>
	<20110614212739.GB2549@chopin.edu.pl>
Message-ID: <BANLkTikh8c8H=WejpMfmXnxjKhvHLvq_bA@mail.gmail.com>

On Tue, Jun 14, 2011 at 5:27 PM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:

> Alex Light dixit (2011-06-14, 13:44):
>
> > >>>            frmglobals = func.__globals__.copy()
> > >>>            func.__globals__.update(localArgs)
> > >>>            ret= func(*args, **kwargs)
> > >>>            func.__globals__.clear()
> > >>>            func.__globals__.update(frmglobals)
>
> Changing global state on each call seems to be both
> concurrency-and-recurrency-unsafe and inefficient.
>

well some of your safety concerns can be allayed, i hope, by replacing this
snipet:

>>>            frmglobals = func.__globals__.copy()
>>>            func.__globals__.update(localArgs)
>>>            ret= func(*args, **kwargs)
>>>            func.__globals__.clear()
>>>            func.__globals__.update(frmglobals)

with this one:

>>>            with _modifyGlobals(func.__globals__, localArgs):
>>>                ret = func(*args, **kwargs)

with  _modifyGlobals defined as:

>>>from contextlib import contextmanager
>>>
>>>@contextmanager
>>>def _modifyGlobals(glbls, additions):
>>>    frmglbls = glbls.copy()
>>>    try:
>>>        glbls.update(additions)
>>>        yield
>>>    finally:
>>>        glbls.clear()
>>>        glbls.update(frmglbls)

as for performance you are correct that it is more efficient to just use
global variables.
the dictionary updates add about 1 x 10**-4 (or, if the check for collisions
with KWargs is removed, 5 x 10**-5)
seconds  to the run time of a function, at least on this computer.
so not terribly significant just be sure to use sparingly

also with the link you mentioned i could not seem to get it to work.
Whenever i tried to use any built-in functions
it would start throwing NameErrors. also that is only useful if you want to
inject all global variables into the function.

--Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110615/d9179001/attachment.html>

From tjreedy at udel.edu  Wed Jun 15 21:04:03 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 15 Jun 2011 15:04:03 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTikh8c8H=WejpMfmXnxjKhvHLvq_bA@mail.gmail.com>
References: <4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>	<20110612222236.GA2540@chopin.edu.pl>	<4DF53DF1.9060804@canterbury.ac.nz>	<it4al9$3c8$1@dough.gmane.org>
	<4DF5B83C.9070206@pearwood.info>	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>	<BANLkTin74200ZuF704Xs27Niu+8weSZenA@mail.gmail.com>	<20110613223027.GB3483@chopin.edu.pl>	<20110614011214.GA5291@chopin.edu.pl>	<BANLkTinwUQx9pNqbck91uC-yJVacYHXvhA@mail.gmail.com>	<20110614212739.GB2549@chopin.edu.pl>
	<BANLkTikh8c8H=WejpMfmXnxjKhvHLvq_bA@mail.gmail.com>
Message-ID: <itavn5$sib$2@dough.gmane.org>

On 6/15/2011 11:48 AM, Alex Light wrote:

>     Alex Light dixit (2011-06-14, 13:44):
>
>      > >>>            frmglobals = func.__globals__.copy()
>      > >>>            func.__globals__.update(localArgs)
>      > >>>            ret= func(*args, **kwargs)
>      > >>>            func.__globals__.clear()
>      > >>>            func.__globals__.update(frmglobals)

>  >>>            frmglobals = func.__globals__.copy()
>  >>>            func.__globals__.update(localArgs)
>  >>>            ret= func(*args, **kwargs)
>  >>>            func.__globals__.clear()
>  >>>            func.__globals__.update(frmglobals)
>
> with this one:
>
>  >>>            with _modifyGlobals(func.__globals__, localArgs):
>  >>>                ret = func(*args, **kwargs)
>
> with  _modifyGlobals defined as:
>
>  >>>from contextlib import contextmanager
>  >>>
>  >>>@contextmanager
>  >>>def _modifyGlobals(glbls, additions):
>  >>>    frmglbls = glbls.copy()
>  >>>    try:
>  >>>        glbls.update(additions)
>  >>>        yield
>  >>>    finally:
>  >>>        glbls.clear()
>  >>>        glbls.update(frmglbls)

Posting code with all the '>>>' prompts added makes it terribly hard to 
cut and paste to try it out.

-- 
Terry Jan Reedy



From scialexlight at gmail.com  Wed Jun 15 21:23:41 2011
From: scialexlight at gmail.com (Alex Light)
Date: Wed, 15 Jun 2011 15:23:41 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <itavn5$sib$2@dough.gmane.org>
References: <4BDDEB96-1EBD-477C-BA08-7750AB42B579@gmail.com>
	<20110612222236.GA2540@chopin.edu.pl>
	<4DF53DF1.9060804@canterbury.ac.nz>
	<it4al9$3c8$1@dough.gmane.org> <4DF5B83C.9070206@pearwood.info>
	<BANLkTi=ROhEAb86V53nJmJoMucgR2GaEoQ@mail.gmail.com>
	<BANLkTin74200ZuF704Xs27Niu+8weSZenA@mail.gmail.com>
	<20110613223027.GB3483@chopin.edu.pl>
	<20110614011214.GA5291@chopin.edu.pl>
	<BANLkTinwUQx9pNqbck91uC-yJVacYHXvhA@mail.gmail.com>
	<20110614212739.GB2549@chopin.edu.pl>
	<BANLkTikh8c8H=WejpMfmXnxjKhvHLvq_bA@mail.gmail.com>
	<itavn5$sib$2@dough.gmane.org>
Message-ID: <BANLkTimRSUD5bY9dmtxjnKGqaZCdxBQq0w@mail.gmail.com>

Here is the source if anyone wants it.
--Alex

On Wed, Jun 15, 2011 at 3:04 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 6/15/2011 11:48 AM, Alex Light wrote:
>
>     Alex Light dixit (2011-06-14, 13:44):
>>
>>     > >>>            frmglobals = func.__globals__.copy()
>>     > >>>            func.__globals__.update(localArgs)
>>     > >>>            ret= func(*args, **kwargs)
>>     > >>>            func.__globals__.clear()
>>     > >>>            func.__globals__.update(frmglobals)
>>
>
>   >>>            frmglobals = func.__globals__.copy()
>>  >>>            func.__globals__.update(localArgs)
>>  >>>            ret= func(*args, **kwargs)
>>  >>>            func.__globals__.clear()
>>  >>>            func.__globals__.update(frmglobals)
>>
>> with this one:
>>
>>  >>>            with _modifyGlobals(func.__globals__, localArgs):
>>  >>>                ret = func(*args, **kwargs)
>>
>> with  _modifyGlobals defined as:
>>
>>  >>>from contextlib import contextmanager
>>  >>>
>>  >>>@contextmanager
>>  >>>def _modifyGlobals(glbls, additions):
>>  >>>    frmglbls = glbls.copy()
>>  >>>    try:
>>  >>>        glbls.update(additions)
>>  >>>        yield
>>  >>>    finally:
>>  >>>        glbls.clear()
>>  >>>        glbls.update(frmglbls)
>>
>
> Posting code with all the '>>>' prompts added makes it terribly hard to cut
> and paste to try it out.
>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110615/19cc4747/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: make_local.py
Type: application/octet-stream
Size: 1776 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110615/19cc4747/attachment.obj>

From zuo at chopin.edu.pl  Thu Jun 16 01:15:36 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Thu, 16 Jun 2011 01:15:36 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF89903.6030202@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl> <4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>
Message-ID: <20110615231536.GA2182@chopin.edu.pl>

Steven D'Aprano dixit (2011-06-15, 21:35):

> This is another reason why function parameters should not be used for
> something that is not a function parameter!
> 
> +1 on the ability to inject locals into a function namespace.
> 
> -1 on having the syntax for that masquerade as function arguments.

OK, so the decorator or decorator-like syntax (using 'inject', 'within',
'owns' or other decorator name...) seems to be the most promising
alternative. If so, next question is: which variant? 

1. Decorator function with closure-like injecting
   (possibly could be implemented using closures):

    @functools.owns(cache=dict(), MAX_CACHE_LEN=100)  
    def calculate(a, b):
        result = cache[(a, b)]
        if result is not None:
            return result
        ...
        # 'cache' identifier cannot be rebound to another object
        # because it was already used above in the function body
        # to refer to the injected object

functools.owns() would be a real decorator function -- to apply either with 
@-syntax or dynamically, e.g.:

    decorated = [functools.owns(func) for func in functions]

One question is whether it is technically possible to avoid introducing
a new keyword (e.g. staticlocal) explicitly marking injected locals.
Using such a keyword would be redundant from user point of view and
non-DRY:

    @functools.owns(cache=dict(), MAX_CACHE_LEN=100)  
    def calculate(a, b):
        staticlocal cache, MAX_CACHE_LEN  # <- redundant and non-DRY :-(
        result = cache[(a, b)]
        if result is not None:
            return result
        ...


2. Decorator function with argument-like injecting.

    @functools.owns(cache=dict(), MAX_CACHE_LEN=100)  
    def calculate(a, b):
        result = cache[(a, b)]
        if result is not None:
            return result
        ...
        # 'cache' identifier *can* be rebound to another object
        # than the injected object -- in the same way arguments can

functools.owns() would be a real decorator function -- to apply either with 
@-syntax or dynamically, e.g.:

    decorated = [functools.owns(func) for func in functions]

To implement such variant -- a new function constructor argument(s)
and/or function/function code attribute(s) (read-only or writable?)
most probably would have to be introduced...


3. Decorator-like language syntax construct:

    @in(cache=dict(), MAX_CACHE_LEN=100)  # or 'owns' or 'inject' or...
    def calculate(a, b):
        result = cache[(a, b)]
        if result is not None:
            return result
        ...
        # 'cache' identifier *can* be rebound to another object
        # than the injected object -- in the same way arguments can

It would not be a real decorator function -- so it would be applicable
only using this syntax, and not dynamically, not after function creation.


Which do you prefer? (or any other?)

Regards.
*j



From steve at pearwood.info  Thu Jun 16 02:46:51 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 16 Jun 2011 10:46:51 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110615231536.GA2182@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>	<20110614101755.GB3754@chopin.edu.pl>	<20110615002825.GA2217@chopin.edu.pl>
	<4DF84F7B.3010700@improva.dk>	<4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
Message-ID: <4DF9527B.9060307@pearwood.info>

Jan Kaliszewski wrote:

> OK, so the decorator or decorator-like syntax (using 'inject', 'within',
> 'owns' or other decorator name...) seems to be the most promising
> alternative. If so, next question is: which variant? 

"owns"? I don't see how that testing for ownership describes what the 
function does.

Likewise for "within", which sounds like it should be a synonym for the 
"in" operator: "if value within range" sort of thing.


> 1. Decorator function with closure-like injecting
>    (possibly could be implemented using closures):
> 
>     @functools.owns(cache=dict(), MAX_CACHE_LEN=100)  
>     def calculate(a, b):
>         result = cache[(a, b)]
>         if result is not None:
>             return result
>         ...
>         # 'cache' identifier cannot be rebound to another object
>         # because it was already used above in the function body
>         # to refer to the injected object

Making locals unrebindable is a change of semantics that is far beyond 
anything I've been discussed here. This will be a big enough change 
without overloading it with changes that will be even more controversial!

(I actually do like the idea of having unrebindable names, but that 
should be kept as a separate issue and not grafted on to this proposal.)


> functools.owns() would be a real decorator function -- to apply either with 
> @-syntax or dynamically, e.g.:
> 
>     decorated = [functools.owns(func) for func in functions]

There shouldn't even be a question about that. Decorator syntax is sugar 
for func = decorator(func). Introducing magic syntax that is recognised 
by the compiler but otherwise is not usable as a function is completely 
unacceptable.

If func is a pre-existing function:

def func(a, b, c):
     pass


then:

new_func = functools.inject(x=1, y=2)(func)


should be the same as:

def new_func(a, b, c):
     # inject locals into the body of the function
     x = 1
     y = 2
     # followed by the body of the original
     pass


except that new_func.__name__ may still reflect the old name "func".


* If the original function previously referenced global or nonlocal x 
and y, the new function must now treat them as local;

* Bindings to x and y should occur once, at function definition time, 
similar to the way default arguments occur once;

* The original function (before the decorator applies) must be untouched 
rather than modified in place.

This implies to me that inject must copy the original function and make 
modifications to the code object. This sounds to me that a 
proof-of-concept implementation would be doable using a byte-code hack.



-- 
Steven


From dstanek at dstanek.com  Thu Jun 16 03:04:15 2011
From: dstanek at dstanek.com (David Stanek)
Date: Wed, 15 Jun 2011 21:04:15 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110611133028.GD2395@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>
Message-ID: <BANLkTikitYhFxQLVW4EF395+Z01HwJvQ9w@mail.gmail.com>

On Sat, Jun 11, 2011 at 9:30 AM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:

>
> == Proposed solutions ==
>
> I see three possibilities:
>
> 1.
> To add a new keyword, e.g. `inject':
>    def do_and_remember(val, verbose=False):
>        inject mem = collections.Counter()
>        ...
> or maybe:
>    def do_and_remember(val, verbose=False):
>        inject collections.Counter() as mem
>        ...
>
> 2. (which personally I would prefer)
> To add `dummy' (or `hidden') keyword arguments, defined after **kwargs
> (and after bare ** if kwargs are not needed; we have already have
> keyword-only arguments after *args or bare *):
>
>    def do_and_remember(val, verbose=False, **, mem=collections.Counter()):
>        ...
>
> do_and_remember(val, False, mem='something') would raise TypeError and
> `mem' shoudn not appear in help() etc. as a function argument.
>
> 3.
> To provide a special decorator, e.g. functools.within:
>    @functools.within(mem=collections.Counter())
>    def do_and_remember(val, verbose=False):
>        ...
>

For these cases I use a class based solution. It's simple and easy to test.


class DoAndRemember:

    def __init__(self):
        self._mem = collections.Counter()

    def __call__(self, val, verbose=False):
        result = do_something(val)
        self.mem[val] += 1
        if verbose:
            print('Done {} times for {!r}'.format(mem[val], val))

do_and_remember = DoAndRemember()


-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek
www: http://dstanek.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110615/26ffc25d/attachment.html>

From ncoghlan at gmail.com  Thu Jun 16 05:41:17 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 16 Jun 2011 13:41:17 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110615231536.GA2182@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl> <4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
Message-ID: <BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>

On Thu, Jun 16, 2011 at 9:15 AM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
> One question is whether it is technically possible to avoid introducing
> a new keyword (e.g. staticlocal) explicitly marking injected locals.
> Using such a keyword would be redundant from user point of view and
> non-DRY:

There has to be *something* that tells the compiler to generate
different bytecode (either a local lookup, a closure lookup or
something new). This lands in the same category as "nonlocal" and
"global" (and no, a "magic" decorator that the compiler recognises is
not a reasonable alternative).

That's why I quite liked the @def idea - it would just define a
sequence of simple statements that the compiler will run at function
*definition* time that is then used to preseed the local namespace at
function *call* time (remember, it is intended to be a mnemonic for
"at definition time" and the use of '@' also reflects the fact that
this code would run just before function decorators are executed).
Just like a class body, the @def code itself would be thrown away and
only the resulting preseeded locals information would be retained. To
allow rebinding to work correctly, this shared state could be
implemented via cell variables rather than ordinary locals.

Possible implementation sketch:

  Compile time:
    - @def statements are compiled in the context of the containing
scope and stored on a new ASDL sequence attribute in the Function AST
    - symtable analysis notes explicitly which names are bound in the
@def statements and this information is stored on the code object
    - code generation produces a cell lookup for any names bound in
@def statements (even if they are also assigned as ordinary locals)
    - Raises a SyntaxError if there is a conflict between parameter
names and names bound in @def statements

  Definition time:
    - @def statements are executed as a suite in the context of the
containing scope but using a *copy* of the locals (so the containing
scope is not modified)
    - names bound in the @def statements (as noted on the code object)
are linked up to the appropriate cells on the function object

  Execution time:
    - Nothing special. The code is executed and references the cell
variables precisely as if they came from a closure.

An API could be provided in functools to provide a clean way to view
(and perhaps modify) the contents of the cells from outside the
function. And yes, I'm aware this blurs the line even further between
functions and classes, but the core difference between "a specific
algorithm with some persistent state" and "persistent state with
optional associated algorithms" remains intact.

And, to repeat the example of how it would look in practice:

   def do_and_remember(val, verbose=False):
         @def mem=collections.Counter()
         # Algorithm that calculates result given val
         mem[val] += 1
         if verbose:
             print('Done {} times for {!r}'.format(_mem[val], val))
         return result

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From zuo at chopin.edu.pl  Thu Jun 16 18:58:03 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Thu, 16 Jun 2011 18:58:03 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DF9527B.9060307@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl> <4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<4DF9527B.9060307@pearwood.info>
Message-ID: <20110616165803.GA2222@chopin.edu.pl>

Steven D'Aprano dixit (2011-06-16, 10:46):

> Making locals unrebindable is a change of semantics that is far
> beyond anything I've been discussed here. This will be a big enough
> change without overloading it with changes that will be even more
> controversial!

That variant (#1) would be simply shortcut for a closure application
-- nothing really new.

    def factory(n):
        """Today's Python example."""
        closuring = 'foo'
        def func(m):
            s = min(n, m) * closuring
            # here you also cannot rebind 'closuring' because is has
            # been referenced above

> Introducing magic syntax that is recognised by the compiler but
> otherwise is not usable as a function is completely unacceptable.

Because of?... And it would not be more 'magic' than any other
language syntax construct -- def, class, decorating with their @, *, **
arguments etc.  The fact that such a new syntax would be similar to
something already known and well settled (decorator function application
syntax) would be rather an andantage than a drawback.

> * If the original function previously referenced global or nonlocal x
> and y, the new function must now treat them as local;
> 
> * Bindings to x and y should occur once, at function definition time,
> similar to the way default arguments occur once;
> 
> * The original function (before the decorator applies) must be
> untouched rather than modified in place.
> 
> This implies to me that inject must copy the original function and
> make modifications to the code object. This sounds to me that a
> proof-of-concept implementation would be doable using a byte-code
> hack.

That's what I propose as variant #2. But that would need byte code
hacking -- or some core language ('magic') modifications.

Cheers.
*j



From zuo at chopin.edu.pl  Thu Jun 16 19:15:25 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Thu, 16 Jun 2011 19:15:25 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl> <4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
Message-ID: <20110616171525.GB2222@chopin.edu.pl>

Nick Coghlan dixit (2011-06-16, 13:41):

> On Thu, Jun 16, 2011 at 9:15 AM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
> > One question is whether it is technically possible to avoid introducing
> > a new keyword (e.g. staticlocal) explicitly marking injected locals.
> > Using such a keyword would be redundant from user point of view and
> > non-DRY:
> 
> There has to be *something* that tells the compiler to generate
> different bytecode (either a local lookup, a closure lookup or
> something new). This lands in the same category as "nonlocal" and
> "global" (and no, a "magic" decorator that the compiler recognises is
> not a reasonable alternative).
> 
> That's why I quite liked the @def idea - it would just define a
> sequence of simple statements that the compiler will run at function
> *definition* time that is then used to preseed the local namespace at
> function *call* time (remember, it is intended to be a mnemonic for
> "at definition time" and the use of '@' also reflects the fact that
> this code would run just before function decorators are executed).
> Just like a class body, the @def code itself would be thrown away and
> only the resulting preseeded locals information would be retained. To
> allow rebinding to work correctly, this shared state could be
> implemented via cell variables rather than ordinary locals.
> 
> Possible implementation sketch:
> 
>   Compile time:
>     - @def statements are compiled in the context of the containing
> scope and stored on a new ASDL sequence attribute in the Function AST
>     - symtable analysis notes explicitly which names are bound in the
> @def statements and this information is stored on the code object
>     - code generation produces a cell lookup for any names bound in
> @def statements (even if they are also assigned as ordinary locals)
>     - Raises a SyntaxError if there is a conflict between parameter
> names and names bound in @def statements
> 
>   Definition time:
>     - @def statements are executed as a suite in the context of the
> containing scope but using a *copy* of the locals (so the containing
> scope is not modified)
>     - names bound in the @def statements (as noted on the code object)
> are linked up to the appropriate cells on the function object
> 
>   Execution time:
>     - Nothing special. The code is executed and references the cell
> variables precisely as if they came from a closure.
> 
> An API could be provided in functools to provide a clean way to view
> (and perhaps modify) the contents of the cells from outside the
> function. And yes, I'm aware this blurs the line even further between
> functions and classes, but the core difference between "a specific
> algorithm with some persistent state" and "persistent state with
> optional associated algorithms" remains intact.
> 
> And, to repeat the example of how it would look in practice:
> 
>    def do_and_remember(val, verbose=False):
>          @def mem=collections.Counter()
>          # Algorithm that calculates result given val
>          mem[val] += 1
>          if verbose:
>              print('Done {} times for {!r}'.format(_mem[val], val))
>          return result

It is not less 'magic' than what I proposed as variant #3. And, in fact,
it is almost the same -- the only important difference is the place:
imho placing it *before* definition better emphasizes that the binding
is an *early* one.

    @inject(mem=collections.Counter(), MAX_MEM=1000)
    def do_and_remember(val, verbose=False):

or even (to stress that it is a language syntax construct:

    @inject mem=collections.Counter(), MAX_MEM=1000
    def do_and_remember(val, verbose=False):

or:

    @inject collections.Counter() as mem, 1000 as MAX_MEM
    def do_and_remember(val, verbose=False):

or something similar...

Also, such placement is imho more appropriate for @-that-starts-a-line-
-syntax (because it would *resemble* decorating syntax anyway -- and
what's wrong with that?) + we avoid misleading clusters such as:

    def do_something(func):
        @def mem=colletions.Counter  # <- looks a bit like another
        @wraps(func)                 #    decorator for wrapper_func()
        @my_decorator(mem)
        def wrapper_func():
            ...

Regards.
*j



From scialexlight at gmail.com  Thu Jun 16 19:21:15 2011
From: scialexlight at gmail.com (Alex Light)
Date: Thu, 16 Jun 2011 13:21:15 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110616165803.GA2222@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl>
	<4DF84F7B.3010700@improva.dk> <4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl> <4DF9527B.9060307@pearwood.info>
	<20110616165803.GA2222@chopin.edu.pl>
Message-ID: <BANLkTikmqpavmRP68jXHmQZz_KGDBdEjxg@mail.gmail.com>

On Thu, Jun 16, 2011 at 12:58 PM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:

> Steven D'Aprano dixit (2011-06-16, 10:46):
> > This implies to me that inject (option 1) must copy the original function
> and
> > make modifications to the code object. This sounds to me that a
> > proof-of-concept implementation would be doable using a byte-code
> > hack.
>
> That's what I propose as variant #2. But that would need byte code
> hacking -- or some core language ('magic') modifications.


I agree with D'Aprano, both option one and two would require modifying the
code object,
bytecode hacks, or core changes to the language, because, although the
result is the
same as your factory example the function is already compiled when it is
given to the
decorator.

my best guess is that to implement this we would need to slightly redefine
the way that
python looks up variables by making it look in a special 'injected'
dictionay after looking
through locals and before globals.

--Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110616/35562d01/attachment.html>

From zuo at chopin.edu.pl  Thu Jun 16 23:45:08 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Thu, 16 Jun 2011 23:45:08 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTikmqpavmRP68jXHmQZz_KGDBdEjxg@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl> <4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<4DF9527B.9060307@pearwood.info>
	<20110616165803.GA2222@chopin.edu.pl>
	<BANLkTikmqpavmRP68jXHmQZz_KGDBdEjxg@mail.gmail.com>
Message-ID: <20110616214508.GA4133@chopin.edu.pl>

Alex Light dixit (2011-06-16, 13:21):

> my best guess is that to implement this we would need to slightly
> redefine the way that python looks up variables by making it look in a
> special 'injected' dictionay after looking through locals and before
> globals.

But one of the default-argument-hack reasons is to optimize variable
access by avoid dictionary loopup (locals are not dictionary-based).

I'd rather stick to the implementation sketch described by Nick
(changing the syntax a bit; as I said, imho @keyword... should be placed
before function def).

Cheers.
*j



From steve at pearwood.info  Fri Jun 17 05:21:03 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 17 Jun 2011 13:21:03 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110616165803.GA2222@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>	<20110614101755.GB3754@chopin.edu.pl>	<20110615002825.GA2217@chopin.edu.pl>
	<4DF84F7B.3010700@improva.dk>	<4DF89903.6030202@pearwood.info>	<20110615231536.GA2182@chopin.edu.pl>	<4DF9527B.9060307@pearwood.info>
	<20110616165803.GA2222@chopin.edu.pl>
Message-ID: <4DFAC81F.2070005@pearwood.info>

Jan Kaliszewski wrote:
> Steven D'Aprano dixit (2011-06-16, 10:46):
> 
>> Making locals unrebindable is a change of semantics that is far
>> beyond anything I've been discussed here. This will be a big enough
>> change without overloading it with changes that will be even more
>> controversial!
> 
> That variant (#1) would be simply shortcut for a closure application
> -- nothing really new.
> 
>     def factory(n):
>         """Today's Python example."""
>         closuring = 'foo'
>         def func(m):
>             s = min(n, m) * closuring
>             # here you also cannot rebind 'closuring' because is has
>             # been referenced above

The error occurs BEFORE the rebinding attempt. You get UnboundLocalError 
when you attempt to execute min(n, m), not when rebinding the closure 
variable. This is a side-effect of the compiler's rule "if you see an 
assignment to a variable, make it a local", that is all.

You can rebind closuring if you tell the compiler that it isn't a local 
variable:

 >>> def factory(n):
...     closuring = 'foo'
...     def func(m):
...         nonlocal closuring
...         s = min(n, m)*closuring
...         closuring = 'spam'
...         return s + closuring
...     return func
...
 >>> f = factory(10)
 >>> f(3)
'foofoofoospam'


In that regard, closure variables are no different from globals. You 
wouldn't say that global are unrebindable because of this:

 >>> def f():
...     print(x)
...     x = 1
...
 >>> f()
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "<stdin>", line 2, in f
UnboundLocalError: local variable 'x' referenced before assignment


The situation is very similar.


>> Introducing magic syntax that is recognised by the compiler but
>> otherwise is not usable as a function is completely unacceptable.
> 
> Because of?... And it would not be more 'magic' than any other
> language syntax construct -- def, class, decorating with their @, *, **
> arguments etc.  The fact that such a new syntax would be similar to
> something already known and well settled (decorator function application
> syntax) would be rather an andantage than a drawback.

But the problem is that it is deceptively similar: it only *seems* 
similar, while the differences are profound.

super() is the only magic function I know of in Python, and that change 
was controversial, hard to implement, and fragile. super() is special 
cased by the compiler and works in ways that no other function can do. 
Hence it is magic. I can't imagine that Guido will agree to a second 
example, at least not without a blindingly obvious benefit.

You can't reason about super()'s behaviour like any other function. 
Things which should work if super() were non-magical break, such as 
aliasing:

my_super = super  # Just another name for the same function.

class MyList(list):
     def __init__(self, *args):
         my_super().__init__(*args)
         self.attr = None

 >>> MyList([])
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "<stdin>", line 3, in __init__
SystemError: super(): __class__ cell not found


And wrapping:


_saved_super = super
def super(*args, **kwargs):
     print(args, kwargs)
     return _saved_super(*args, **kwargs)

class MyList(list):
     def __init__(self, *args):
         super().__init__(*args)
         self.attr = None


 >>> MyList([])
() {}
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "<stdin>", line 3, in __init__
   File "<stdin>", line 3, in super
SystemError: super(): no arguments



Only the exact incantation of built-in super() inside a method of a 
class works. As I said: magic. (Although you can supply all the 
arguments for super manually, which is tricky to get right but non-magic.)


You are proposing that inject should also be magic: only the exact 
incantation @inject(...) directly above a function will work. We won't 
be able to wrap inject in another function, or alias it, or use it 
without the @ syntax. inject() isn't really a decorator, although it 
superficially looks like one. It's actually a compiler directive.

If you want to propose #pragma for Python, do so, but don't call it a 
decorator!

Most importantly, we won't be able to apply it to functions that already 
exist:

list_of_functions = [spam, ham, cheese]  # defined elsewhere
decorator = inject(a=1)
decorated = [decorator(f) for f in list_of_functions]

will fail. I consider this completely unacceptable.




-- 
Steven



From ncoghlan at gmail.com  Fri Jun 17 06:39:50 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 17 Jun 2011 14:39:50 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110616171525.GB2222@chopin.edu.pl>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl> <4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
Message-ID: <BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>

On Fri, Jun 17, 2011 at 3:15 AM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
> or even (to stress that it is a language syntax construct:
>
> ? ?@inject mem=collections.Counter(), MAX_MEM=1000
> ? ?def do_and_remember(val, verbose=False):

While that would require the from__future__ dance to make "inject" a
new keyword, I like it much better than the
looks-like-a-decorator-but-isn't syntax.

The '@def' keyword would also technically work with that positioning,
but @inject provides a better mnemonic for what is going on when the
assignments are positioned outside the function.

> Also, such placement is imho more appropriate for @-that-starts-a-line-
> -syntax (because it would *resemble* decorating syntax anyway -- and
> what's wrong with that?) + we avoid misleading clusters such as:
>
> ? ?def do_something(func):
> ? ? ? ?@def mem=colletions.Counter ?# <- looks a bit like another
> ? ? ? ?@wraps(func) ? ? ? ? ? ? ? ? # ? ?decorator for wrapper_func()
> ? ? ? ?@my_decorator(mem)
> ? ? ? ?def wrapper_func():
> ? ? ? ? ? ?...

The advantage of putting the '@def' lines *inside* the function is
that it makes it clearer which namespace they're affecting. Examples
like the above are readily addressed via style rules that say "don't
do that, it's hard to read - leave a blank line between the @def code
and the subsequent function decorators"

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From steve at pearwood.info  Fri Jun 17 07:37:46 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 17 Jun 2011 15:37:46 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>	<20110614101755.GB3754@chopin.edu.pl>	<20110615002825.GA2217@chopin.edu.pl>
	<4DF84F7B.3010700@improva.dk>	<4DF89903.6030202@pearwood.info>	<20110615231536.GA2182@chopin.edu.pl>	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
Message-ID: <4DFAE82A.8040103@pearwood.info>

Nick Coghlan wrote:
> On Fri, Jun 17, 2011 at 3:15 AM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
>> or even (to stress that it is a language syntax construct:
>>
>>    @inject mem=collections.Counter(), MAX_MEM=1000
>>    def do_and_remember(val, verbose=False):
> 
> While that would require the from__future__ dance to make "inject" a
> new keyword, I like it much better than the
> looks-like-a-decorator-but-isn't syntax.

What benefit is there in making inject a keyword? Even super isn't a 
keyword.

As far as I'm concerned, inject need only be a function in the functools 
module, not even a built-in, let alone a keyword.

Here's a quick and dirty version that comes close to the spirit of 
inject, as I see it. Thanks to Alex Light's earlier version.


# Credit to Alex Light.
from contextlib import contextmanager
from functools import wraps

def inject(**localArgs):
     def decorator(func):
         glbs = func.__globals__
         @wraps(func)
         def inner(*args, **kwargs):
             with _modifyGlobals(glbs, localArgs):
                 ret = func(*args, **kwargs)
             return ret
         return inner
     return decorator

@contextmanager
def _modifyGlobals(glbls, additions):
     frmglbls = glbls.copy()
     try:
         glbls.update(additions)
         yield
     finally:
         glbls.clear()
         glbls.update(frmglbls)



And demonstrating it in use:


 >>> def func(obj):
...     print(len)
...     return len(obj)+1
...
 >>> import builtins
 >>> newfunc = inject(len=lambda o: print(o) or builtins.len(o))(func)
 >>>
 >>> func([])  # Original function unchanged, still uses uninjected len.
<built-in function len>
1
 >>> newfunc([])  # New function uses injected len.
<function <lambda> at 0xb7c2f86c>
[]
1


And as a decorator:

 >>> @inject(a=1)
... def spam():
...     print(a)
...
 >>> a = 42
 >>> spam()
1


Unfortunately, this proof-of-concept inject function doesn't actually 
inject into locals, hence the "import builtins" work-around. But it 
demonstrates the intent, and the API.



-- 
Steven


From ericsnowcurrently at gmail.com  Fri Jun 17 07:48:25 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 16 Jun 2011 23:48:25 -0600
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl> <4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
Message-ID: <BANLkTinF9fcQGmgdF1NMRiC4tW=MaguOkA@mail.gmail.com>

On Thu, Jun 16, 2011 at 10:39 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Fri, Jun 17, 2011 at 3:15 AM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
>> or even (to stress that it is a language syntax construct:
>>
>> ? ?@inject mem=collections.Counter(), MAX_MEM=1000
>> ? ?def do_and_remember(val, verbose=False):
>
> While that would require the from__future__ dance to make "inject" a
> new keyword, I like it much better than the
> looks-like-a-decorator-but-isn't syntax.
>

Even still, at first glance that looks like a decorator.  Also, going
multiline makes it worse:

 ? ?@inject (mem=collections.Counter(),
                 MAX_MEM=1000)
 ? ?def do_and_remember(val, verbose=False):
        ...

>
> The advantage of putting the '@def' lines *inside* the function is
> that it makes it clearer which namespace they're affecting. Examples
> like the above are readily addressed via style rules that say "don't
> do that, it's hard to read - leave a blank line between the @def code
> and the subsequent function decorators"
>

In my mind, putting it inside the function is similar to docstrings
being inside, especially if the simple statements are evaluated at
definition time.  Also, I kind of like the idea of combing @ with the
keyword since it distinguishes the context.

What about when you have several of these and they get long?  Could
you use parentheses?  With the def keyword and parentheses it looks
different enough from a function that I don't mind it, but maybe the
keyword could be different (I do like that it reuses a keyword):

  def f(a,b):
      """Do something...

      """
      @def (x=[name for name in names if name != None],
               y=something_else)

      print(a, b)
      print([y(name) for name in x])

(And a given statement would fit in nicely instead of those parentheses. ;)

-eric

> Cheers,
> Nick.
>
> --
> Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From ericsnowcurrently at gmail.com  Fri Jun 17 07:49:36 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 16 Jun 2011 23:49:36 -0600
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTinF9fcQGmgdF1NMRiC4tW=MaguOkA@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl> <4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
	<BANLkTinF9fcQGmgdF1NMRiC4tW=MaguOkA@mail.gmail.com>
Message-ID: <BANLkTi=z4qm-UBa_7iL=h4r7uzJOn6C2TA@mail.gmail.com>

> definition time. ?Also, I kind of like the idea of combing @ with the

s/combing/combining/

-eric


From ericsnowcurrently at gmail.com  Fri Jun 17 08:13:57 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 17 Jun 2011 00:13:57 -0600
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DFAE82A.8040103@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl> <4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
	<4DFAE82A.8040103@pearwood.info>
Message-ID: <BANLkTingnPvM8-X5PqvqiGPRmha7OpVSfg@mail.gmail.com>

On Thu, Jun 16, 2011 at 11:37 PM, Steven D'Aprano <steve at pearwood.info> wrote:
>
> What benefit is there in making inject a keyword? Even super isn't a
> keyword.
>
> As far as I'm concerned, inject need only be a function in the functools
> module, not even a built-in, let alone a keyword.
>

If inject is a decorator then it would be used to inject into the
locals of any function at runtime.  This is in contrast to Nick's
proposal where it's strictly tied to definition time and injection is
internal to to the subject of the injection, the function body.

In the latter case, the resulting code object from the function body
could incorporate the injection right there.  To accomplish the same
thing with a decorator, the function definition would have to know
about any possible decoration before the code object is compiled (good
luck), or the decorator would replace/modify the compiled code.  Seems
like that's not far off from what Jan was proposing.

The alternative is to have the injection handled externally to the
compiled code object, like a co_static on the code object of a
__static__ on the function object.  Then the execution of the function
code object would pull that in.

-eric


From ncoghlan at gmail.com  Fri Jun 17 09:02:52 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 17 Jun 2011 17:02:52 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DFAE82A.8040103@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl> <4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
	<4DFAE82A.8040103@pearwood.info>
Message-ID: <BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>

On Fri, Jun 17, 2011 at 3:37 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> What benefit is there in making inject a keyword? Even super isn't a
> keyword.

Given the magic effects they have on the compiler, super and __class__
probably *should* be keywords. The only reason they aren't is that
their effect (automatically defining __class__ as a local when inside
a function in a class scope) is relatively harmless in the event that
super has actually been rebound to refer to something other than the
builtin:

class C:
  def f(self):
    print(locals())
  def g(self):
    __class__
    print(locals())
  def h(self):
    super
    print(locals())

>>> C().f()
{'self': <__main__.C object at 0xde60d0>}
>>> C().g()
{'self': <__main__.C object at 0xde60d0>, '__class__': <class '__main__.C'>}
>>> C().h()
{'self': <__main__.C object at 0xde60d0>, '__class__': <class '__main__.C'>}

> As far as I'm concerned, inject need only be a function in the functools
> module, not even a built-in, let alone a keyword.
>
> Here's a quick and dirty version that comes close to the spirit of inject,
> as I see it. Thanks to Alex Light's earlier version.
>
>
> # Credit to Alex Light.
> from contextlib import contextmanager
> from functools import wraps
>
> def inject(**localArgs):
> ? ?def decorator(func):
> ? ? ? ?glbs = func.__globals__
> ? ? ? ?@wraps(func)
> ? ? ? ?def inner(*args, **kwargs):
> ? ? ? ? ? ?with _modifyGlobals(glbs, localArgs):
> ? ? ? ? ? ? ? ?ret = func(*args, **kwargs)
> ? ? ? ? ? ?return ret
> ? ? ? ?return inner
> ? ?return decorator

Sorry, I meant to point out why this was a bad idea when Alex first posted it.

The __globals__ reference on a function object refers to the globals
of the module where the function is defined. Modify the contents of
that dictionary and you modify the contents of that module. So this
"injection" approach not only affects the function being decorated,
but every other function in the module. Thread safety is completely
non-existent and cannot be handled locally within the decorated
function.

The reason something like @inject or @def is needed as a language
construct is because the object of the exercise is to define a new
kind of scope (call it "shared locals" for lack of a better name) and
we need the compiler's help to do it properly.

Currently, the closest equivalent to a shared locals scope is the
default argument namespace, which is why people use it that way: the
names are assigned values at function definition time, and they are
automatically copied into the frame locals whenever the function is
invoked. A shared namespace can also be created explicitly by using a
closure or a class, but both of those suffer from serious verbosity
(and hence readability) problems when the design intent you are aiming
to express is a single algorithm with some persistent state.

As noted in Jan's original message, using the default argument
namespace has its own flaws (rebinding of immutable targets not
working properly, cluttering the function signature on introspection,
risk of inadvertent replacement in the call), but if it didn't address
a genuine design need, it wouldn't be so popular. Hence the current
discussion, which reminds me a lot of the PEP 308 (ternary
expressions) discussion. Developers have proven they want this
functionality by coming up with a hack that does it, but the hack is
inherently flawed. Telling them "don't do that" is never going to
work, so the best way to eliminate usage of the hack is to provide a
way to do it *right*. (Relating back to PEP 308: how often do you see
the and/or hack in modern Python code written by anyone that learned
the language post Python 2.4?)

The runtime *semantics* of my implementation sketch (an additional set
of cells stored on the function object that are known to the compiler
and accessed via closure ) are almost certainly the right way to go:
it's a solution that cleanly handles rebinding of immutable targets
and avoids cluttering the externall visible function signature with
additional garbage. The only question is how to tell the compiler
about it, and there are three main options for that:

1. Embedded in the function header, modelled on the handling of
keyword-only arguments:

  def example(arg, **, cache=set(), invocations=0):
    """Record and return arguments seen and count the number of times
the function has been invoked"""
    invocations += 1
    cache.add(arg)
    return arg

  Pros: no bikeshedding about the keyword for the new syntax,
namespace for execution is clearly the same as that for default
arguments (i.e. the containing namespace)
  Cons: look like part of the argument namespace (when they really
aren't), no mnemonic to assist new users in remembering what they're
for, no open questions

2. Inside the function as a new statement type (bikeshed colour
options: @def, @shared, shared)

  def example(arg, **, cache=set(), invocations=0):
    """Record and return arguments seen and count the number of times
the function has been invoked"""
    @def cache=set(), invocations=0
    invocations += 1
    cache.add(arg)
    return arg

  Pros: implementation detail of shared state is hidden inside the
function where it belongs, keyword choice can provide a good mnemonic
for functionality
  Cons: needs new style rules on appropriate placements of @def/shared
statements (similar to nonlocal and global), use of containing
namespace for execution may be surprising
  Open Questions: whether to allow only one line with a tuple of
assignments or multiple lines, whether to allow simple assignments
only or any simple non-flow control statement

3. After the decorators and before the function definition (bikeshed
colour options: @def, @inject, @shared)

  @def cache=set(), invocations=0
  def example(arg)
    """Record and return arguments seen and count the number of times
the function has been invoked"""
    invocations += 1
    cache.add(arg)
    return arg

  Pros: keyword choice can provide a good mnemonic for functionality,
namespace for execution is clearly the same as that for decorator
expressions (i.e. the containing namespace)
  Cons: puts private implementation details ahead of the public
signature information, looks too much like an ordinary decorator
  Open Questions: whether to allow only one line with a tuple of
assignments or multiple lines

I already have too much on my to-do list to champion a PEP for this,
but I'd be happy to help someone else with the mechanics of writing
one and getting it published on python.org (hint, hint Jan!).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From steve at pearwood.info  Fri Jun 17 14:12:58 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 17 Jun 2011 22:12:58 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>	<20110614101755.GB3754@chopin.edu.pl>	<20110615002825.GA2217@chopin.edu.pl>	<4DF84F7B.3010700@improva.dk>	<4DF89903.6030202@pearwood.info>	<20110615231536.GA2182@chopin.edu.pl>	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>	<20110616171525.GB2222@chopin.edu.pl>	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>	<4DFAE82A.8040103@pearwood.info>
	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>
Message-ID: <4DFB44CA.1050605@pearwood.info>

Nick Coghlan wrote:
> On Fri, Jun 17, 2011 at 3:37 PM, Steven D'Aprano <steve at pearwood.info> wrote:

>> Here's a quick and dirty version that comes close to the spirit of inject,
>> as I see it. Thanks to Alex Light's earlier version.
[code snipped]
> 
> Sorry, I meant to point out why this was a bad idea when Alex first posted it.
> 
> The __globals__ reference on a function object refers to the globals
> of the module where the function is defined. Modify the contents of
> that dictionary and you modify the contents of that module. So this
> "injection" approach not only affects the function being decorated,
> but every other function in the module. Thread safety is completely
> non-existent and cannot be handled locally within the decorated
> function.

I believe that you may have missed that the _modifyGlobals context 
manager makes a copy of globals before modifying it.

But even if you are correct, and the implementation as given is broken, 
I had written:

[quote]
Unfortunately, this proof-of-concept inject function DOESN'T ACTUALLY 
INJECT INTO LOCALS [emphasis added], hence the "import builtins" 
work-around. But it demonstrates the intent, and the API.
[end quote]

There was no intention for this to be the working implementation, just 
to demonstrate the API. As I described earlier, "close to the spirit of 
inject".

I agree with much of the rest of your post (snipped for brevity), with a 
few additional points below:


> The only question is how to tell the compiler
> about it, and there are three main options for that:

Four actually.


> 1. Embedded in the function header, modelled on the handling of
> keyword-only arguments:
> 
>   def example(arg, **, cache=set(), invocations=0):

>   Cons: look like part of the argument namespace (when they really
> aren't), no mnemonic to assist new users in remembering what they're
> for, no open questions

Additional con: you can only inject such locals at the time you write 
the function. Cannot take an existing function and make a runtime 
modification of it.

This is, essentially, a compiler directive masquerading as function 
parameters.


> 2. Inside the function as a new statement type (bikeshed colour
> options: @def, @shared, shared)

>   Pros: implementation detail of shared state is hidden inside the
> function where it belongs, keyword choice can provide a good mnemonic
> for functionality

This proposal shouldn't be just about shared state. That is just one 
use-case out of a number, and not all use-cases should be hidden.


>   Cons: needs new style rules on appropriate placements of @def/shared
> statements (similar to nonlocal and global), use of containing
> namespace for execution may be surprising

Additional cons: looks too much like a decorator, particularly if the 
function contains an inner function; also looks too much like a function 
definition. Can only be performed when the function is written, and 
cannot be applied to existing functions.

This too is a compiler directive, this time masquerading as a decorator.


>   Open Questions: whether to allow only one line with a tuple of
> assignments or multiple lines, whether to allow simple assignments
> only or any simple non-flow control statement

Whether the @def line must appear at the start of the function (before 
or after the docstring), or like globals, can it appear anywhere in the 
function?


> 3. After the decorators and before the function definition (bikeshed
> colour options: @def, @inject, @shared)

This too is a compiler directive masquerading as a decorator. It too 
suffers from much the same cons as putting @def inside the function body.


You have missed a fourth option, which I have been championing: make 
inject an ordinary function, available from the functools module. The 
*implementation* of inject almost certainly will require support from 
the compiler, but that doesn't mean the interface should!

Pros:

- "Inject" is the obvious name, because that's what it does: inject the 
given keyword arguments into a (copy of a) function as locals.

- Not a compiler directive, but an ordinary function that operates at 
runtime like any other function.

- Hence it works like ordinary decorators.

- Doesn't require a new keyword or new syntax.

- Can be applied to any Python function at any time, not just when the 
function is written.

Cons:

- The implementation will require unsupported bytecode hacks, or 
compiler support, but so will any other solution. This is only a 
negative when compared to the alternative "do nothing".

- Some people may disagree that "inject" is the obvious name. There may 
still be room for bikeshedding.

Open questions:

- Should injected locals go directly into the locals, as if executed in 
the body of the function, or into a new "shared/injected locals" 
namespace as suggested by Nick?




-- 
Steven


From ncoghlan at gmail.com  Fri Jun 17 15:26:43 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 17 Jun 2011 23:26:43 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DFB44CA.1050605@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl> <4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
	<4DFAE82A.8040103@pearwood.info>
	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>
	<4DFB44CA.1050605@pearwood.info>
Message-ID: <BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>

On Fri, Jun 17, 2011 at 10:12 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> You have missed a fourth option, which I have been championing: make inject
> an ordinary function, available from the functools module. The
> *implementation* of inject almost certainly will require support from the
> compiler, but that doesn't mean the interface should!

No, I didn't miss it, I left it out on purpose because I think messing
with the runtime name lookup semantics is a terrible idea. You and
others seem fond of it, but namespace semantics are the heart and soul
of why functions are so much faster than module level code and we
shouldn't be touching that logic with a 10 foot pole.

Adding a new cell-based shared namespace that uses the same runtime
lookup semantics as closures to replace *existing* uses of the default
argument hack? Sure, that's a reasonable proposal (it may still get
rejected due to devils in the details, but it has at least as much
going for it as PEP 308 did). Messing with normal locals from outside
a function, or providing an officially sanctioned way to convert
global references to some other kind of reference *after* the function
has already been defined? Hell no, that's a solution looking for a
problem and the concept of eliminating the default argument hack
shouldn't be burdened with that kind of overreaching.

The secret to the speed of functions lies in the fact that the
compiler knows all the names at compile time so it can generate
appropriate load/store operations for the different scopes (array
lookup for locals, cell dereference for closure variables,
global-or-builtin lookup for everything else). This benefits not just
CPython, but all Python implementations: inside a function, they're
allowed to assume that the *only* code changing the state of the
locals is the function code itself. Cell dereferencing allows for the
fact that closure variables might change (but are still reasonably
close to locals in speed, since the *cells* are referenced from an
array), and global and builtin lookup is the slowest of all (since it
involves actually looking up identifiers in namespace dictionaries).
Even a JIT compiler like PyPy can be more aggressive about optimising
local and cell access than it can be about the officially shifting
sands that are the global and builtin namespaces.

This is why the nonlocal and global directives exist: to tell the
compiler to change how it treats certain names. Arguments (including
the associated default values) are given additional special treatment
due to their placement in the function header. If we want to create a
new namespace that is given special treatment by the compiler, those
are the two options that are even remotely viable: placement in the
function header (after the ** entry) or flagged via a new compiler
directive (and the precedent of "nonlocal" and "global" suggests that
directive should occur inside the function body rather than anywhere
else). "@def" is primarily a proposal to avoid having to do the from
__future__ dance in defining a new keyword, so I'll modify it to the
more explicit "atdef" to avoid confusion with decorators).

A new compiler directive is my own preference (due to the major
semantic differences between how shared variables will be handled and
how default arguments are handled), and I now believe it makes sense
to use nonlocal, global and default arguments as the model for how
that would work:

  atdef VAR=EXPR [, VAR=EXPR]*

As with nonlocal and global, definition time statements could
technically appear anywhere in the function body (with their full
effect), but style guidelines would recommend placing them at the
beginning of the function, just after the docstring. Parentheses
around the var list would not be permitted - use multiple shared
statements instead (parentheses would, however, naturally permit the
expressions themselves to span multiple lines).

Such a statement would readily cover the speed enhancement,
early-binding and shared state use cases for the default argument hack
(indeed, the compiler could conceivably detect if a shared value was
never rebound and simply load the cell contents into each frame as a
local variable in that case, avoiding even the cell dereference
overhead relative to the speed hack).

The 'atdef' phrasing slightly emphasises the early-binding use case,
but still seems reasonable for the speed enhancement and shared state
use cases. In contrast, a keyword like 'shared' which emphasised the
shared state use case, would feel far more out of place when used for
speed enhancement or early binding (as well as being far more likely
to conflict with existing variables names).

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From scialexlight at gmail.com  Fri Jun 17 17:22:42 2011
From: scialexlight at gmail.com (Alex Light)
Date: Fri, 17 Jun 2011 11:22:42 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl>
	<4DF84F7B.3010700@improva.dk> <4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
	<4DFAE82A.8040103@pearwood.info>
	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>
	<4DFB44CA.1050605@pearwood.info>
	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
Message-ID: <BANLkTimhKqhptta5DUQ0NJJ_y4019+jMnw@mail.gmail.com>

On Fri, Jun 17, 2011 at 9:26 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

>

atdef VAR=EXPR [, VAR=EXPR]*



Such a statement would readily cover the speed enhancement,
>
early-binding and shared state use cases for the default argument hack
>

Doing things this way however removes one of the chief benefits of the
inject statement,
the ability to create multiple versions of the same function with different
sets of shared data.

You earlier gave an example similar to this as an example of what the syntax
 under your proposal would look like.

 def example(arg):
   """Record and return arguments seen"""
   atdef cache=set()

   #do something

   cache.add(arg)
   return arg

under proposal 4 (inject as full blown function, usable like any decorator)
it would look like this instead

@inject(cache = set())
def example(arg):
    """Record and return arguments seen"""

    #do something

    cache.add(arg)
    return arg

but what if, later, you decided you needed 10 different 'example' functions
each with its own cache as well
as the ability to make more.

under your proposal we would have to make extensive changes.
turning the code into

def make_example():
    def example(arg):
        """Record and return arguments seen"""

        #do something

        cache.add(arg)
        return arg
    return example

example_list = [make_example() for i in range(10)]

this code is rather difficult to read and it is difficult at first glance to
figure out what is actually happening

under proposal 4 the result would be much simpler. we could just remove the
@inject and put that on in the
list comprehension, like so.

def _example_func(arg):
    """Record and return arguments seen"""

    #do something

    cache.add(arg)
    return arg

example_list = [inject(cache=set())( _example_func) for i in range(10)]
make_example = lambda: inject(cache=set())( _example_func)

this is IMO far easier to read and understand.
Furthermore it gives the added benefit in that you can chose to run
'example' so that it uses
a true global variable, instead of an injected one.

--Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110617/a3741e69/attachment.html>

From jimjjewett at gmail.com  Fri Jun 17 17:59:01 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Fri, 17 Jun 2011 11:59:01 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl>
	<4DF84F7B.3010700@improva.dk> <4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
Message-ID: <BANLkTinves-9kdBrVHa5gWfyGOPnjNFarA@mail.gmail.com>

On Fri, Jun 17, 2011 at 12:39 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Fri, Jun 17, 2011 at 3:15 AM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
>> or even (to stress that it is a language syntax construct:

>> ? ?@inject mem=collections.Counter(), MAX_MEM=1000
>> ? ?def do_and_remember(val, verbose=False):

> While that would require the from__future__ dance to make "inject" a
> new keyword, I like it much better than the
> looks-like-a-decorator-but-isn't syntax.

This is reminding me of the once (or final or static) discussion a few
years ago, but I can't seem to find that PEP.

-jJ


From ethan at stoneleaf.us  Fri Jun 17 19:31:22 2011
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 17 Jun 2011 10:31:22 -0700
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTimhKqhptta5DUQ0NJJ_y4019+jMnw@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>	<20110614101755.GB3754@chopin.edu.pl>	<20110615002825.GA2217@chopin.edu.pl>	<4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>	<20110615231536.GA2182@chopin.edu.pl>	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>	<20110616171525.GB2222@chopin.edu.pl>	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>	<4DFAE82A.8040103@pearwood.info>	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>	<4DFB44CA.1050605@pearwood.info>	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
	<BANLkTimhKqhptta5DUQ0NJJ_y4019+jMnw@mail.gmail.com>
Message-ID: <4DFB8F6A.50506@stoneleaf.us>

Alex Light wrote:
> On Fri, Jun 17, 2011 at 9:26 AM, Nick Coghlan wrote:  
> 
>     atdef VAR=EXPR [, VAR=EXPR]*
> 
>     Such a statement would readily cover the speed enhancement, 
> 
>     early-binding and shared state use cases for the default argument hack
> 
>  
> Doing things this way however removes one of the chief benefits of the 
> inject statement, the ability to create multiple versions of the same function with
> different sets of shared data.

[snip]

> Furthermore it gives the added benefit in that you can chose to run 
> 'example' so that it uses a true global variable, instead of an injected one. 

That answers my question about why you would want to inject into an
already defined function.

~Ethan~



From ericsnowcurrently at gmail.com  Fri Jun 17 20:43:28 2011
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 17 Jun 2011 12:43:28 -0600
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTimhKqhptta5DUQ0NJJ_y4019+jMnw@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl> <4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
	<4DFAE82A.8040103@pearwood.info>
	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>
	<4DFB44CA.1050605@pearwood.info>
	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
	<BANLkTimhKqhptta5DUQ0NJJ_y4019+jMnw@mail.gmail.com>
Message-ID: <BANLkTina4i6_SiSVOb+3FCVuAuzHLOHuwA@mail.gmail.com>

On Fri, Jun 17, 2011 at 9:22 AM, Alex Light <scialexlight at gmail.com> wrote:
>
> Doing things this way however removes one of the chief benefits of the
> inject statement,
> the ability to create multiple versions of the same function with different
> sets of shared data.


The only way I could see runtime injection work is if you limited the
injection to names already tied to the locals, in the same way
parameters are.  This would require one of the 3 solutions that Nick
outlined.

Let's assume the injection values were stored on the function object,
like closures and defaults are, perhaps in an attribute named
__atdef__.  Then runtime injection could be used to replace __atdef__,
like you can with the defaults, if it were not read-only.

However, If __atdef__ were read-only, like __closure__, the runtime
injection would have to do some trickery, like you have to do if you
are going to mess with the closures [1].  This is a hack, in my mind,
since being read-only indicates to me that the expectation is you
shouldn't touch it!

With that said, in that case a runtime injection function would have
to generate a new function.  The new function would have to have the
desired __atdef__, and match any other read-only attribute.  Then the
injection function would copy into the new function object all the
remaining attributes of the old function, including the code object.

Here's an example of what I mean:

  def inject(f, *args):
      if len(args) != len(f.__atdef__):
          raise TypeError("__atdef__ mismatch")
      func = FunctionType(f.__code__, f.__globals__, f.__name__,
                                        f.__defaults__, f.__closure__,
tuple(args))
      # copy in the remaining attributes, like __doc__
      return func

You can already do this with closures and defaults.  If the elements
of __atdef__ are cells, like with __closure__ then you would have to
throw in a little more logic.  If you wanted to do kwargs, you would
have to introspect the names corresponding to __atdef__.  I don't know
if this would have a performance impact on the new function, in case
__defaults__ or __closure__ are more than just attributes on the
function object.

Like I said, this is a hack around the read-only attribute, but it
shows that with the solutions Nick outlined, you can still have an
injection function (could be used as a decorator, I suppose) .  The
only catch is that the names for __atdef__ would be tied into the
function body at definition time, which I think is a good thing.

Finally, regardless of if __atdef__ were read-only or not, I think a
runtime injection function should return a new function and leave the
old one untouched.  That seems to meet the use-case that you
presented.

-eric


[1] Here's an example:

from types import FunctionType
INJECTEDKEY = "injected_{}"
OUTERLINE = "    outer_{0} = injected_{0}"
INNERLINE = "        inner_{0} = outer_{0}"
SOURCE= ("def not_important():",
                 "    def also_not_important():",
                 "    return also_not_important")
def inject_closure(f, *args):
      injected = {}
      source = list(SOURCE)
      for i in range(len(args)):
          source.insert(1, OUTERLINE.format(i))
          source.insert(-1, INNERLINE.format(i))
          injected[INJECTEDKEY.format(i)] = args[i]
      exec("\n".join(source), injected, injected)
      closure = injected["not_important"]().__closure__
      func = FunctionType(f.__code__, f.__globals__, f.__name__,
                                        f.__defaults__, closure)
      func.__annotations__ = f.__annotations__
      func.__doc__ = f.__doc__
      func.__kwdefaults__ = f.__kwdefaults__
      func.__module__ = f.__module__
      return func


From ronaldoussoren at mac.com  Fri Jun 17 16:30:00 2011
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Fri, 17 Jun 2011 16:30:00 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DFAE82A.8040103@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl>
	<4DF84F7B.3010700@improva.dk> <4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
	<4DFAE82A.8040103@pearwood.info>
Message-ID: <C46C76D4-6D89-4C8B-B8F9-20891E246B5C@mac.com>





On 17 Jun, 2011, at 7:37, Steven D'Aprano wrote:

> Nick Coghlan wrote:
>> On Fri, Jun 17, 2011 at 3:15 AM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
>>> or even (to stress that it is a language syntax construct:
>>> 
>>>   @inject mem=collections.Counter(), MAX_MEM=1000
>>>   def do_and_remember(val, verbose=False):
>> While that would require the from__future__ dance to make "inject" a
>> new keyword, I like it much better than the
>> looks-like-a-decorator-but-isn't syntax.
> 
> What benefit is there in making inject a keyword? Even super isn't a keyword.
> 
> As far as I'm concerned, inject need only be a function in the functools module, not even a built-in, let alone a keyword.
> 
> Here's a quick and dirty version that comes close to the spirit of inject, as I see it. Thanks to Alex Light's earlier version.

FYI <http://code.activestate.com/recipes/277940-decorator-for-bindingconstants-at-compile-time/> implements a "bind all globals" variant of the injection using byte-code hacks. Changing that to only bind specific globals should be easy enough ;-)

Is the inject functionality needed by other Python implementations, and in particular by PyPy?   I use the keyword arguments hack mostly for two reasons: slightly higher speed in tight loops and a workaround for accessing globals in __del__ methods. Both are primairily CPython hacks,  AFAIK the PyPy jit is smart enough to optimize globals access close to the speed of local variable acces.

A stdlib function that implements the activestate recipe would be good enough if the functionality is only needed for CPython (with a fallback to an function that doesn't change the function body for the other Python implementations)

Ronald


From ethan at stoneleaf.us  Fri Jun 17 23:00:00 2011
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 17 Jun 2011 14:00:00 -0700
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTimhKqhptta5DUQ0NJJ_y4019+jMnw@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>	<20110614101755.GB3754@chopin.edu.pl>	<20110615002825.GA2217@chopin.edu.pl>	<4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>	<20110615231536.GA2182@chopin.edu.pl>	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>	<20110616171525.GB2222@chopin.edu.pl>	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>	<4DFAE82A.8040103@pearwood.info>	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>	<4DFB44CA.1050605@pearwood.info>	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
	<BANLkTimhKqhptta5DUQ0NJJ_y4019+jMnw@mail.gmail.com>
Message-ID: <4DFBC050.4000906@stoneleaf.us>

Alex Light wrote:
> Furthermore it gives the added benefit in that you can chose to run 
> 'example' so that it uses
> a true global variable, instead of an injected one. 

Only if the global is mutable.  Rebinding an immutable global requires 
the 'global' keyword, which would then clash with the injected names.

~Ethan~


From tjreedy at udel.edu  Fri Jun 17 22:58:56 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 17 Jun 2011 16:58:56 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>	<20110614101755.GB3754@chopin.edu.pl>	<20110615002825.GA2217@chopin.edu.pl>
	<4DF84F7B.3010700@improva.dk>	<4DF89903.6030202@pearwood.info>	<20110615231536.GA2182@chopin.edu.pl>	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>	<20110616171525.GB2222@chopin.edu.pl>	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>	<4DFAE82A.8040103@pearwood.info>	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>	<4DFB44CA.1050605@pearwood.info>
	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
Message-ID: <itgf6i$qqg$1@dough.gmane.org>

On 6/17/2011 9:26 AM, Nick Coghlan wrote:

> This is why the nonlocal and global directives exist: to tell the
> compiler to change how it treats certain names. Arguments (including
> the associated default values) are given additional special treatment
> due to their placement in the function header. If we want to create a
> new namespace that is given special treatment by the compiler,

I do not really want a new namespace and for the purpose of the OP, 
named local constants (for speed or freezing the meaning of an 
expression or both), we do not need one. There is already a fourth 
'namespace' for constants, a tuple f.__code__.co_consts, whose 'names' 
are indexes, just as with the locals array. Given

def f(a, **, b=1001, len = len): return 2001 # one possible spelling

def f(a):                                    # alternate
     constant b = 1001, len = len
     return 2001

the compiler should put 1001 and len into co.consts and convert 'b' and 
'len' into the corresponding indexes, just like it does with 'a', and 
use the LOAD_CONST bytecode just as with literal constants like 2001 in 
the body. Constant names would not go into .co_names and not increment 
.co_argcount. This would make named constants as fast and def-time 
frozen as default args without the disadvantages of being included in 
the signature and over-writable on calls.

-- 
Terry Jan Reedy



From steve at pearwood.info  Sat Jun 18 06:03:34 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 18 Jun 2011 14:03:34 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>	<20110614101755.GB3754@chopin.edu.pl>	<20110615002825.GA2217@chopin.edu.pl>	<4DF84F7B.3010700@improva.dk>	<4DF89903.6030202@pearwood.info>	<20110615231536.GA2182@chopin.edu.pl>	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>	<20110616171525.GB2222@chopin.edu.pl>	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>	<4DFAE82A.8040103@pearwood.info>	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>	<4DFB44CA.1050605@pearwood.info>
	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
Message-ID: <4DFC2396.1060905@pearwood.info>

Nick Coghlan wrote:
> On Fri, Jun 17, 2011 at 10:12 PM, Steven D'Aprano <steve at pearwood.info> wrote:
>> You have missed a fourth option, which I have been championing: make inject
>> an ordinary function, available from the functools module. The
>> *implementation* of inject almost certainly will require support from the
>> compiler, but that doesn't mean the interface should!
> 
> No, I didn't miss it, I left it out on purpose because I think messing
> with the runtime name lookup semantics is a terrible idea.

Isn't changing name lookup semantics at runtime precisely what JIT 
compilers do? But it doesn't really matter, because that's not what I'm 
proposing. I'm not suggesting that the lookup semantics should be 
changed when the function is called. I'm saying that a new function 
should be created, based on the original function, with the desired 
semantics.

In principle, this could be as simple as:

- make a copy of the function object
- in the copy, add cells for any injected variable
- and modify the copied code object to change the appropriate 
LOAD_GLOBAL opcodes to LOAD_DEREF (and similarly for rebindings).

although I dare say that in practice there'll be a certain amount of 
book-keeping required to make it work reliably.



-- 
Steven



From jh at improva.dk  Sat Jun 18 13:04:45 2011
From: jh at improva.dk (Jacob Holm)
Date: Sat, 18 Jun 2011 13:04:45 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DFC2396.1060905@pearwood.info>
References: <20110611133028.GD2395@chopin.edu.pl>	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>	<20110614101755.GB3754@chopin.edu.pl>	<20110615002825.GA2217@chopin.edu.pl>	<4DF84F7B.3010700@improva.dk>	<4DF89903.6030202@pearwood.info>	<20110615231536.GA2182@chopin.edu.pl>	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>	<20110616171525.GB2222@chopin.edu.pl>	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>	<4DFAE82A.8040103@pearwood.info>	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>	<4DFB44CA.1050605@pearwood.info>	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
	<4DFC2396.1060905@pearwood.info>
Message-ID: <4DFC864D.1000200@improva.dk>

On 2011-06-18 06:03, Steven D'Aprano wrote:
> Nick Coghlan wrote:
>> On Fri, Jun 17, 2011 at 10:12 PM, Steven D'Aprano
>> <steve at pearwood.info> wrote:
>>> You have missed a fourth option, which I have been championing: make
>>> inject
>>> an ordinary function, available from the functools module. The
>>> *implementation* of inject almost certainly will require support from
>>> the
>>> compiler, but that doesn't mean the interface should!
>>
>> No, I didn't miss it, I left it out on purpose because I think messing
>> with the runtime name lookup semantics is a terrible idea.
> 

I think that depends on what you count as part of the runtime name
lookup semantics.


> Isn't changing name lookup semantics at runtime precisely what JIT
> compilers do? But it doesn't really matter, because that's not what I'm
> proposing. I'm not suggesting that the lookup semantics should be
> changed when the function is called. I'm saying that a new function
> should be created, based on the original function, with the desired
> semantics.
> 
> In principle, this could be as simple as:
> 
> - make a copy of the function object
> - in the copy, add cells for any injected variable
> - and modify the copied code object to change the appropriate
> LOAD_GLOBAL opcodes to LOAD_DEREF (and similarly for rebindings).
> 
> although I dare say that in practice there'll be a certain amount of
> book-keeping required to make it work reliably.
> 

If you want the injected values to also affect functions that are
defined within the decorated function it gets a lot more complicated.
But yes, in theory it could work.

One thing that would make it a *lot* easier to write such an "inject"
function would be if we could replace the way globals are looked up to
use cells as well.  I am thinking of a different "kind" of cell that
wouldn't hold its value itself but get it from the module globals the
way LOAD_GLOBAL does today.  This cell would be in the __closures__ of
the function and could be replaced using a decorator like Steven proposed.

A consequence of this would be that you could optionally allow
"nonlocal" to bind global names when there are no suitable nonlocal
names to bind to (e.g. in a top-level function).  It has always slightly
bothered me that you couldn't do that, because it makes it harder to
move code between levels.

As written, this would probably slow down access to globals a little
bit.  However I have another idea (basically a more backward-compatible
variation of PEP 280) that would let us use cells or cell-like objects
for almost all accesses, at the cost of changing <module>.__dict__ to be
a dict *subclass*.

Best regards

 - Jacob


From cmjohnson.mailinglist at gmail.com  Sat Jun 18 23:22:51 2011
From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson)
Date: Sat, 18 Jun 2011 11:22:51 -1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <4DFC864D.1000200@improva.dk>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl>
	<4DF84F7B.3010700@improva.dk> <4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
	<4DFAE82A.8040103@pearwood.info>
	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>
	<4DFB44CA.1050605@pearwood.info>
	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
	<4DFC2396.1060905@pearwood.info> <4DFC864D.1000200@improva.dk>
Message-ID: <BANLkTimLnwjL3YjPybZzv2QGfr1F93LbNw@mail.gmail.com>

On Sat, Jun 18, 2011 at 1:04 AM, Jacob Holm <jh at improva.dk> wrote:

> A consequence of this would be that you could optionally allow
> "nonlocal" to bind global names when there are no suitable nonlocal
> names to bind to (e.g. in a top-level function). ?It has always slightly
> bothered me that you couldn't do that, because it makes it harder to
> move code between levels.

I will say that I was surprised to discover that `nonlocal` can be
used only for outer function locals and not for globals, since the
name "nonlocal" seems very general, as if it encompassed all things
not local. If it could be changed, I think it would be a little more
intuitive. In addition, as you mention, it's slightly better for
refactoring.


From tjreedy at udel.edu  Sun Jun 19 07:09:20 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 19 Jun 2011 01:09:20 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTimLnwjL3YjPybZzv2QGfr1F93LbNw@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>	<20110614101755.GB3754@chopin.edu.pl>	<20110615002825.GA2217@chopin.edu.pl>	<4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>	<20110615231536.GA2182@chopin.edu.pl>	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>	<20110616171525.GB2222@chopin.edu.pl>	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>	<4DFAE82A.8040103@pearwood.info>	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>	<4DFB44CA.1050605@pearwood.info>	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>	<4DFC2396.1060905@pearwood.info>
	<4DFC864D.1000200@improva.dk>
	<BANLkTimLnwjL3YjPybZzv2QGfr1F93LbNw@mail.gmail.com>
Message-ID: <itk0a0$h98$1@dough.gmane.org>

On 6/18/2011 5:22 PM, Carl M. Johnson wrote:

> I will say that I was surprised to discover that `nonlocal` can be
> used only for outer function locals and not for globals, since the
> name "nonlocal" seems very general, as if it encompassed all things
> not local.

We were aware of that when it was selected, after much discussion.
Closures were originally read-only partly because it was uncertain how 
to spell 'write'.

> If it could be changed, I think it would be a little more
> intuitive.

It cannot, I hope for obvious reasons.

I should hope that the docs make clear enought that names in a function 
are *partitioned* into module, closure, and local and that written names 
are local by default or one of the other 2 if declared.

Actually, I think improving the nonlocal doc is part of some issue.

-- 
Terry Jan Reedy



From raymond.hettinger at gmail.com  Sun Jun 19 07:41:30 2011
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 19 Jun 2011 06:41:30 +0100
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTimLnwjL3YjPybZzv2QGfr1F93LbNw@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl>
	<4DF84F7B.3010700@improva.dk> <4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
	<4DFAE82A.8040103@pearwood.info>
	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>
	<4DFB44CA.1050605@pearwood.info>
	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
	<4DFC2396.1060905@pearwood.info> <4DFC864D.1000200@improva.dk>
	<BANLkTimLnwjL3YjPybZzv2QGfr1F93LbNw@mail.gmail.com>
Message-ID: <F92C0277-8D7E-4647-BAB7-01C926F5C910@gmail.com>


On Jun 18, 2011, at 10:22 PM, Carl M. Johnson wrote:

> On Sat, Jun 18, 2011 at 1:04 AM, Jacob Holm <jh at improva.dk> wrote:
> 
>> A consequence of this would be that you could optionally allow
>> "nonlocal" to bind global names when there are no suitable nonlocal
>> names to bind to (e.g. in a top-level function).  It has always slightly
>> bothered me that you couldn't do that, because it makes it harder to
>> move code between levels.
> 
> I will say that I was surprised to discover that `nonlocal` can be
> used only for outer function locals and not for globals, since the
> name "nonlocal" seems very general, as if it encompassed all things
> not local. If it could be changed, I think it would be a little more
> intuitive. In addition, as you mention, it's slightly better for
> refactoring.

We should put a stop the notion that any time someone says,
"I was surprised" that there needs to be a change to the language.
If surprise happens because someone skipped reading the docs
and made an incorrect guess about how a keyword  behaves
(i.e. using a new feature without reading about what it actually does),
then "I was surprised" means very little.

No matter what was implemented for "nonlocal", someone
was going to be surprised that it didn't match their intuition.
If nonlocal meant, "first match in the chain of enclosing scopes",
then would you expect "nonlocal int" to write into the builtin scope?
If nonlocal included globals, would it be a surprise that "global x"
and "nonlocal x" would do exactly the same thing, but only if x
already existed in the global scope?  Whatever the answers,
the important point is that it is hard to eliminate surprise when
surprise is based on someone's guess about how a keyword
is implemented.   One the Python Weekly URL's quotes of the
week last month was:  "When did we suddenly come to expect
that people could program in a language without actually learning it?"

ISTM, it would be much better if language change proposals came
in the form of:  "change X makes the following code better and is
worth the breaking of code Y and making person Z relearn what
the feature does."  That would get to essentials while taking the
unpersuasive "I was surprised" off the table.

my-two-cents-ly,


Raymond





From ncoghlan at gmail.com  Sun Jun 19 08:56:30 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 19 Jun 2011 16:56:30 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <F92C0277-8D7E-4647-BAB7-01C926F5C910@gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl> <4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
	<4DFAE82A.8040103@pearwood.info>
	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>
	<4DFB44CA.1050605@pearwood.info>
	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
	<4DFC2396.1060905@pearwood.info> <4DFC864D.1000200@improva.dk>
	<BANLkTimLnwjL3YjPybZzv2QGfr1F93LbNw@mail.gmail.com>
	<F92C0277-8D7E-4647-BAB7-01C926F5C910@gmail.com>
Message-ID: <BANLkTik_RMv3VktMtCVMVtNT7S8G37enDg@mail.gmail.com>

On Sun, Jun 19, 2011 at 3:41 PM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
> ISTM, it would be much better if language change proposals came
> in the form of: ?"change X makes the following code better and is
> worth the breaking of code Y and making person Z relearn what
> the feature does." ?That would get to essentials while taking the
> unpersuasive "I was surprised" off the table.

Indeed. If anyone was curious as to why I've been recently trying to
steer the discussion back specifically towards Jan's original idea of
replacing current (ab)uses of the default argument hack (and nothing
more), this is pretty much it.

We *know* those use cases exist because there is code in the wild that
uses them (including in the standard library). Saying "don't do that"
isn't an adequate response as, for anyone that knows about the hack
and how it works, the alternatives are far harder to read (in addition
to being far more verbose and error prone to write in the first
place).

Extrapolating to additional functionality that is difficult or
impossible in current Python code is sheer speculation that is likely
to weigh down any proposal with unnecessary cruft that gets it
rejected.

What is needed now is one or more volunteers that are willing and able to:

1. Write a PEP that:
  - distils the core discussion in this thread down into a specific
proposal that can be brought up on python-dev
  - explains *why* the default argument hack is undesirable in general
(unintuitive, hard to look up, harmful to introspection, invites
errors when calling affected functions)
  - clearly articulates the uses of the default argument hack that the
proposal aims to eliminate (early binding, shared locals, performance)
  - include real world example of such uses from the standard library
(and potentially other code bases)
  - optionally, also describe the potential "function factories" that
could be developed based on the semantics I proposed (see Eric Snow's
post for a sketch of how such factories might work once given a
template function to work with)
2. Create a reference implementation targeting Python 3.3

(with step 1 being significantly more important at this stage - while
the implementation can't be called *easy*, it should be a reasonably
straightforward combination of the existing code that handles nonlocal
statements and that which handles the calculation and storage of
default arguments).

I'm not going to do it myself (I already have a couple of open PEPs
that I need to make the time to follow up on), but I'm more than happy
to provide pointers and advice to someone else that steps up to do so.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From tjreedy at udel.edu  Sun Jun 19 20:26:44 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 19 Jun 2011 14:26:44 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTik_RMv3VktMtCVMVtNT7S8G37enDg@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>	<20110614101755.GB3754@chopin.edu.pl>	<20110615002825.GA2217@chopin.edu.pl>
	<4DF84F7B.3010700@improva.dk>	<4DF89903.6030202@pearwood.info>	<20110615231536.GA2182@chopin.edu.pl>	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>	<20110616171525.GB2222@chopin.edu.pl>	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>	<4DFAE82A.8040103@pearwood.info>	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>	<4DFB44CA.1050605@pearwood.info>	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>	<4DFC2396.1060905@pearwood.info>
	<4DFC864D.1000200@improva.dk>	<BANLkTimLnwjL3YjPybZzv2QGfr1F93LbNw@mail.gmail.com>	<F92C0277-8D7E-4647-BAB7-01C926F5C910@gmail.com>
	<BANLkTik_RMv3VktMtCVMVtNT7S8G37enDg@mail.gmail.com>
Message-ID: <itlf14$hub$1@dough.gmane.org>

On 6/19/2011 2:56 AM, Nick Coghlan wrote:

> Indeed. If anyone was curious as to why I've been recently trying to
> steer the discussion back specifically towards Jan's original idea of
> replacing current (ab)uses of the default argument hack (and nothing
> more), this is pretty much it.

Along that line, I posted the following concrete suggestion a couple of 
days ago, but have seen no response:
'''
for the purpose of the OP, named local constants (for speed or freezing 
the meaning of an expression or both), we do not need [a new namespace]. 
There is already a fourth 'namespace' for constants, a tuple 
f.__code__.co_consts, whose 'names' are indexes, just as with the locals 
array. Given

def f(a, **, b=1001, len = len): return 2001 # one possible spelling

def f(a):                                    # alternate
     constant b = 1001, len = len
     return 2001

the compiler should put 1001 and len into co.consts and convert 'b' and 
'len' into the corresponding indexes, just like it does with 'a', and 
use the LOAD_CONST bytecode just as with literal constants like 2001 in 
the body. Constant names would not go into .co_names and not increment 
.co_argcount. This would make named constants as fast and def-time 
frozen as default args without the disadvantages of being included in 
the signature and over-writable on calls.
'''
Did this not actually go through?

> We *know* those use cases exist because there is code in the wild that
> uses them (including in the standard library). Saying "don't do that"
> isn't an adequate response as, for anyone that knows about the hack
> and how it works, the alternatives are far harder to read (in addition
> to being far more verbose and error prone to write in the first
> place).
>
> Extrapolating to additional functionality that is difficult or
> impossible in current Python code is sheer speculation that is likely
> to weigh down any proposal with unnecessary cruft that gets it
> rejected.
>
> What is needed now is one or more volunteers that are willing and able to:
>
> 1. Write a PEP that:
>    - distils the core discussion in this thread down into a specific
> proposal that can be brought up on python-dev

The above is a specific proposal (with two possible syntax spellings) 
and an outline of a specific implementation.

>    - explains *why* the default argument hack is undesirable in general
> (unintuitive, hard to look up, harmful to introspection, invites
> errors when calling affected functions)
>    - clearly articulates the uses of the default argument hack that the
> proposal aims to eliminate (early binding, shared locals, performance)

By 'shared locals' do you mean nonlocals? That is not quite what Jan was 
requesting.

>    - include real world example of such uses from the standard library
> (and potentially other code bases)
>    - optionally, also describe the potential "function factories" that
> could be developed based on the semantics I proposed (see Eric Snow's
> post for a sketch of how such factories might work once given a
> template function to work with)
> 2. Create a reference implementation targeting Python 3.3
>
> (with step 1 being significantly more important at this stage - while
> the implementation can't be called *easy*, it should be a reasonably
> straightforward combination of the existing code that handles nonlocal
> statements and that which handles the calculation and storage of
> default arguments).

I do not see what nonlocals has to do or needs to have to do with 
*local* constants. My proposal is that to store named constants, we 
reuse the current code to recognize and calculate named defaulted locals 
with the current code to store and retrieve anonymous constants,

-- 
Terry Jan Reedy



From jimjjewett at gmail.com  Sun Jun 19 21:28:52 2011
From: jimjjewett at gmail.com (Jim Jewett)
Date: Sun, 19 Jun 2011 15:28:52 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <itgf6i$qqg$1@dough.gmane.org>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl>
	<4DF84F7B.3010700@improva.dk> <4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
	<4DFAE82A.8040103@pearwood.info>
	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>
	<4DFB44CA.1050605@pearwood.info>
	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
	<itgf6i$qqg$1@dough.gmane.org>
Message-ID: <BANLkTim5TT9TDOBUkxGuOBWB+UxSpHqa8w@mail.gmail.com>

On Fri, Jun 17, 2011 at 4:58 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 6/17/2011 9:26 AM, Nick Coghlan wrote:

>> This is why the nonlocal and global directives exist: to tell the
>> compiler to change how it treats certain names. Arguments (including
>> the associated default values) are given additional special treatment
>> due to their placement in the function header. If we want to create a
>> new namespace that is given special treatment by the compiler,

> I do not really want a new namespace and for the purpose of the OP,
> named local constants (for speed or freezing the meaning of an
> expression or both), we do not need one.  There is already a fourth
> 'namespace' for constants, a tuple f.__code__.co_consts, whose
> 'names' are indexes, just as with the locals array.

I really like this idea.

The only concerns I can see are losing the name for use in debugging
or embedded functions, and I assume that those can be dealt with.

-jJ


From tjreedy at udel.edu  Sun Jun 19 23:10:38 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 19 Jun 2011 17:10:38 -0400
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTim5TT9TDOBUkxGuOBWB+UxSpHqa8w@mail.gmail.com>
References: <20110611133028.GD2395@chopin.edu.pl>	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>	<20110614101755.GB3754@chopin.edu.pl>	<20110615002825.GA2217@chopin.edu.pl>	<4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>	<20110615231536.GA2182@chopin.edu.pl>	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>	<20110616171525.GB2222@chopin.edu.pl>	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>	<4DFAE82A.8040103@pearwood.info>	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>	<4DFB44CA.1050605@pearwood.info>	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>	<itgf6i$qqg$1@dough.gmane.org>
	<BANLkTim5TT9TDOBUkxGuOBWB+UxSpHqa8w@mail.gmail.com>
Message-ID: <itlokd$3bl$1@dough.gmane.org>

On 6/19/2011 3:28 PM, Jim Jewett wrote:
> On Fri, Jun 17, 2011 at 4:58 PM, Terry Reedy<tjreedy at udel.edu>  wrote:
>> On 6/17/2011 9:26 AM, Nick Coghlan wrote:
>
>>> This is why the nonlocal and global directives exist: to tell the
>>> compiler to change how it treats certain names. Arguments (including
>>> the associated default values) are given additional special treatment
>>> due to their placement in the function header. If we want to create a
>>> new namespace that is given special treatment by the compiler,
>
>> I do not really want a new namespace and for the purpose of the OP,
>> named local constants (for speed or freezing the meaning of an
>> expression or both), we do not need one.  There is already a fourth
>> 'namespace' for constants, a tuple f.__code__.co_consts, whose
>> 'names' are indexes, just as with the locals array.
>
> I really like this idea.
>
> The only concerns I can see are losing the name for use in debugging
> or embedded functions, and I assume that those can be dealt with.

I consider that a secondary detail. If the names are recorded, in a 
method that distinguishes them from parameter names, then they can be 
introspected, just like other local names. They could be included in 
locals(), though I hardly ever use that and am not familiar with the 
details of its runtime construction.

The main problem of my idea as originally conceived is that it only 
really works everywhere for constant expressions limited to literals and 
builtin names (as were my examples). While that would be useful and 
eliminate one category of default arg misuse, it probably is not enough 
for new syntax.

For general expressions, it is only guaranteed to work as intended for 
top-level def statements in interactive mode, which are compiled 
immediately before execution. Otherwise, there is a gap between creation 
of the code-object and first execution of the def statement. So 
name-resolution may be too early or even impossible (as it is for .pyc 
files). Or some new mechanism would be needed to patch the code object.

This gets to the point that compilation time and hence code-object 
creation time seems more an implementation detail than part of the 
language def. CPython creates code objects just once for each function 
body, and reuses them for each def or lambda invocation, but this may be 
an optimization rather than language requirement.

-- 
Terry Jan Reedy



From zuo at chopin.edu.pl  Mon Jun 20 13:34:11 2011
From: zuo at chopin.edu.pl (Jan Kaliszewski)
Date: Mon, 20 Jun 2011 13:34:11 +0200
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <BANLkTik_RMv3VktMtCVMVtNT7S8G37enDg@mail.gmail.com>
References: <BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
	<4DFAE82A.8040103@pearwood.info>
	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>
	<4DFB44CA.1050605@pearwood.info>
	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
	<4DFC2396.1060905@pearwood.info> <4DFC864D.1000200@improva.dk>
	<BANLkTimLnwjL3YjPybZzv2QGfr1F93LbNw@mail.gmail.com>
	<F92C0277-8D7E-4647-BAB7-01C926F5C910@gmail.com>
	<BANLkTik_RMv3VktMtCVMVtNT7S8G37enDg@mail.gmail.com>
Message-ID: <20110620113411.GA2177@chopin.edu.pl>

Nick Coghlan dixit (2011-06-17, 17:02):

[...snip...]
> I already have too much on my to-do list to champion a PEP for this,
> but I'd be happy to help someone else with the mechanics of writing
> one and getting it published on python.org (hint, hint Jan!).

Nick Coghlan dixit (2011-06-19, 16:56):

[...snip...]
> What is needed now is one or more volunteers that are willing and able to:
> 
> 1. Write a PEP that:
>   - distils the core discussion in this thread down into a specific
> proposal that can be brought up on python-dev
>   - explains *why* the default argument hack is undesirable in general
> (unintuitive, hard to look up, harmful to introspection, invites
> errors when calling affected functions)
>   - clearly articulates the uses of the default argument hack that the
> proposal aims to eliminate (early binding, shared locals, performance)
>   - include real world example of such uses from the standard library
> (and potentially other code bases)
>   - optionally, also describe the potential "function factories" that
> could be developed based on the semantics I proposed (see Eric Snow's
> post for a sketch of how such factories might work once given a
> template function to work with)
> 2. Create a reference implementation targeting Python 3.3
> 
> (with step 1 being significantly more important at this stage - while
> the implementation can't be called *easy*, it should be a reasonably
> straightforward combination of the existing code that handles nonlocal
> statements and that which handles the calculation and storage of
> default arguments).
> 
> I'm not going to do it myself (I already have a couple of open PEPs
> that I need to make the time to follow up on), but I'm more than happy
> to provide pointers and advice to someone else that steps up to do so.

I could do it with pleasure, at least step #1 -- i.e. writing a PEP
(step #2 seems to be quite non-trivial :), though for sure would be very
instructive...).

*But* -- only provided that it would be known and accepted that *I could
not act urgently nor quickly* at all (I got a new job recently and even
don't know yet how much spare time per week would I be able to save up
during incoming months).

Would that be OK?

Cheers.
*j



From mikegraham at gmail.com  Mon Jun 20 16:03:57 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Mon, 20 Jun 2011 10:03:57 -0400
Subject: [Python-ideas] Simpler namespace packages
Message-ID: <BANLkTikd3PTL2RYKWxs5LSi-1PQtBOqFVw@mail.gmail.com>

I was reading over PEP 382 and am not sure I really understand how it
is a worthwhile step forward. Namespace packages are currently a mess
and personally if anyone asks me I recommend not using the
currently-available approach at all.

Would it be possible to allow a more simple definition, for example
putting the dot itself in the filename? Where these would be similar?

site-packages/
    foo/
        __init__.py
    foo.bar.py
    foo.baz/
        __init__.py
        qux.py
     foo.spam.eggs.py

and

site-packages/
    foo/
        __init__.py
        bar.py
        baz/
            __init__.py
            qux.py
        spam/
            __init__.py #empty
            eggs.py

Obviously this is not fully-defined, but would this kind of approach
fill the same need and be nicer to use, create, find, and understand
then what is proposed in PEP 382?


From scialexlight at gmail.com  Mon Jun 20 16:26:37 2011
From: scialexlight at gmail.com (Alex Light)
Date: Mon, 20 Jun 2011 10:26:37 -0400
Subject: [Python-ideas] Simpler namespace packages
In-Reply-To: <BANLkTikd3PTL2RYKWxs5LSi-1PQtBOqFVw@mail.gmail.com>
References: <BANLkTikd3PTL2RYKWxs5LSi-1PQtBOqFVw@mail.gmail.com>
Message-ID: <BANLkTi=LT9yB-cC5N0LqAkY9AgPc_r7cbQ@mail.gmail.com>

On Mon, Jun 20, 2011 at 10:03 AM, Mike Graham <mikegraham at gmail.com> wrote:

> Would it be possible to allow a more simple definition, for example
> putting the dot itself in the filename? Where these would be similar?
>
> site-packages/
>    foo/
>        __init__.py
>    foo.bar.py
>    foo.baz/
>        __init__.py
>        qux.py
>     foo.spam.eggs.py
>
> and
>
> site-packages/
>    foo/
>        __init__.py
>        bar.py
>        baz/
>            __init__.py
>            qux.py
>        spam/
>            __init__.py #empty
>            eggs.py
>

I fail to see how the first example is clearer than the second. Indeed
the opposite seems to be true.
What i think you need to understand is that the name of the module is the
same as the path to it.
IMO the current system emphasizes that very strongly and this proposal would
only make the
concept less clear.

--Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110620/f114d058/attachment.html>

From ncoghlan at gmail.com  Mon Jun 20 16:56:34 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 21 Jun 2011 00:56:34 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <itlf14$hub$1@dough.gmane.org>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl> <4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
	<4DFAE82A.8040103@pearwood.info>
	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>
	<4DFB44CA.1050605@pearwood.info>
	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
	<4DFC2396.1060905@pearwood.info> <4DFC864D.1000200@improva.dk>
	<BANLkTimLnwjL3YjPybZzv2QGfr1F93LbNw@mail.gmail.com>
	<F92C0277-8D7E-4647-BAB7-01C926F5C910@gmail.com>
	<BANLkTik_RMv3VktMtCVMVtNT7S8G37enDg@mail.gmail.com>
	<itlf14$hub$1@dough.gmane.org>
Message-ID: <BANLkTi=gN35_OgBcBegJWEN=+xuf+BpZjA@mail.gmail.com>

On Mon, Jun 20, 2011 at 4:26 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> Did this not actually go through?

It did, I just forgot to reply. Sorry about that.

co_consts is not a namespace - it's just a cache where the compiler
stashes values (usually literals) that can be calculated at compile
time (note, NOT definition time - it happens earlier than that,
potentially even prior to the current application invocation if the
module was cached in a .pyc file).

Using it for mutable state is not possible, and thus would completely
miss the point of some of the significant uses of the default argument
hack. It is also quite possible for a single code object to be shared
amongst multiple function definitions (e.g. when a function
declaration is inside a loop). Accordingly, any new definition time
state needs to go on the function object, for all the same reasons
that current definition time state (i.e. annotations and default
parameter values) is stored there.

> I do not see what nonlocals has to do or needs to have to do with *local* constants. My proposal is that to store named constants, we reuse the current code to recognize and calculate named defaulted locals with the current code to store and retrieve anonymous constants,

The nonlocals handling code is relevant because creating state that is
shared between function invocations is what happens with closures, and
the nonlocal statement serves to tell the compiler that names that
would otherwise be considered local should instead be considered
closure references. We want the new statement to do something similar:
the nominated names will exist in the new definition time namespace
rather than being looked up in any of the existing locations (locals,
outer scopes, globals/builtins).

And, as noted above, the const calculation code isn't useful because
it happens at the wrong time (compilation time instead of definition
time) and because code objects may be shared amongst multiple function
definitions.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Mon Jun 20 17:15:51 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 21 Jun 2011 01:15:51 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <itlokd$3bl$1@dough.gmane.org>
References: <20110611133028.GD2395@chopin.edu.pl>
	<BANLkTim9DXqit0fUnHTsvpy5HD6rSAYtfw@mail.gmail.com>
	<20110614101755.GB3754@chopin.edu.pl>
	<20110615002825.GA2217@chopin.edu.pl> <4DF84F7B.3010700@improva.dk>
	<4DF89903.6030202@pearwood.info>
	<20110615231536.GA2182@chopin.edu.pl>
	<BANLkTikPbqOU7__0QBNjXe58FARnsDZgjg@mail.gmail.com>
	<20110616171525.GB2222@chopin.edu.pl>
	<BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
	<4DFAE82A.8040103@pearwood.info>
	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>
	<4DFB44CA.1050605@pearwood.info>
	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
	<itgf6i$qqg$1@dough.gmane.org>
	<BANLkTim5TT9TDOBUkxGuOBWB+UxSpHqa8w@mail.gmail.com>
	<itlokd$3bl$1@dough.gmane.org>
Message-ID: <BANLkTinoyKC-JjA29XFOEPLHZq1eSScV+A@mail.gmail.com>

On Mon, Jun 20, 2011 at 7:10 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> The main problem of my idea as originally conceived is that it only really
> works everywhere for constant expressions limited to literals and builtin
> names (as were my examples). While that would be useful and eliminate one
> category of default arg misuse, it probably is not enough for new syntax.

Oops, I should have finished reading the thread before replying :)

> For general expressions, it is only guaranteed to work as intended for
> top-level def statements in interactive mode, which are compiled immediately
> before execution. Otherwise, there is a gap between creation of the
> code-object and first execution of the def statement. So name-resolution may
> be too early or even impossible (as it is for .pyc files). Or some new
> mechanism would be needed to patch the code object.
>
> This gets to the point that compilation time and hence code-object creation
> time seems more an implementation detail than part of the language def.
> CPython creates code objects just once for each function body, and reuses
> them for each def or lambda invocation, but this may be an optimization
> rather than language requirement.

The compilation time/definition time/execution time split is part of
the language definition, as is the immutability of code objects
(although, interestingly enough, the compile time distinction is
mentioned in the language reference, but not well defined - it is
mostly implicit in the definition of the compile() builtin). An
implementation doesn't *have* to reuse the code objects when multiple
function objects share the same definition, but there's no real reason
not to.

I created an issue pointing out that these semantics should really be
clarified in the execution model section of the language reference:
http://bugs.python.org/issue12374

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Mon Jun 20 17:19:44 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 21 Jun 2011 01:19:44 +1000
Subject: [Python-ideas] 'Injecting' objects as function-local constants
In-Reply-To: <20110620113411.GA2177@chopin.edu.pl>
References: <BANLkTikqq1q73JD8UW+sMV0kmJcM7sXtHQ@mail.gmail.com>
	<4DFAE82A.8040103@pearwood.info>
	<BANLkTim28jdGUg5-rzr0D4cWd47PoVCp6w@mail.gmail.com>
	<4DFB44CA.1050605@pearwood.info>
	<BANLkTimdi1qCanuGjHgc-LnGtFeofXsQfw@mail.gmail.com>
	<4DFC2396.1060905@pearwood.info> <4DFC864D.1000200@improva.dk>
	<BANLkTimLnwjL3YjPybZzv2QGfr1F93LbNw@mail.gmail.com>
	<F92C0277-8D7E-4647-BAB7-01C926F5C910@gmail.com>
	<BANLkTik_RMv3VktMtCVMVtNT7S8G37enDg@mail.gmail.com>
	<20110620113411.GA2177@chopin.edu.pl>
Message-ID: <BANLkTikmTU2WkpJAZ7X1C4J7n9gL7UnpwA@mail.gmail.com>

On Mon, Jun 20, 2011 at 9:34 PM, Jan Kaliszewski <zuo at chopin.edu.pl> wrote:
> I could do it with pleasure, at least step #1 -- i.e. writing a PEP
> (step #2 seems to be quite non-trivial :), though for sure would be very
> instructive...).
>
> *But* -- only provided that it would be known and accepted that *I could
> not act urgently nor quickly* at all (I got a new job recently and even
> don't know yet how much spare time per week would I be able to save up
> during incoming months).
>
> Would that be OK?

That's fine, there's still quite a lot of time before the first alpha
of 3.3. The main goal is to get something captured in the form of a
PEP so even if nothing happens for a while, the discussion doesn't
have to restart from scratch when the topic comes up again.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From mikegraham at gmail.com  Mon Jun 20 18:24:06 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Mon, 20 Jun 2011 12:24:06 -0400
Subject: [Python-ideas] Simpler namespace packages
In-Reply-To: <BANLkTi=LT9yB-cC5N0LqAkY9AgPc_r7cbQ@mail.gmail.com>
References: <BANLkTikd3PTL2RYKWxs5LSi-1PQtBOqFVw@mail.gmail.com>
	<BANLkTi=LT9yB-cC5N0LqAkY9AgPc_r7cbQ@mail.gmail.com>
Message-ID: <BANLkTi=eZgeWPP-chmebwc1dRRusQywsNA@mail.gmail.com>

On Mon, Jun 20, 2011 at 10:26 AM, Alex Light <scialexlight at gmail.com> wrote:
> On Mon, Jun 20, 2011 at 10:03 AM, Mike Graham <mikegraham at gmail.com> wrote:
>>
>> Would it be possible to allow a more simple definition, for example
>> putting the dot itself in the filename? Where these would be similar?
>>
>> site-packages/
>> ? ?foo/
>> ? ? ? ?__init__.py
>> ? ?foo.bar.py
>> ? ?foo.baz/
>> ? ? ? ?__init__.py
>> ? ? ? ?qux.py
>> ? ? foo.spam.eggs.py
>>
>> and
>>
>> site-packages/
>> ? ?foo/
>> ? ? ? ?__init__.py
>> ? ? ? ?bar.py
>> ? ? ? ?baz/
>> ? ? ? ? ? ?__init__.py
>> ? ? ? ? ? ?qux.py
>> ? ? ? ?spam/
>> ? ? ? ? ? ?__init__.py #empty
>> ? ? ? ? ? ?eggs.py
>
> I fail to see how the first example is clearer than the second. Indeed
> the?opposite?seems to be true.
> What i think you need to understand is that the name of the module is the
> same as the path to it.
> IMO the current system emphasizes that very strongly and this proposal would
> only make the
> concept less clear.
> --Alex

The first example ISN'T clearer than the second. I think the second is
more easily understood to be sure. The first example, I'm saying, may
be clearer than other implementations of namespace packages, not
clearer than a normal package.

A namespace package is one where the top level(s) aren't for a normal
package. If foo.bar and foo.baz and foo.qux were distributed
completely separately, foo would be a namespace package.

Currently, a namespace package might look like

site-packages/
    foo/ # No __init__.py in foo
        bar/
            __init__.py
            spam.py
    foo.bar-1.2.3-py2.7-nspkg.pth # A pth file to tell Python how to
import the package
    foo.bar-1.2.3-py2.7.egg-info/
        namespace_packages.txt # This would say "foo" in it
        ....other stuff

and PEP382 tries to improve the situation a bit. I'm hoping we can
come up with something that is easily understood.


Mike


From scialexlight at gmail.com  Tue Jun 21 03:24:27 2011
From: scialexlight at gmail.com (Alex Light)
Date: Mon, 20 Jun 2011 21:24:27 -0400
Subject: [Python-ideas] Simpler namespace packages
In-Reply-To: <BANLkTi=eZgeWPP-chmebwc1dRRusQywsNA@mail.gmail.com>
References: <BANLkTikd3PTL2RYKWxs5LSi-1PQtBOqFVw@mail.gmail.com>
	<BANLkTi=LT9yB-cC5N0LqAkY9AgPc_r7cbQ@mail.gmail.com>
	<BANLkTi=eZgeWPP-chmebwc1dRRusQywsNA@mail.gmail.com>
Message-ID: <BANLkTikRQ5-UEzYKLWgwzZtB6n3kyWdNDQ@mail.gmail.com>

Sorry mixed up first and second examples

-alex
 On Jun 20, 2011 12:24 PM, "Mike Graham" <mikegraham at gmail.com> wrote:
> On Mon, Jun 20, 2011 at 10:26 AM, Alex Light <scialexlight at gmail.com>
wrote:
>> On Mon, Jun 20, 2011 at 10:03 AM, Mike Graham <mikegraham at gmail.com>
wrote:
>>>
>>> Would it be possible to allow a more simple definition, for example
>>> putting the dot itself in the filename? Where these would be similar?
>>>
>>> site-packages/
>>>    foo/
>>>        __init__.py
>>>    foo.bar.py
>>>    foo.baz/
>>>        __init__.py
>>>        qux.py
>>>     foo.spam.eggs.py
>>>
>>> and
>>>
>>> site-packages/
>>>    foo/
>>>        __init__.py
>>>        bar.py
>>>        baz/
>>>            __init__.py
>>>            qux.py
>>>        spam/
>>>            __init__.py #empty
>>>            eggs.py
>>
>> I fail to see how the first example is clearer than the second. Indeed
>> the opposite seems to be true.
>> What i think you need to understand is that the name of the module is the
>> same as the path to it.
>> IMO the current system emphasizes that very strongly and this proposal
would
>> only make the
>> concept less clear.
>> --Alex
>
> The first example ISN'T clearer than the second. I think the second is
> more easily understood to be sure. The first example, I'm saying, may
> be clearer than other implementations of namespace packages, not
> clearer than a normal package.
>
> A namespace package is one where the top level(s) aren't for a normal
> package. If foo.bar and foo.baz and foo.qux were distributed
> completely separately, foo would be a namespace package.
>
> Currently, a namespace package might look like
>
> site-packages/
> foo/ # No __init__.py in foo
> bar/
> __init__.py
> spam.py
> foo.bar-1.2.3-py2.7-nspkg.pth # A pth file to tell Python how to
> import the package
> foo.bar-1.2.3-py2.7.egg-info/
> namespace_packages.txt # This would say "foo" in it
> ....other stuff
>
> and PEP382 tries to improve the situation a bit. I'm hoping we can
> come up with something that is easily understood.
>
>
> Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110620/0b2821e1/attachment.html>

From sven at marnach.net  Thu Jun 23 19:53:29 2011
From: sven at marnach.net (Sven Marnach)
Date: Thu, 23 Jun 2011 18:53:29 +0100
Subject: [Python-ideas] A few suggestions for the random module
Message-ID: <20110623175329.GF4696@pantoffel-wg.de>

I'd like to suggest what I consider a few minor improvements to
Python's random module.

1. random.expovariate(lambd)

   The current implementation [1] is

       random = self.random
       u = random()
       while u <= 1e-7:
           u = random()
       return -_log(u)/lambd

   I'd suggest to simplify this to

       return -log(1.0 - self.random())/lambd

   self.random() returns a float in the half-open interval [0.0, 1.0),
   so 1.0 - self.random() is in the half-open interval (0.0, 1.0],
   which is exactly what we need here.  Even if the random number gets
   as close to 1.0 as possible within the limits of double precision,
   taking the logarithm won't be any problem (the lowest occuring
   value of log(...) being roughly -36.7).

   The suggested implementation is not only simpler and faster, it is
   also more correct.  The current implementation will never return
   0.0, although this is a perfectly valid return value, and it
   chooses the arbitrary cut-off value 1e7, while the suggested
   implementation is only limited by floating point precision and the
   range of random.random().

   [1]: http://hg.python.org/cpython/file/54fb77e0762c/Lib/random.py#l392

2. random.sample(population, k)

   The current implementation [2] chooses one of two algorithms,
   depending on whether the number of samples k is relatively small
   compared to the size n of the population.  I suggest to add
   reservoir sampling [3] as a third algorithm, used in either of the
   following cases:

    * k is "close" to n (for example k > .75 * n)

    * population is not a sequence (but of course an iterable)

   Reservoir sampling would only use O(1) additional memory, compared
   to O(n) for the current algorithm for relativly large values of k.
   It would also facilitate to select samples from iterables without
   storing a complete list of all items in memory at any time.  It
   could also be used for smaller values of k, trading reduced memory
   usage for increased execution time.

   Example implementation:

       it = iter(population)
       result = list(_islice(it, k))
       self.shuffle(result)
       for i, x in enumerate(it, k + 1):
           j = randbelow(i)
           if j < k:
               result[j] = x

   We need to call self.shuffle(result) here to ensure that all
   subslices are valid random samples again.

   An alternative to adding this algorithm to the current sample()
   function is to add a new function sample_iterable() or similar.
   (This new function probably shouldn't call shuffle(), since the
   caller can always do this if required.)

   [2]: http://hg.python.org/cpython/file/54fb77e0762c/Lib/random.py#l268
   [3]: http://en.wikipedia.org/wiki/Reservoir_sampling

3. random.choice(seq)

   Similarly to the above explanation, random.choice() could be
   generalised to arbitrary iterables.  Of course the current O(1)
   alogrithm should be used if a sequence is passed.

In case of positive feedback on one or more of those points, I'd be
happy to prepare a patch.

 -- Sven


From dickinsm at gmail.com  Thu Jun 23 22:43:32 2011
From: dickinsm at gmail.com (Mark Dickinson)
Date: Thu, 23 Jun 2011 22:43:32 +0200
Subject: [Python-ideas] A few suggestions for the random module
In-Reply-To: <20110623175329.GF4696@pantoffel-wg.de>
References: <20110623175329.GF4696@pantoffel-wg.de>
Message-ID: <BANLkTimwL1bccebdEkS4Rv+MW=JfUkg+AA@mail.gmail.com>

On Thu, Jun 23, 2011 at 7:53 PM, Sven Marnach <sven at marnach.net> wrote:
> I'd like to suggest what I consider a few minor improvements to
> Python's random module.
> [...]

Agreed on point 1 (implementation of random.expovariate).

For all three issues, I suggest filing a report in the issue tracker.
(Probably one separate report per item, though perhaps items 2 and 3
could be combined.)

Mark


From stutzbach at google.com  Thu Jun 23 23:21:19 2011
From: stutzbach at google.com (Daniel Stutzbach)
Date: Thu, 23 Jun 2011 14:21:19 -0700
Subject: [Python-ideas] A few suggestions for the random module
In-Reply-To: <20110623175329.GF4696@pantoffel-wg.de>
References: <20110623175329.GF4696@pantoffel-wg.de>
Message-ID: <BANLkTikOr7TqNyYn2O6=Z-9+wUNjDnkOw+cdxy1KYJUGRvtTxw@mail.gmail.com>

+1 on all three suggestions.

As Mark said, please file these ideas at http://bugs.python.org so we can
keep track of them.  I think the second and third suggestions can be
combined as one issue.  It would also be helpful if you could supply a patch
for the changes (and for the second and third suggestions, unit tests to
exercise the new functionality).

-- 
Daniel Stutzbach
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110623/c217b634/attachment.html>

From raymond.hettinger at gmail.com  Fri Jun 24 00:02:20 2011
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Fri, 24 Jun 2011 00:02:20 +0200
Subject: [Python-ideas] A few suggestions for the random module
In-Reply-To: <20110623175329.GF4696@pantoffel-wg.de>
References: <20110623175329.GF4696@pantoffel-wg.de>
Message-ID: <346EAC9B-7A22-4EAA-B15A-F8C33F280E97@gmail.com>


On Jun 23, 2011, at 7:53 PM, Sven Marnach wrote:

> I'd like to suggest what I consider a few minor improvements to
> Python's random module.

You can create a feature request on the bug tracker and assign to me.

> 
> 1. random.expovariate(lambd)

This seems reasonable
> 
> 2. random.sample(population, k)

This may be a unnecessary optimization (not worth the complexity),
but I will look at it further.

> 
> 3. random.choice(seq)

It could be generalized to arbitrary iterables (Bentley provides an example of how to do this) but it is fragile (i.e. falls apart badly with weak random number generators) and doesn't correspond well with real use cases.  


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20110624/d6686dd1/attachment.html>

From sven at marnach.net  Sat Jun 25 20:37:16 2011
From: sven at marnach.net (Sven Marnach)
Date: Sat, 25 Jun 2011 19:37:16 +0100
Subject: [Python-ideas] A few suggestions for the random module
In-Reply-To: <346EAC9B-7A22-4EAA-B15A-F8C33F280E97@gmail.com>
References: <20110623175329.GF4696@pantoffel-wg.de>
	<346EAC9B-7A22-4EAA-B15A-F8C33F280E97@gmail.com>
Message-ID: <20110625183716.GA3377@pantoffel-wg.de>

Raymond Hettinger schrieb am Fr, 24. Jun 2011, um 00:02:20 +0200:
>      1. random.expovariate(lambd)
> 
>    This seems reasonable

When I started to prepare a patch, I noticed you already implemented
this one.  Thanks!

>      2. random.sample(population, k)
> 
>    This may be a unnecessary optimization (not worth the complexity),
>    but I will look at it further.

The main point was the generalization to arbitrary iterables, the
reduced memory usage being just a positive side effect.  I'm not
really convinced of the idea any more since it doesn't really fit well
in the current `sample()` implementation.  If someone wants to save
some memory in the case that k is not much less than n, she can just
copy the population, `shuffle()` it and delete the unneeded part.

The only question that remains is if it is worthwhile to introduce a
new function for this purpose.  While there definitely are use cases
for the algorithm, I simply don't know if they are common enough to
justify a function in the standard library.

>      3. random.choice(seq)
> 
>    It could be generalized to arbitrary iterables (Bentley provides an
>    example of how to do this) but it is fragile (i.e. falls apart badly with
>    weak random number generators) and doesn't correspond well with real use
>    cases.

Again, there definitely are real world use cases.  If the above
mentioned function would be introduced, this one would simply be the
special case k = 1.

 -- Sven


From g.rodola at gmail.com  Wed Jun 29 19:00:01 2011
From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=)
Date: Wed, 29 Jun 2011 19:00:01 +0200
Subject: [Python-ideas] Adding shutil.disk_usage()
In-Reply-To: <op.vwx9drqcd4rfj9@efcore>
References: <BANLkTimBgYw=XmNQFDbFOH2y5P_qS248yQ@mail.gmail.com>
	<op.vwx9drqcd4rfj9@efcore>
Message-ID: <BANLkTimrT8b_qxmxVQYBKyB=93SUU75BVQ@mail.gmail.com>

This is now tracked at http://bugs.python.org/issue12442

Regards,

--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/


From sturla at molden.no  Thu Jun 30 13:38:07 2011
From: sturla at molden.no (Sturla Molden)
Date: Thu, 30 Jun 2011 13:38:07 +0200
Subject: [Python-ideas] dir with a glob?
Message-ID: <4E0C601F.5010001@molden.no>

Often when exploring an object with the 'dir' function, particularly in 
large packages like SciPy, I find that I need to filter the outout. 
Since a dir reminds me of a dos 'dir' or linux 'ls', a glob feels like 
the most natural to use.

For example, none of these would work:

 >>> dir(sp.fft.i*)  # syntax error
 >>> dir('sp.fft.i*') # returns the attributes a string

I believe a new dir functions is needed, or a change in the behviour of 
the current version.

Of course a 'glob aware dir function' can be implemented by monitoring 
the call stack (cf. sys._getframe().f_back), but I think the problem is 
general enough to warrant an official sultion.

Sturla


From ben+python at benfinney.id.au  Thu Jun 30 14:29:41 2011
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 30 Jun 2011 22:29:41 +1000
Subject: [Python-ideas] dir with a glob?
References: <4E0C601F.5010001@molden.no>
Message-ID: <87liwjlcy2.fsf@benfinney.id.au>

Sturla Molden <sturla at molden.no> writes:

> Often when exploring an object with the 'dir' function, particularly
> in large packages like SciPy, I find that I need to filter the outout.

We have list comprehensions and generator expressions to filter a
sequence, and I suspect they would serve you well for this purpose.

> Since a dir reminds me of a dos 'dir' or linux 'ls', a glob feels like
> the most natural to use.
>
> For example, none of these would work:
>
> >>> dir(sp.fft.i*)  # syntax error

What would you expect this to return? The ?dir? function is specifically
for inspecting *one* object. Do you just want all the attributes of all
the matching objects mixed up together, or what?

> I believe a new dir functions is needed, or a change in the behviour
> of the current version.

This is ?python-ideas?. What is your idea for the desired behaviour?

What about using a list comprehension or generator expression to get
what you want?

-- 
 \        ?A learning experience is one of those things that say, ?You |
  `\    know that thing you just did? Don't do that.?? ?Douglas Adams, |
_o__)                                                       2000-04-05 |
Ben Finney



From masklinn at masklinn.net  Thu Jun 30 14:38:09 2011
From: masklinn at masklinn.net (Masklinn)
Date: Thu, 30 Jun 2011 14:38:09 +0200
Subject: [Python-ideas] dir with a glob?
In-Reply-To: <87liwjlcy2.fsf@benfinney.id.au>
References: <4E0C601F.5010001@molden.no> <87liwjlcy2.fsf@benfinney.id.au>
Message-ID: <64A24A6E-634B-46BB-AB68-2F4579B861CA@masklinn.net>

On 2011-06-30, at 14:29 , Ben Finney wrote:
> Sturla Molden <sturla at molden.no> writes:
>> I believe a new dir functions is needed, or a change in the behviour
>> of the current version.
> 
> This is ?python-ideas?. What is your idea for the desired behaviour?
> 
> What about using a list comprehension or generator expression to get
> what you want?
Or a good ol' `filter`:

    filter(methodcaller('startswith', 'i'), dir(sp.fft))

From jsbueno at python.org.br  Thu Jun 30 14:41:15 2011
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Thu, 30 Jun 2011 08:41:15 -0400
Subject: [Python-ideas] dir with a glob?
In-Reply-To: <87liwjlcy2.fsf@benfinney.id.au>
References: <4E0C601F.5010001@molden.no>
	<87liwjlcy2.fsf@benfinney.id.au>
Message-ID: <BANLkTi=OHA7-qnJPCHENCL7dk0dQ3OAOuw@mail.gmail.com>

On Thu, Jun 30, 2011 at 8:29 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
> Sturla Molden <sturla at molden.no> writes:
>
>> Often when exploring an object with the 'dir' function, particularly
>> in large packages like SciPy, I find that I need to filter the outout.
>
> We have list comprehensions and generator expressions to filter a
> sequence, and I suspect they would serve you well for this purpose.

The problem is that dir is generally used on a live section,
and a list comprehension to filter dir results is way to verbose
to type when you need it.

>
>> Since a dir reminds me of a dos 'dir' or linux 'ls', a glob feels like
>> the most natural to use.
>>
>> For example, none of these would work:
>>
>> >>> dir(sp.fft.i*) ?# syntax error
>
> What would you expect this to return? The ?dir? function is specifically
> for inspecting *one* object. Do you just want all the attributes of all
> the matching objects mixed up together, or what?
>
>> I believe a new dir functions is needed, or a change in the behviour
>> of the current version.
>
> This is ?python-ideas?. What is your idea for the desired behaviour?
>
> What about using a list comprehension or generator expression to get
> what you want?


I believe a second parameter to dir, being  a glob filter string, would be
fine. Being able to type:

dir (gtk.Window, "set*")

, for example
instead of

[attrib for attrib in dir(gtk.Window) if attrib.startswith("set")]


would be indeed a good thing to have.

  js
 -><-


> --
> ?\ ? ? ? ??A learning experience is one of those things that say, ?You |
> ?`\ ? ?know that thing you just did? Don't do that.?? ?Douglas Adams, |
> _o__) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 2000-04-05 |
> Ben Finney
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>


From grosser.meister.morti at gmx.net  Thu Jun 30 15:09:46 2011
From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=)
Date: Thu, 30 Jun 2011 15:09:46 +0200
Subject: [Python-ideas] dir with a glob?
In-Reply-To: <4E0C601F.5010001@molden.no>
References: <4E0C601F.5010001@molden.no>
Message-ID: <4E0C759A.4000904@gmx.net>

I would rather add a grep method to regular expression objects:

	def grep(self,sequence):
		for s in sequence:
			if self.search(s):
				yield s

And a global function to the re module:

	def grep(regex,sequence,flags=0):
		return re.compile(regex,flags).grep(sequence)


Then you could do:

	re.grep("^i", dir(sp.fft))

Yes, it would be more to type but it would be more general.

OT: I think it would be good if regular expressions would be callable with this method:

	def __call__(self,s):
		return self.search(s) is not None

Then one could use it in a filter expression:

	filter(re.compile("^i"), dir(sp.fft))

On 06/30/2011 01:38 PM, Sturla Molden wrote:
> Often when exploring an object with the 'dir' function, particularly in large packages like SciPy, I
> find that I need to filter the outout. Since a dir reminds me of a dos 'dir' or linux 'ls', a glob
> feels like the most natural to use.
>
> For example, none of these would work:
>
>  >>> dir(sp.fft.i*) # syntax error
>  >>> dir('sp.fft.i*') # returns the attributes a string
>
> I believe a new dir functions is needed, or a change in the behviour of the current version.
>
> Of course a 'glob aware dir function' can be implemented by monitoring the call stack (cf.
> sys._getframe().f_back), but I think the problem is general enough to warrant an official sultion.
>
> Sturla



From sturla at molden.no  Thu Jun 30 15:14:59 2011
From: sturla at molden.no (Sturla Molden)
Date: Thu, 30 Jun 2011 15:14:59 +0200
Subject: [Python-ideas] dir with a glob?
In-Reply-To: <BANLkTi=OHA7-qnJPCHENCL7dk0dQ3OAOuw@mail.gmail.com>
References: <4E0C601F.5010001@molden.no> <87liwjlcy2.fsf@benfinney.id.au>
	<BANLkTi=OHA7-qnJPCHENCL7dk0dQ3OAOuw@mail.gmail.com>
Message-ID: <4E0C76D3.1060606@molden.no>

Den 30.06.2011 14:41, skrev Joao S. O. Bueno:
> dir (gtk.Window, "set*")

Yes, something like that would be conscise enough for an interactive 
session.

Sturla




From sturla at molden.no  Thu Jun 30 15:27:16 2011
From: sturla at molden.no (Sturla Molden)
Date: Thu, 30 Jun 2011 15:27:16 +0200
Subject: [Python-ideas] dir with a glob?
In-Reply-To: <87liwjlcy2.fsf@benfinney.id.au>
References: <4E0C601F.5010001@molden.no> <87liwjlcy2.fsf@benfinney.id.au>
Message-ID: <4E0C79B4.3030209@molden.no>

Den 30.06.2011 14:29, skrev Ben Finney:
> We have list comprehensions and generator expressions to filter a 
> sequence, and I suspect they would serve you well for this purpose.

Dir is often used to explore objects or modules interactively, which 
means we need a very short syntax for this to be handy. Perhaps 
something like this:

dir(object, "foo*")

Making an utility function like this with glob or regex is of course 
extremely simple. Anyone can do it for themselves, how ever they like.

What I am asking is if the need to filter the output from dir is so 
common that it could warrant a change to Python?

Sturla






From mikegraham at gmail.com  Thu Jun 30 15:30:29 2011
From: mikegraham at gmail.com (Mike Graham)
Date: Thu, 30 Jun 2011 09:30:29 -0400
Subject: [Python-ideas] dir with a glob?
In-Reply-To: <4E0C79B4.3030209@molden.no>
References: <4E0C601F.5010001@molden.no> <87liwjlcy2.fsf@benfinney.id.au>
	<4E0C79B4.3030209@molden.no>
Message-ID: <BANLkTin0ODEQGfodKe5KcU5=UUYzFf+m-A@mail.gmail.com>

On Thu, Jun 30, 2011 at 9:27 AM, Sturla Molden <sturla at molden.no> wrote:
> Dir is often used to explore objects or modules interactively,

This strikes me as a mistake. If something is made more featureful, it
should be help, not dir.


From ncoghlan at gmail.com  Thu Jun 30 15:34:00 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 30 Jun 2011 23:34:00 +1000
Subject: [Python-ideas] dir with a glob?
In-Reply-To: <4E0C601F.5010001@molden.no>
References: <4E0C601F.5010001@molden.no>
Message-ID: <BANLkTimAymaXoVVwMDd_U+mj21fjxS3Jdw@mail.gmail.com>

On Thu, Jun 30, 2011 at 9:38 PM, Sturla Molden <sturla at molden.no> wrote:
> Often when exploring an object with the 'dir' function, particularly in
> large packages like SciPy, I find that I need to filter the outout. Since a
> dir reminds me of a dos 'dir' or linux 'ls', a glob feels like the most
> natural to use.
>
> For example, none of these would work:
>
>>>> dir(sp.fft.i*) ?# syntax error
>>>> dir('sp.fft.i*') # returns the attributes a string

import fnmatch
def glob_dir(obj, pattern):
    return fnmatch.filter(dir(obj), pattern)

>>> print(glob_dir(str, "c*"))
['capitalize', 'center', 'count']

If it's something you use often, drop it into a utility module or PYTHONSTARTUP

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ncoghlan at gmail.com  Thu Jun 30 15:36:39 2011
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 30 Jun 2011 23:36:39 +1000
Subject: [Python-ideas] dir with a glob?
In-Reply-To: <4E0C79B4.3030209@molden.no>
References: <4E0C601F.5010001@molden.no> <87liwjlcy2.fsf@benfinney.id.au>
	<4E0C79B4.3030209@molden.no>
Message-ID: <BANLkTimwsdO-x_JEA1eJJhnNnMsJVyQ+eg@mail.gmail.com>

On Thu, Jun 30, 2011 at 11:27 PM, Sturla Molden <sturla at molden.no> wrote:
> What I am asking is if the need to filter the output from dir is so common
> that it could warrant a change to Python?

No, if an object is complicated enough that pprint(dir(obj)) and
help(obj) aren't adequate to explore it, then it is time to go read
the documentation (or the source, if the documentation is lacking).

The interactive prompt is just one of the available tools for code
exploration, it doesn't make sense to try to make it do everything.

Cheers,
Nick.

-- 
Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia


From ben+python at benfinney.id.au  Thu Jun 30 15:41:29 2011
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 30 Jun 2011 23:41:29 +1000
Subject: [Python-ideas] dir with a glob?
References: <4E0C601F.5010001@molden.no> <87liwjlcy2.fsf@benfinney.id.au>
	<4E0C79B4.3030209@molden.no>
Message-ID: <87hb77l9me.fsf@benfinney.id.au>

Sturla Molden <sturla at molden.no> writes:

> dir(object, "foo*")

I ask again: what would you expect (give an example) the output of this
to be?

> What I am asking is if the need to filter the output from dir is so
> common that it could warrant a change to Python?

Given that we already have ways to filter a sequence built into the
language, I doubt the need for a special way to filter the output from
some particular function.

-- 
 \      ?I used to be a proofreader for a skywriting company.? ?Steven |
  `\                                                            Wright |
_o__)                                                                  |
Ben Finney



From p.f.moore at gmail.com  Thu Jun 30 15:43:17 2011
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 30 Jun 2011 14:43:17 +0100
Subject: [Python-ideas] dir with a glob?
In-Reply-To: <4E0C76D3.1060606@molden.no>
References: <4E0C601F.5010001@molden.no> <87liwjlcy2.fsf@benfinney.id.au>
	<BANLkTi=OHA7-qnJPCHENCL7dk0dQ3OAOuw@mail.gmail.com>
	<4E0C76D3.1060606@molden.no>
Message-ID: <BANLkTimfd-ukfSfujxHH7z2oZbRE3807pw@mail.gmail.com>

On 30 June 2011 14:14, Sturla Molden <sturla at molden.no> wrote:
> Den 30.06.2011 14:41, skrev Joao S. O. Bueno:
>>
>> dir (gtk.Window, "set*")
>
> Yes, something like that would be conscise enough for an interactive
> session.

Sounds like what you're really after is a richer interactive
environment. Have you looked at IPython? (Disclaimer: I don't use it
myself, but I know it adds *lots* of features to make the interactive
experience better and more productive).

Paul.


From __peter__ at web.de  Thu Jun 30 16:54:03 2011
From: __peter__ at web.de (Peter Otten)
Date: Thu, 30 Jun 2011 16:54:03 +0200
Subject: [Python-ideas] dir with a glob?
References: <4E0C601F.5010001@molden.no> <87liwjlcy2.fsf@benfinney.id.au>
	<4E0C79B4.3030209@molden.no> <87hb77l9me.fsf@benfinney.id.au>
Message-ID: <iui2m4$qf0$1@dough.gmane.org>

Ben Finney wrote:

> Sturla Molden <sturla at molden.no> writes:
> 
>> dir(object, "foo*")
> 
> I ask again: what would you expect (give an example) the output of this
> to be?

If I were to guess:

>>> import __builtin__, fnmatch         
>>> def dir(obj, glob=None):            
...     names = __builtin__.dir(obj)
...     if glob is not None:
...             names = fnmatch.filter(names, glob)
...     return names                               
...                                                

Example usage: what was the name of the function to remove a directory and 
all empty parents again?

>>> import os
>>> dir(os)

[snip list with more than 200 names]

>>> dir(os, "r*d*")
['read', 'readlink', 'removedirs', 'rmdir']

Ah, I think I remember it now...

>> What I am asking is if the need to filter the output from dir is so
>> common that it could warrant a change to Python?

At the moment I'm writing list comprehensions in cases like the above, but 
I'd welcome the addition of a glob or regex argument to dir().
 
> Given that we already have ways to filter a sequence built into the
> language, I doubt the need for a special way to filter the output from
> some particular function.
 




From steve at pearwood.info  Thu Jun 30 17:54:29 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 01 Jul 2011 01:54:29 +1000
Subject: [Python-ideas] dir with a glob?
In-Reply-To: <BANLkTimwsdO-x_JEA1eJJhnNnMsJVyQ+eg@mail.gmail.com>
References: <4E0C601F.5010001@molden.no>
	<87liwjlcy2.fsf@benfinney.id.au>	<4E0C79B4.3030209@molden.no>
	<BANLkTimwsdO-x_JEA1eJJhnNnMsJVyQ+eg@mail.gmail.com>
Message-ID: <4E0C9C35.4080405@pearwood.info>

Nick Coghlan wrote:
> On Thu, Jun 30, 2011 at 11:27 PM, Sturla Molden <sturla at molden.no> wrote:
>> What I am asking is if the need to filter the output from dir is so common
>> that it could warrant a change to Python?
> 
> No, if an object is complicated enough that pprint(dir(obj)) and
> help(obj) aren't adequate to explore it, then it is time to go read
> the documentation (or the source, if the documentation is lacking).

Do you think that *reading the source* is to be preferred over a simple 
tool like running a filter over the output of dir()? I'm not exactly 
sure what point you're trying to make, but I don't think it's a good one.


> The interactive prompt is just one of the available tools for code
> exploration, it doesn't make sense to try to make it do everything.

But the interactive prompt already does everything. Anything you can do 
in Python can be done at the prompt. The question we are asking is not 
should users be able to filter the output of dir, because of course they 
can. The question we are asking is, should there be One Obvious Way to 
filter the output of dir, rather than the status quo (people do without, 
reinvent the wheel, or struggle with non-obvious, inconvenient and 
verbose ways).

I'm +1 on adding a glob filter to dir.



-- 
Steven



From steve at pearwood.info  Thu Jun 30 18:31:38 2011
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 01 Jul 2011 02:31:38 +1000
Subject: [Python-ideas] dir with a glob?
In-Reply-To: <87hb77l9me.fsf@benfinney.id.au>
References: <4E0C601F.5010001@molden.no>
	<87liwjlcy2.fsf@benfinney.id.au>	<4E0C79B4.3030209@molden.no>
	<87hb77l9me.fsf@benfinney.id.au>
Message-ID: <4E0CA4EA.2010705@pearwood.info>

Ben Finney wrote:
> Sturla Molden <sturla at molden.no> writes:
> 
>> dir(object, "foo*")
> 
> I ask again: what would you expect (give an example) the output of this
> to be?

That specific example should return an empty list, since object has no 
attributes that match the glob foo*


>> What I am asking is if the need to filter the output from dir is so
>> common that it could warrant a change to Python?
> 
> Given that we already have ways to filter a sequence built into the
> language, I doubt the need for a special way to filter the output from
> some particular function.

There is precedence though.

Many string methods have special way to limit their output to some 
subset of results, rather than relying on a generic, built-in way to do 
the same thing:

mystr.find("spam", 23, 42) vs mystr[23:42].find("spam")


dir itself already duplicates functionality available elsewhere:

 >>> dir() == sorted(globals())
True

dir with an argument is a little trickier, but it's not far off a one-liner:

dir(d) == sorted(
   set(sum((o.__dict__.keys() for o in ((d,)+type(d).__mro__)), []))
   )

(at least for new-style objects with no __slots__).


dir is a convenience function, designed for interactive use. The docs 
make it explicit:

[quote]
Because dir() is supplied primarily as a convenience for use at an 
interactive prompt, it tries to supply an interesting set of names more 
than it tries to supply a rigorously or consistently defined set of names...

http://docs.python.org/library/functions.html#dir


Given that, I see no downside to making dir more convenient for 
interactive use. That's what it's for.



-- 
Steven



From g.brandl at gmx.net  Thu Jun 30 19:22:39 2011
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 30 Jun 2011 19:22:39 +0200
Subject: [Python-ideas] dir with a glob?
In-Reply-To: <4E0CA4EA.2010705@pearwood.info>
References: <4E0C601F.5010001@molden.no>
	<87liwjlcy2.fsf@benfinney.id.au>	<4E0C79B4.3030209@molden.no>
	<87hb77l9me.fsf@benfinney.id.au> <4E0CA4EA.2010705@pearwood.info>
Message-ID: <iuibbu$l7d$1@dough.gmane.org>

On 30.06.2011 18:31, Steven D'Aprano wrote:
> Ben Finney wrote:
>> Sturla Molden <sturla at molden.no> writes:
>> 
>>> dir(object, "foo*")
>> 
>> I ask again: what would you expect (give an example) the output of this
>> to be?
> 
> That specific example should return an empty list, since object has no 
> attributes that match the glob foo*
> 
> 
>>> What I am asking is if the need to filter the output from dir is so
>>> common that it could warrant a change to Python?
>> 
>> Given that we already have ways to filter a sequence built into the
>> language, I doubt the need for a special way to filter the output from
>> some particular function.
> 
> There is precedence though.
> 
> Many string methods have special way to limit their output to some 
> subset of results, rather than relying on a generic, built-in way to do 
> the same thing:
> 
> mystr.find("spam", 23, 42) vs mystr[23:42].find("spam")
> 
> 
> dir itself already duplicates functionality available elsewhere:
> 
>  >>> dir() == sorted(globals())
> True
> 
> dir with an argument is a little trickier, but it's not far off a one-liner:
> 
> dir(d) == sorted(
>    set(sum((o.__dict__.keys() for o in ((d,)+type(d).__mro__)), []))
>    )
> 
> (at least for new-style objects with no __slots__).

And don't forget about __dir__...

> dir is a convenience function, designed for interactive use. The docs 
> make it explicit:
> 
> [quote]
> Because dir() is supplied primarily as a convenience for use at an 
> interactive prompt, it tries to supply an interesting set of names more 
> than it tries to supply a rigorously or consistently defined set of names...
> 
> http://docs.python.org/library/functions.html#dir
> 
> 
> Given that, I see no downside to making dir more convenient for 
> interactive use. That's what it's for.

I agree.  I often am looking for a specific member that I know exists, but
don't recall the exact name (and in particular, not what the name starts
with: at least dir() output is sorted).  Searching through one screenful
of members isn't pretty.

So, +1 for the second argument.

Georg



From g.brandl at gmx.net  Thu Jun 30 19:27:12 2011
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 30 Jun 2011 19:27:12 +0200
Subject: [Python-ideas] dir with a glob?
In-Reply-To: <4E0C759A.4000904@gmx.net>
References: <4E0C601F.5010001@molden.no> <4E0C759A.4000904@gmx.net>
Message-ID: <iuibk9$l7d$2@dough.gmane.org>

On 30.06.2011 15:09, Mathias Panzenb?ck wrote:
> I would rather add a grep method to regular expression objects:
> 
> 	def grep(self,sequence):
> 		for s in sequence:
> 			if self.search(s):
> 				yield s
> 
> And a global function to the re module:
> 
> 	def grep(regex,sequence,flags=0):
> 		return re.compile(regex,flags).grep(sequence)
> 
> 
> Then you could do:
> 
> 	re.grep("^i", dir(sp.fft))
> 
> Yes, it would be more to type but it would be more general.

No you couldn't, since this would return <generator ...> -- an explicit
list() would be required.

> OT: I think it would be good if regular expressions would be callable with this method:
> 
> 	def __call__(self,s):
> 		return self.search(s) is not None
> 
> Then one could use it in a filter expression:
> 
> 	filter(re.compile("^i"), dir(sp.fft))

What's wrong with

   filter(re.compile("^i").search, dir(sp.fft))

?  And for non-interactive use, you almost always have a compiled regex object
already, so that this becomes

   filter(rex.search, seq)

which is not sufficiently harder than

   rex.grep(seq)

to justify a new regex method.  (Also, people won't be able to remember if
grep() uses match() or search().)

Georg