why cannot assign to function call
rdmurray at bitdance.com
rdmurray at bitdance.com
Sat Feb 28 14:46:00 EST 2009
Mark Wooding <mdw at distorted.org.uk> wrote:
> Ethan Furman <ethan at stoneleaf.us> writes:
>
> > Mark Wooding wrote:
> >> Here's what I think is the defining property of pass-by-value [...]:
> >>
> >> The callee's parameters are /new variables/, initialized /as if by
> >> assignment/ from the values of caller's argument expressions.
> >>
> >> My soundbite definition for pass-by-reference is this:
> >>
> >> The callee's parameters are merely /new names/ for the caller's
> >> argument variables -- as far as that makes sense.
> >
> > Greetings, Mark!
>
> You posted to the whole group -- probably using a wide-reply option
> while reading the mailing list. Still, I'll give an answer here in
> order to help any other readers of the list or newsgroup. However, as
> far as I'm concerned, this discussion is basically at an end and I'm not
> really interested in starting it up all over again. To that end, I've
> set followups to `poster'. I shall continue reply to private email
> which seems interested in a sensible discussion.
I just popped in to this thread, and read the whole thing in a marathon
session (why, I'm not quite sure, but somehow I found it interesting :).
I'm going to follow up here at the risk of annoying Mark, because I
think there may be a way to reconcile the language in a way that is
helpful in explaining things to Python beginners.
> > I was hoping you might be able to clarify those last two sound bites
> > for me -- I think I understand what you are saying, but I'm confused
> > about how they relate to Python...
I think this is the key point. I am an experienced Python programmer,
and I had to think long and hard about what you were saying, Mark, in
order to understand how it applied to Python. I think this means that
your model and/or your level of abstraction is not "natural" when trying
to understand Python programs, and I think I may have figured out why.
> > Specifically, how is a new name (pbr) different, in Python, from a new
> > name initialized as if by assignment (pbv)? It seems to me than you
> > end up with the same thing in either case (in Python, at least),
> > making the distinction non-existent.
>
> You've missed a level of indirection. In particular, `names' aren't
> things that you initialize. They're things that you /bind/ to
> variables. The crucial difference is that, in pass-by-value, new
> variables are created as an intrinsic part of the process, whereas in
> pass-by-reference, new variables are not (usually) created, and instead
> the formal parameter names are bound to the caller's pre-existing
> argument variables.
I think that the reason Ethan missed that level of indirection is that
Python hides that level of indirection from the programmer. And I think
that this is where the conversation between you and Steven got hung
up, Mark. You are firmly grounded in how languages work in general,
so your conceptual model naturally includes details that Steven's
conceptual model can ignore, because he's dealing only with the model
used by Python. He can (and wants to!) ignore the level of indirection
that Python doesn't even give him access to.
In short, there is a reason why you almost never hear the word 'variable'
from an experienced Python programmer when talking about the Python model.
That level detail is abstracted into something else by the Python
conceptual model: it becomes a namespace mapping names to objects.
> It's worth noting that I use the terms `name' and `binding' in different
> ways from most of the Python community. This is unfortunate. The
> discrepancy is actually because the Python meanings of these words are
> not the same as the meanings in the wider computer science and
> mathematical communities. For example, many Python users seem to use
> `binding' to mean `assignment', which is a shame because it leaves the
> concept that is usually called `binding' without a name. So I'll stick
> with the wider meanings.
It seems to me from reading the thread that everyone is actually in
pretty good agreement that 'name' is the symbol one types in the
program source to denote...something. It's the something where things
get tricky. So let's leave name itself alone.
But let's replace 'assignment' and 'binding' with the concept that is
more natural to Python's model, that of a "mapping". A namespace maps
names to objects. If I'm understanding you correctly, Mark, in your
model what I'm calling a namespace is the set of variable bindings in a
particular environment. Because Python does not allow the programmer
to actually rebind a variable (Python handles the association between
name and variable completely behind the scenes), Python programmers
naturally skip past that name-to-variable binding step, and conceptually
think about the name being bound to the _value_ instead. After all,
that's the only thing they can change. This may be unfortunate from
a precisionist viewpoint, but it is both natural and, IMO, inevitable,
because it makes thinking about Python programs more efficient.
> A while ago, I posted an essay -- to this group -- which may help
> explain the concepts:
>
> Message-ID: <8763k14nc6.fsf.mdw at metalzone.distorted.org.uk>
> http://groups.google.com/group/comp.lang.python/msg/f6f1a321f819d02b
>
> On with the question.
>
> > def func(bar):
> > bar.pop()
> >
> > Pass-by-reference:
> > foo = ['Ethan','Furman']
> > func(foo) # bar = foo
> >
> > Pass-by-value:
> > foo = ['Python','Rocks!']
> > func(foo) # bar is new name for foo
> > # is this any different from above?
> >
> > If I have this right, in both cases foo will be reduced to a
> > single-item list after func.
>
> You're correct. So: we can conclude that the above test is not
> sufficient to distinguish the two cases.
Suppose we recast this by saying that the function has a namespace,
and in that namespace 'bar' maps to something. What that something is
depends on what is passed in that argument position when func is called.
The calling code's namespace maps 'foo' to something. When func is
called, a mapping to that same object is set up in func's namespace,
under the name 'bar'.
In Python's conceptual model it doesn't make any sense to ask if
the argument is passed by value or by reference. In reality, if I'm
understanding you correctly Mark, you are absolutely correct that the
'variable' (the thing actually pointed to by 'name', which is a storage
location that contains a pointer to an object) is passed by value.
That is, the contents of the storage location that the caller's 'foo'
is pointing to is copied into a new storage location that the callee's
'bar' points to. Conceptually, however, that is an implementation detail.
Conceptually, the important thing is that a _new mapping_ in a _different
namespace_ is set up to the _same object_. That is the semantics that
Python the language defines.
As Steven said, talking about variables seems to be drilling down into
a level of abstraction that is more confusing than it is helpful when
you are actually _programming_ in Python. It can be a useful level of
abstraction to talk about when trying to explain what is happening to
someone with previous experience of other languages, but it isn't useful
(in my experience) for everyday Python programming.
The reason that is true is because in Python there is _no way_ to get
a second reference to the 'variable' that is pointed to by 'foo' or by
'bar'. (Unless you walk into that secret back room, of course, and even
then it is hard, I think.) As I said, Python hides that level of detail
from the programmer. This means that it effectively does not exist when
you program Python, and _this_ is why we get all tangled up in language
when we try to discuss it. You, Mark, had to go to great lengths to
define your terms, and still a lot of people didn't get it, because in
actual, on-the-ground Python programming, variables just don't enter in to
it, because they can not be directly manipulated from the Python program.
I suspect that the longer one has been a Python programmer, the harder
it is to even see this blind spot, and thus the harder it is to respond
sensibly to someone new to Python whose conceptual model includes
variables that _can_ be accessed and manipulated from the program. I know
_I_ had to read and re-read your posts before I finally figured it out
(assuming I have!). (FYI, my programming learning path was roughly
Basic->FORTRAN ->LISP->assembler->C->Python, and I've been programming
Python since 1993 or thereabouts).
(BTW, I am not claiming Python is unique in this regard, I'm just talking
about how it looks from inside the Python programming mindset.)
> > Any further explanation you care to provide will be greatly
> > appreciated!
>
> This test is sufficient to distinguish:
>
> def test(x):
> x = 'clobbered'
>
> y = 'virgin'
> test(y)
> print y
>
> If it prints `virgin' then you have call-by-value. If it prints
> `clobbered' then you have call-by-reference.
Or if you think in terms of namespace mappings, 'test' cannot change what
'y' is mapped to in the caller's namespace, because (in this function)
it has no mapping to that namespace object. It _can_ make changes to
the thing that y is mapped _to_, if that thing is mutable. That's the
most useful way to think about it that I've found. Your reasoning about
variables and call-by-reference and call-by-value, while correct and
enlightening at the computer science level, confuses me and makes me
less efficient if I think about it while writing a Python program.
> Let's examine the two cases, as I did in the essay I cited above. I'll
> do call-by-value first. First, we define a function `test'. Then, we
> initialize `y'. It's worth examining this process in detail. The name
> `y' is initially unbound. so it is implicitly bound to a fresh
> variable. Then, (a reference to) the string object 'virgin' is stored
> in this variable. We can show this diagrammatically as follows.
>
> y (in global env.) ====> [VAR] ---> 'virgin'
>
> (In the diagrams, ===> denotes a binding relationship, between names and
> variables; and ---> denotes a reference relationship, between variables
> and values.)
This diagram is IMO accurate and correct. However, in Python we can
never actually touch [VAR] except insofar as we change the right hand
side element of the diagram. And we can only do _that_ if we have access
to the object that is the global name space, and we mutate that object.
So for reasoning about Python programs, I think like this:
y (in global namespace) ---> 'virgin'
> Next, we call the `test' function. Call-by-value says that we must
> evaluate the argument expressions. There's only one: `x'. The value of
> a name is obtained by (a) finding which variable is bound to the name,
> and (b) extracting the value from this variable. Well, the variable is
> the one we just bound, and the value stored is (the reference to) the
> string 'virgin'. So the result of evaluating the argument expressions
> is simply (the reference to) that string.
>
> The function has one parameter, `y'. A new environment is constructed
> by extending the global environment. In this new environment, the name
> `y' is bound to a fresh variable -- distinct from all others, and
> especially from the variable bound to `x' -- and in that variable we
> store the value of the corresponding argument expression. Result: the
> function body is executed in an environment which is like the global
> environment except that `y' is bound to a fresh variable containing
> 'virgin'.
>
> y (in global env.) ====> [VAR] ---> 'virgin'
> ^
> |
> x (in function `test') ====> [VAR] -------'
I would describe this (in Python) as follows: we call the test function,
passing it x. This means we find the object that x is mapped to in
the caller's namespace, and we map 'y' in the function's namespace
to the same object:
y (in global env.) ---> 'virgin'
^
|
x (in function 'test') ---'
> Now there's an assignment
>
> x = 'clobbered'
>
> The name `x' is already bound to a variable. So we modify that variable
> so that it stores (a reference to) the string 'clobbered'.
>
> y (in global env.) ====> [VAR] ---> 'virgin'
>
> x (in function `test') ====> [VAR] ---> 'clobbered'
Assignment in Python changes a mapping (mutates a namespace object). In
this case the mapping being changed is the local function's namespace
mapping. We can't affect the caller's namespace mapping, because (in
this function) we don't have access to the caller's namespace object.
So I would diagram it like this:
y (in global env.) -----> 'virgin'
x (in function 'test') ----> 'clobbered'
> And then the function ends. The environment we constructed is
> forgotten. The variable bound to `x' is lost forever, since it wasn't
> bound to any other name. Since modifying that variable was the only
> action carried out in the function, and the variable is now lost, there
> is no externally observable effect. When we finally print `y', we see
> `virgin', because the variable bound to `y' was unchanged.
>
> y (in global env.) ====> [VAR] ---> 'virgin'
The function's local namespace is not accessible to the caller, so the
caller can't access the object 'clobbered'. Its own namespace mapping
is unchanged. The object 'virgin' is unchanged. So the caller sees no
change as a result of calling the function.
Now, to demonstrate that this model of namespaces-as-mappings (and
first level objects themselves) is more useful to the Python programmer
than the accurate but too-low-level description in terms of variables,
consider this:
>>> def test(x):
... x = 'clobbered'
... return locals()
>>> y = 'virgin'
>>> ns = test(y)
>>> print y
virgin
>>> print ns['x']
clobbered
In other words, while thinking about 'variables' and 'pass by value' is
_accurate_, it does not add anything to the conceptual model a programmer
needs in order to understand and write Python code. Namespaces as first
level objects that map names to values, however, is an extremely useful
and powerful mental model for programming in Python. The two concepts are
conceptually equivalent regardless of implementation details (assuming we
say a namespace immutably associates names with variables, which in turn
mutably point to values), but the namespace, IMO, is the more useful of
the two when reasoning about Python programs.
> So much for call-by-value. How about call-by-reference? Well,
> everything is the same until the actual call. But then everything
> changes.
>
> Firstly, call-by-reference /doesn't/ evaluate the argument expressions.
> Instead, we just note that `y' is bound to a particular variable in the
> global environment. The function has a single parameter `x'. A new
> environment is constructed by extending the global environment (again):
> in this new environment, the name `x' is bound to /the same variable/
> that `y' is bound to in the global environment.
>
> y (in global env.) ====> [VAR] <===== x (in function `test')
> |
> |
> v
> 'virgin'
And this is something that you can _never_ do in Python (short of
mindbogglingly bad hackery). I know you know that, but the fact that
you _cannot_ do it is why people start getting confused when talking
about call-by-reference vs call-by-value in a Python context.
> Now we assign to `x'. The detailed rules are the same: `x' is bound, so
> we modify the variable it's bound to, so that it stores 'clobbered'.
> But this time, the variable being clobbered is the /same/ variable that
> `y' is bound to. So finally, when we print `y', we see the string
> 'clobbered.
>
>
> y (in global env.) ====> [VAR] <===== x (in function `test')
> |
> |
> v
> 'clobbered'
>
> Now, let's look at your example.
>
> > def func(bar):
> > bar.pop()
> >
> > foo = ['Ethan','Furman']
> > func(foo) # bar = foo
>
> I won't describe this in the same excruciating detail as I did for the
> one above; but I will make some observations.
>
> In call-by-reference, the environment in which the body of `func' is
> executed is constructed by extending the global environment with a
> binding of the name `bar' to the same variable as is bound to `foo' in
> the global environment. There is only the one variable, so obviously
> both names (considered as expressions) must evaluate to the same value.
> I won't go into the details of method invocation, which in Python is
> quite complicated; but `bar.pop()' mutates this value: it modifies it in
> place. So, when the function returns, `foo' can be seen to print
> differently. The value is, in some sense, the /same/ value as it was
> before, in that it occupies the same storage locations, but the contents
> of those storage locations has been altered.
>
> foo =====> [VAR] <===== bar
> (in global env) | (in function `func')
> |
> v
> ['Ethan', 'Furman']
And just to be crystal clear, the above case is _not_ the one Python does.
> In call-by-value, `func' executes in an environment constructed by
> extending the global environment with a binding of `bar' to a /fresh/
> variable. This fresh variable is then initialized: we store the value
> of the argument expression `foo' into it. This value is a reference to
> the list ['Ethan', 'Furman']. (See, I've stopped parenthesizing the
> reference stuff, because here it really matters.) So, we have two
> variables, bound to `foo' and `bar', but both variables refer to the
> same list object. The body of `func' doesn't modify any variables;
> rather, it mutates the list object in place (as above). Since both
> variables refer to the /same/ list object, this mutation is still
> observable outside of the function.
>
> foo =====> [VAR] ---> ['Ethan', 'Furman']
> (in global env) ^
> |
> bar =====> [VAR'] ------------'
> (in function `func')
And for reasoning about Python programs, I would write this diagram
(which _is_ the way Python does it) as:
foo ---> ['Ethan', 'Furman']
(in global namespace) ^
|
bar ------------'
(in 'func' namespace)
> So the reason that your example doesn't distinguish the two cases is
> because, in both cases, there's still only one value, which is mutated.
> But there is a conceptual difference, unobservable in this instance,
Unobservable in Python because Python gives the programmer no way to
create two pointers to the same '[VAR]'.
> because call-by-reference has two names bound to the same variable,
> while call-by-value has two names, bound to /distinct/ variables but
> which both refer to the same value.
>
> Finally, it may be instructive to remove the issue of function calling
> altogether. Consider:
>
> foo = ['Ethan', 'Furman']
> bar = foo
> bar.pop()
>
> What is the final value of `foo'?
>
> Now:
>
> x = 'virgin'
> y = x
> y = 'clobbered'
>
> What is the final value of `x'?
>
> Are you enlightened?
Hopefully so! Your explanation of the underlying mechanics of call by
reference and call by value is the best I've ever seen, and all that
effort you put in to defining your terms earlier in this thread has
ultimately paid off.
Although this has been long enough already (especially since I quoted
your excellent article in full), I still want to go on a bit more.
I want to demonstrate the explanatory power of the namespace model,
which was one of your arguments with Steven: that his model of Python
working directly with values couldn't explain certain things.
On of the examples you gave (and I'm doing this from memory) was this:
>>> l = [ lambda: x for x in [1,2,3] ]
>>> [ f() for f in l ]
[3, 3, 3]
Your explanation was in terms of variables and environments, and frankly I
had a hard time following it (though I get it now). Here's how I explain
it in terms of namespaces: lambda creates a closure, and part of the
closure is a reference to the locals namespace object that exists when the
closure is formed. The (top level of the) list comprehension is operating
in the local namespace, so it is this namespace that gets captured inside
the closure. When we execute the functions later, what happens is that
'x' is looked up in this namespace, and the object that _it maps to at
that time_ is returned. Since the last assignment (mapping) to X was
to the object '3', '3' is what is returned by all three functions calls.
The code above is equivalent to the following code, which makes is
clearer what is happening:
>>> l2 = []
>>> for x in 1,2,3:
... l2.append(locals())
>>> [ ns['x'] for ns in l2 ]
[3, 3, 3]
So in terms of namespaces it is crystal clear why we get three 3s,
and why the following produces three 5s:
>>> x = 5
>>> [ ns['x'] for ns in l2 ]
[5, 5, 5]
>>> [ f() for f in l ]
[5, 5, 5]
This is conceptually equivalent to your variables and environments model,
but I think it is easier to understand and reason about, when talking
only about Python code, because it elides that level of indirection to
which the Python programmer has no access. At least, I find it so!
To take the final step toward unifying your view with Steven's view,
we have to admit that Python objects that reference other objects can be
viewed either as having [VAR] elements containing pointers to objects,
or as namespaces referencing other objects, and that these two views
are conceptually equivalent.
Thus, take your circular list example:
>>> a [1, 2, 3]
>>> a[1] = a
>>> a
[1, [...], 3]
Your view was that this list must contain a pointer variable which points
to the list itself (a reference), while Steven held that Python lists
reference other objects directly, and had no problem with this Tardis-like
construct. You are obviously correct that what a list is is a sequence of
variable slots that can point to any object, including the list itself.
But we can also view the list as a namespace whose name elements are a
zero-origined, monotonically increasing set of integers, and that within
this name space these names map directly to python objects.
In Python (or at least in CPython), the two views are equivalent. But I
argue, with Steven, that the latter view is more _useful_ for a Python
programmer, and that it is in fact Python's conceptual model.
I would also argue that the latter view is the one that it is more
helpful to explain to newcomers to Python, with due acknowledgement
being made to the reality of the behind-the-scenes variables for those
coming from a background where that will be helpful. The trick is to
get the long-time Python programmers to remember, when talking to such
a newcomer, that those variables even exist!
--RDM
More information about the Python-list
mailing list