[Tutor] List comprehension + lambdas - strange behaviour

[Steven D'Aprano]

On Fri, 7 May 2010 05:11:38 pm spir ☣ wrote:
> On Thu, 6 May 2010 22:15:34 +0100
>
> "Alan Gauld" <alan.gauld at btinternet.com> wrote:
> > As others have pointed out you are returning a reference not a
> > value.

Others have said that, but it's not true. Python does not 
have "references". The Python implementation under the hood uses 
references all the time, but they aren't visible to Python code. In 
Python you have names, and objects, and nothing else.

The classic test of languages with references is, can you write a 
function that swaps two variables? That is, something like this:

>>> x = 1
>>> y = 2
>>> swap(x, y)
>>> print x, y
2, 1

But you can't do this in Python. This proves beyond all doubt that 
Python code does not have references.



> Yes. (I have said that, too.) But still there is a mystery for me.
> Better explained byt the following:
>
> x = 0 ; print id(x)	# an address

No, id(x) is an ID number, not an address. It happens to be that for 
CPython, the address of the object in the underlying implementation is 
used as the ID number, but that's an accident of implementation. They 
could have very easily decided to add 7 to the address, or multiply it 
by 3. Jython uses a completely different scheme, where ID numbers have 
nothing to do with addresses. They go 1, 2, 3, 4, ...

Python guarantees that no two objects will have the same ID number *at 
the same time*, but it makes no other promises. CPython, for example, 
re-uses ID numbers. Jython probably doesn't.


> def f() : print x	# 0
> x = 1 ; print id(x)	# another one
> f()			# 1
>
> This shows, I guess, that the reference of the upvalue x is *not* an
> address. But the key (maybe the name itself ?) used by python to 
> lookup a symbol's value, in a given scope, at runtime. Indeed f must
> find its upvalue in the global scope. 

Yes. The *name* "x" is looked up in the global scope at runtime. The 
address of the object bound to x is never used by Python, except that 
CPython uses it as a ID number. 


> Note the scope must also be referenced:
>
> def f():
> 	# not the global scope
> 	x = 0
> 	def g():
> 		print x
> 	x = 1
> 	return g
> # global scope
> f()()			# 1
>
> I guess the above example also shows that upvalues can be
> "finalised", since here the scope is lost xwhen f runs.
>
> Does anyone know if this reasoning is correct, and how this is done?


Python has the concept of names spaces. Names are strings 
like "x", "foo", "math" and so forth. When you refer to a name, Python 
searches each name space in turn:

* the local name space of the current function (if any)
* the name space of each parent function (if any) in turn
* the global name space
* the built-in name space

and then raises NameError if it doesn't find a match at all. Inside a 
class, it's a little different, but essentially follows a similar 
pattern:

* the local name space of the method
* the global name space
* built-ins
* raise NameError

Notice that the class name space is deliberately left out.

When looking up an attribute of an item, it goes something like this:


* if the class has a __getattribute__ method, call it
* the instance __slots__ (if any)
* the instance __dict__ (if it exists)
* the class __dict__
* the __dict__ of any superclasses
* if the class has a __getattr__ method, call it
* raise AttributeError


Regardless of how you look up a name or attribute, once Python finds it, 
it passes the object back to you. Actually, because objects are large 
complicated objects, the Python implementation actually passes some 
sort of internal short-cut to the object, for speed. In CPython, that 
is a pointer. In Jython, it is a reference or safe-pointer. In other 
Python implementations (IronPython, PyPy, etc.) some other mechanism is 
used.

The mechanism isn't important. From your Python code, you have no way of 
telling what the mechanism is, all you see is names ("math") and 
objects (the math module object).

In CPython, we can say some other things about the implementation of 
name spaces. In the global (top level) scope of a module, in the 
interactive interpreter, and inside a class, the name space is a 
dictionary with strings as keys. Inside functions, the name space is a 
bit more complicated: for speed it is turned into a C-level table. 
(Actually, it's a bit more complicated for classes too, with slots and 
other optimizations.) This is why Python classes have a __dict__ 
attribute, and why the dict you get back from globals() is writable, 
but changing the dict you get back from locals() has no effect inside a 
function.

Objects can have any number of names. If you do this:

a = b = c = d = 42
e = c

then you have given the object 42 five distinct names. But if you do 
this:

mylist = []
mylist.append(20 + 3)

the object 23 is created and stored inside a list, but it has no name. 
It is an anonymous object, but you can still refer to it:

mylist[0]  # gives 23

Some operations do a re-binding, that it, they associate a new object 
with a name:

x = "something special"

def f(): pass

class K: pass

create a new binding between an object and the names x, f and K. The 
command del x will remove the *name* x from the name space. What 
happens next varies -- if the string used somewhere else, then nothing 
happens. But if the string is no longer used, then Python will 
automatically delete the object and reuse its memory. This is why 
destructors (__del__ methods) don't necessarily run when you expect 
them to.

>>> class Example:
...     def __del__(self):
...             print "Goodbye!"
...
>>> x = Example()
>>> mylist = [x]
>>> del x
>>> mylist[0] = None
Goodbye!



Hope this helps,



-- 
Steven D'Aprano