To clarify how Python handles two equal objects

Fri Jan 13 21:56:48 EST 2023

Jen,

Can a compiler, or spot compiler, always know if something that was a reference is changed?

Obvious examples may be if a change happens in non-deterministic ways such as in a fork chosen at random or from user input but also sometimes levels of indirection such as deleting an object that internally contains a reference the other, perhaps even more indirectly.

I know programmers often have their code overhauled and their assumptions go away such as someone deciding to create a variable name in an inner scope and thus hiding the same variable they were using in an outer scope. So even if your code is currently valid, after it changes, a later compiler if it detected some change might not want to do your speedup. 

I think the subject line of the message we keep exchanging is now a bit misleading. It is not about two objects nor really about how python handles them. There seem to be one object and possibly multiple views of it and you may not want to pass the entire object around or manipulate it a certain way. I am not so certain your methods necessarily speed things up as certain views simply do calculations on the many places they need to read or change to supply what you want.

From: Jen Kris <jenkris at tutanota.com> 
Sent: Friday, January 13, 2023 10:58 AM
To: avi.e.gross at gmail.com
Cc: python-list at python.org
Subject: RE: To clarify how Python handles two equal objects

Avi,

Thanks for your comments.  You make a good point. 

Going back to my original question, and using your slice() example: 

middle_by_two = slice(5, 10, 2)

nums = [n for n in range(12)]

q = nums[middle_by_two]

x = id(q)

b = q

y = id(b)

If I assign "b" to "q", then x and y match – they point to the same memory until "b" OR "q" are  reassigned to something else.  If "q" changes during the lifetime of "b" then it’s not safe to use the pointer to "q" for "b", as in:

nums = [n for n in range(2, 14)]

q = nums[middle_by_two]

x = id(q)

y = id(b)

Now "x" and "y" are different, as we would expect.  So when writing a spot speed up in a compiled language, you can see in the Python source if either is reassigned, so you’ll know how to handle it.  The motivation behind my question was that in a compiled extension it’s faster to borrow a pointer than to move an entire array if it’s possible, but special care must be taken. 

Jen

Jan 12, 2023, 20:51 by avi.e.gross at gmail.com <mailto:avi.e.gross at gmail.com> :

Jen,

It is dangerous territory you are treading as there are times all or parts of objects are copied, or changed in place or the method you use to make a view is not doing quite what you want.

As an example, you can create a named slice such as:

middle_by_two = slice(5, 10, 2)

The above is not in any sense pointing at anything yet. But given a long enough list or other such objects, it will take items (starting at index 0) starting with item that are at indices 5 then 7 then 9 as in this:

nums = [n for n in range(12)]

nums[middle_by_two]

[5, 7, 9]

The same slice will work on anything else:

list('abcdefghijklmnopqrstuvwxyz')[middle_by_two]

['f', 'h', 'j']

So although you may think the slice is bound to something, it is not. It is an object that only later is briefly connected to whatever you want to apply it to.

If I later change nums, above, like this:

nums = [-3, -2, -1] + nums

nums

[-3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]

nums[middle_by_two]

[2, 4, 6]

In the example, you can forget about whether we are talking about pointers directly or indirectly or variable names and so on. Your "view" remains valid ONLY as long as you do not change either the slice or the underlying object you are applying to -- at least not the items you want to extract.

Since my example inserted three new items at the start using negative numbers for illustration, you would need to adjust the slice by making a new slice designed to fit your new data. The example below created an adjusted slice that adds 3 to the start and stop settings of the previous slice while copying the step value and then it works on the elongated object:

middle_by_two_adj = slice(middle_by_two.start + 3, middle_by_two.stop + 3, middle_by_two.step)

nums[middle_by_two_adj]

[5, 7, 9]

A suggestion is that whenever you are not absolutely sure that the contents of some data structure might change without your participation, then don't depend on various kinds of aliases to keep the contents synchronized. Make a copy, perhaps a deep copy and make sure the only thing ever changing it is your code and later, if needed, copy the result back to any other data structure. Of course, if anything else is accessing the result in the original in between, it won't work.

Just FYI, a similar analysis applies to uses of the numpy and pandas and other modules if you get some kind of object holding indices to a series such as integers or Booleans and then later try using it after the number of items or rows or columns have changed. Your indices no longer match.

Avi

-----Original Message-----

From: Python-list <python-list-bounces+avi.e.gross=gmail.com at python.org <mailto:python-list-bounces+avi.e.gross=gmail.com at python.org> > On Behalf Of Jen Kris via Python-list

Sent: Wednesday, January 11, 2023 1:29 PM

To: Roel Schroeven <roel at roelschroeven.net <mailto:roel at roelschroeven.net> >

Cc: python-list at python.org <mailto:python-list at python.org> 

Subject: Re: To clarify how Python handles two equal objects

Thanks for your comments. After all, I asked for clarity so it’s not pedantic to be precise, and you’re helping to clarify. 

Going back to my original post,

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]

arr1 = mx1[2]

Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer). Similarly, if I write "mx1[2][1] += 5" then again both names will be updated. 

That’s what I meant by "an operation on one is an operation on the other." To be more precise, an operation on one name will be reflected in the other name. The difference is in the names, not the pointers. Each name has the same pointer in my example, but operations can be done in Python using either name. 

Jan 11, 2023, 09:13 by roel at roelschroeven.net <mailto:roel at roelschroeven.net> :

Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list:

Yes, I did understand that. In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they’re the same memory block).

Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they’re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two objects, and "an operation on one" and "an operation on the other" make it sound like there are two operations.

Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise.

So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that change will of course be visible through all of them.

Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments.

The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off.

-- 

"Peace cannot be kept by force. It can only be achieved through understanding."

-- Albert Einstein

-- 

https://mail.python.org/mailman/listinfo/python-list

-- 

https://mail.python.org/mailman/listinfo/python-list