Some examples for what I have in mind: def read_and_close_file(file: Move[FileIO]): """ File will be closed, so we can not use after """ file.read() file.close() @dataclass class Model: x: List[int] def make_list_from_model(model: Move[Model]) -> List[int]: """ This is not a pure function, which is bad, but if we move model here and won't use it after, then it's kinda pure """ model.x.append(1) # x is big and we don't want to copy it, instead we just push another element return model.x def run_thread_unsafe_process(writer: Move[BytesIO]) -> Thread: thread = Thread(target=thread_unsafe_process, args=(writer,)) thread.start() return thread def thread_unsafe_process(writer: BytesIO): while True: sleep(1) writer.write(b'Hello!') file = open('move.py') read_and_close_file(file) file.read() # Mistake, file is already closed, static analyzer cah check that model = Model([1, 2, 3]) y = make_list_from_model(model) print(model.x) # Using of corrupted value, it might be unexpected for caller and cause some mistakes writer = open('buffer.tmp', 'wb') run_thread_unsafe_process(writer) writer.write(b'world') # mistake, can lead to data-race (in this case, two messages, "hello" and "world", might mix up)

All, Most of rust is based around the idea of move semantics. It's really useful in cases wherever you want the object to be mutable, want to avoid side effects from something else still having a reference unintentionally, and avoid a lot of deep copies of the underlying resources. On the project I work on (rubin observatory) we created something that lets us somewhat emulate this in python and it has been useful for debugging and some safety. We used it on a method-chaining interface on query objects; it comes up when you have a system where you want to convert one complex object into a slightly different complex object, and you could do that more efficiently when you know the original doesn't need to exist anymore - e.g. by taking an internal list and appending to it, rather than copying the whole thing and then appending. On Thu, Nov 26, 2020, 12:35 PM William Pickard <lollol222gg@gmail.com> wrote:

C++ has official move syntax via the R-Value Reference (&&). C#/.NET is the only other language I know of where it can be emulated (via Constructors/Assignment operators).

I'm not fluent in Perl and Ruby to try to guess if they can support it or not. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/BATDZV... Code of Conduct: http://python.org/psf/codeofconduct/

Yeah, in most places we just use python as you described. In a few localized places it turned out to be useful for detecting issues and working with external resources that were exposed through python objects. I would not say if this proposal were included with typing that it should be taught as common practice. There are certainly other ways to do this as well, but sometimes it would be convenient. On Thu, Nov 26, 2020, 1:08 PM Antoine Pitrou <solipsis@pitrou.net> wrote:

Why are you trying to replicate move semantics in Python? The Python ownership model is entirely different from C++. In C++ terms, every Python object is like a shared_ptr<> (but with additional support for tracking and collecting reference cycles).

Generally, when a Python function wants to "take ownership" of a mutable object (say, to mutate it without the caller being surprised), it makes a copy of the object (e.g. dict.copy()).

Regards

Antoine.

On Thu, 26 Nov 2020 17:34:52 -0000 "William Pickard" <lollol222gg@gmail.com> wrote:

...

C++ has official move syntax via the R-Value Reference (&&). C#/.NET is the only other language I know of where it can be emulated (via Constructors/Assignment operators).

I'm not fluent in Perl and Ruby to try to guess if they can support it or not. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/BATDZV... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7GHJS3... Code of Conduct: http://python.org/psf/codeofconduct/

I agree that is how python is implemented, and from a memory perspective, this is not needed. What I was talking about was more from a conceptual resource point of view, and not just memory as a resource. Consider a generator, you can have many different bindings to the object, but when it is exhausted, it may not make sense to access that that generator though one of the other bindings. Of course accessing it can raise an exception, or you can have objects you can ask of the resources are still "valid" for whatever the resource is (in this case a generator), rebind a name after a function call with additional checks, etc. It would be convenient if you could mark in source code that you intended a resource to be "moved" and any further access through other bindings are not what was intended. This would help catch logic bugs during type checking, instead of hitting this issue at runtime. On Thu, Nov 26, 2020, 1:42 PM Richard Damon <Richard@damon-family.org> wrote:

On 11/26/20 12:07 PM, Guido van Rossum wrote:

...

This reminds me of something in C++. Does it exist in other languages?

A key point is that C++ needs 'move' behavior as its variables are buckets of bits, and assigning one variable to another, like in a call, requires copying that whole bucket of bits, and if it refers to other buckets of bits, may need to copy them too.

One thing that can be done when making this assignment, if the original isn't going to be needed is to 'steal' those buckets that the first object pointed to, possibly saving a lot of work.

Python doesn't need to do this, as it isn't based on a bucket of bits type model, but its names are just bound to objects, and can readily share.

Another way to think of the issue is it is mostly an issue with using Call By Value, which is the basic way to pass parameters in C and C++ (there is also a 'Call By Reverence' which is really just a Call by Value, where the value is a pointer, maybe with syntactic sugar to hide the pointerness)

Python doesn't use Call by Value, but its method is better described as Call by Binding, the parameters in the function are bound to the pbjects specified in the call. If those objects are mutable, the function can change the object that the caller gave, but can't 'rebind' any variable names in the call statement to new objects.

-- Richard Damon _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5VKCC6... Code of Conduct: http://python.org/psf/codeofconduct/

Steve, I take your point, and like indicated in another part of this thread, for most parts of our stack its standard practices. It is only when managing certain external resources that we take a bit of extra care, and that is in a library that is not outwardly exposed. I agree with you also that I don't even know if this is a thing that is possible in python unless the type checker could assume that a caller was meaning to move a variable when they called a function declared with move. And of course this would still be at the type checking level, and not at execution time. For our use case, we put a move indicator at the point the function is called, and not in the signature of the function itself (because of what you indicated above), signaling the intention was to move it. It's not really much different than f(x); del x (though of course there are details). And so python does provide everything needed already to ensure one logical binding (and of course there are N other solutions people can think of, guards, exceptions, etc). My point was more that if there was a way I could indicate to a type checker that it may be unsafe for anything to use a reference to this object after this function call it would be helpful(as I might have forgotten behavior of functions, or simply not been paying attention). On Thu, Nov 26, 2020 at 3:53 PM Steven D'Aprano <steve@pearwood.info> wrote:

On Thu, Nov 26, 2020 at 02:15:50PM -0500, nate lust wrote:

...

It would be convenient if you could mark in source code that you intended a resource to be "moved" and any further access through other bindings are not what was intended. This would help catch logic bugs during type checking, instead of hitting this issue at runtime.

I can't help but feel that if your code relies on the concept of "moving" an object, your concept is already at odds with Python's object and execution model.

But putting that aside, is this is even technically possible? Can type checkers track the "owner" of a resource in this way?

obj = {} # obj is owned by this code block process(obj) # still owned by this block print(obj) # so this is okay capture(obj) # but now ownership is stolen by capture() print(obj) # and this is a type-error (?!)

Maybe I'm just not familiar enough with the type-checking state of art, but I would be amazed if this was possible.

By the way, I have intentionally used the term "stolen" rather than "taken" or "transfered to" with respect to moving ownership of `obj`. At the caller's site, there is no obvious difference between the calls to process() which doesn't move obj, and capture() which does.

Any innocuous-looking function could "move" ownership, with *or without* the knowledge of the caller. It's true that the Python interpreter itself has no concept of "ownership" in this sense, but to the degree that the coder cares about such ownership, having it moved without their knowledge and consent is surely "theft" :-)

-- Steve _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/MNWGVX... Code of Conduct: http://python.org/psf/codeofconduct/

-- Nate Lust, PhD. Astrophysics Dept. Princeton University

26.11.20 19:07, Guido van Rossum пише:

This reminds me of something in C++. Does it exist in other languages?

Indeed, this is a term from C++. In older C++ you could pass argument by value, which makes a copy, and by reference, which is equivalent to passing a pointer by value plus syntax sugar. Passing by value may be inefficient if you need to copy more than several bytes. Passing by reference has other problems. It was a source of errors, so passing object created on stack by non-constant reference was forbidden. If you need to modify argument passed by constant reference in the function you need to create a copy, even if the object is never used after passing to the function (like in foo(bar()), the returned value of bar() cannot be used outside of foo(), so it could be allow to modify it). To resolve this problem the move semantic was introduced.

Do you have a more realistic example where this would catch an important type error?

I seen and used the following example (with variations) multiple times: class Client: def send(self, msg, token, headers=None): if headers is None: headers = {} # headers = headers.copy() headers["Authorization"] = f"Bearer {token}" headers["Content-Type"] = "text/plain" headers["X-Hash"] = md5sum(msg.encode()) with self.request('POST', self.url, headers=headers, data=msg) as resp: ... client.send('Hello!', mytoken, headers={'X-Type': 'Spam'}) The function sets some standard headers, but you can pass also additional headers. If you only pass a just created dictionary, the function can modify it in-place because its value is not used outside. But if you create the dict once and use it in other places, modifying it inside the function is a non-desired effect. You need to make a copy in the function just because you don't know how it will be used in the user code. If you document that the argument cannot be used after calling the function and add some marks for static type checking, you could avoid copying. Personally I think Python has a little need in such feature. Copied dicts and lists are usually small, and it adds not much overhead in comparison with other overhead that Python does have. And if you need to pass and modify a large collection, documenting this should be enough.

On 2020-11-27 at 10:38:06 +0200, Serhiy Storchaka <storchaka@gmail.com> wrote:

26.11.20 19:07, Guido van Rossum пише:

...

This reminds me of something in C++. Does it exist in other languages?

Indeed, this is a term from C++. In older C++ you could pass argument by value, which makes a copy, and by reference, which is equivalent to passing a pointer by value plus syntax sugar. Passing by value may be inefficient if you need to copy more than several bytes. Passing by reference has other problems. It was a source of errors, so passing object created on stack by non-constant reference was forbidden. If you need to modify argument passed by constant reference in the function you need to create a copy, even if the object is never used after passing to the function (like in foo(bar()), the returned value of bar() cannot be used outside of foo(), so it could be allow to modify it). To resolve this problem the move semantic was introduced.

...

Do you have a more realistic example where this would catch an important type error?

I seen and used the following example (with variations) multiple times:

class Client: def send(self, msg, token, headers=None): if headers is None: headers = {} # headers = headers.copy() headers["Authorization"] = f"Bearer {token}" headers["Content-Type"] = "text/plain" headers["X-Hash"] = md5sum(msg.encode()) with self.request('POST', self.url, headers=headers, data=msg) as resp: ...

client.send('Hello!', mytoken, headers={'X-Type': 'Spam'})

The function sets some standard headers, but you can pass also additional headers. If you only pass a just created dictionary, the function can modify it in-place because its value is not used outside. But if you create the dict once and use it in other places, modifying it inside the function is a non-desired effect. You need to make a copy in the function just because you don't know how it will be used in the user code. If you document that the argument cannot be used after calling the function and add some marks for static type checking, you could avoid copying.

Personally I think Python has a little need in such feature. Copied dicts and lists are usually small, and it adds not much overhead in comparison with other overhead that Python does have. And if you need to pass and modify a large collection, documenting this should be enough.

For an opposing point of view, why not write that method as follows: class Client: def send(self, msg, token, additional_headers=None): headers = {"Authorization" : f"Bearer {token}", "Content-Type" : "text/plain", "X-Hash" : md5sum(msg.encode()), **(additional_headers or {})} ... Then I only modify a just created dictionary and never some dictionary of questionable origin and/or future. Obviously, there's a difference in behavior if additional_headers contains Authorization or X-Hash or Content-Type, but I can't say which behavior is correct without more context, and I can't say which is more efficient (in time or in space) without knowing how often additional_headers is not None or how big it usually is. I come from old(er) school (1980s, 1990s) embedded systems, and who "owns" a particular mutable data structure and how/where it gets mutated always came up long before we wrote any code. No, I'm not claiming that pre-ansi C and assembler are more productive or less runtime error prone than newer languages, but is this feature only necessary because "modern" software development no longer includes a design phase or adequate documentation? I understand the advantages of automated systems (the compiler and/or static analyzer) over manual ones (code review, documentation, writing unit tests), but ISTM that if you're writing code and don't know whether or not you can mutate a given dictionary, or you're calling a function and don't know whether or not that function might mutate that dictionary, then the battle is already lost. Memory management implementation details is a long way from executable pseudo code. (30 years is a long time, too.)

On 11/27/20 8:32 AM, Antoine Pitrou wrote:

On Fri, 27 Nov 2020 07:32:17 -0500 2QdxY4RzWzUUiLuE@potatochowder.com wrote:

...

I come from old(er) school (1980s, 1990s) embedded systems, and who "owns" a particular mutable data structure and how/where it gets mutated always came up long before we wrote any code. No, I'm not claiming that pre-ansi C and assembler are more productive or less runtime error prone than newer languages, but is this feature only necessary because "modern" software development no longer includes a design phase or adequate documentation? "Modern" software development is just like older software development in that regard: sometimes it includes a design phase and/or adequate (i.e. sufficiently precise) documentation, sometimes it doesn't.

...

Memory management implementation details is a long way from executable pseudo code. (30 years is a long time, too.) This isn't really about memory management, though.

Maybe it would help to clarify what it *is* about.  The original proposal makes no mention of the problem being solved. --Ned.

On 2020-11-27 at 14:32:11 +0100, Antoine Pitrou <solipsis@pitrou.net> wrote:

On Fri, 27 Nov 2020 07:32:17 -0500 2QdxY4RzWzUUiLuE@potatochowder.com wrote:

...

I come from old(er) school (1980s, 1990s) embedded systems, and who "owns" a particular mutable data structure and how/where it gets mutated always came up long before we wrote any code. No, I'm not claiming that pre-ansi C and assembler are more productive or less runtime error prone than newer languages, but is this feature only necessary because "modern" software development no longer includes a design phase or adequate documentation?

"Modern" software development is just like older software development in that regard: sometimes it includes a design phase and/or adequate (i.e. sufficiently precise) documentation, sometimes it doesn't.

Fair enough. :-) (In my case, memory was a scarce resource and we emphasized documentation about it.)

...

Memory management implementation details is a long way from executable pseudo code. (30 years is a long time, too.)

This isn't really about memory management, though.

It reminds me of "use after free" errors. Whenever Rust comes up, I think of memory management. Call me biased. :-) Also, in Serhiy's example,¹ if Client.send "moved" headers, then it could also free the memory associated with that dictionary at the end (although I realize that that may not always be the case). Many times, freeing memory sooner is better. ¹ https://mail.python.org/archives/list/python-ideas@python.org/message/VBB3XG...

On Thu, Nov 26, 2020 at 1:51 PM Bruce Leban <bruce@leban.us> wrote:

What might be useful are declarations that:

- The object is not modified (i.e., read only). If a function that declares a parameter as read-only passes that parameter to one that does not use that declaration, that can be identified as an error. - The object is destroyed or consumed by the function. Any use of the object after calling the function could be identified as an error.

I don't think Python is ready for const arguments (the first bullet). "Const propagation" can be a real problem. The second seems to be what the OP is after, and I now can see the use case -- in fact in libraries for scientific computing and/or machine learning I think Iv'e seen are a bunch of hacks to imply transfer of ownership that might benefit from this. But I'm not yet sufficiently familiar with that field to be able to point you to examples. Hopefully there are readers here who can. (Nate?) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

On 2020-11-27 01:12, Ned Batchelder wrote:

On 11/26/20 6:44 AM, 3mir33@gmail.com wrote:

...

Add something like Move type hint to typing module. It will tell the analyzer that the input parameter of the function is moved and can not be used after. For example: ``` def f(d: Move[dict]) -> dict: d['a'] = 2 return d

d = {1: 2} f(d) print(d[1]) # mistake, using of moved value

Maybe I'm behind the times on the latest thinking in programming languages.  I'm not familiar with "move semantics". Can someone explain what is wrong with the code above?  What is the mistake? `print(d[1])` will work perfectly fine.  The function changes the 'a' key, and then we access the 1 key. Should the example have used the same key in both places?

d is moved into function f. f does return d when it exits, moving it out back again, but as the caller doesn't bind to it, it's discarded. Move semantics is a way of ensuring that there's only one "owner", which makes garbage collection simpler; when an object is discarded (dereferenced) by its the owner, it can be garbage-collected.

On 2020-11-27 02:30, Ned Batchelder wrote:

On 11/26/20 9:02 PM, MRAB wrote:

...

On 2020-11-27 01:12, Ned Batchelder wrote:

...

On 11/26/20 6:44 AM, 3mir33@gmail.com wrote:

...

Add something like Move type hint to typing module. It will tell the analyzer that the input parameter of the function is moved and can not be used after. For example: ``` def f(d: Move[dict]) -> dict:      d['a'] = 2      return d

d = {1: 2} f(d) print(d[1])  # mistake, using of moved value

Maybe I'm behind the times on the latest thinking in programming languages.  I'm not familiar with "move semantics". Can someone explain what is wrong with the code above?  What is the mistake? `print(d[1])` will work perfectly fine.  The function changes the 'a' key, and then we access the 1 key. Should the example have used the same key in both places?

d is moved into function f. f does return d when it exits, moving it out back again, but as the caller doesn't bind to it, it's discarded.

It's not discarded, it's still referenced by d in the outer scope.

No, it's not any more, and that's the point. It was _moved_ into the function, and although the function returned it, it was discarded because the caller didn't bind it to keep hold of it.

On Thu, Nov 26, 2020 at 7:45 PM MRAB <python@mrabarnett.plus.com> wrote:

...

It's not discarded, it's still referenced by d in the outer scope.

No, it's not any more, and that's the point. It was _moved_ into the function, and although the function returned it, it was discarded because the caller didn't bind it to keep hold of it.

Sounds like one of you is describing current semantics and the other is explaining the proposed new semantics. :-) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

On 11/26/20 11:45 PM, Guido van Rossum wrote: Yes, I see that now.  As Chris points out elsewhere in the thread, this proposal would have the type annotations change the actual behavior of the code. No, it will not change the runtime behavior. But it can make a restriction which was in the documentation only to be enforced by the

27.11.20 13:25, Ned Batchelder пише: linter. If it is documented, that you should never use a dict after passing it as argument to f(), the code that uses it has a bug. With the proposed feature MyPy could warn you about this bug. It is discussable how much useful this feature is. Python is not C++, and some microoptimizations important in C++ are not worth in Python.

On Fri, Nov 27, 2020 at 2:46 PM MRAB <python@mrabarnett.plus.com> wrote:

On 2020-11-27 02:30, Ned Batchelder wrote:

...

On 11/26/20 9:02 PM, MRAB wrote:

...

On 2020-11-27 01:12, Ned Batchelder wrote:

...

On 11/26/20 6:44 AM, 3mir33@gmail.com wrote:

...

Add something like Move type hint to typing module. It will tell the analyzer that the input parameter of the function is moved and can not be used after. For example: ``` def f(d: Move[dict]) -> dict: d['a'] = 2 return d

d = {1: 2} f(d) print(d[1]) # mistake, using of moved value

Maybe I'm behind the times on the latest thinking in programming languages. I'm not familiar with "move semantics". Can someone explain what is wrong with the code above? What is the mistake? `print(d[1])` will work perfectly fine. The function changes the 'a' key, and then we access the 1 key. Should the example have used the same key in both places?

d is moved into function f. f does return d when it exits, moving it out back again, but as the caller doesn't bind to it, it's discarded.

It's not discarded, it's still referenced by d in the outer scope.

No, it's not any more, and that's the point. It was _moved_ into the function, and although the function returned it, it was discarded because the caller didn't bind it to keep hold of it.

Currently, type hints and annotations don't change actual behaviour like this. Is this proposal to have it actually get discarded, or merely to have a type checker flag a warning if you continue to use it? I strongly disagree with the former. When you call a function, it doesn't matter what the function is, you just call it. It's the function's responsibility to do anything special. That means there can't be an unbind operation in the caller on the basis of a flag on the function. The latter is also more closely aligned with the way that Python currently handles "oops you used this after it should have been discarded". Consider:

...

...

d = {1:2, 3:4} i = iter(d) next(i) 1 del d[1] next(i) Traceback (most recent call last): File "<stdin>", line 1, in <module> RuntimeError: dictionary changed size during iteration i <dict_keyiterator object at 0x7f28c19244f0>

The iterator still exists, but has been internally flagged as "hey, this thing is broken now, don't use it". If that's the meaning of "move semantics" (imagine that "del d[1]" is replaced with some function that is listed as 'consuming' i), then that does at least make reasonable sense. ChrisA

Your example is specific to C++ only. In Rust it is not just an optimization, it's about ownership. Example from Rust: fn f1(vec: Vec<i32>) {} fn f2() { let mut vec = vec![1]; f1(vec); vec.push(2); // This will not compile (borrow of moved value), because f1 took ownership of vec }