I am not sure if this has been suggested before, so my apologies if it has. I would like to propose adding lazy types for casting builtins in a lazy fashion. e.g. `lazy_tuple` which creates a reference to the source iterable and a morally immutable sequence but only populates the tupular container when it or the source is used. Note that if the original object is not a built-in, will not be used after casting _or_ is immutable, this is trivial to implement. The difficulty arises when this it not the case, since the lazy object needs to also freeze any values that are mutated in the original to avoid side-effects. An alternative to adding lazy types / lazy type casting (making it possible to implement these oneself) would be to add method call hooks to python, since this would allow having a "freeze value" callback hooked into the __setitem__ and __getitem__ methods. Hooks may be a more useful solution for wider use cases as well.
Hello, On Tue, 08 Dec 2020 11:46:59 -0000 "Mathew Elman" <mathew.elman@ocado.com> wrote:
You pinpointed the problem exactly. So, the only reliable way to create "lazy" version is to store a copy *eagerly*, which makes the point moot. Lazy-evaluation languages usually deal with that problem by disallowing mutation in the first place (i.e. being purely functional). You can use that solution in Python too. (Without "no mutation" error checking, but it should be possible to implement the alternative API which is "immutable", and I bet someone even did that (PoC-style of course).)
[] -- Best regards, Paul mailto:pmiscml@gmail.com
On Tue, Dec 08, 2020 at 11:46:59AM -0000, Mathew Elman wrote:
What are your use-cases for this? Does this include things like `lazy_list`, `lazy_float`, `lazy_bool`, `lazy_str`, `lazy_bytearray` etc?
Nope, sorry, I don't see how that would work. Here I have a list: L = [50, 40, 30, 20, 10] Suppose these hooks exist. I want to make a "lazy tuple": t = lazy_tuple(L) How do these hooks freeze the list? What if I have more than one lazy object pointing at the same source? s = lazy_str(L) And then follow with a different method call? L.insert(2, "surprise!") I just can't see how this will work. -- Steve
Steven D'Aprano wrote:
I would say yes, it should include these types as well. The use case is for when you are casting between types a high number of times to pass them around, especially into another type and back.
they don't freeze the list, they freeze the values in the lazy_tuple when the list is mutated, so when mutating the list, the values in the tuple are set (if they have not been already), so that they aren't also mutated in the lazy_tuple. Of course for delete and insert this would mean you might have to "freeze" from where the insert/delete occurs to the end of the tuple but that is no different than using a normal tuple i.e. iterate over all elements. However for setitem where only position i is replaced, the tuple can set its value to the original without having to evaluate anything extra.
I don't follow what your point is, sorry. Are you saying that insert is another method that can update a list so delitem and setitem hooks would miss it? The point I was making with hooks is that you _could_ write a lazy class that creates freezing hooks for any mutation method. But the hooks would be up to the coder to implement and hook into the correct mutation methods. I also don't see an issue with multiple lazy types pointing to the same "source". It would just freeze the values in both of them when updating anything in the original.
I agree that if you are using them as iterables, then the type is usually not important because you are treating their type as just iter-able. The lazy iterable would more or less just be the same as passing in `iter(sequence)`. This is for other use cases where the use of the object is specific to the type, e.g. a case where in the outer scope you construct a list and mutate it but when it is passed to an inner scope, you want to enforce that it can be accessed but not mutated. Likewise if lazy built in types were implemented in python then getting a slice of a sequence could also be done lazily, whereas my understanding at the moment is that it has to create a whole new sequence.
That makes sense, so these are kind of 'views' over a sequence, but ones that implement a copy-on-write when the underlying object is modified in any way? I can see this being super hard/impossible to implement reliably, but would be a pretty nice addition if it can be done. On Wed, Dec 9, 2020 at 12:25 PM Mathew Elman <mathew.elman@ocado.com> wrote:
On Wed, 9 Dec 2020 at 09:27, Mathew Elman <mathew.elman@ocado.com> wrote:
If you need this for annotations/typing alone, can't you just use `typing.cast` in the inner scope? (or before calling it for that matter) Anyway, mypy at least wil error if you annotate the inner scope as a "Sequence" (in contrast with MutableSequence), and will error if you try to change the Sequence - and it stlll remain compatible with incoming "MutableSequences". For the cases it does not cover, there is still "cast" - and it feels _a lot_ simpler than having actual runtime lazy objetcs as primitives in the language.
On Wed, Dec 09, 2020 at 12:05:17PM -0000, Mathew Elman wrote:
The use case is for when you are casting between types a high number of times to pass them around, especially into another type and back.
I would say, *don't do that*. If you have a list, and you pass it to another function after converting to a tuple, why would you take that tuple and convert back to a list when you already have a list? For types like casting back and forth between float and int, there will be data loss. Even if there is not, the overhead of creating a "lazy proxy" to the float is probably greater than just converting.
How is this supposed to work in practice? Where does the hook live? When is it called? What does it do? How does the lazy tuple know that the list's setitem has been called? -- Steve
Steven D'Aprano wrote:
How is this supposed to work in practice? A method e.g. `create_hook(method, callback)` would return a concrete reference to the hook, and uses a weak reference to know if it needs to execute the callback (meaning that it would only add the overhead of a check for callbacks if the concrete reference still existed).
It knows because of a hook? This is my whole point about including hooks in this thread, i.e. if it was possible to add a callback on a method, it would execute when it is called. I am not sure I understand your question.
Hello, On Tue, 08 Dec 2020 11:46:59 -0000 "Mathew Elman" <mathew.elman@ocado.com> wrote:
You pinpointed the problem exactly. So, the only reliable way to create "lazy" version is to store a copy *eagerly*, which makes the point moot. Lazy-evaluation languages usually deal with that problem by disallowing mutation in the first place (i.e. being purely functional). You can use that solution in Python too. (Without "no mutation" error checking, but it should be possible to implement the alternative API which is "immutable", and I bet someone even did that (PoC-style of course).)
[] -- Best regards, Paul mailto:pmiscml@gmail.com
On Tue, Dec 08, 2020 at 11:46:59AM -0000, Mathew Elman wrote:
What are your use-cases for this? Does this include things like `lazy_list`, `lazy_float`, `lazy_bool`, `lazy_str`, `lazy_bytearray` etc?
Nope, sorry, I don't see how that would work. Here I have a list: L = [50, 40, 30, 20, 10] Suppose these hooks exist. I want to make a "lazy tuple": t = lazy_tuple(L) How do these hooks freeze the list? What if I have more than one lazy object pointing at the same source? s = lazy_str(L) And then follow with a different method call? L.insert(2, "surprise!") I just can't see how this will work. -- Steve
Steven D'Aprano wrote:
I would say yes, it should include these types as well. The use case is for when you are casting between types a high number of times to pass them around, especially into another type and back.
they don't freeze the list, they freeze the values in the lazy_tuple when the list is mutated, so when mutating the list, the values in the tuple are set (if they have not been already), so that they aren't also mutated in the lazy_tuple. Of course for delete and insert this would mean you might have to "freeze" from where the insert/delete occurs to the end of the tuple but that is no different than using a normal tuple i.e. iterate over all elements. However for setitem where only position i is replaced, the tuple can set its value to the original without having to evaluate anything extra.
I don't follow what your point is, sorry. Are you saying that insert is another method that can update a list so delitem and setitem hooks would miss it? The point I was making with hooks is that you _could_ write a lazy class that creates freezing hooks for any mutation method. But the hooks would be up to the coder to implement and hook into the correct mutation methods. I also don't see an issue with multiple lazy types pointing to the same "source". It would just freeze the values in both of them when updating anything in the original.
I agree that if you are using them as iterables, then the type is usually not important because you are treating their type as just iter-able. The lazy iterable would more or less just be the same as passing in `iter(sequence)`. This is for other use cases where the use of the object is specific to the type, e.g. a case where in the outer scope you construct a list and mutate it but when it is passed to an inner scope, you want to enforce that it can be accessed but not mutated. Likewise if lazy built in types were implemented in python then getting a slice of a sequence could also be done lazily, whereas my understanding at the moment is that it has to create a whole new sequence.
That makes sense, so these are kind of 'views' over a sequence, but ones that implement a copy-on-write when the underlying object is modified in any way? I can see this being super hard/impossible to implement reliably, but would be a pretty nice addition if it can be done. On Wed, Dec 9, 2020 at 12:25 PM Mathew Elman <mathew.elman@ocado.com> wrote:
On Wed, 9 Dec 2020 at 09:27, Mathew Elman <mathew.elman@ocado.com> wrote:
If you need this for annotations/typing alone, can't you just use `typing.cast` in the inner scope? (or before calling it for that matter) Anyway, mypy at least wil error if you annotate the inner scope as a "Sequence" (in contrast with MutableSequence), and will error if you try to change the Sequence - and it stlll remain compatible with incoming "MutableSequences". For the cases it does not cover, there is still "cast" - and it feels _a lot_ simpler than having actual runtime lazy objetcs as primitives in the language.
On Wed, Dec 09, 2020 at 12:05:17PM -0000, Mathew Elman wrote:
The use case is for when you are casting between types a high number of times to pass them around, especially into another type and back.
I would say, *don't do that*. If you have a list, and you pass it to another function after converting to a tuple, why would you take that tuple and convert back to a list when you already have a list? For types like casting back and forth between float and int, there will be data loss. Even if there is not, the overhead of creating a "lazy proxy" to the float is probably greater than just converting.
How is this supposed to work in practice? Where does the hook live? When is it called? What does it do? How does the lazy tuple know that the list's setitem has been called? -- Steve
Steven D'Aprano wrote:
How is this supposed to work in practice? A method e.g. `create_hook(method, callback)` would return a concrete reference to the hook, and uses a weak reference to know if it needs to execute the callback (meaning that it would only add the overhead of a check for callbacks if the concrete reference still existed).
It knows because of a hook? This is my whole point about including hooks in this thread, i.e. if it was possible to add a callback on a method, it would execute when it is called. I am not sure I understand your question.
participants (5)
-
Joao S. O. Bueno -
Mathew Elman -
Paul Sokolovsky -
Stestagg -
Steven D'Aprano