I am not sure if this has been suggested before, so my apologies if it has. I would like to propose adding lazy types for casting builtins in a lazy fashion. e.g. `lazy_tuple` which creates a reference to the source iterable and a morally immutable sequence but only populates the tupular container when it or the source is used. Note that if the original object is not a built-in, will not be used after casting _or_ is immutable, this is trivial to implement. The difficulty arises when this it not the case, since the lazy object needs to also freeze any values that are mutated in the original to avoid side-effects. An alternative to adding lazy types / lazy type casting (making it possible to implement these oneself) would be to add method call hooks to python, since this would allow having a "freeze value" callback hooked into the __setitem__ and __getitem__ methods. Hooks may be a more useful solution for wider use cases as well.
Hello, On Tue, 08 Dec 2020 11:46:59 -0000 "Mathew Elman" <mathew.elman@ocado.com> wrote:
I am not sure if this has been suggested before, so my apologies if it has.
I would like to propose adding lazy types for casting builtins in a lazy fashion. e.g. `lazy_tuple` which creates a reference to the source iterable and a morally immutable sequence but only populates the tupular container when it or the source is used.
Note that if the original object is not a built-in, will not be used after casting _or_ is immutable, this is trivial to implement. The difficulty arises when this it not the case, since the lazy object needs to also freeze any values that are mutated in the original to avoid side-effects.
You pinpointed the problem exactly. So, the only reliable way to create "lazy" version is to store a copy *eagerly*, which makes the point moot. Lazy-evaluation languages usually deal with that problem by disallowing mutation in the first place (i.e. being purely functional). You can use that solution in Python too. (Without "no mutation" error checking, but it should be possible to implement the alternative API which is "immutable", and I bet someone even did that (PoC-style of course).)
An alternative to adding lazy types / lazy type casting (making it possible to implement these oneself) would be to add method call hooks to python, since this would allow having a "freeze value" callback hooked into the __setitem__ and __getitem__ methods. Hooks may be a more useful solution for wider use cases as well.
[] -- Best regards, Paul mailto:pmiscml@gmail.com
On Tue, Dec 08, 2020 at 11:46:59AM -0000, Mathew Elman wrote:
I would like to propose adding lazy types for casting builtins in a lazy fashion. e.g. `lazy_tuple` which creates a reference to the source iterable and a morally immutable sequence but only populates the tupular container when it or the source is used.
What are your use-cases for this? Does this include things like `lazy_list`, `lazy_float`, `lazy_bool`, `lazy_str`, `lazy_bytearray` etc?
An alternative to adding lazy types / lazy type casting (making it possible to implement these oneself) would be to add method call hooks to python, since this would allow having a "freeze value" callback hooked into the __setitem__ and __getitem__ methods. Hooks may be a more useful solution for wider use cases as well.
Nope, sorry, I don't see how that would work. Here I have a list: L = [50, 40, 30, 20, 10] Suppose these hooks exist. I want to make a "lazy tuple": t = lazy_tuple(L) How do these hooks freeze the list? What if I have more than one lazy object pointing at the same source? s = lazy_str(L) And then follow with a different method call? L.insert(2, "surprise!") I just can't see how this will work. -- Steve
Steven D'Aprano wrote:
On Tue, Dec 08, 2020 at 11:46:59AM -0000, Mathew Elman wrote:
I would like to propose adding lazy types for casting builtins in a lazy fashion. e.g. lazy_tuple which creates a reference to the source iterable and a morally immutable sequence but only populates the tupular container when it or the source is used. What are your use-cases for this? Does this include things like lazy_list, lazy_float, lazy_bool, lazy_str, lazy_bytearray etc?
I would say yes, it should include these types as well. The use case is for when you are casting between types a high number of times to pass them around, especially into another type and back.
An alternative to adding lazy types / lazy type casting (making it possible to implement these oneself) would be to add method call hooks to python, since this would allow having a "freeze value" callback hooked into the __setitem__ and __getitem__ methods. Hooks may be a more useful solution for wider use cases as well. Nope, sorry, I don't see how that would work. Here I have a list: L = [50, 40, 30, 20, 10]
Suppose these hooks exist. I want to make a "lazy tuple": t = lazy_tuple(L)
How do these hooks freeze the list?
they don't freeze the list, they freeze the values in the lazy_tuple when the list is mutated, so when mutating the list, the values in the tuple are set (if they have not been already), so that they aren't also mutated in the lazy_tuple. Of course for delete and insert this would mean you might have to "freeze" from where the insert/delete occurs to the end of the tuple but that is no different than using a normal tuple i.e. iterate over all elements. However for setitem where only position i is replaced, the tuple can set its value to the original without having to evaluate anything extra.
What if I have more than one lazy object pointing at the same source? s = lazy_str(L)
And then follow with a different method call? L.insert(2, "surprise!")
I just can't see how this will work.
I don't follow what your point is, sorry. Are you saying that insert is another method that can update a list so delitem and setitem hooks would miss it? The point I was making with hooks is that you _could_ write a lazy class that creates freezing hooks for any mutation method. But the hooks would be up to the coder to implement and hook into the correct mutation methods. I also don't see an issue with multiple lazy types pointing to the same "source". It would just freeze the values in both of them when updating anything in the original.
On Wed, Dec 9, 2020 at 12:05 PM Mathew Elman <mathew.elman@ocado.com> wrote:
Steven D'Aprano wrote:
On Tue, Dec 08, 2020 at 11:46:59AM -0000, Mathew Elman wrote:
I would like to propose adding lazy types for casting builtins in a lazy fashion. e.g. lazy_tuple which creates a reference to the source iterable and a morally immutable sequence but only populates the tupular container when it or the source is used. What are your use-cases for this? Does this include things like lazy_list, lazy_float, lazy_bool, lazy_str, lazy_bytearray etc?
I would say yes, it should include these types as well. The use case is for when you are casting between types a high number of times to pass them around, especially into another type and back.
Out of morbid curiosity, why would you need to be casting between iterables many times? Don't most consumers treat iterables roughly equally?
I agree that if you are using them as iterables, then the type is usually not important because you are treating their type as just iter-able. The lazy iterable would more or less just be the same as passing in `iter(sequence)`. This is for other use cases where the use of the object is specific to the type, e.g. a case where in the outer scope you construct a list and mutate it but when it is passed to an inner scope, you want to enforce that it can be accessed but not mutated. Likewise if lazy built in types were implemented in python then getting a slice of a sequence could also be done lazily, whereas my understanding at the moment is that it has to create a whole new sequence.
That makes sense, so these are kind of 'views' over a sequence, but ones that implement a copy-on-write when the underlying object is modified in any way? I can see this being super hard/impossible to implement reliably, but would be a pretty nice addition if it can be done. On Wed, Dec 9, 2020 at 12:25 PM Mathew Elman <mathew.elman@ocado.com> wrote:
I agree that if you are using them as iterables, then the type is usually not important because you are treating their type as just iter-able. The lazy iterable would more or less just be the same as passing in `iter(sequence)`.
This is for other use cases where the use of the object is specific to the type, e.g. a case where in the outer scope you construct a list and mutate it but when it is passed to an inner scope, you want to enforce that it can be accessed but not mutated. Likewise if lazy built in types were implemented in python then getting a slice of a sequence could also be done lazily, whereas my understanding at the moment is that it has to create a whole new sequence. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ZQIXX5... Code of Conduct: http://python.org/psf/codeofconduct/
Stestagg wrote:
That makes sense, so these are kind of 'views' over a sequence, but ones that implement a copy-on-write when the underlying object is modified in any way?
yes, a lazy sequence type would be exactly that, that is a nice way of putting it
I can see this being super hard/impossible to implement reliably, but would be a pretty nice addition if it can be done.
On Wed, 9 Dec 2020 at 09:27, Mathew Elman <mathew.elman@ocado.com> wrote:
I agree that if you are using them as iterables, then the type is usually not important because you are treating their type as just iter-able. The lazy iterable would more or less just be the same as passing in `iter(sequence)`.
This is for other use cases where the use of the object is specific to the type, e.g. a case where in the outer scope you construct a list and mutate it but when it is passed to an inner scope, you want to enforce that it can be accessed but not mutated. Likewise if lazy built in types were implemented in python then getting a slice of a sequence could also be done lazily, whereas my understanding at the moment is that it has to create a whole new sequence.
If you need this for annotations/typing alone, can't you just use `typing.cast` in the inner scope? (or before calling it for that matter) Anyway, mypy at least wil error if you annotate the inner scope as a "Sequence" (in contrast with MutableSequence), and will error if you try to change the Sequence - and it stlll remain compatible with incoming "MutableSequences". For the cases it does not cover, there is still "cast" - and it feels _a lot_ simpler than having actual runtime lazy objetcs as primitives in the language.
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ZQIXX5... Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Dec 09, 2020 at 12:05:17PM -0000, Mathew Elman wrote:
Steven D'Aprano wrote:
On Tue, Dec 08, 2020 at 11:46:59AM -0000, Mathew Elman wrote:
I would like to propose adding lazy types for casting builtins in a lazy fashion. e.g. lazy_tuple which creates a reference to the source iterable and a morally immutable sequence but only populates the tupular container when it or the source is used. What are your use-cases for this? Does this include things like lazy_list, lazy_float, lazy_bool, lazy_str, lazy_bytearray etc?
I would say yes, it should include these types as well.
The use case is for when you are casting between types a high number of times to pass them around, especially into another type and back.
I would say, *don't do that*. If you have a list, and you pass it to another function after converting to a tuple, why would you take that tuple and convert back to a list when you already have a list? For types like casting back and forth between float and int, there will be data loss. Even if there is not, the overhead of creating a "lazy proxy" to the float is probably greater than just converting.
How do these hooks freeze the list?
they don't freeze the list, they freeze the values in the lazy_tuple when the list is mutated, so when mutating the list, the values in the tuple are set (if they have not been already), so that they aren't also mutated in the lazy_tuple.
How is this supposed to work in practice? Where does the hook live? When is it called? What does it do? How does the lazy tuple know that the list's setitem has been called? -- Steve
Steven D'Aprano wrote:
On Wed, Dec 09, 2020 at 12:05:17PM -0000, Mathew Elman wrote:
Steven D'Aprano wrote: On Tue, Dec 08, 2020 at 11:46:59AM -0000, Mathew Elman wrote: I would like to propose adding lazy types for casting builtins in a lazy fashion. e.g. lazy_tuple which creates a reference to the source iterable and a morally immutable sequence but only populates the tupular container when it or the source is used. What are your use-cases for this? Does this include things like lazy_list, lazy_float, lazy_bool, lazy_str, lazy_bytearray etc? I would say yes, it should include these types as well. The use case is for when you are casting between types a high number of times to pass them around, especially into another type and back. I would say, don't do that. If you have a list, and you pass it to another function after converting to a tuple, why would you take that tuple and convert back to a list when you already have a list? For types like casting back and forth between float and int, there will be data loss. Even if there is not, the overhead of creating a "lazy proxy" to the float is probably greater than just converting. I see what you mean here, I think my initial thought was more towards container types rather than ints and floats, I said these should be included because I saw no reason why not. Thinking about it, for int and float types it makes less sense, because you have no great saving, whereas for a container type, you can increase the reference count by 1 instead of N.
How do these hooks freeze the list? they don't freeze the list, they freeze the values in the lazy_tuple when the list is mutated, so when mutating the list, the values in the tuple are set (if they have not been already), so that they aren't also mutated in the lazy_tuple. You seem to be referring to the lazy_tuple as the hook, I am saying that _I_ could implement a lazy class if hooks existed. i.e. I could make a class `LazyTuple` that when instanced with a list, creates and stores the hooks on itself.
How is this supposed to work in practice? A method e.g. `create_hook(method, callback)` would return a concrete reference to the hook, and uses a weak reference to know if it needs to execute the callback (meaning that it would only add the overhead of a check for callbacks if the concrete reference still existed).
Where does the hook live? When is it called? What does it do? How does the lazy tuple know that the list's setitem has been called?
It knows because of a hook? This is my whole point about including hooks in this thread, i.e. if it was possible to add a callback on a method, it would execute when it is called. I am not sure I understand your question.
participants (5)
-
Joao S. O. Bueno -
Mathew Elman -
Paul Sokolovsky -
Stestagg -
Steven D'Aprano