weakref.link: Keep A alive while B is alive (and/or update WeakKeyDictionary)
Proposal: Have a weakref.link (not at all attached to the naming) primitive that allows one to keep object A alive while object B is alive. This would be functionally similar to adding A as an attribute of B from the GC's perspective without making any assumptions about what attributes are available in B. Here's a really course example of how this might be used: # Associate "extra_data" with "obj" where "extra_data" will lose its reference when "obj" is no longer reachable through strong refs. my_link = weakref.link(obj, extra_data) ... obj, extra_data = my_link() if obj is not None: do_something(obj, extra_data) This would also allow us to fix or create a new version of WeakKeyDictionary where values can have strong references back to their key without preventing their key from being reclaimed. This has been an issue for tensorflow in the past (see https://github.com/tensorflow/tensorflow/issues/42022) and is likely to be a subtle issue in many more libraries. Even the suggested use case in the docs "This can be especially useful with objects that override attribute accesses." is likely to be broken because of this issue. While I'd be weary of changing the semantics of an existing data structure it seems highly unlikely anything depends on this behavior since it makes WeakKeyDictionary degrade to behaving like a normal dictionary for that key. Notes: * Not attached to this specific approach or API * Mostly interesting in updating/creating new WeakKeyDictionary as mentioned
On Mon, 2021-07-19 at 22:24 +0000, Mark Gordon wrote:
Proposal:
Have a weakref.link (not at all attached to the naming) primitive that allows one to keep object A alive while object B is alive. This would be functionally similar to adding A as an attribute of B from the GC's perspective without making any assumptions about what attributes are available in B. Here's a really course example of how this might be used:
Probably, the "stake holders" on this list are well aware. But, as far as I understand, this is (or requires) an "Ephemeron": https://en.wikipedia.org/wiki/Ephemeron While I might be tempted to try/use it if Python had it (and was available from C), I doubt that it actually matters in practice, for my "use case". The "use case", why I was curious about it is for multiple-dispatching (to allow users to extend NumPy). If you dispatch based on input types and the input type can be deleted, the same problem arises: The (type specific) implementation is expected to – and sometimes must – hold a strong reference to the input types. Thus creating an unbreakable circle. (You can avoid this sometimes, but I don't think you can in general) An Ephemeron should be able to describe this correctly. I expect even a situation like: func(A, B) -> (A, B, C) which can be deleted if either `A` or `B` is "almost collectable", but must keep alive `C` if both `A` and `B` are alive. Cheers, Sebastian
# Associate "extra_data" with "obj" where "extra_data" will lose its reference when "obj" is no longer reachable through strong refs. my_link = weakref.link(obj, extra_data)
...
obj, extra_data = my_link() if obj is not None: do_something(obj, extra_data)
This would also allow us to fix or create a new version of WeakKeyDictionary where values can have strong references back to their key without preventing their key from being reclaimed. This has been an issue for tensorflow in the past (see https://github.com/tensorflow/tensorflow/issues/42022) and is likely to be a subtle issue in many more libraries. Even the suggested use case in the docs "This can be especially useful with objects that override attribute accesses." is likely to be broken because of this issue. While I'd be weary of changing the semantics of an existing data structure it seems highly unlikely anything depends on this behavior since it makes WeakKeyDictionary degrade to behaving like a normal dictionary for that key.
Notes: * Not attached to this specific approach or API * Mostly interesting in updating/creating new WeakKeyDictionary as mentioned _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4KBLRV... Code of Conduct: http://python.org/psf/codeofconduct/
Would fixing this help? https://bugs.python.org/issue44140 On Tue, 20 Jul 2021, 02:16 Sebastian Berg, <sebastian@sipsolutions.net> wrote:
On Mon, 2021-07-19 at 22:24 +0000, Mark Gordon wrote:
Proposal:
Have a weakref.link (not at all attached to the naming) primitive that allows one to keep object A alive while object B is alive. This would be functionally similar to adding A as an attribute of B from the GC's perspective without making any assumptions about what attributes are available in B. Here's a really course example of how this might be used:
Probably, the "stake holders" on this list are well aware. But, as far as I understand, this is (or requires) an "Ephemeron":
https://en.wikipedia.org/wiki/Ephemeron
While I might be tempted to try/use it if Python had it (and was available from C), I doubt that it actually matters in practice, for my "use case".
The "use case", why I was curious about it is for multiple-dispatching (to allow users to extend NumPy). If you dispatch based on input types and the input type can be deleted, the same problem arises: The (type specific) implementation is expected to – and sometimes must – hold a strong reference to the input types. Thus creating an unbreakable circle.
(You can avoid this sometimes, but I don't think you can in general)
An Ephemeron should be able to describe this correctly. I expect even a situation like:
func(A, B) -> (A, B, C)
which can be deleted if either `A` or `B` is "almost collectable", but must keep alive `C` if both `A` and `B` are alive.
Cheers,
Sebastian
# Associate "extra_data" with "obj" where "extra_data" will lose its reference when "obj" is no longer reachable through strong refs. my_link = weakref.link(obj, extra_data)
...
obj, extra_data = my_link() if obj is not None: do_something(obj, extra_data)
This would also allow us to fix or create a new version of WeakKeyDictionary where values can have strong references back to their key without preventing their key from being reclaimed. This has been an issue for tensorflow in the past (see https://github.com/tensorflow/tensorflow/issues/42022) and is likely to be a subtle issue in many more libraries. Even the suggested use case in the docs "This can be especially useful with objects that override attribute accesses." is likely to be broken because of this issue. While I'd be weary of changing the semantics of an existing data structure it seems highly unlikely anything depends on this behavior since it makes WeakKeyDictionary degrade to behaving like a normal dictionary for that key.
Notes: * Not attached to this specific approach or API * Mostly interesting in updating/creating new WeakKeyDictionary as mentioned _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/4KBLRV...
Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7C7EVG... Code of Conduct: http://python.org/psf/codeofconduct/
Looks like a separate issue to me. That issue can be fixed by updating the Python implementation of WeakKeyDictioary. The fix for the WeakKeyDictionary issue I mentioned and the Ephemeron primitive (as Sebastian identified it) are equivalant (you can implement one from the other) and cannot be reduced to any existing primitive as far as I can tell. Such a change would need to happen within the Python engine itself.
participants (3)
-
Mark Gordon -
Sebastian Berg -
Thomas Grainger