Get the value of an attribute of an object hierarchy with a path expression
This proposal is about adding a builtin function or a package to the standard library to access an element in an object hierarchy with default behavior i.e. if at any level the attribute/key/index does not exist in the path the function returns a default value. Motivation: mapping an object tree to a tuple or a flat object cannot be achieved compactly with proper error handling. Need: I have found a half dozen of packages implementing this functionality on github and there is a high number of loosely related libraries. Here is a motivating example: animal = { 'name': 'Wombat', 'avg_properties': { 'height': {'value': 66, 'unit': 'cm'}, 'length':{'value': 108, 'unit': 'cm'}, 'weight': {'value': 27, 'unit': 'kg'} } } assert getpath(animal, ('avg_properties', 'length', 'value'), default='-') == 108 assert getpath(animal, ('min_properties', 'length', 'value'), default='-') == '-' assert getpaths(animal, 'avg_properties.length.value', default='-') == 108 assert getpaths(animal, 'avg_properties/width/value', sep='/') == None I created a package to demonstrate the idea: https://github.com/axonmatics/quickpath
On Wed, 4 May 2022 at 12:27, Dr. Guta Gábor <gabor.guta@proinbox.com> wrote:
This proposal is about adding a builtin function or a package to the standard library to access an element in an object hierarchy with default behavior i.e. if at any level the attribute/key/index does not exist in the path the function returns a default value.
[...]
I created a package to demonstrate the idea: https://github.com/axonmatics/quickpath
There's another PyPI library "glom" that does this sort of thing. I'm not sure there's enough need for this that it should be in the stdlib. And I'm certain it isn't sufficiently common to justify being a builtin. Paul
On Wed, 4 May 2022 at 22:32, Dr. Guta Gábor <gabor.guta@proinbox.com> wrote:
You may find PEP 505 interesting, then. https://peps.python.org/pep-0505/ Also possibly of interest is PEP 463 for another approach, where you just let the exception happen where it happens, then catch it and provide a value: https://peps.python.org/pep-0463/ For the simple case of "here's a series of attribute/item lookups, if you ever get AttributeError/LookupError, return this default", it's not too hard to make your own function. "Enough need" can be measured in a few places, but here are some questions that might guide that measurement: 1) How frequently do people write (some version of) this function? 1a) Can all variants of the function be cleanly provided by a single stdlib function? 2) Does the feature have subtleties that are easy to get wrong, and thus people write buggy versions of it without realising it? 3) Does the lack of the feature cause code to be significantly less readable? 4) Is code less secure and/or does it suffer from encapsulation breakage when this is done manually? (See, for instance, f-strings vs .format_map().) In this particular case, I've certainly made versions of this function before. But they often have slightly different rules, like "if not found, insert an empty dict at that point and keep going", or "after making any change, save everything to a JSON file" etc. So it has some weak justification on point 1, but loses part of that on 1a. Overall, I think this is something that has enough variations to be best done as a personal toolkit function, and it doesn't need any special syntax or anything. The broader (and also narrower) proposals in PEP 505 and 463 have stronger justifications, but also greater consequences. ChrisA
1) How frequently do people write (some version of) this function? I think it is pretty frequent to check that a variable is not None before accessing it's attributes or element. 1a) Can all variants of the function be cleanly provided by a single stdlib function? 2 functions can reasonably cover the main use cases: access hierarchy with a sequence of keys and access hierarchy with string of keys. I think this could be a package which can extended with various other helper function like getlistpath which could gather the leaves from a tree like hierarchy. 2) Does the feature have subtleties that are easy to get wrong, and thus people write buggy versions of it without realising it? I think, no. 3) Does the lack of the feature cause code to be significantly less readable? Yes, this can save several lines of code. 4) Is code less secure and/or does it suffer from encapsulation breakage when this is done manually? (See, for instance, f-strings vs .format_map().) Yes, error handling is often omitted. I feel this functionality has a bit of chicken & egg problem: if these functions would be available in stdlib, people would use it; but as they do not provide such a complex functionality which would make programmers start to look for such a library on pypi.
On 5/05/22 3:53 am, gabor.guta@proinbox.com wrote:
I think it is pretty frequent to check that a variable is not None before accessing it's attributes or element.
While that's probably true, in my experience I'm usually performing the test once on a particular object and then accessing a bunch of attributes. I'm not doing it for individual attribute accesses. Also I don't find that I'm doing deeply nested sequences of accesses very often. It gives me the feeling that I'm mixing up levels of abstraction in my code in a way that's going to cause problems later. -- Greg
On Wed, 4 May 2022 at 13:29, Dr. Guta Gábor <gabor.guta@proinbox.com> wrote:
Well, there are at least two libraries on PyPI, and at least one of them (glom) has been around for some time, but neither is particularly well known. This suggests to me that this isn't an issue that people are routinely asking "how do I solve this problem?" (or if they are, their requirements are sufficiently different that no one solution has emerged as "the obvious one"). So I struggle to imagine that there's a significant group of users who are all waiting for a solution in the stdlib, and for whom installing a library from PyPI is a problem.
The other way to solve this problem is to have null-conditional access operators ?. and ?[] like in C# (i.e. a.b.c can be written to a?.b?.c to be prone to None values).
Well, that's not the *only* other way to solve the problem. Using a library from PyPI is an entirely reasonable solution as well. You seem to be discounting that as an option. Can you explain what's wrong with using a library off PyPI? Just to be clear, I'm one of the stronger supporters of having an extensive stdlib, so when I say this, I'm *not* saying that everything should be on PyPI. But it would be good if you could articulate your reasons for why this specific functionality is less usable as an external library, that justify it being in the stdlib.
I don't think that null-conditional operators are Pythonic, but I guess if an other language has a complete set of operators around a problem, it could be something that interest lots of people.
I agree with you somewhat on this point, I don't think anyone has successfully demonstrated that library-based solutions aren't sufficient here. But library-based doesn't necessarily mean "in the stdlib". Paul
On Wed, 4 May 2022 at 12:27, Dr. Guta Gábor <gabor.guta@proinbox.com> wrote:
This proposal is about adding a builtin function or a package to the standard library to access an element in an object hierarchy with default behavior i.e. if at any level the attribute/key/index does not exist in the path the function returns a default value.
[...]
I created a package to demonstrate the idea: https://github.com/axonmatics/quickpath
There's another PyPI library "glom" that does this sort of thing. I'm not sure there's enough need for this that it should be in the stdlib. And I'm certain it isn't sufficiently common to justify being a builtin. Paul
On Wed, 4 May 2022 at 22:32, Dr. Guta Gábor <gabor.guta@proinbox.com> wrote:
You may find PEP 505 interesting, then. https://peps.python.org/pep-0505/ Also possibly of interest is PEP 463 for another approach, where you just let the exception happen where it happens, then catch it and provide a value: https://peps.python.org/pep-0463/ For the simple case of "here's a series of attribute/item lookups, if you ever get AttributeError/LookupError, return this default", it's not too hard to make your own function. "Enough need" can be measured in a few places, but here are some questions that might guide that measurement: 1) How frequently do people write (some version of) this function? 1a) Can all variants of the function be cleanly provided by a single stdlib function? 2) Does the feature have subtleties that are easy to get wrong, and thus people write buggy versions of it without realising it? 3) Does the lack of the feature cause code to be significantly less readable? 4) Is code less secure and/or does it suffer from encapsulation breakage when this is done manually? (See, for instance, f-strings vs .format_map().) In this particular case, I've certainly made versions of this function before. But they often have slightly different rules, like "if not found, insert an empty dict at that point and keep going", or "after making any change, save everything to a JSON file" etc. So it has some weak justification on point 1, but loses part of that on 1a. Overall, I think this is something that has enough variations to be best done as a personal toolkit function, and it doesn't need any special syntax or anything. The broader (and also narrower) proposals in PEP 505 and 463 have stronger justifications, but also greater consequences. ChrisA
1) How frequently do people write (some version of) this function? I think it is pretty frequent to check that a variable is not None before accessing it's attributes or element. 1a) Can all variants of the function be cleanly provided by a single stdlib function? 2 functions can reasonably cover the main use cases: access hierarchy with a sequence of keys and access hierarchy with string of keys. I think this could be a package which can extended with various other helper function like getlistpath which could gather the leaves from a tree like hierarchy. 2) Does the feature have subtleties that are easy to get wrong, and thus people write buggy versions of it without realising it? I think, no. 3) Does the lack of the feature cause code to be significantly less readable? Yes, this can save several lines of code. 4) Is code less secure and/or does it suffer from encapsulation breakage when this is done manually? (See, for instance, f-strings vs .format_map().) Yes, error handling is often omitted. I feel this functionality has a bit of chicken & egg problem: if these functions would be available in stdlib, people would use it; but as they do not provide such a complex functionality which would make programmers start to look for such a library on pypi.
On 5/05/22 3:53 am, gabor.guta@proinbox.com wrote:
I think it is pretty frequent to check that a variable is not None before accessing it's attributes or element.
While that's probably true, in my experience I'm usually performing the test once on a particular object and then accessing a bunch of attributes. I'm not doing it for individual attribute accesses. Also I don't find that I'm doing deeply nested sequences of accesses very often. It gives me the feeling that I'm mixing up levels of abstraction in my code in a way that's going to cause problems later. -- Greg
On Wed, 4 May 2022 at 13:29, Dr. Guta Gábor <gabor.guta@proinbox.com> wrote:
Well, there are at least two libraries on PyPI, and at least one of them (glom) has been around for some time, but neither is particularly well known. This suggests to me that this isn't an issue that people are routinely asking "how do I solve this problem?" (or if they are, their requirements are sufficiently different that no one solution has emerged as "the obvious one"). So I struggle to imagine that there's a significant group of users who are all waiting for a solution in the stdlib, and for whom installing a library from PyPI is a problem.
The other way to solve this problem is to have null-conditional access operators ?. and ?[] like in C# (i.e. a.b.c can be written to a?.b?.c to be prone to None values).
Well, that's not the *only* other way to solve the problem. Using a library from PyPI is an entirely reasonable solution as well. You seem to be discounting that as an option. Can you explain what's wrong with using a library off PyPI? Just to be clear, I'm one of the stronger supporters of having an extensive stdlib, so when I say this, I'm *not* saying that everything should be on PyPI. But it would be good if you could articulate your reasons for why this specific functionality is less usable as an external library, that justify it being in the stdlib.
I don't think that null-conditional operators are Pythonic, but I guess if an other language has a complete set of operators around a problem, it could be something that interest lots of people.
I agree with you somewhat on this point, I don't think anyone has successfully demonstrated that library-based solutions aren't sufficient here. But library-based doesn't necessarily mean "in the stdlib". Paul
participants (5)
-
Chris Angelico -
Dr. Guta Gábor -
gabor.guta＠proinbox.com
-
Greg Ewing -
Paul Moore