Paul,

I agree that the mapping to a flat data structure might have limited use cases. However, like you said it is only one example. What I am proposing is a nested iteration tool that can map both keys and values to new keys and new values. It only considers nesting in dictionaries, lists, and tuples by default, but allows users to pass another function to continue searching for pointers in other types. For example:

class Node:
    name: str=""
    next: 'Node' = None
    val: Any = None

def newDefaultMapper(o) -> o:
    '''Converts the object o and returns it'''
    if isinstance(o, dict): # This would be automatically implemented in itermapper.Mapper if arg 'dictMapper' was defined
        return {k+"Mapped": itermapper.Mapper(v) for k, v in o.items()}
    if isinstance(o, Node):
        o.name = 'm' + o.name[1]
        if o.next:
             return itermapper.Mapper(o.next) # resorts to newDefaultMapper if it cannot handle, then the oldDefaultMapper
        return o
     return itermapper.oldDefaultMapper(o)

n1 = Node(name='n1')
n2 = Node(name='n2')
n1.next = n2

d = {'a': 1, 'b': n1}

itermapper(d, default=newDefaultMapper)

print(d)
>> {'aMapped': 1, 'bMapped': <__main__.Node at 0x....>}

print(n1.name, n2.name)
>> m1 m2


This functionality partially exists in the current versions of python in the JSON dumps function. Dumps iterates over all values in a nested data structure and it also accepts a JSONEncoder with a new default function to handle arbitrary types. No such functionality exists in other parts of python to map values in a nested data structure into values of another identical nested data structure.

What I am proposing goes beyond this though to also handle keys. It also doesn't return JSON strings, but a new mapped data structure.

Best,
Sven

On Wed, Jul 21, 2021 at 9:30 AM Paul Moore <p.f.moore@gmail.com> wrote:
You've given some examples of what flatten() does, but I still don't
really understand how it's expected to work. Could you provide a
specification, rather than just examples? Also, I've never needed a
function like this in anything I've ever done - that may mean nothing,
but it does suggest that you need to demonstrate that it's a generally
useful tool (maybe by pointing to examples of "real world" code in
various contexts that would be improved by using it). There's a lot of
pressure to keep the stdlib for things that are widely applicable, and
the burden of demonstrating that lies with the proposer.

As the proposal stands here, I honestly don't see anything that
suggests this wouldn't be better as a standalone function published on
PyPI.

Paul

On Wed, 21 Jul 2021 at 13:43, Sven Voigt <svenpvoigt@gmail.com> wrote:
>
> Hello Python-ideas,
>
> I am working with lots of JSON objects in my work and need to obtain JSON pointers to particular components of a JSON object. These pointers are defined as part of the JSON-LD specifications as relative IRIs and as part of the JSON-Schema specifications. However, I am not aware of these pointers being part of the JSON specification. The pointers are really straightforward and follow the same syntax as other IRIs (URLs) or file paths. For example, the following flatten function illustrates the mapping of nested dictionaries and arrays to a single flat dictionary, where each nested key is mapped to a unique path.
>
> d = {"a": 1, "b": {"ba": 2, "bb": [{"bba": 3, "bbb": None}]}}
> flatten(d)
> >> {'a': 1, 'b/ba': 2, 'b/bb/0/bba': 3, 'b/bb/0/bbb': None}
>
> Not only does this conversion help in generating JSON pointers, but it helps with logic for nested data structures. Specifically, assume there is a dictionary, which includes nested dictionaries, lists, and tuples, and any of these elements are contained within another dictionary or that other dictionaries elements. Then it is not sufficient to simply compare the items of the two dictionaries to check whether the first is a subset of the other. However, the flat structures can be compared directly.
>
> 1. A nested dictionary is a subset of another nested dictionary. The flatten function ensures the nested dictionaries are not checked for equivalence but subset of.
> # Current
> {"a": 1, "c": {"d": 4}}.items() <= {"a": 1, "b": 2, "c": {"d": 4, "e": 5}}.items()
> >> False
> # Proposed
> flatten({"a": 1, "c": {"d": 4}}).items() <= flatten({"a": 1, "b": 2, "c": {"d": 4, "e": 5}}).items()
> >> True
>
> 2. A nested list or tuple is a subset of a dictionary.
> # Current
> {"a": 1, "c": [3]}.items() <= {"a": 1, "b": 2, "c": [3,3]}.items()
> >> False
> # Proposed
> flatten({"a": 1, "c": [3]}).items() <= flatten({"a": 1, "b": 2, "c": [3,3]}).items()
> >> True
>
> Note that these examples only have one level of nesting, but the flatten function must handle any level of nesting. For the current version of the flatten function, I essentially borrowed the JSON "dumps" source code. However, the dumps code doesn't utilize any other functions and I believe there may be a missing "nestedIter" function in python. This function could be used for JSON dumps or flatten or anytime someone wants to visit all nested elements of a dictionary and perform some operation. Therefore, I think this function could be included in the JSON library if python-ideas thinks that the pointers are very specific to JSON or the itertools library, where the nested iteration function could be generally used for all nested data types.
>
> Let me know what you think of this proposal and I am looking forward to your responses.
>
> Best,
> Sven
> _______________________________________________
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-leave@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/V2DNTLXII2QNYSE3FJXRO45ZFTCMBZ5G/
> Code of Conduct: http://python.org/psf/codeofconduct/