I've seen this pattern a lot at a past employer, and despite the obvious convenience I've come to see it as an anti-pattern: for people expecting Python semantics it's quite surprising to read code that writes foo.bar and

Guido van Rossum wrote: then reads back foo['bar']. Would it be significantly less surprising if it were to be exclusively allowed for the JSON module rather than anything that accepts custom dicts? To me, that seems like the most useful and intuitive location for the dot notation, and it would keep the Python semantics intact everywhere else. I think it's fairly unlikely that most users of JSON would be surprised by it, but I can see that it could be surprising elsewhere. On Tue, Apr 14, 2020 at 11:33 PM Guido van Rossum <guido@python.org> wrote:

I've seen this pattern a lot at a past employer, and despite the obvious convenience I've come to see it as an anti-pattern: for people expecting Python semantics it's quite surprising to read code that writes foo.bar and then reads back foo['bar']. We should not try to import JavaScript's object model into Python.

On Tue, Apr 14, 2020 at 8:06 PM Raymond Hettinger < raymond.hettinger@gmail.com> wrote:

...

SimpleNamespace() is really good at giving attribute style-access. I would like to make that functionality available to the JSON module (or just about anything else that accepts a custom dict) by adding the magic methods for mappings so that this works:

catalog = json.load(f, object_hook=SimpleNamespace)

print(catalog['clothing']['mens']['shoes']['extra_wide']['quantity']) # currently possible with dict() print(catalog.clothing.mens.shoes.extra_wide.quantity]) # proposed with SimpleNamespace() print(catalog.clothing.boys['3t'].tops.quantity # would also be supported

I've already seen something like this in production; however, people are having to write custom subclasses to do it. This is kind of bummer because the custom subclasses are a pain to write, are non-standard, and are generally somewhat slow. I would like to see a high-quality version this made more broadly available.

The core idea is keep the simple attribute access but make it easier to load data programmatically:

>>> ns = SimpleNamespace(roses='red', violets='blue') >>> thing = input() sugar >>> quality = input() sweet >>> setattr(ns, thing, quality) # current >>> ns['sugar'] = 'sweet' # proposed

If the PEP 584 __ior__ method were supported, updating a SimpleNamespace would be much cleaner:

ns |= some_dict

I posted an issue on the tracker: https://bugs.python.org/issue40284 . There was a suggestion to create a different type for this, but I don't see the point in substantially duplicating everything SimpleNamespace already does just so we can add some supporting dunder methods. Please add more commentary so we can figure-out the best way to offer this powerful functionality.

Raymond

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JOMND56P... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EJ6XJW6S... Code of Conduct: http://python.org/psf/codeofconduct/

The "hey I'm a dict, but look over hear and now you can treat my like like any other object" duality doesn't sit well with me. I do understand the idea of wanting to convert something to convert JSON data into a more object-like access pattern, though. So if something were to be brought in, for me the question would be whether it's an object that wraps a dict internally w/o re-exposing __getitem__() to the dict like a new veneer, or the creation of a new object that deep-copies the dict so it's more of an object transformation.

I will also say that when I do want such a shift from JSON or dict data structures to a more concrete one I write my own classmethod to provide some error checking, so I haven't personally needed a package to do this for me.

On Wed, Apr 15, 2020 at 7:37 AM Victor Stinner <vstinner@python.org> wrote:

Le mer. 15 avr. 2020 à 05:02, Raymond Hettinger <raymond.hettinger@gmail.com> a écrit :

...

I would like to make that functionality available to the JSON module (or just about anything else that accepts a custom dict) by adding the magic methods for mappings so that this works:

catalog = json.load(f, object_hook=SimpleNamespace)

print(catalog['clothing']['mens']['shoes']['extra_wide']['quantity']) # currently possible with dict()

...

print(catalog.clothing.mens.shoes.extra_wide.quantity])

# proposed with SimpleNamespace()

...

print(catalog.clothing.boys['3t'].tops.quantity

# would also be supported

It seems like there are many projects on PyPI which provide such features. A few examples:

* https://pypi.org/project/attrdict/ "mapping objects that allow their elements to be accessed both as keys and as attributes" * https://pypi.org/project/dictlib/ "Dictionary keys as object attributes" * https://pypi.org/project/objdict/ "An ObjDict supports all normal dict operations, but adds support for accessing and setting entries as attributes" * https://pypi.org/project/mydict/ "dict subclass which tries to act like JavaScript objects, so you can use the dot notation (d.foo) to access members of the object" * https://pypi.org/project/dpath/ "accessing and searching dictionaries via /slashed/paths ala xpath". Example: dpath.util.search(x, "a/b/[cd]") * etc.

There are many variants:

* Support nested dictionaries or not * "Path" as a string ala XPath using "/" or "." separator -versus- don't give a direct access into nested dictionaries * Optional default value when a key/attribute is missing * etc.

Is there a project on PyPI which already already been battle-tested and gained enough popularity to consider that it's now mature enough to enter the stdlib?

+1 that was my first reaction as well. We don't need this battery. If you wanted to make it a more general thing, introduce a special operator for it instead of overloading ".". Ex: copy Perl and adopt -> as the "lookup the name in a dictionary without typing quotes around the constant value" operator. (please don't!) by the time we're saying "JSON should decode into a structured format" we can also say very true things like "JSON data should be validated against a schema" at which point we're ultimately going to wind up reinventing https://googleapis.dev/python/protobuf/latest/google/protobuf/json_format.ht... and equivalent other things. -gps The problem is that so far, I see no clear "winner" of a "standard"

API. So maybe leaving such feature on PyPI is better for now. It seems like these projects are opinionated about what should be the "right" behavior :-)

And yeah, there are even more advanced tools like https://glom.readthedocs.io/

As I wrote in the issue, I would prefer to leave SimpleNamespace unchanged. If such feature lands into the stdlib, I would prefer it to be a new class.

Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XDFTMWK5... Code of Conduct: http://python.org/psf/codeofconduct/

First of all, be aware of the limitations of this approach (which will need to be clearly documented if we go this way):

- It only allows valid Python identifiers -- while JSON accepts any sequence of Unicode characters for keys ( http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf, p.5) - It will clash with any regular attributes of whatever type implements this senantic

Agreed, these are all problems that lead to long term code maintenance

On Wed, Apr 15, 2020 at 12:49 PM Ivan Pozdeev via Python-Dev < python-dev@python.org> wrote: headaches (add a new method and a bunch of existing code mysteriously breaks). The dict ['key'] lookup / get('key') syntax must always remain for access to those as a result. this is mere syntactic sugar convenience. Unfortunately it's a slippery slope; do this just for JSON and people will ask why it can't be done for any dict and why it isn't the default available everywhere. Similar to -> in Perl.

- - It's not very scalable and will upset static code checkers and IDEs as it's impossible to determine if the attribute name is valid or provide typing hints

Not necessarily. If you've given the JSON a schema, you can use that to teach tools attr names and types. otherwise anything doing such checking should see that this is an instance of a type with a __getattr__ implemented and thus all attributes must be deemed valid.

-

------------------------------ Now, I think it's possible to make a class that would *wrap* an existing mapping. This would completely decouple the convenience syntax from the storage semantics and any specific modules and types.

When writing a proof-of-concept implementation, however, I bumped into the need to distinguish which of the child objects are containers (thus need to be wrapped as well) and which are the leaves (thus need to be returned as is). I guess this is going to be application-specific. And the same problem will arise with any implementation, not only a wrapper. On 15.04.2020 5:59, Raymond Hettinger wrote:

SimpleNamespace() is really good at giving attribute style-access. I would like to make that functionality available to the JSON module (or just about anything else that accepts a custom dict) by adding the magic methods for mappings so that this works:

catalog = json.load(f, object_hook=SimpleNamespace) print(catalog['clothing']['mens']['shoes']['extra_wide']['quantity']) # currently possible with dict() print(catalog.clothing.mens.shoes.extra_wide.quantity]) # proposed with SimpleNamespace() print(catalog.clothing.boys['3t'].tops.quantity # would also be supported

I've already seen something like this in production; however, people are having to write custom subclasses to do it. This is kind of bummer because the custom subclasses are a pain to write, are non-standard, and are generally somewhat slow. I would like to see a high-quality version this made more broadly available.

The core idea is keep the simple attribute access but make it easier to load data programmatically:

>>> ns = SimpleNamespace(roses='red', violets='blue') >>> thing = input() sugar >>> quality = input() sweet >>> setattr(ns, thing, quality) # current >>> ns['sugar'] = 'sweet' # proposed

If the PEP 584 __ior__ method were supported, updating a SimpleNamespace would be much cleaner:

ns |= some_dict

I posted an issue on the tracker: https://bugs.python.org/issue40284 . There was a suggestion to create a different type for this, but I don't see the point in substantially duplicating everything SimpleNamespace already does just so we can add some supporting dunder methods. Please add more commentary so we can figure-out the best way to offer this powerful functionality.

Raymond

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.orghttps://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JOMND56P... Code of Conduct: http://python.org/psf/codeofconduct/

-- Regards, Ivan

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/IEKXDY7H... Code of Conduct: http://python.org/psf/codeofconduct/

On 16.04.2020 0:34, Glenn Linderman wrote:

On 4/15/2020 12:47 PM, Ivan Pozdeev via Python-Dev wrote:

...

When writing a proof-of-concept implementation, however, I bumped into the need to distinguish which of the child objects are containers (thus need to be wrapped as well) and which are the leaves (thus need to be returned as is). I guess this is going to be application-specific. And the same problem will arise with any implementation, not only a wrapper.

Do the child objects truly need to be wrapped, or just accessed?

They need to be wrapped on the fly, to support nested  ".key" accesses.

Thanks for your comments though, they inspired a thought.

The problem with the glom syntax versus the dotted syntax is that the minimal syntax is bulky.

obj.abc.def.ghi     versus   glom( obj, 'abc.def.ghi')

The problem with attribute syntax is the conflict with regular attributes, and the limitation of valid identifier characters. Glom probably doesn't have these problems.

"Glom syntax" still excludes the delimiter, whatever it is, from use in keys. So it's still a further limitation compared to the JSON spec.

So my new thought is that we need a new syntax.  The minimal existing syntax might be

obj._('abc.def.ghi')     or maybe   obj['abc.def.ghi']

although the latter would conflict with regular dict lookups, but obj isn't a regular dict, so that might not matter.

A new syntax might be:

obj.'abc.def.ghi'

In any of these syntaxes, one could enhance it to use variables in some spots similarly to f-strings (but from the beginning, so no 'f' would be required)

foo = 'def' obj.'abc.{foo}.ghi'

would access the same item as previous examples. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3A5V62LY... Code of Conduct: http://python.org/psf/codeofconduct/ -- Regards, Ivan

On Wed, Apr 15, 2020 at 2:59 PM Ivan Pozdeev via Python-Dev <python-dev@python.org> wrote:

"Glom syntax" still excludes the delimiter, whatever it is, from use in keys. So it's still a further limitation compared to the JSON spec.

Glom does let you be specific about the exact lookup keys if you want, to handle keys that contain embedded periods, or non-string keys. The syntax looks like: from glom import glom, Path glom(obj, Path("a", "b.c", 2)) https://glom.readthedocs.io/en/latest/api.html#specifier-types For a simple case like this t's a bit wordier than obj["a"]["b.c"][2], but OTOH you get better error message on failed lookups, null-coalescing support by using default=, etc. -n -- Nathaniel J. Smith -- https://vorpus.org

Here's another revolutionary thought:  add a new operator and associated dunder method (to object?) whose meaning is *undefined*.  Its default implementation would do *nothing* useful (raise an error? return None?). E.g. suppose the operator were `..` Then in a specific class you could implement x..y to mean x['y'] and then you could write     obj..abc..def..ghi Still fairly concise, but warns that what is happening is not normal attribute lookup. On 15/04/2020 22:34, Glenn Linderman wrote:

On 4/15/2020 12:47 PM, Ivan Pozdeev via Python-Dev wrote:

...

When writing a proof-of-concept implementation, however, I bumped into the need to distinguish which of the child objects are containers (thus need to be wrapped as well) and which are the leaves (thus need to be returned as is). I guess this is going to be application-specific. And the same problem will arise with any implementation, not only a wrapper.

Do the child objects truly need to be wrapped, or just accessed?

Thanks for your comments though, they inspired a thought.

The problem with the glom syntax versus the dotted syntax is that the minimal syntax is bulky.

obj.abc.def.ghi     versus   glom( obj, 'abc.def.ghi')

The problem with attribute syntax is the conflict with regular attributes, and the limitation of valid identifier characters. Glom probably doesn't have these problems.

So my new thought is that we need a new syntax.  The minimal existing syntax might be

obj._('abc.def.ghi')     or maybe   obj['abc.def.ghi']

although the latter would conflict with regular dict lookups, but obj isn't a regular dict, so that might not matter.

A new syntax might be:

obj.'abc.def.ghi'

In any of these syntaxes, one could enhance it to use variables in some spots similarly to f-strings (but from the beginning, so no 'f' would be required)

foo = 'def' obj.'abc.{foo}.ghi'

would access the same item as previous examples. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3A5V62LY... Code of Conduct: http://python.org/psf/codeofconduct/