Proposal for __encode__ method for json encoder and dataclasses
In response to the additional work required to convert the new python dataclass using the json encoder I propose an __encode__ method that will be included in the default dataclass attributes. This would then be picked up by the default json encoder and called to encode the object. Using a common __encode__ tag would allow the solution to also be applied to custom objects and other new objects without needing to update all json encoders. The __encode__ method would work similar to __repr__ but output a json string of the object.
On Thu, Dec 10, 2020 at 12:38 AM sam bland <sbland.co.uk@gmail.com> wrote:
In response to the additional work required to convert the new python dataclass using the json encoder I propose an __encode__ method that will be included in the default dataclass attributes. This would then be picked up by the default json encoder and called to encode the object. Using a common __encode__ tag would allow the solution to also be applied to custom objects and other new objects without needing to update all json encoders. The __encode__ method would work similar to __repr__ but output a json string of the object.
There's actually a pretty easy way to encode a dataclass - assuming you just want all of its attributes, which is what you'd get from a default implementation. Just encode its __dict__:
It'll quietly ignore any methods you've added, and just output the attributes. ChrisA
On 12/9/2020 10:39 AM, Chris Angelico wrote:
The dataclass metadata feature was added for this type of use, or at least experimenting with it: ----- import json import dataclasses @dataclasses.dataclass class Demo: name: str rank: int value: int = dataclasses.field(metadata={"json": False}) def serialize(o): return json.dumps( { k: getattr(o, k) for k, v in o.__dataclass_fields__.items() if v.metadata.get("json", True) } ) print(serialize(Demo("foo", 1, 23))) ---- Prints: {"name": "foo", "rank": 1}
On 12/10/2020 10:08 AM, Eric V. Smith wrote:
The dataclass metadata feature was added for this type of use, or at least experimenting with it:
... I realize this isn't helpful in general, where a dataclass is a member of another object hierarchy. But it could be leveraged by a json.JSONEncoder (though I haven't tried it). Eric
I'm a bit confused here. there have been multiple discussions on this list about having a magic methods (maybe __json__) that could be called by the json module (or any other) -- so that objects could define for themselves how to be encoded as JSON. Is this the same idea? in which case, if you do a search on this list and you'll find a lot of discussion. Or is this something specific to dataclasses? if so -- why? Anyway, I've written a dataclass json encoder/decoder, and I'm sure others have too. And it wasn't very hard -- I'm not sure what the "extra work" you are referring to is here. If the data classes have only attributes that can be encoded in json anyway (strings, dicts, numbers, lists....) then it's pretty trivial, as Chris A. pointed out. The less trivial part is decoding. What I've done that is unique (or at least less common) is make a point of not requiring any special key in the JSON at all -- so the JSON you get is plain, pure JSON. To decode arbitrary objects, it relies on the type of that object to know how to decode itself. This was easy with dataclasses, because __dataclass_fields__ holds the fields, and the type of the fields. So what I did was add a special method that it could look for, if it was there, then it would be used to decode the json (actually JSON-compatible Python). Anyway, as Guido suggested this was very doable with a class decorator. Whether it's a good idea to build it into dataclasses, I don't know -- I suspect not. I'm wary of giving JSON this kind of special place in Python. On the other hand, a "standard" collection of extra tools designed for datacalsses would be pretty nice, and this could belong there. -Chris NOTE: This is all part of a technically open-source, but not published project. But if folks are interested, maybe I could find the time to publish it on it's own. PS: if this does get anywhere -- please call it something like __json__ or __json_encode__ and not plain __encode__ there are any number of ways one might encode something! On Wed, Dec 9, 2020 at 5:38 AM sam bland <sbland.co.uk@gmail.com> wrote:
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Dec 10, 2020 at 12:38 AM sam bland <sbland.co.uk@gmail.com> wrote:
In response to the additional work required to convert the new python dataclass using the json encoder I propose an __encode__ method that will be included in the default dataclass attributes. This would then be picked up by the default json encoder and called to encode the object. Using a common __encode__ tag would allow the solution to also be applied to custom objects and other new objects without needing to update all json encoders. The __encode__ method would work similar to __repr__ but output a json string of the object.
There's actually a pretty easy way to encode a dataclass - assuming you just want all of its attributes, which is what you'd get from a default implementation. Just encode its __dict__:
It'll quietly ignore any methods you've added, and just output the attributes. ChrisA
On 12/9/2020 10:39 AM, Chris Angelico wrote:
The dataclass metadata feature was added for this type of use, or at least experimenting with it: ----- import json import dataclasses @dataclasses.dataclass class Demo: name: str rank: int value: int = dataclasses.field(metadata={"json": False}) def serialize(o): return json.dumps( { k: getattr(o, k) for k, v in o.__dataclass_fields__.items() if v.metadata.get("json", True) } ) print(serialize(Demo("foo", 1, 23))) ---- Prints: {"name": "foo", "rank": 1}
On 12/10/2020 10:08 AM, Eric V. Smith wrote:
The dataclass metadata feature was added for this type of use, or at least experimenting with it:
... I realize this isn't helpful in general, where a dataclass is a member of another object hierarchy. But it could be leveraged by a json.JSONEncoder (though I haven't tried it). Eric
I'm a bit confused here. there have been multiple discussions on this list about having a magic methods (maybe __json__) that could be called by the json module (or any other) -- so that objects could define for themselves how to be encoded as JSON. Is this the same idea? in which case, if you do a search on this list and you'll find a lot of discussion. Or is this something specific to dataclasses? if so -- why? Anyway, I've written a dataclass json encoder/decoder, and I'm sure others have too. And it wasn't very hard -- I'm not sure what the "extra work" you are referring to is here. If the data classes have only attributes that can be encoded in json anyway (strings, dicts, numbers, lists....) then it's pretty trivial, as Chris A. pointed out. The less trivial part is decoding. What I've done that is unique (or at least less common) is make a point of not requiring any special key in the JSON at all -- so the JSON you get is plain, pure JSON. To decode arbitrary objects, it relies on the type of that object to know how to decode itself. This was easy with dataclasses, because __dataclass_fields__ holds the fields, and the type of the fields. So what I did was add a special method that it could look for, if it was there, then it would be used to decode the json (actually JSON-compatible Python). Anyway, as Guido suggested this was very doable with a class decorator. Whether it's a good idea to build it into dataclasses, I don't know -- I suspect not. I'm wary of giving JSON this kind of special place in Python. On the other hand, a "standard" collection of extra tools designed for datacalsses would be pretty nice, and this could belong there. -Chris NOTE: This is all part of a technically open-source, but not published project. But if folks are interested, maybe I could find the time to publish it on it's own. PS: if this does get anywhere -- please call it something like __json__ or __json_encode__ and not plain __encode__ there are any number of ways one might encode something! On Wed, Dec 9, 2020 at 5:38 AM sam bland <sbland.co.uk@gmail.com> wrote:
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
participants (5)
-
Chris Angelico -
Christopher Barker -
Eric V. Smith -
Guido van Rossum -
sam bland