On Mon, Apr 6, 2020 at 8:27 AM Wes Turner wrote:

This thread discusses a __json__ encoder protocol:

"adding support for a "raw output" in JSON serializer"

https://mail.python.org/archives/list/python-ideas@python.org/thread/WT6Z6YJ...

and a numbe rof other issues :-) This would be a really good time for someone to go through this list and maybe others), and summarize all the discussions around __json__ FWIW, the idea of a __json__ protocol has support, but it's not clear to me if there wasn't interest by core devs, or if it's just that no one has cleaned the discussion up and made it into a clean proposal to be considered. Note that there IS notable overhead to adding a new dunder like that -- there will be (legitimate) resistance.

JSON5 allows things like ±inf and nan.

For my part, I would REALLY like to see the stdlib json module support JSON5 -- it better matches Python. (inf, comments, and trailing commas -- ALL things I really miss with JSON) Second on my list would be support for Decimal, as that matches teh JSON data model (nothing in there about binary floating point). As for __json__ -- I get it, but I think I'm -0.5: because it's a one way street -- of you want to go from custom object => JSON, iot's great, but the other way doesn't work, so you'll need to do SOMETHING custom there anyway. So having the JSON module fully support a mapping between the JSON types and Python seems enough to me. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Mon, Apr 6, 2020 at 9:18 AM Dan Cojocaru wrote:

Addressing your last concern, about __json__ being only class -> JSON, not JSON -> class, classes implementing __str__ only go class -> str, not str -> class. Just because we can't fully solve a problem, it doesn't mean we shouldn't attempt to solve (or provide a more convenient to solve) half of it.

That's why I'm only -0.5 :-) But __str__ is pretty key to a lot of Python usage, for every type of object, etc. JSON is a far more specialized use case. Which I think is the argument against it: it's adding overhead to a lot, in order to support one particular special use case. Anyway, my primary suggestion at this point, if someone wants to pursue this, is to go back and gather all the arguments for and against that have already been made -- there really has been a LOT of discussion about this already! -CHB

--- Dan Cojocaru On 6 Apr 2020, 19:02 +0300, Christopher Barker , wrote:

On Mon, Apr 6, 2020 at 8:27 AM Wes Turner wrote:

...

This thread discusses a __json__ encoder protocol:

"adding support for a "raw output" in JSON serializer"

https://mail.python.org/archives/list/python-ideas@python.org/thread/WT6Z6YJ...

and a numbe rof other issues :-)

This would be a really good time for someone to go through this list and maybe others), and summarize all the discussions around __json__

FWIW, the idea of a __json__ protocol has support, but it's not clear to me if there wasn't interest by core devs, or if it's just that no one has cleaned the discussion up and made it into a clean proposal to be considered.

Note that there IS notable overhead to adding a new dunder like that -- there will be (legitimate) resistance.

...

JSON5 allows things like ±inf and nan.

For my part, I would REALLY like to see the stdlib json module support JSON5 -- it better matches Python. (inf, comments, and trailing commas -- ALL things I really miss with JSON)

Second on my list would be support for Decimal, as that matches teh JSON data model (nothing in there about binary floating point).

As for __json__ -- I get it, but I think I'm -0.5: because it's a one way street -- of you want to go from custom object => JSON, iot's great, but the other way doesn't work, so you'll need to do SOMETHING custom there anyway. So having the JSON module fully support a mapping between the JSON types and Python seems enough to me.

-CHB

-- Christopher Barker, PhD

Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On 7/04/20 4:57 am, Wes Turner wrote:

Python object > JSON  > object requires type information to be serialized.

Not necessarily -- Java's json library uses reflection on compile time type information to deserialise json into an object hierarchy. You tell it the class corresponding to the top level and it figures out the rest. Something similar could be done in Python using type annotations. -- Greg

On Mon, Apr 6, 2020 at 9:17 PM Greg Ewing wrote:

Something similar could be done in Python using type annotations.

sure can -- I've done this wrapping dataclasses -- each object in the hierarchy knows how to serialize/deserialize itself from JSON -- so there JSON can be totally standard. -CHB

-- Greg _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/C4DMLE... Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

I totally agree, but the JSONEncoder/JSONDecoder mechanism already exist. I'm certainly removing stuff. I'm just proposing adding a simpler way to do one of the things - serialising. An example use case would be creating a simple class for something like a REST API and that class having a __json__ function. Python doesn't need to deserialise the JSON, so you only need one half of the story, the half that __json__ would make easier. --- Dan Cojocaru On 6 Apr 2020, 21:48 +0300, Andrew Barnert , wrote:

On Apr 6, 2020, at 09:22, Dan Cojocaru wrote:

...

Addressing your last concern, about __json__ being only class -> JSON, not JSON -> class, classes implementing __str__ only go class -> str, not str -> class.

Sure, but str is inherently one-way. There’s no unambiguous mapping from human-readable text to data.

That’s not true for JSON; the entire point of it is data interchange. You expect to be able to dump an object, send it over the wire or store it to a file, load it (or even parse it in JS or ObjC or Go or whatever) and get back an equivalent object. It’s easy to come up with ways to build on top of JSON to interchange things like time points or raw binary strings or higher-level structured objects, but they require doing something on both the encode side and the decode side. Just being able to encode them to something human-readable is useless—if I encode a datetime object, I need to get back a datetime (or Date or NSDate or whatever) on the other end, not a str (or string or NSString or whatever) that a human could tell is probably meant to be a datetime but will raise an exception when I try to subtract it from now().

(Of course JSON isn’t perfect, as anyone who’s tried to interchange, say, int64 values discovers… but it’s good enough for many applications.)

On Tue, Apr 7, 2020 at 4:48 AM Andrew Barnert via Python-ideas wrote:

That’s not true for JSON; the entire point of it is data interchange. You expect to be able to dump an object, send it over the wire or store it to a file, load it (or even parse it in JS or ObjC or Go or whatever) and get back an equivalent object. It’s easy to come up with ways to build on top of JSON to interchange things like time points or raw binary strings or higher-level structured objects, but they require doing something on both the encode side and the decode side. Just being able to encode them to something human-readable is useless—if I encode a datetime object, I need to get back a datetime (or Date or NSDate or whatever) on the other end, not a str (or string or NSString or whatever) that a human could tell is probably meant to be a datetime but will raise an exception when I try to subtract it from now().

(Of course JSON isn’t perfect, as anyone who’s tried to interchange, say, int64 values discovers… but it’s good enough for many applications.)

The trouble with using JSON to interchange things that aren't in the JSON spec is that you then have to build another layer on top of it - a non-standard layer. That means that whatever you do, it's inherently app-specific. For instance, I might choose to encode a datetime as a string in ISO format, but how is the other end going to know that it was meant to be a date? (Usually, if I'm sending something to some front-end JavaScript, I'll just hard-code the JS to know that thing[0].created is a timestamp, and should be parsed accordingly.) So if it's app-specific, then the best way to handle it is in your app, not in the data type you're encoding. Subclassing JSONEncoder works for this; adding a __json__ method doesn't really, unless there is some single canonical encoding for a particular object. The two become close to equivalent when you're only asking about your own custom classes. You can, in fact, create your own private __json__ protocol (although, since it's private to you, it'd probably be better to call the method "to_json" rather than "__json__"), and have a subclass of JSONEncoder that calls it. It'd work fine because you don't NEED to interoperate with other libraries. It's only when you try to standardize something that's inherently nonstandard that things get problematic :) ChrisA

On Mon, Apr 6, 2020, 3:37 PM Dan Cojocaru wrote:

I don't see why the need for standardisation exists.

Here's another example because I can explain myself best by giving examples:

I have an API. I can write some classes that correspond to JSON inputs my API expects and define __json__ in them to convey how they are expected to be serialised. For example, a Person class with first_name and last_name fields being serialised as a dict with the fields in camel case instead. (Sure, normally I'd write a serialisation function as well as part of my package in this case but I'm just coming up with examples on the spot).

DRF (Django REST API framework) serializers have a .to_representation() method:

Takes the object instance that requires serialization, and should return a primitive representation. Typically this means returning a structure of built-in Python datatypes. The exact types that can be handled will depend on the render classes you have configured for your API.

May be overridden in order to modify the representation style https://www.django-rest-framework.org/api-guide/serializers/#overriding-seri...

For non standard things like date and time, many implementations of JSON already added support for it using a format I can't recall at the moment. And even if a class defines __json__ in a particular format but you would need another one, the method of making a custom JSONEncoder still remains, as __json__ functions would be attempted only after the custom default implementation. Therefore, __json__ would enable sensible defaults to be chosen while not harming the possibility of using a custom format if it is so desired.

If specified, default should be a function that gets called for objects

In trying to do this (again), I've realized that you *can't* just check for hasattr(obj, '__json__') in a JSONEncoder.default method because default only get's called for types it doesn't know how to serialize; so if you e.g. subclass a dict, default() never gets called for the dict subclass. https://docs.python.org/3/library/json.html : that can’t otherwise be serialized. https://github.com/python/cpython/blob/master/Lib/json/encoder.py https://stackoverflow.com/questions/16405969/how-to-change-json-encoding-beh... specifies how to overload _make_iterencode *in pure Python*, but AFAICS that NOPs use of the C-optimized json encoder. The python json module originally came from simplejson. Here's simplejson's for_json implementation in pure Python: https://github.com/simplejson/simplejson/blob/288e4e005c39a2eb855b5225c5dc8e... : for_json = _for_json and getattr(value, 'for_json', None)

if for_json and callable(for_json): chunks = _iterencode(for_json(), _current_indent_level)

And simplejson's for_json implementation in C: https://github.com/simplejson/simplejson/blob/288e4e005c39a2eb855b5225c5dc8e... Is there a strong reason that the method would need to be called __json__ instead of 'for_json'? Passing a spec=None kwarg through to the __json__()/for_json() method would not be compatible with simplejson's existing implementation. Passing a spec=None kwarg would be necessary to support different JSON standards within the same method; which I think is desirable because JSON/JSON5/JSONLD/JSON_future.

--- Dan Cojocaru On 6 Apr 2020, 22:08 +0300, Chris Angelico , wrote:

On Tue, Apr 7, 2020 at 4:48 AM Andrew Barnert via Python-ideas wrote:

That’s not true for JSON; the entire point of it is data interchange. You expect to be able to dump an object, send it over the wire or store it to a file, load it (or even parse it in JS or ObjC or Go or whatever) and get back an equivalent object. It’s easy to come up with ways to build on top of JSON to interchange things like time points or raw binary strings or higher-level structured objects, but they require doing something on both the encode side and the decode side. Just being able to encode them to something human-readable is useless—if I encode a datetime object, I need to get back a datetime (or Date or NSDate or whatever) on the other end, not a str (or string or NSString or whatever) that a human could tell is probably meant to be a datetime but will raise an exception when I try to subtract it from now().

(Of course JSON isn’t perfect, as anyone who’s tried to interchange, say, int64 values discovers… but it’s good enough for many applications.)

The trouble with using JSON to interchange things that aren't in the JSON spec is that you then have to build another layer on top of it - a non-standard layer. That means that whatever you do, it's inherently app-specific. For instance, I might choose to encode a datetime as a string in ISO format, but how is the other end going to know that it was meant to be a date? (Usually, if I'm sending something to some front-end JavaScript, I'll just hard-code the JS to know that thing[0].created is a timestamp, and should be parsed accordingly.)

So if it's app-specific, then the best way to handle it is in your app, not in the data type you're encoding. Subclassing JSONEncoder works for this; adding a __json__ method doesn't really, unless there is some single canonical encoding for a particular object.

The two become close to equivalent when you're only asking about your own custom classes. You can, in fact, create your own private __json__ protocol (although, since it's private to you, it'd probably be better to call the method "to_json" rather than "__json__"), and have a subclass of JSONEncoder that calls it. It'd work fine because you don't NEED to interoperate with other libraries. It's only when you try to standardize something that's inherently nonstandard that things get problematic :)

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4ZEQKT... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7LTMRW... Code of Conduct: http://python.org/psf/codeofconduct/