If for_json is true (not the default), objects with a for_json() method will use the return value of that method for encoding as JSON instead of

This thread discusses a __json__ encoder protocol: "adding support for a "raw output" in JSON serializer" https://mail.python.org/archives/list/python-ideas@python.org/thread/WT6Z6YJ... You actually don't have to subclass JSONEncoder; you can specify a default= method that uses isinstance(), hasattr(), or singledispatch. https://simplejson.readthedocs.io/en/latest/ - "Specializing JSON object encoding:" - https://simplejson.readthedocs.io/en/latest/#simplejson.dump the object. "use __json__ attribute to encode custom objects" https://github.com/simplejson/simplejson/issues/52 Simplejson supports object.for_json(), but only checks objects if for_json=True is specified. FWIW, Jupyter notebook supports object._repr_json_() and e.g. object._repr_pretty_() Dunder methods are name-mangled, and, by convention reserved for internal Python use: if CPython decides that object.__json__() should return a bool for whether a string is already JSON-encoded, and that breaks your excessively-presumptive dunder json method implementation, tough luck. Singledispatch (or, indeed, creating your own utils.json.dump that's a functools.partial that just specifies a default default=) is probably fastest. JSON5 allows things like ±inf and nan. On Mon, Apr 6, 2020, 9:32 AM Dan Cojocaru wrote:

Hello,

I'm not really sure how suggesting improvements to Python goes and I've found this mailing list, so I thought I'd email it.

My suggestion is related to the json built in package, and more exactly about serialising custom classes.

As of now, the process involves using a JSONEncoder class which, by default, supports the following types/values: dict, list, tuple, str, int, float, True, False, None and int- & float-derived Enums.

In order to dump a custom class to JSON, the process involves creating a new class inheriting JSONEncoder and defining the default function in order to handle custom classes.

I say that this approach is very complex and perhaps unpythonic, definitely very weird compared to other approaches.

An alternative that comes to mind and would feel more natural is adding support for recognising a __json__ function. If a class would contain such a function, the default JSONEncoder should take that function's return value instead for encoding, assuming that such returned value will be able to be then serialised.

If such a functionality would be added, many custom functions will most likely return a dict from their __json__ function, skipping the need for creating a class inheriting JSONEncoder altogether.

I hope this idea is a good one and could perhaps even get implemented in the next version of Python.

Respectfully, Dan Cojocaru. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/B5RUXM... Code of Conduct: http://python.org/psf/codeofconduct/

Addressing your last concern, about __json__ being only class -> JSON, not JSON -> class, classes implementing __str__ only go class -> str, not str -> class. Just because we can't fully solve a problem, it doesn't mean we shouldn't attempt to solve (or provide a more convenient to solve) half of it. --- Dan Cojocaru On 6 Apr 2020, 19:02 +0300, Christopher Barker , wrote:

On Mon, Apr 6, 2020 at 8:27 AM Wes Turner wrote:

...

...

This thread discusses a __json__ encoder protocol:

"adding support for a "raw output" in JSON serializer" https://mail.python.org/archives/list/python-ideas@python.org/thread/WT6Z6YJ...

and a numbe rof other issues :-)

This would be a really good time for someone to go through this list and maybe others), and summarize all the discussions around __json__

FWIW, the idea of a __json__ protocol has support, but it's not clear to me if there wasn't interest by core devs, or if it's just that no one has cleaned the discussion up and made it into a clean proposal to be considered.

Note that there IS notable overhead to adding a new dunder like that -- there will be (legitimate) resistance.

...

JSON5 allows things like ±inf and nan.

For my part, I would REALLY like to see the stdlib json module support JSON5 -- it better matches Python. (inf, comments, and trailing commas -- ALL things I really miss with JSON)

Second on my list would be support for Decimal, as that matches teh JSON data model (nothing in there about binary floating point).

As for __json__ -- I get it, but I think I'm -0.5: because it's a one way street -- of you want to go from custom object => JSON, iot's great, but the other way doesn't work, so you'll need to do SOMETHING custom there anyway. So having the JSON module fully support a mapping between the JSON types and Python seems enough to me.

-CHB

-- Christopher Barker, PhD

Python Language Consulting   - Teaching   - Scientific Software Development   - Desktop GUI and Web Development   - wxPython, numpy, scipy, Cython

Python object > JSON > object requires type information to be serialized. Jsonpickle is two-way; with support for numpy, pandas, JS, datetimes, UUIDs, and IDK about e.g. complex fractions https://github.com/jsonpickle/jsonpickle JSONLD can specify type information in a space-efficient way with @context (rather than having to specify e.g. @type=xsd:boolean for every boolean value) JSONLD includes @type URIs that can be e.g. XSD datatypes or https://schema.org types. - JSON is an ECMA spec. - jsonpickle has a python and a js implementation - JSON5 is an unofficial spec with many implementations - JSONLD 1.1 is a W3C candidate recommendation with many implementations (pyld, rdflib-jsonld, ) In order to optionally serialize to any of {JSON, JSON5, JSONLD, JSON_future}, we could: A) accept an optional parameter on a __json__ method: __json__(obj, spec=None) B) establish a convention for registering functions to call with singledispatch in a default method ?) Other ideas? On Mon, Apr 6, 2020, 12:22 PM Dan Cojocaru wrote:

Addressing your last concern, about __json__ being only class -> JSON, not JSON -> class, classes implementing __str__ only go class -> str, not str -> class. Just because we can't fully solve a problem, it doesn't mean we shouldn't attempt to solve (or provide a more convenient to solve) half of it.

--- Dan Cojocaru On 6 Apr 2020, 19:02 +0300, Christopher Barker , wrote:

On Mon, Apr 6, 2020 at 8:27 AM Wes Turner wrote:

...

This thread discusses a __json__ encoder protocol:

"adding support for a "raw output" in JSON serializer"

https://mail.python.org/archives/list/python-ideas@python.org/thread/WT6Z6YJ...

and a numbe rof other issues :-)

This would be a really good time for someone to go through this list and maybe others), and summarize all the discussions around __json__

FWIW, the idea of a __json__ protocol has support, but it's not clear to me if there wasn't interest by core devs, or if it's just that no one has cleaned the discussion up and made it into a clean proposal to be considered.

Note that there IS notable overhead to adding a new dunder like that -- there will be (legitimate) resistance.

...

JSON5 allows things like ±inf and nan.

For my part, I would REALLY like to see the stdlib json module support JSON5 -- it better matches Python. (inf, comments, and trailing commas -- ALL things I really miss with JSON)

Second on my list would be support for Decimal, as that matches teh JSON data model (nothing in there about binary floating point).

As for __json__ -- I get it, but I think I'm -0.5: because it's a one way street -- of you want to go from custom object => JSON, iot's great, but the other way doesn't work, so you'll need to do SOMETHING custom there anyway. So having the JSON module fully support a mapping between the JSON types and Python seems enough to me.

-CHB

-- Christopher Barker, PhD

Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/JYQTHW... Code of Conduct: http://python.org/psf/codeofconduct/

I totally agree, but the JSONEncoder/JSONDecoder mechanism already exist. I'm certainly removing stuff. I'm just proposing adding a simpler way to do one of the things - serialising. An example use case would be creating a simple class for something like a REST API and that class having a __json__ function. Python doesn't need to deserialise the JSON, so you only need one half of the story, the half that __json__ would make easier. --- Dan Cojocaru On 6 Apr 2020, 21:48 +0300, Andrew Barnert , wrote:

On Apr 6, 2020, at 09:22, Dan Cojocaru wrote:

...

Addressing your last concern, about __json__ being only class -> JSON, not JSON -> class, classes implementing __str__ only go class -> str, not str -> class.

Sure, but str is inherently one-way. There’s no unambiguous mapping from human-readable text to data.

That’s not true for JSON; the entire point of it is data interchange. You expect to be able to dump an object, send it over the wire or store it to a file, load it (or even parse it in JS or ObjC or Go or whatever) and get back an equivalent object. It’s easy to come up with ways to build on top of JSON to interchange things like time points or raw binary strings or higher-level structured objects, but they require doing something on both the encode side and the decode side. Just being able to encode them to something human-readable is useless—if I encode a datetime object, I need to get back a datetime (or Date or NSDate or whatever) on the other end, not a str (or string or NSString or whatever) that a human could tell is probably meant to be a datetime but will raise an exception when I try to subtract it from now().

(Of course JSON isn’t perfect, as anyone who’s tried to interchange, say, int64 values discovers… but it’s good enough for many applications.)

I don't see why the need for standardisation exists. Here's another example because I can explain myself best by giving examples: I have an API. I can write some classes that correspond to JSON inputs my API expects and define __json__ in them to convey how they are expected to be serialised. For example, a Person class with first_name and last_name fields being serialised as a dict with the fields in camel case instead. (Sure, normally I'd write a serialisation function as well as part of my package in this case but I'm just coming up with examples on the spot). For non standard things like date and time, many implementations of JSON already added support for it using a format I can't recall at the moment. And even if a class defines __json__ in a particular format but you would need another one, the method of making a custom JSONEncoder still remains, as __json__ functions would be attempted only after the custom default implementation. Therefore, __json__ would enable sensible defaults to be chosen while not harming the possibility of using a custom format if it is so desired. --- Dan Cojocaru On 6 Apr 2020, 22:08 +0300, Chris Angelico , wrote:

On Tue, Apr 7, 2020 at 4:48 AM Andrew Barnert via Python-ideas wrote:

...

That’s not true for JSON; the entire point of it is data interchange. You expect to be able to dump an object, send it over the wire or store it to a file, load it (or even parse it in JS or ObjC or Go or whatever) and get back an equivalent object. It’s easy to come up with ways to build on top of JSON to interchange things like time points or raw binary strings or higher-level structured objects, but they require doing something on both the encode side and the decode side. Just being able to encode them to something human-readable is useless—if I encode a datetime object, I need to get back a datetime (or Date or NSDate or whatever) on the other end, not a str (or string or NSString or whatever) that a human could tell is probably meant to be a datetime but will raise an exception when I try to subtract it from now().

(Of course JSON isn’t perfect, as anyone who’s tried to interchange, say, int64 values discovers… but it’s good enough for many applications.)

The trouble with using JSON to interchange things that aren't in the JSON spec is that you then have to build another layer on top of it - a non-standard layer. That means that whatever you do, it's inherently app-specific. For instance, I might choose to encode a datetime as a string in ISO format, but how is the other end going to know that it was meant to be a date? (Usually, if I'm sending something to some front-end JavaScript, I'll just hard-code the JS to know that thing[0].created is a timestamp, and should be parsed accordingly.)

So if it's app-specific, then the best way to handle it is in your app, not in the data type you're encoding. Subclassing JSONEncoder works for this; adding a __json__ method doesn't really, unless there is some single canonical encoding for a particular object.

The two become close to equivalent when you're only asking about your own custom classes. You can, in fact, create your own private __json__ protocol (although, since it's private to you, it'd probably be better to call the method "to_json" rather than "__json__"), and have a subclass of JSONEncoder that calls it. It'd work fine because you don't NEED to interoperate with other libraries. It's only when you try to standardize something that's inherently nonstandard that things get problematic :)

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4ZEQKT... Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, Apr 6, 2020, 3:37 PM Dan Cojocaru wrote:

I don't see why the need for standardisation exists.

Here's another example because I can explain myself best by giving examples:

I have an API. I can write some classes that correspond to JSON inputs my API expects and define __json__ in them to convey how they are expected to be serialised. For example, a Person class with first_name and last_name fields being serialised as a dict with the fields in camel case instead. (Sure, normally I'd write a serialisation function as well as part of my package in this case but I'm just coming up with examples on the spot).

DRF (Django REST API framework) serializers have a .to_representation() method:

Takes the object instance that requires serialization, and should return a primitive representation. Typically this means returning a structure of built-in Python datatypes. The exact types that can be handled will depend on the render classes you have configured for your API.

May be overridden in order to modify the representation style https://www.django-rest-framework.org/api-guide/serializers/#overriding-seri...

For non standard things like date and time, many implementations of JSON already added support for it using a format I can't recall at the moment. And even if a class defines __json__ in a particular format but you would need another one, the method of making a custom JSONEncoder still remains, as __json__ functions would be attempted only after the custom default implementation. Therefore, __json__ would enable sensible defaults to be chosen while not harming the possibility of using a custom format if it is so desired.

If specified, default should be a function that gets called for objects

In trying to do this (again), I've realized that you *can't* just check for hasattr(obj, '__json__') in a JSONEncoder.default method because default only get's called for types it doesn't know how to serialize; so if you e.g. subclass a dict, default() never gets called for the dict subclass. https://docs.python.org/3/library/json.html : that can’t otherwise be serialized. https://github.com/python/cpython/blob/master/Lib/json/encoder.py https://stackoverflow.com/questions/16405969/how-to-change-json-encoding-beh... specifies how to overload _make_iterencode *in pure Python*, but AFAICS that NOPs use of the C-optimized json encoder. The python json module originally came from simplejson. Here's simplejson's for_json implementation in pure Python: https://github.com/simplejson/simplejson/blob/288e4e005c39a2eb855b5225c5dc8e... : for_json = _for_json and getattr(value, 'for_json', None)

if for_json and callable(for_json): chunks = _iterencode(for_json(), _current_indent_level)

And simplejson's for_json implementation in C: https://github.com/simplejson/simplejson/blob/288e4e005c39a2eb855b5225c5dc8e... Is there a strong reason that the method would need to be called __json__ instead of 'for_json'? Passing a spec=None kwarg through to the __json__()/for_json() method would not be compatible with simplejson's existing implementation. Passing a spec=None kwarg would be necessary to support different JSON standards within the same method; which I think is desirable because JSON/JSON5/JSONLD/JSON_future.

--- Dan Cojocaru On 6 Apr 2020, 22:08 +0300, Chris Angelico , wrote:

On Tue, Apr 7, 2020 at 4:48 AM Andrew Barnert via Python-ideas wrote:

That’s not true for JSON; the entire point of it is data interchange. You expect to be able to dump an object, send it over the wire or store it to a file, load it (or even parse it in JS or ObjC or Go or whatever) and get back an equivalent object. It’s easy to come up with ways to build on top of JSON to interchange things like time points or raw binary strings or higher-level structured objects, but they require doing something on both the encode side and the decode side. Just being able to encode them to something human-readable is useless—if I encode a datetime object, I need to get back a datetime (or Date or NSDate or whatever) on the other end, not a str (or string or NSString or whatever) that a human could tell is probably meant to be a datetime but will raise an exception when I try to subtract it from now().

(Of course JSON isn’t perfect, as anyone who’s tried to interchange, say, int64 values discovers… but it’s good enough for many applications.)

The trouble with using JSON to interchange things that aren't in the JSON spec is that you then have to build another layer on top of it - a non-standard layer. That means that whatever you do, it's inherently app-specific. For instance, I might choose to encode a datetime as a string in ISO format, but how is the other end going to know that it was meant to be a date? (Usually, if I'm sending something to some front-end JavaScript, I'll just hard-code the JS to know that thing[0].created is a timestamp, and should be parsed accordingly.)

So if it's app-specific, then the best way to handle it is in your app, not in the data type you're encoding. Subclassing JSONEncoder works for this; adding a __json__ method doesn't really, unless there is some single canonical encoding for a particular object.

The two become close to equivalent when you're only asking about your own custom classes. You can, in fact, create your own private __json__ protocol (although, since it's private to you, it'd probably be better to call the method "to_json" rather than "__json__"), and have a subclass of JSONEncoder that calls it. It'd work fine because you don't NEED to interoperate with other libraries. It's only when you try to standardize something that's inherently nonstandard that things get problematic :)

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4ZEQKT... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7LTMRW... Code of Conduct: http://python.org/psf/codeofconduct/

Type information (the TBox) can be serialized along with instances or in a separate schema definition. https://en.wikipedia.org/wiki/Tbox On Tue, Apr 7, 2020, 12:17 AM Greg Ewing wrote:

On 7/04/20 4:57 am, Wes Turner wrote:

...

Python object > JSON > object requires type information to be serialized.

Not necessarily -- Java's json library uses reflection on compile time type information to deserialise json into an object hierarchy. You tell it the class corresponding to the top level and it figures out the rest.

Something similar could be done in Python using type annotations.

-- Greg _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/C4DMLE... Code of Conduct: http://python.org/psf/codeofconduct/