Propouse add context to json module.
JSON serialization used in many different libraries without the ability for configuration (Example https://github.com/aio-libs/aioredis/blob/8a207609b7f8a33e74c7c8130d97186e78...). Propose to add something like the context in the decimal module, wIch will contain all global configs for dump and load functions. Thanks!
Based on experience with the decimal module, I think this would open a can of worms. To match what decimal does, we would need a Context() object with methods for dump, dumps, load, loads. There would need to be a thread-local or contextvar instance accessed by getcontext and setcontext, and perhaps a decorator as well. We would need a few pre-made instances for common cases. Also, the decimal module was context aware from the outset. For JSON, we have large body of pre-existing client code that was created and tested without the concept of a context. Should existing code use the new context and possibly break assumed invariants? If the existing code had explicit parameters (such as indent=4), would the context override the parameter, take a backseat to the parameter, or raise an exception?
I agree with Raymond. The "implicit context" pattern is really only desirable in a small subset of situations, and I don't think configuring JSON serialization is one of those. (it may even be an active nuisance, because some libraries may be using JSON as an implementation detail and a "JSON context" might break those libraries by changing important settings they rely on) Regards Antoine. On Thu, 06 Aug 2020 20:15:27 -0000 raymond.hettinger@gmail.com wrote:
Maybe use context as a context manager. For example ``` with json.Context(ensure_ascii=False): json.dumps(...) ``` Implementation can be done via contextlib.
On Fri, Aug 7, 2020 at 5:22 PM Kazantcev Andrey <heckad@yandex.ru> wrote:
If all you want is a way to parameterize naive calls to json.dumps(), you could monkeypatch it. import json import contextlib @contextlib.contextmanager def monkeypatch_json(**defaults): orig = json.dumps try: def dumps(*a, **kw): return orig(*a, **{**defaults, **kw}) json.dumps = dumps yield finally: json.dumps = orig Not gonna be 100% reliable and I don't think it belongs in the stdlib, but might be useful. ChrisA
On Fri, Aug 7, 2020 at 5:50 PM Kazantcev Andrey <heckad@yandex.ru> wrote:
I don't see that often, do you have a specific example? Worst case, you could monkeypatch json.dumps with a version that respects a set of defaults, prior to importing the library in question. import json orig = json.dumps def dumps(*a, **kw): return orig(*a, **{**json.defaults, **kw}) json.dumps = dumps json.defaults = {} Definitely hacky though (and you absolutely have to get your imports in the right order) and I would much prefer not to do this. But it's possible. ChrisA
The problem in this code lib.py ``` from json import dumps def some_func(): # do something res = dumps(...) # do something ``` If I patch dumps like you propose lib doesn't see any change. Also, it's all hacks. I wish dump and load themselves could take parameters from somewhere else, and that was the standard behaviour.
On Fri, Aug 7, 2020 at 6:24 PM Kazantcev Andrey <heckad@yandex.ru> wrote:
PLEASE include context when you reply. Yes, I'm aware that my original demo wouldn't work with this kind of thing. But (1) does that actually happen? and (2) my second example will work, as long as you do it before this library imports anything from json. Why do you want dump and load to take parameters from "somewhere else"? As a general rule, it's better for them to get their parameters entirely from, well, their parameters. What you're trying to do is override someone else's code, and that basically means monkeypatching the library, or monkeypatching the json module. Take your pick. ChrisA
Chris Angelico wrote:
Why do you want dump and load to take parameters from "somewhere else"?
Because developers of libraries don't think about configuring json.dump method in mos of cases. I would like to have a mechanism that would allow tweaking the behaviour for the entire program. I just don't know how best to do it. The idea with the configuration via the context manager seems to me to be good. I know exactly what code will be patched and I can catch the bug.
On Fri, 07 Aug 2020 08:54:37 -0000 "Kazantcev Andrey" <heckad@yandex.ru> wrote:
Then you should report a bug to them to give access to such configuration, where that makes sense.
I would like to have a mechanism that would allow tweaking the behaviour for the entire program.
That's an anti-pattern IMO. Regards Antoine.
On Fri, 7 Aug 2020 at 09:26, Kazantcev Andrey <heckad@yandex.ru> wrote:
That's such a general statement, you could probably use it for *any* function, at some point. Which makes your argument pretty weak, IMO. And to be honest, the standard Python way of providing that behaviour is monkeypatching. It feels like a hack because the general view is that needing to do it is a sign of a badly designed API, is all. For your example, if wanting to change the format of dumps is an important feature, lib.py "should" have been designed with a configuration mechanism that allows users to choose that format. The fact that it doesn't either implies that the author of lib.py doesn't want you to do that, or that they didn't think of it. Monkeypatching allows you to address that limitation, so in that sense it's a flexible but advanced mechanism, not a hack. It's not particularly hard (as I'm sure you realise): def custom_dumps(obj): json.dumps(obj, param1=preference1, ...) old_val = lib.dumps try: lib.some_func() finally: lib.dumps = old_val That needs an understanding of how lib.py works, and it needs you to accept the risk of using the library in a way that's not supported, but it's a perfectly reasonable approach for an unexpected case, IMO. Paul
06.08.20 11:03, Kazantcev Andrey пише:
JSON serialization used in many different libraries without the ability for configuration (Example https://github.com/aio-libs/aioredis/blob/8a207609b7f8a33e74c7c8130d97186e78...). Propose to add something like the context in the decimal module, wIch will contain all global configs for dump and load functions.
Changing global defaults is a bad idea, because it would break all libraries which use the json module. As for a local context which incorporates configuration for serializing and deserializing, you already have two context classes. They are called JSONEncoder and JSONDecoder.
Based on experience with the decimal module, I think this would open a can of worms. To match what decimal does, we would need a Context() object with methods for dump, dumps, load, loads. There would need to be a thread-local or contextvar instance accessed by getcontext and setcontext, and perhaps a decorator as well. We would need a few pre-made instances for common cases. Also, the decimal module was context aware from the outset. For JSON, we have large body of pre-existing client code that was created and tested without the concept of a context. Should existing code use the new context and possibly break assumed invariants? If the existing code had explicit parameters (such as indent=4), would the context override the parameter, take a backseat to the parameter, or raise an exception?
I agree with Raymond. The "implicit context" pattern is really only desirable in a small subset of situations, and I don't think configuring JSON serialization is one of those. (it may even be an active nuisance, because some libraries may be using JSON as an implementation detail and a "JSON context" might break those libraries by changing important settings they rely on) Regards Antoine. On Thu, 06 Aug 2020 20:15:27 -0000 raymond.hettinger@gmail.com wrote:
Maybe use context as a context manager. For example ``` with json.Context(ensure_ascii=False): json.dumps(...) ``` Implementation can be done via contextlib.
On Fri, Aug 7, 2020 at 5:22 PM Kazantcev Andrey <heckad@yandex.ru> wrote:
If all you want is a way to parameterize naive calls to json.dumps(), you could monkeypatch it. import json import contextlib @contextlib.contextmanager def monkeypatch_json(**defaults): orig = json.dumps try: def dumps(*a, **kw): return orig(*a, **{**defaults, **kw}) json.dumps = dumps yield finally: json.dumps = orig Not gonna be 100% reliable and I don't think it belongs in the stdlib, but might be useful. ChrisA
On Fri, Aug 7, 2020 at 5:50 PM Kazantcev Andrey <heckad@yandex.ru> wrote:
I don't see that often, do you have a specific example? Worst case, you could monkeypatch json.dumps with a version that respects a set of defaults, prior to importing the library in question. import json orig = json.dumps def dumps(*a, **kw): return orig(*a, **{**json.defaults, **kw}) json.dumps = dumps json.defaults = {} Definitely hacky though (and you absolutely have to get your imports in the right order) and I would much prefer not to do this. But it's possible. ChrisA
The problem in this code lib.py ``` from json import dumps def some_func(): # do something res = dumps(...) # do something ``` If I patch dumps like you propose lib doesn't see any change. Also, it's all hacks. I wish dump and load themselves could take parameters from somewhere else, and that was the standard behaviour.
On Fri, Aug 7, 2020 at 6:24 PM Kazantcev Andrey <heckad@yandex.ru> wrote:
PLEASE include context when you reply. Yes, I'm aware that my original demo wouldn't work with this kind of thing. But (1) does that actually happen? and (2) my second example will work, as long as you do it before this library imports anything from json. Why do you want dump and load to take parameters from "somewhere else"? As a general rule, it's better for them to get their parameters entirely from, well, their parameters. What you're trying to do is override someone else's code, and that basically means monkeypatching the library, or monkeypatching the json module. Take your pick. ChrisA
Chris Angelico wrote:
Why do you want dump and load to take parameters from "somewhere else"?
Because developers of libraries don't think about configuring json.dump method in mos of cases. I would like to have a mechanism that would allow tweaking the behaviour for the entire program. I just don't know how best to do it. The idea with the configuration via the context manager seems to me to be good. I know exactly what code will be patched and I can catch the bug.
On Fri, 07 Aug 2020 08:54:37 -0000 "Kazantcev Andrey" <heckad@yandex.ru> wrote:
Then you should report a bug to them to give access to such configuration, where that makes sense.
I would like to have a mechanism that would allow tweaking the behaviour for the entire program.
That's an anti-pattern IMO. Regards Antoine.
On Fri, 7 Aug 2020 at 09:26, Kazantcev Andrey <heckad@yandex.ru> wrote:
That's such a general statement, you could probably use it for *any* function, at some point. Which makes your argument pretty weak, IMO. And to be honest, the standard Python way of providing that behaviour is monkeypatching. It feels like a hack because the general view is that needing to do it is a sign of a badly designed API, is all. For your example, if wanting to change the format of dumps is an important feature, lib.py "should" have been designed with a configuration mechanism that allows users to choose that format. The fact that it doesn't either implies that the author of lib.py doesn't want you to do that, or that they didn't think of it. Monkeypatching allows you to address that limitation, so in that sense it's a flexible but advanced mechanism, not a hack. It's not particularly hard (as I'm sure you realise): def custom_dumps(obj): json.dumps(obj, param1=preference1, ...) old_val = lib.dumps try: lib.some_func() finally: lib.dumps = old_val That needs an understanding of how lib.py works, and it needs you to accept the risk of using the library in a way that's not supported, but it's a perfectly reasonable approach for an unexpected case, IMO. Paul
06.08.20 11:03, Kazantcev Andrey пише:
JSON serialization used in many different libraries without the ability for configuration (Example https://github.com/aio-libs/aioredis/blob/8a207609b7f8a33e74c7c8130d97186e78...). Propose to add something like the context in the decimal module, wIch will contain all global configs for dump and load functions.
Changing global defaults is a bad idea, because it would break all libraries which use the json module. As for a local context which incorporates configuration for serializing and deserializing, you already have two context classes. They are called JSONEncoder and JSONDecoder.
participants (7)
-
Antoine Pitrou -
Chris Angelico -
Kazantcev Andrey -
Marco Sulla -
Paul Moore -
raymond.hettinger＠gmail.com -
Serhiy Storchaka