Python Object Notation (PyON)
I have added support for naming objects in dumps, cross-references and declarative support in dumps/loads too. I can't now commit changes. This night I will commit changes to sources and update project pages. Here are examples of how it works now: == Examples with loads== ===Example with recursive list===
lst = pyon.loads(""" ... lst = ['foo', lst] ... lst""") lst is lst[1] True
===Example with recursive dict===
d = pyon.loads(""" ... d = {'a':'foo', 'b':d} ... d""") d is d['b'] True
===Example with cross-references===
ob = pyon.loads(""" ... lst = ['foo', lst, d] ... d = {'a':'foo', 'b':d, 'c':lst} ... [d, lst]""") lst = ob[1] d = ob[0] lst[1] is lst True d is d['b'] True lst[2] is d True d['c'] is lst True
===Example with recursive class and cross-references===
ob = pyon.loads(""" ... c = C(parent=c, lst=lst, d=d) ... lst = ['foo', lst, d, c] ... d = {'a':'foo', 'b':d, 'c':lst, 'd':c} ... [d, lst, c]""") lst = ob[1] d = ob[0] c = ob[2] c.parent is c True c.lst is lst True c.d is d True d is d['b'] True lst[2] is d True d['c'] is lst True d['d'] is c True lst[3] is c True
p1 = (1,2) p2 = [1,2] pyon.dumps([p1,p2,p1,p2], fast=False) _p__0=(1,2) _p__1=[1,2] [_p__0,_p__1,_p__0,_p__1] pyon.dumps([p1,p2,p1,p2], fast=False, p1=p1,p2=p2)
==Examples with dumps== ===Example with naming and assignments== p1=(1,2) p2=[1,2] [p1,p2,p1,p2] ===Example with recursive list===
lst = ['foo'] lst.append(lst) pyon.dumps(lst, fast=False) _p__0=['foo',_p__0] _p__0 pyon.dumps(lst, fast=False, lst=lst) lst=['foo',lst] lst
===Example with recursive dict===
d = {'a':'foo'} d['b'] = d pyon.dumps(d, fast=False) _p__0={'a':'foo', 'b':_p__0} _p__0 pyon.dumps(d, fast=False, d=d) d={'a':'foo', 'b':d} d
===Example with recursion in class instance== class C(object): def __reduce__(self): return C, (), self.__dict__
c= C() c.parent = c pyon.dumps(c, fast=False) _p__0=C(parent=_p__0) _p__0 pyon.dumps(c, fast=False, c=c) c=C(parent=c) c
===Example with cross-refernce=== class C(object): def __reduce__(self): return C, (), self.__dict__
lst = ['foo'] lst.append(lst) d = {'a':'bar','b':lst} d['c'] = d c = C() c.lst = lst c.d = d c.parent = c d['d'] = c pyon.dumps([lst,d,c], fast = False, lst=lst, d=d) lst=['foo',lst] _p__1=C(lst=lst,d=d,parent=_p__1) d={'a':'bar','c':d,'b':lst,'d':_p__1} [lst,d,_p__1]
Best regards, Zaur
On 05/11/2008, Zaur Shibzoukhov <szport@gmail.com> wrote:
I have added support for naming objects in dumps, cross-references and declarative support in dumps/loads too.
This is great stuff!
I can't now commit changes. This night I will commit changes to sources and update project pages.
Here are examples of how it works now: [...]
pyon.dumps([lst,d,c], fast = False, lst=lst, d=d)
How about changing the syntax slightly so that the given names are all in a dictionary:
pyon.dumps([lst, d, c], fast=False, given={'lst':lst, 'd':d}) (or one could use given=dict(lst=lst, d=d))
This would have two advantages: * eliminate the rist of keyword argument name collision * one could put all the 'given' objects in a dictionary and then 'pickle' expressions as needed using this method. Later pyon.loads could be passed this dictionary so that the objects can be unpickled correctly. I think this idea is good as it would make it possible to pickle some objects that contain unpicklable objects just by declaring them as 'given'. Also, what happens with types? E.g.
pyon.dumps([int, float, str])
I think it would be good if typenames were considered literals (like numbers and strings) so that the above returns '[int, float, str]' (and the same for user-defined types maybe). -- Arnaud
I've been following this discussion and it seems like implementation is way ahead of design. I'm looking at the syntax below and seeing it has no advantage over text pickle format to me. Sure it's a bit more readable, but as it stands, this is really only going to be useful for python-python operations and the existing pickle formats do that fine. Why would we need another one-off markup language? All that you've said so far about Yaml was "I didn't yet made comparison with yaml. I should do that too. Now I can only say that yaml is much complicated notation (IMHO)." In fact, I think a subset of Yaml is all that is necessary for this, just as Json is (now) a subset of Yaml. My suggestion is that Json < Pyon < Yaml is a good goal. I agree Yaml is complicated and the docs don't help as much as they could with too much emphasis on syntax and not enough on semantics. However, Yaml has already addressed issues like typing and defining objects to be reused. I'm not a Yaml expert so I may have gotten some of the following wrong, but it should be enough to give a general idea: [int, float, str] == [!type int, !type float, !type str] or maybe [!py!type int, !py!type float, !py!type str] and _p__0=(1,2) _p__1=[1,2] [_p__0,_p__1,_p__0,_p__1] == [&_p__0 !tuple [1,2], &_p__1 [1,2], *_p__0, *_p__1] or - &_p__0 (1,2) - &_p__1 [1,2] - *_p__0 - *_p__1 and lst=['foo',lst] _p__1=C(lst=lst,d=d,parent=_p__1) d={'a':'bar','c':d,'b':lst,'d':_p__1} [lst,d,_p__1] ==? [&lst ['foo', *lst]], &d{'a','bar','c':*d,'b':*lst,'d':&_p__1 !!C {lst: *lst, d: *d, parent: *_p__1}}] or in a multi-line format: - &lst ['foo, *lst] - &d 'a': 'bar' 'c': *d 'b': *lst 'd': &_p__1 !!C lst: *lst d: *d parent: *_p__1 The only thing I see missing from Yaml is forward references which might be nice for producing more readable output in some cases but I don't think is technically necessary. --- Bruce On Wed, Nov 5, 2008 at 2:57 AM, Arnaud Delobelle <arnodel@googlemail.com>wrote:
On 05/11/2008, Zaur Shibzoukhov <szport@gmail.com> wrote:
I have added support for naming objects in dumps, cross-references and declarative support in dumps/loads too.
This is great stuff!
I can't now commit changes. This night I will commit changes to sources and update project pages.
Here are examples of how it works now: [...]
pyon.dumps([lst,d,c], fast = False, lst=lst, d=d)
How about changing the syntax slightly so that the given names are all in a dictionary:
pyon.dumps([lst, d, c], fast=False, given={'lst':lst, 'd':d}) (or one could use given=dict(lst=lst, d=d))
This would have two advantages:
* eliminate the rist of keyword argument name collision * one could put all the 'given' objects in a dictionary and then 'pickle' expressions as needed using this method. Later pyon.loads could be passed this dictionary so that the objects can be unpickled correctly.
I think this idea is good as it would make it possible to pickle some objects that contain unpicklable objects just by declaring them as 'given'.
<snip>
I think main difference between JSON, PyON and YAML is that both JSON and PyON are notations, YAML is markup language. I consider PyON as an extension of JSON that better reflects python object model such as JSON reflects Javascript object model. If object has no cycles or cross-references then it's representation is clear. In other cases representation similar to "object equation". Best regards, Zaur 2008/11/5 Bruce Leban <bruce@leapyear.org>:
I've been following this discussion and it seems like implementation is way ahead of design. I'm looking at the syntax below and seeing it has no advantage over text pickle format to me. Sure it's a bit more readable, but as it stands, this is really only going to be useful for python-python operations and the existing pickle formats do that fine. Why would we need another one-off markup language? All that you've said so far about Yaml was "I didn't yet made comparison with yaml. I should do that too. Now I can only say that yaml is much complicated notation (IMHO)." In fact, I think a subset of Yaml is all that is necessary for this, just as Json is (now) a subset of Yaml. My suggestion is that Json < Pyon < Yaml is a good goal. I agree Yaml is complicated and the docs don't help as much as they could with too much emphasis on syntax and not enough on semantics. However, Yaml has already addressed issues like typing and defining objects to be reused. I'm not a Yaml expert so I may have gotten some of the following wrong, but it should be enough to give a general idea:
[int, float, str] == [!type int, !type float, !type str] or maybe [!py!type int, !py!type float, !py!type str]
and
_p__0=(1,2) _p__1=[1,2] [_p__0,_p__1,_p__0,_p__1]
==
[&_p__0 !tuple [1,2], &_p__1 [1,2], *_p__0, *_p__1]
or
- &_p__0 (1,2) - &_p__1 [1,2] - *_p__0 - *_p__1
and
lst=['foo',lst] _p__1=C(lst=lst,d=d,parent=_p__1) d={'a':'bar','c':d,'b':lst,'d':_p__1} [lst,d,_p__1]
==?
[&lst ['foo', *lst]], &d{'a','bar','c':*d,'b':*lst,'d':&_p__1 !!C {lst: *lst, d: *d, parent: *_p__1}}]
or in a multi-line format:
- &lst ['foo, *lst] - &d 'a': 'bar' 'c': *d 'b': *lst 'd': &_p__1 !!C lst: *lst d: *d parent: *_p__1
The only thing I see missing from Yaml is forward references which might be nice for producing more readable output in some cases but I don't think is technically necessary.
--- Bruce
On Wed, Nov 5, 2008 at 2:57 AM, Arnaud Delobelle <arnodel@googlemail.com> wrote:
On 05/11/2008, Zaur Shibzoukhov <szport@gmail.com> wrote:
I have added support for naming objects in dumps, cross-references and declarative support in dumps/loads too.
This is great stuff!
I can't now commit changes. This night I will commit changes to sources and update project pages.
Here are examples of how it works now: [...]
pyon.dumps([lst,d,c], fast = False, lst=lst, d=d)
How about changing the syntax slightly so that the given names are all in a dictionary:
pyon.dumps([lst, d, c], fast=False, given={'lst':lst, 'd':d}) (or one could use given=dict(lst=lst, d=d))
This would have two advantages:
* eliminate the rist of keyword argument name collision * one could put all the 'given' objects in a dictionary and then 'pickle' expressions as needed using this method. Later pyon.loads could be passed this dictionary so that the objects can be unpickled correctly.
I think this idea is good as it would make it possible to pickle some objects that contain unpicklable objects just by declaring them as 'given'.
<snip>
Has anyone in this discussion used either Yaml or Json? I've used both. And I much prefer Json. The libraries to encode and decode in most languages worth mentioning are *very* fast (comparable to cPickle). Yaml...not so much. Never mind that Yaml has some ugly bits that make it's use as a configuration language (which isn't uncommon) have some very unpleasant gotchas. Adding *yet another* Python-centric serialization language seems more than a little silly to me; it seems like a waste of time. Of course you are free to develop this method as you see fit, but don't be surprised if you bring it to python-dev and they say, "no." - Josiah On Wed, Nov 5, 2008 at 6:25 AM, Bruce Leban <bruce@leapyear.org> wrote:
I've been following this discussion and it seems like implementation is way ahead of design. I'm looking at the syntax below and seeing it has no advantage over text pickle format to me. Sure it's a bit more readable, but as it stands, this is really only going to be useful for python-python operations and the existing pickle formats do that fine. Why would we need another one-off markup language? All that you've said so far about Yaml was "I didn't yet made comparison with yaml. I should do that too. Now I can only say that yaml is much complicated notation (IMHO)." In fact, I think a subset of Yaml is all that is necessary for this, just as Json is (now) a subset of Yaml. My suggestion is that Json < Pyon < Yaml is a good goal. I agree Yaml is complicated and the docs don't help as much as they could with too much emphasis on syntax and not enough on semantics. However, Yaml has already addressed issues like typing and defining objects to be reused. I'm not a Yaml expert so I may have gotten some of the following wrong, but it should be enough to give a general idea:
[int, float, str] == [!type int, !type float, !type str] or maybe [!py!type int, !py!type float, !py!type str]
and
_p__0=(1,2) _p__1=[1,2] [_p__0,_p__1,_p__0,_p__1]
==
[&_p__0 !tuple [1,2], &_p__1 [1,2], *_p__0, *_p__1]
or
- &_p__0 (1,2) - &_p__1 [1,2] - *_p__0 - *_p__1
and
lst=['foo',lst] _p__1=C(lst=lst,d=d,parent=_p__1) d={'a':'bar','c':d,'b':lst,'d':_p__1} [lst,d,_p__1]
==?
[&lst ['foo', *lst]], &d{'a','bar','c':*d,'b':*lst,'d':&_p__1 !!C {lst: *lst, d: *d, parent: *_p__1}}]
or in a multi-line format:
- &lst ['foo, *lst] - &d 'a': 'bar' 'c': *d 'b': *lst 'd': &_p__1 !!C lst: *lst d: *d parent: *_p__1
The only thing I see missing from Yaml is forward references which might be nice for producing more readable output in some cases but I don't think is technically necessary.
--- Bruce
On Wed, Nov 5, 2008 at 2:57 AM, Arnaud Delobelle <arnodel@googlemail.com> wrote:
On 05/11/2008, Zaur Shibzoukhov <szport@gmail.com> wrote:
I have added support for naming objects in dumps, cross-references and declarative support in dumps/loads too.
This is great stuff!
I can't now commit changes. This night I will commit changes to sources and update project pages.
Here are examples of how it works now: [...]
pyon.dumps([lst,d,c], fast = False, lst=lst, d=d)
How about changing the syntax slightly so that the given names are all in a dictionary:
pyon.dumps([lst, d, c], fast=False, given={'lst':lst, 'd':d}) (or one could use given=dict(lst=lst, d=d))
This would have two advantages:
* eliminate the rist of keyword argument name collision * one could put all the 'given' objects in a dictionary and then 'pickle' expressions as needed using this method. Later pyon.loads could be passed this dictionary so that the objects can be unpickled correctly.
I think this idea is good as it would make it possible to pickle some objects that contain unpicklable objects just by declaring them as 'given'.
<snip> _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
2008/11/5 Josiah Carlson <josiah.carlson@gmail.com>:
Adding *yet another* Python-centric serialization language seems more than a little silly to me; it seems like a waste of time. Of course you are free to develop this method as you see fit, but don't be surprised if you bring it to python-dev and they say, "no." PyON is just that-born. The PyON concept and it's implementation is at an early stage. Therefore, it too early offer python-dev.
Personally I don't consider PyON as another serialization language. I would like to see it as literal object notation based on python syntax. Currently, this is not possible, because it is necessary to expand the syntax of the python language. I don't expect this soon. At this stage PyON uses the existing syntax of python language for the human readable/writeble literal representation of objects. I hope that one day someone will propose to expand the syntax of the language and introduce literal notation for objects representation embedded into python language. Best regards, Zaur
On Wed, Nov 5, 2008 at 11:10 AM, Zaur Shibzoukhov <szport@gmail.com> wrote:
2008/11/5 Josiah Carlson <josiah.carlson@gmail.com>:
Adding *yet another* Python-centric serialization language seems more than a little silly to me; it seems like a waste of time. Of course you are free to develop this method as you see fit, but don't be surprised if you bring it to python-dev and they say, "no." PyON is just that-born. The PyON concept and it's implementation is at an early stage. Therefore, it too early offer python-dev.
Personally I don't consider PyON as another serialization language. I would like to see it as literal object notation based on python syntax. Currently, this is not possible, because it is necessary to expand the syntax of the python language. I don't expect this soon.
At this stage PyON uses the existing syntax of python language for the human readable/writeble literal representation of objects. I hope that one day someone will propose to expand the syntax of the language and introduce literal notation for objects representation embedded into python language.
Python already has a literal syntax for describing objects in-line in the language. It's the syntax itself, and repr(obj) (given proper __repr__ methods) can give you everything you need, except for recursive and multi-referenced objects. I guess I really don't understand the purpose of PyON. Syntactically it doesn't fit between json and yaml. It supports features that are more useful for a serialization language for RPC, etc., rather than configuration/inlining. And it doesn't really offer a reverse of representation -> object without going through the standard Python parser and executing the result (which has security implications). Again, json is very human readable/writable, is very close to literal Python syntax, and already has support in just about every language worth discussing (and Python offers a json loading module in the standard library). Can you give me a good reason why someone would want to choose PyON over json in 6 months? - Josiah
2008/11/6 Josiah Carlson <josiah.carlson@gmail.com>:
I guess I really don't understand the purpose of PyON. Syntactically it doesn't fit between json and yaml. It supports features that are more useful for a serialization language for RPC, etc., rather than configuration/inlining. And it doesn't really offer a reverse of representation -> object without going through the standard Python parser and executing the result (which has security implications). PyON dosn't use exec in order to reconstruct obbject. It uses python parser for constructing AST and then builds object using pickle-like protocol from AST. I guess PyON could be used as standard reconstructable representation of python objects.
Again, json is very human readable/writable, is very close to literal Python syntax, and already has support in just about every language worth discussing (and Python offers a json loading module in the standard library). Can you give me a good reason why someone would want to choose PyON over json in 6 months? If someone need standard way for reconstructable representation of almost every python object based on python syntax then PyON is a possible choice (probably in near future). Also maybe in the future someone will offer better solution for reconstructable representation of python objects based on python syntax.
Also I do not think that PyON should be used instead of JSON. Best regards, Zaur
On 05/11/2008, Josiah Carlson <josiah.carlson@gmail.com> wrote:
Has anyone in this discussion used either Yaml or Json? I've used both. And I much prefer Json. The libraries to encode and decode in most languages worth mentioning are *very* fast (comparable to cPickle). Yaml...not so much. Never mind that Yaml has some ugly bits that make it's use as a configuration language (which isn't uncommon) have some very unpleasant gotchas.
Josiah, A bit OT, but I'm wondering, what kind of unpleasant gotchas are we talking about? I've used YAML a very little bit, but it looked very interesting to me. Could you give a pointer to what unpleasantness to expect? Jan
I ran into some empty string issues, some binary string issues, and I've heard from Guido that he's had some issues with empty lists. These are all trivial cases in json. I also don't know how yaml handles unicode data, but I do know that a good json library will have no issues with unicode. - Josiah On Fri, Nov 7, 2008 at 3:54 AM, Jan Kanis <jan.kanis@phil.uu.nl> wrote:
On 05/11/2008, Josiah Carlson <josiah.carlson@gmail.com> wrote:
Has anyone in this discussion used either Yaml or Json? I've used both. And I much prefer Json. The libraries to encode and decode in most languages worth mentioning are *very* fast (comparable to cPickle). Yaml...not so much. Never mind that Yaml has some ugly bits that make it's use as a configuration language (which isn't uncommon) have some very unpleasant gotchas.
Josiah,
A bit OT, but I'm wondering, what kind of unpleasant gotchas are we talking about? I've used YAML a very little bit, but it looked very interesting to me. Could you give a pointer to what unpleasantness to expect?
Jan
participants (5)
-
Arnaud Delobelle -
Bruce Leban -
Jan Kanis -
Josiah Carlson -
Zaur Shibzoukhov