Python - remote object protocols and security

Burak Arslan burak.arslan at
Mon Jul 15 17:42:07 CEST 2013

On 07/15/13 13:51, Chris Angelico wrote:
> So the only bit you still need is: How do you transmit this across the
> network? Since it's now all just bytes, that's easy enough to do, eg
> with TCP. But that depends on the rest of your system, and is a quite
> separate question - and quite probably one you already have the answer
> to.

For Json, you need to have a way of delimiting messages -- to my
knowledge, Python's json library does not support parsing streams.

You can send the json document in the body of a Http POST, or a ZeroMQ
message, or in a UDP datagram (if you can guarantee it fits inside one)
or in a simple TCP-based encapsulation mechanism that e.g. prepends the
length of the message to the document.



As MessagePack already does this, you can send MessagePack documents via
an ordinary TCP socket and easily recover them on the other side of the

>>> import msgpack; from StringIO import StringIO
>>> s = StringIO(msgpack.dumps({"a":1}) + msgpack.dumps({"b":2}))
>>> for doc in msgpack.Unpacker(s):
...     print doc
{'a': 1}
{'b': 2}

This won't work with Json:

>>> import json; from StringIO import StringIO
>>> s = StringIO(json.dumps({"a":1}) + json.dumps({"b":2}))
>>> for doc in json.load(s): # or whatever ???
...     print doc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/json/", line 290, in load
  File "/usr/lib64/python2.7/json/", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/", line 368, in decode
    raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 9 - line 1 column 17 (char 8 - 16)

Note that this is a limitation of python's Json parser, not Json itself.

There seems to be a json.scanner module that *sounds* like it provides
this functionality,
but I couldn't find any documentation about it.

Alternatively, PyYaml can also parse streams. yaml.{dump,load}_all()
provide pickle-like unsafe (de)serialization support and
yaml.safe_{dump,load}_all provide msgpack-like safe-but-limited stream
parsing support.


On 07/15/13 13:57, Chris Angelico wrote:
> But what I meant was that the [Json] protocol itself is designed with
> security restrictions in mind. It's designed not to fetch additional
> content from the network (as XML can),

Can you explain how parsing XML can fetch data from the network?


More information about the Python-list mailing list