On Mon, Jul 13, 2020 at 12:03 PM Edwin Zimmerman <edwin@211mainstreet.net> wrote:

I would have interest in it.


OK -- I'll see what I can do about pulling it out and putting it on gitHub.

Not sure I'll have the time to clean it up and make a nice package out of it, but maybe there's some ideas in there worth sharing.

-CHB


 

 

--Edwin

 

 

I'm no security expert, but we've got a big pile of serialization code that is kind of like JSON-pickly, but it will only deserialize known objects. it's a bit of pain to declare what you want to work with, but it seems safer.

 

I also have a newer system (built on top of dataclasses) that serializes to JSON, but it's "pure" JSON -- it does not store any info about object types or anything in the JSON. Rather, the deserializer knows what it expects, and can only unpack that. e.g., you set a class attribute to the type "List_of_Object_A" -- and then it will use Object_A's deserializer to try to unpack the JSON.

 

It does mean you can't "just save" an arbitrary object, but you can use a decorator on a datacalsss to make any object savable.

 

I did it this way because I want the JSON to be plain old JSON, but I also think it's substantially more secure.

 

part of a non-released code base, but iof folks think it would be handy, I can pull it out into its own package.

 

-CHB

 

 

 

On Mon, Jul 13, 2020 at 9:54 AM Wes Turner <wes.turner@gmail.com> wrote:

Looks like pyro4 (python remote objects) has moved to the serpent library (as.literal_eval) [1]

 

> defaults to a safe serializer (serpent https://pypi.python.org/pypi/serpent ) that supports many Python data types.

> supports different serializers (serpent, json, marshal, msgpack, pickle, cloudpickle, dill)

 

And pyro5 has removed support for unsafe serializers [5]:

 

> no support for unsafe serializers AT ALL (pickle, dill, cloudpickle) - only safe serializers (serpent, marshal, json, msgpack)

> 

> for now, requires msgpack to be installed as well as serpent.

 

TBH, I'm not sure how much any of the serialization protocols have been fuzzed for safety purposes if at all.

 

 

 

(FWIW, PyArrow has very fast support for "Streaming, Serialization, and IPC" 

 https://arrow.apache.org/docs/python/ipc.html and support for "Arbitrary Object Serialization" https://arrow.apache.org/docs/python/ipc.html#arbitrary-object-serialization w/ pyarrow.serialize() and pyarrow.deserialize())

 

On Mon, Jul 13, 2020, 7:57 AM Chris Angelico <rosuav@gmail.com> wrote:

On Mon, Jul 13, 2020 at 8:58 PM Edwin Zimmerman <edwin@211mainstreet.net> wrote:
>
> On 7/11/2020 11:17 PM, Greg Ewing wrote:
>
> On 12/07/20 1:01 pm, Edwin Zimmerman wrote:
>
> As I see it, the unsafe callables (eval, exec, os.system, etc) are generally functions, and safe ones(int, list, dict) are generally classes, though there certainly would be exceptions.
>
>
> Where security is concerned, "there certainly would be exceptions"
> are not words you want to hear.
>
> Agreed, that is why pickle should almost never be used.  In the past, I have looked long and hard at using pickle in my own projects, but was always turned away because of its potential for security issues.  I've thought for years that pickle is a major security foot gun, and I think that not allowing this by default:
> >>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
> would be a step in the right direction.
>

A pickle file (or equivalent blob in a database, or whatever) should
be considered equally as trusted as your source code. If you're
writing out a file that has the exact same access permissions as your
own source code, and then reading it back, you shouldn't have to worry
about pickle's safety any more than you worry about your code's safety
- anyone who could maliciously craft something for you to unpickle
could equally just edit the source code directly.

ChrisA
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-leave@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/HRJVLRL4FDF4MQ6EPEHC36P6CHZSQBZ3/
Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-leave@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7CLM57BK3YRFOOZBSEIWV65EKPQLZXPJ/
Code of Conduct: http://python.org/psf/codeofconduct/



--

Christopher Barker, PhD

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython



--
Christopher Barker, PhD

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython