[Patches] protection for marshalling recursive data structures

Vladimir Marangozov Vladimir.Marangozov@inrialpes.fr
Wed, 28 Jun 2000 08:10:41 +0200 (CEST)


Michael Hudson wrote:
> 
> As I really do not have anything better to do at the moment, I've written
> a patch to Python/marshal.c that prevents Python dumping core when trying
> to marshal stack bustingly deep (or recursive) data structure.
> 
> It just throws an exception; even slightly clever handling of recursive
> data is what pickle is for...

I have submitted a patch for this either; see "marshalling recursive objects"
dated May 12-13 on the patches list. The solution I proposed is more
heavyweight than this one because it effectively detects cyclic references.

As Guido noted at that time, my version changed the existing marshal
semantics because I wanted to marshal recursive objects a la pickle.
To which I replied that I have a corrected version which preserves the
existing semantics and asked for advice on what version to resubmit, out
of two: a) accept marshalling recursive objects with backwards compatibility
or b) reject recursive objects by raising an exception. And this is where
the story ended at that time.

So it boils down to which one of these three is most appropriate:

1. Your solution (lightweight) based on a fixed watermark value without
   effectively detecting cyclic references
2. Detect cyclic objects and reject them by raising an exception
3. Detect cyclic objects and accept them by introducing a new marshal
   type code 'o' just like pickle does.

#2 & #3 are more heavy as they require tracking the currently marshalled
objects in a dict (or a list). A priori, #3 is out of scope as it does too
much and overlaps pickle.

So my guess is that Guido has to decide which one he prefers (#1 or #2).
If you're not Guido, you may want to chime in too with an opinion.

#2 hasn't been posted anywhere as I waited for Guido's response...



> 
> Index: marshal.c
> ===================================================================
> RCS file: /cvsroot/python/python/dist/src/Python/marshal.c,v
> retrieving revision 1.47
> diff -u -r1.47 marshal.c
> --- marshal.c	2000/05/03 23:44:39	1.47
> +++ marshal.c	2000/06/19 00:18:44
> @@ -58,6 +58,7 @@
>  typedef struct {
>  	FILE *fp;
>  	int error;
> +	int depth;
>  	/* If fp == NULL, the following are valid: */
>  	PyObject *str;
>  	char *ptr;
> @@ -144,8 +145,13 @@
>  {
>  	int i, n;
>  	PyBufferProcs *pb;
> +
> +	p->depth++;
>  	
> -	if (v == NULL) {
> +	if (p->depth > 5000) {
> +		p->error = 2;
> +	} 
> +	else if (v == NULL) {
>  		w_byte(TYPE_NULL, p);
>  	}
>  	else if (v == Py_None) {
> @@ -301,6 +307,7 @@
>  	WFILE wf;
>  	wf.fp = fp;
>  	wf.error = 0;
> +	wf.depth = 0;
>  	w_long(x, &wf);
>  }
>  
> @@ -690,6 +697,7 @@
>  	wf.ptr = PyString_AS_STRING((PyStringObject *)wf.str);
>  	wf.end = wf.ptr + PyString_Size(wf.str);
>  	wf.error = 0;
> +	wf.depth = 0;
>  	w_object(x, &wf);
>  	if (wf.str != NULL)
>  		_PyString_Resize(&wf.str,
> @@ -697,7 +705,9 @@
>  			   PyString_AS_STRING((PyStringObject *)wf.str)));
>  	if (wf.error) {
>  		Py_XDECREF(wf.str);
> -		PyErr_SetString(PyExc_ValueError, "unmarshallable object");
> +		PyErr_SetString(PyExc_ValueError, 
> +				(wf.error==1)?"unmarshallable object"
> +				:"object too deeply nested to marshal");
>  		return NULL;
>  	}
>  	return wf.str;
> @@ -724,9 +734,12 @@
>  	wf.str = NULL;
>  	wf.ptr = wf.end = NULL;
>  	wf.error = 0;
> +	wf.depth = 0;
>  	w_object(x, &wf);
>  	if (wf.error) {
> -		PyErr_SetString(PyExc_ValueError, "unmarshallable object");
> +		PyErr_SetString(PyExc_ValueError, 
> +				(wf.error==1)?"unmarshallable object"
> +				:"object too deeply nested to marshal");
>  		return NULL;
>  	}
>  	Py_INCREF(Py_None);
> 
> On my machine, it seems an unpatched marshal blows up at a depth of about
> 13000, so I hope 5000 is OK almost everywhere.
> 
> I confirm that, to the best of my knowledge and belief, this
> contribution is free of any claims of third parties under
> copyright, patent or other rights or interests ("claims").  To
> the extent that I have any such claims, I hereby grant to CNRI a
> nonexclusive, irrevocable, royalty-free, worldwide license to
> reproduce, distribute, perform and/or display publicly, prepare
> derivative versions, and otherwise use this contribution as part
> of the Python software and its related documentation, or any
> derivative versions thereof, at no cost to CNRI or its licensed
> users, and to authorize others to do so.
> 
> I acknowledge that CNRI may, at its sole discretion, decide
> whether or not to incorporate this contribution in the Python
> software and its related documentation.  I further grant CNRI
> permission to use my name and other identifying information
> provided to CNRI by me for use in connection with the Python
> software and its related documentation.
> 
> Cheers,
> M.
> 
> 
> _______________________________________________
> Patches mailing list
> Patches@python.org
> http://www.python.org/mailman/listinfo/patches
> 


-- 
       Vladimir MARANGOZOV          | Vladimir.Marangozov@inrialpes.fr
http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252