Pickle in a POST/GET request give EOFError
Michael Ricordeau
michael.ricordeau at gmail.com
Thu Nov 18 05:26:00 EST 2010
When loading from json datas with simplejson, strings are unicode .
Le Thu, 18 Nov 2010 11:22:30 +0100,
Romaric DEFAUX <rde at audaxis.com> a écrit :
> Hi again,
>
> I try simplejson, but I have a problem. (I just replace pickle
> everywhere by simplejson)
>
> Here's a test I did :
> # python
> Python 2.5.2 (r252:60911, Jan 20 2010, 21:48:48)
> [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import simplejson
> >>> disk_list = []
> >>> disk = {}
> >>> disk['name'] = "disk1"
> >>> disk['size'] = "52428800"
> >>> disk_list.append(disk)
> >>> print disk_list
> [{'name': 'disk1', 'size': '52428800'}]
> >>> disk = {}
> >>> disk['name'] = "disk2"
> >>> disk['size'] = "647648"
> >>> disk_list.append(disk)
> >>> print disk_list
> [{'name': 'disk1', 'size': '52428800'}, {'name': 'disk2', 'size': '647648'}]
> >>> disk_list_json = simplejson.dumps(disk_list)
> >>> print disk_list_json
> [{"name": "disk1", "size": "52428800"}, {"name": "disk2", "size": "647648"}]
> >>> disk_list2 = simplejson.loads(disk_list_json)
> >>> print disk_list2
> [{u'name': u'disk1', u'size': u'52428800'}, {u'name': u'disk2', u'size':
> u'647648'}]
> >>> print repr(disk_list_json)
> '[{"name": "disk1", "size": "52428800"}, {"name": "disk2", "size":
> "647648"}]'
>
> Explanation :
> I "dumps" with json a list of dictionnaries into disk_list_json.
> When I "loads" it, I don't get my list of dictionnaries like before (see
> disk_list2). It adds "u" letter everywhere.
> Why ?
> Thanks for help !
>
> Romaric Defaux
>
> Le 18/11/2010 09:43, Michael Ricordeau a écrit :
> > Hi,
> >
> > you can use json for passing list and dict .
> > Pickle is dangerous .
> >
> > Instead of pickle.loads/pickle.dumps use json.loads and json.dumps
> > (using stdlib json in python>= 2.6 or simplejson in python< 2.6)
> >
> > Regards
> >
> >
> >
> > Le Thu, 18 Nov 2010 09:29:00 +0100,
> > Romaric DEFAUX<rde at audaxis.com> a écrit :
> >
> >
> >> Le 17/11/2010 18:52, geremy condra a écrit :
> >>
> >>> On Wed, Nov 17, 2010 at 6:44 AM, Romaric DEFAUX<rde at audaxis.com> wrote:
> >>>
> >>>
> >>>> Le 16/11/2010 17:47, Romaric DEFAUX a écrit :
> >>>>
> >>>>
> >>>>> Hi everybody !
> >>>>>
> >>>>> First time I write to this mailing list :)
> >>>>> I started writing in python last week, that's probably why I can't
> >>>>> understand the following problem...
> >>>>>
> >>>>>
> >>>>> I create a list called web_site_list.
> >>>>> This list contain dictionaries called web_site.
> >>>>> And some values in this dictionaries are list too.
> >>>>>
> >>>>> I do that in a function and I return this :
> >>>>> return pickle.dumps(web_site_list)
> >>>>>
> >>>>> This is working fine :)
> >>>>>
> >>>>> If I do :
> >>>>> print "%s" % pickle.loads(system.get_web_site_list())
> >>>>>
> >>>>> I've got the right stuffs. For example it returns :
> >>>>> [{'documentroot_size': '120', 'servername': '---default---', 'client':
> >>>>> 'undefined', 'documentroot': '/var/www/', 'client_contact': 'undefined',
> >>>>> 'serveralias': []}]
> >>>>>
> >>>>> I send this to a web service. I send it like that :
> >>>>> #I put it in params
> >>>>> def system_updateweb_site(server, login, password):
> >>>>> params = {}
> >>>>> params['login'] = login
> >>>>> params['password'] = password
> >>>>> params['action'] = 'updateweb_site'
> >>>>> params['servername'] = get_servername()
> >>>>> params['hosted_web_site'] = get_web_site_list()
> >>>>> return call_system_ws(server, params)
> >>>>>
> >>>>> #Here's how I send it (I tried in GET and POST)
> >>>>> def call_system_ws(host, params):
> >>>>> query_string = urllib.urlencode(params)
> >>>>> #GET
> >>>>> # f = urllib.urlopen("http://%s/ws?%s" % (host, query_string))
> >>>>> #POST
> >>>>> f = urllib.urlopen("http://%s/ws" % (host), query_string)
> >>>>> result = f.readline().strip()
> >>>>> if result == 'ERROR':
> >>>>> msg = f.readline().strip()
> >>>>> return (False, msg)
> >>>>> return (True, result)
> >>>>>
> >>>>>
> >>>>> On the server side :
> >>>>> if action == 'updateweb_site':
> >>>>> if not (fields.has_key('servername') and
> >>>>> fields.has_key('hosted_web_site')):
> >>>>> raise WSError('missing parameter :
> >>>>> servername or hosted_web_site')
> >>>>> log ('ERROR : missing parameter :
> >>>>> servername or hosted_web_site')
> >>>>> else:
> >>>>>
> >>>>> servername=g.db.escape_string(fields['servername'])
> >>>>>
> >>>>> hosted_web_site=g.db.escape_string(fields['hosted_web_site'])
> >>>>> output =
> >>>>> systemserver.updateweb_site(cursor, servername, hosted_web_site)
> >>>>>
> >>>>> In systemserver.py :
> >>>>> def updateweb_site(cursor, host, hosted_web_site):
> >>>>> web_site_list = pickle.loads(hosted_web_site)
> >>>>> return "%s" % (web_site_list)
> >>>>>
> >>>>> I catch this error :*
> >>>>>
> >>>>> <type 'exceptions.EOFError'>*:
> >>>>>
> >>>>> args = ()
> >>>>> message = ''
> >>>>>
> >>>>> Why ?
> >>>>>
> >>>>> If I just print hosted_web_site, I get this on my web page :
> >>>>>
> >>>>>
> >>>>> (lp0\n(dp1\nS\'documentroot_size\'\np2\nS\'120\'\np3\nsS\'servername\'\np4\nS\'default\'\np5\nsS\'client\'\np6\nS\'undefined\'\np7\nsS\'documentroot\'\np8\nS\'/var/www/\'\np9\nsS\'client_contact\'\np10\ng7\nsS\'serveralias\'\np11\n(lp12\nsa.
> >>>>>
> >>>>> It's the "pickled view" of
> >>>>> [{'documentroot_size': '120', 'servername': '---default---', 'client':
> >>>>> 'undefined', 'documentroot': '/var/www/', 'client_contact': 'undefined',
> >>>>> 'serveralias': []}]
> >>>>>
> >>>>> Can someone help me please ? I spend my afternoon to google to try to find
> >>>>> a solution...
> >>>>>
> >>>>>
> >>>>> Thanks in advance !!!
> >>>>>
> >>>>> Romaric Defaux
> >>>>>
> >>>>>
> >>>>>
> >>>> After entirely rewrite my code to not use Web service but socket (a real
> >>>> client/server program) I finally found the problem... And it's not linked to
> >>>> the POST or GET method...
> >>>> It's because of that :
> >>>> g.db.escape_string(fields['hosted_web_site'])
> >>>> (escape_string is the function in MySQLdb library)
> >>>> It escapes the simple quote of the pickled object, and break it...
> >>>>
> >>>> It's good to know, NEVER escape a pickled object :)
> >>>>
> >>>> Romaric Defaux
> >>>>
> >>>>
> >>> I'm not sure I understand what you're doing here, but I trust you've
> >>> read about and understand the security problems with pickle?
> >>>
> >>> Geremy Condra
> >>>
> >>>
> >> I read quickly the security problems with pickle. But I don't feel
> >> concern about that because I run my program in a private network, not
> >> over internet. And now I use socket to communicate on a non-standard
> >> port, not anymore web service on the 80 port. If I plan to run it
> >> through wan, I will encrypt datas for sure with SSL or something like
> >> that :)
> >>
> >> Romaric Defaux
> >>
> >>
>
>
More information about the Python-list
mailing list