Pickle in a POST/GET request give EOFError

Romaric DEFAUX rde at audaxis.com
Thu Nov 18 12:27:32 CET 2010


Hi answer to myself :)

With str(unicode_string) !

Le 18/11/2010 12:23, Romaric DEFAUX a écrit :
> Sorry for this stupid question, but how can I convert unicode to string.
> I ask that because I hit an error (perhaps a bug) in mod_wsgi, and 
> that crash the apache process :
>
> TypeError: sequence of string values expected, value of type unicode 
> found
> RuntimeError: generator ignored GeneratorExit
> child pid 16211 exit signal Segmentation fault (11)
>
> Romaric Defaux
>
> Le 18/11/2010 11:26, Michael Ricordeau a écrit :
>> When loading from json datas with simplejson, strings are unicode .
>>
>>
>> Le Thu, 18 Nov 2010 11:22:30 +0100,
>> Romaric DEFAUX<rde at audaxis.com>  a écrit :
>>
>>> Hi again,
>>>
>>> I try simplejson, but I have a problem. (I just replace pickle
>>> everywhere by simplejson)
>>>
>>> Here's a test I did :
>>> # python
>>> Python 2.5.2 (r252:60911, Jan 20 2010, 21:48:48)
>>> [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
>>> Type "help", "copyright", "credits" or "license" for more information.
>>> >>>  import simplejson
>>> >>>  disk_list = []
>>> >>>  disk = {}
>>> >>>  disk['name'] = "disk1"
>>> >>>  disk['size'] = "52428800"
>>> >>>  disk_list.append(disk)
>>> >>>  print disk_list
>>> [{'name': 'disk1', 'size': '52428800'}]
>>> >>>  disk = {}
>>> >>>  disk['name'] = "disk2"
>>> >>>  disk['size'] = "647648"
>>> >>>  disk_list.append(disk)
>>> >>>  print disk_list
>>> [{'name': 'disk1', 'size': '52428800'}, {'name': 'disk2', 'size': 
>>> '647648'}]
>>> >>>  disk_list_json = simplejson.dumps(disk_list)
>>> >>>  print disk_list_json
>>> [{"name": "disk1", "size": "52428800"}, {"name": "disk2", "size": 
>>> "647648"}]
>>> >>>  disk_list2 = simplejson.loads(disk_list_json)
>>> >>>  print disk_list2
>>> [{u'name': u'disk1', u'size': u'52428800'}, {u'name': u'disk2', 
>>> u'size':
>>> u'647648'}]
>>> >>>  print repr(disk_list_json)
>>> '[{"name": "disk1", "size": "52428800"}, {"name": "disk2", "size":
>>> "647648"}]'
>>>
>>> Explanation :
>>> I "dumps" with json a list of dictionnaries into disk_list_json.
>>> When I "loads" it, I don't get my list of dictionnaries like before 
>>> (see
>>> disk_list2). It adds "u" letter everywhere.
>>> Why ?
>>> Thanks for help !
>>>
>>> Romaric Defaux
>>>
>>> Le 18/11/2010 09:43, Michael Ricordeau a écrit :
>>>> Hi,
>>>>
>>>> you can use json for passing list and dict .
>>>> Pickle is dangerous .
>>>>
>>>> Instead of pickle.loads/pickle.dumps use json.loads and json.dumps
>>>> (using stdlib json in python>= 2.6 or simplejson in python<   2.6)
>>>>
>>>> Regards
>>>>
>>>>
>>>>
>>>> Le Thu, 18 Nov 2010 09:29:00 +0100,
>>>> Romaric DEFAUX<rde at audaxis.com>   a écrit :
>>>>
>>>>
>>>>> Le 17/11/2010 18:52, geremy condra a écrit :
>>>>>
>>>>>> On Wed, Nov 17, 2010 at 6:44 AM, Romaric 
>>>>>> DEFAUX<rde at audaxis.com>    wrote:
>>>>>>
>>>>>>
>>>>>>> Le 16/11/2010 17:47, Romaric DEFAUX a écrit :
>>>>>>>
>>>>>>>
>>>>>>>> Hi everybody !
>>>>>>>>
>>>>>>>> First time I write to this mailing list :)
>>>>>>>> I started writing in python last week, that's probably why I can't
>>>>>>>> understand the following problem...
>>>>>>>>
>>>>>>>>
>>>>>>>> I create a list called web_site_list.
>>>>>>>> This list contain dictionaries called web_site.
>>>>>>>> And some values in this dictionaries are list too.
>>>>>>>>
>>>>>>>> I do that in a function and I return this :
>>>>>>>> return pickle.dumps(web_site_list)
>>>>>>>>
>>>>>>>> This is working fine :)
>>>>>>>>
>>>>>>>> If I do :
>>>>>>>> print "%s" % pickle.loads(system.get_web_site_list())
>>>>>>>>
>>>>>>>> I've got the right stuffs. For example it returns :
>>>>>>>> [{'documentroot_size': '120', 'servername': '---default---', 
>>>>>>>> 'client':
>>>>>>>> 'undefined', 'documentroot': '/var/www/', 'client_contact': 
>>>>>>>> 'undefined',
>>>>>>>> 'serveralias': []}]
>>>>>>>>
>>>>>>>> I send this to a web service. I send it like that :
>>>>>>>> #I put it in params
>>>>>>>> def system_updateweb_site(server, login, password):
>>>>>>>>           params = {}
>>>>>>>>           params['login'] = login
>>>>>>>>           params['password'] = password
>>>>>>>>           params['action'] = 'updateweb_site'
>>>>>>>>           params['servername'] = get_servername()
>>>>>>>>           params['hosted_web_site'] = get_web_site_list()
>>>>>>>>           return call_system_ws(server, params)
>>>>>>>>
>>>>>>>> #Here's how I send it (I tried in GET and POST)
>>>>>>>> def call_system_ws(host, params):
>>>>>>>>           query_string = urllib.urlencode(params)
>>>>>>>> #GET
>>>>>>>> #       f = urllib.urlopen("http://%s/ws?%s" % (host, 
>>>>>>>> query_string))
>>>>>>>> #POST
>>>>>>>>           f = urllib.urlopen("http://%s/ws" % (host), 
>>>>>>>> query_string)
>>>>>>>>           result = f.readline().strip()
>>>>>>>>           if result == 'ERROR':
>>>>>>>>                   msg = f.readline().strip()
>>>>>>>>                   return (False, msg)
>>>>>>>>           return (True, result)
>>>>>>>>
>>>>>>>>
>>>>>>>> On the server side :
>>>>>>>>                           if action == 'updateweb_site':
>>>>>>>>                                   if not 
>>>>>>>> (fields.has_key('servername') and
>>>>>>>> fields.has_key('hosted_web_site')):
>>>>>>>>                                           raise 
>>>>>>>> WSError('missing parameter :
>>>>>>>> servername or hosted_web_site')
>>>>>>>>                                           log ('ERROR : missing 
>>>>>>>> parameter :
>>>>>>>> servername or hosted_web_site')
>>>>>>>>                                   else:
>>>>>>>>
>>>>>>>>     servername=g.db.escape_string(fields['servername'])
>>>>>>>>
>>>>>>>>     hosted_web_site=g.db.escape_string(fields['hosted_web_site'])
>>>>>>>>                                           output =
>>>>>>>> systemserver.updateweb_site(cursor, servername, hosted_web_site)
>>>>>>>>
>>>>>>>> In systemserver.py :
>>>>>>>> def updateweb_site(cursor, host, hosted_web_site):
>>>>>>>>           web_site_list = pickle.loads(hosted_web_site)
>>>>>>>>           return "%s" % (web_site_list)
>>>>>>>>
>>>>>>>> I catch this error :*
>>>>>>>>
>>>>>>>> <type 'exceptions.EOFError'>*:
>>>>>>>>
>>>>>>>> args = ()
>>>>>>>> message = ''
>>>>>>>>
>>>>>>>> Why ?
>>>>>>>>
>>>>>>>> If I just print hosted_web_site, I get this on my web page :
>>>>>>>>
>>>>>>>>
>>>>>>>> (lp0\n(dp1\nS\'documentroot_size\'\np2\nS\'120\'\np3\nsS\'servername\'\np4\nS\'default\'\np5\nsS\'client\'\np6\nS\'undefined\'\np7\nsS\'documentroot\'\np8\nS\'/var/www/\'\np9\nsS\'client_contact\'\np10\ng7\nsS\'serveralias\'\np11\n(lp12\nsa. 
>>>>>>>>
>>>>>>>>
>>>>>>>> It's the "pickled view" of
>>>>>>>> [{'documentroot_size': '120', 'servername': '---default---', 
>>>>>>>> 'client':
>>>>>>>> 'undefined', 'documentroot': '/var/www/', 'client_contact': 
>>>>>>>> 'undefined',
>>>>>>>> 'serveralias': []}]
>>>>>>>>
>>>>>>>> Can someone help me please ? I spend my afternoon to google to 
>>>>>>>> try to find
>>>>>>>> a solution...
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks in advance !!!
>>>>>>>>
>>>>>>>> Romaric Defaux
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> After entirely rewrite my code to not use Web service but socket 
>>>>>>> (a real
>>>>>>> client/server program) I finally found the problem... And it's 
>>>>>>> not linked to
>>>>>>> the POST or GET method...
>>>>>>> It's because of that :
>>>>>>> g.db.escape_string(fields['hosted_web_site'])
>>>>>>> (escape_string is the function in MySQLdb library)
>>>>>>> It escapes the simple quote of the pickled object, and break it...
>>>>>>>
>>>>>>> It's good to know, NEVER escape a pickled object :)
>>>>>>>
>>>>>>> Romaric Defaux
>>>>>>>
>>>>>>>
>>>>>> I'm not sure I understand what you're doing here, but I trust you've
>>>>>> read about and understand the security problems with pickle?
>>>>>>
>>>>>> Geremy Condra
>>>>>>
>>>>>>
>>>>> I read quickly the security problems with pickle. But I don't feel
>>>>> concern about that because I run my program in a private network, not
>>>>> over internet. And now I use socket to communicate on a non-standard
>>>>> port, not anymore web service on the 80 port. If I plan to run it
>>>>> through wan, I will encrypt datas for sure with SSL or something like
>>>>> that :)
>>>>>
>>>>> Romaric Defaux
>>>>>
>>>>>
>>>
>

-- 
Romaric Defaux
Administrateur Systèmes et Réseaux
Audaxis
+33.820 00 70 70


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5361 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://mail.python.org/pipermail/python-list/attachments/20101118/2bc274db/attachment.bin>


More information about the Python-list mailing list