dataReceived gets called with any data that is available on the socket.  That might not be all data you sent on the other side.  To ensure complete "messages" are delivered your application has to specify some framing, such as Netstrings.  

See: http://twistedmatrix.com/documents/current/api/twisted.protocols.basic.NetstringReceiver.html and the original specification of netstrings http://cr.yp.to/proto/netstrings.txt

That being said, it's a very bad idea to send pickles over the network because unpickling can result in arbitrary code execution.

Peruse some of the results of https://www.google.com/search?q=pickle+execute+arbitrary+code for examples of these dangers.

-David


On Thu, Mar 28, 2013 at 6:24 PM, succer110@tiscali.it <succer110@tiscali.it> wrote:
I have made 2 application: 
The client extract data from a sql server (10k lines), and send every line pickled to a "collector" server via socket.
The server uses twisted and receive every line, unpikle it and store the data in another sql server.

Everytime i start sending data from client to server, in the first 200 line (everytime a different line) **the server** throws an exception:
SOMETIMES it something like:

    Traceback (most recent call last):
      File "collector2.py", line 81, in dataReceived
        self.count,account = pickle.loads(data)
      File "/usr/lib/python2.6/pickle.py", line 1374, in loads
        return Unpickler(file).load()
      File "/usr/lib/python2.6/pickle.py", line 858, in load
        dispatch[key](self)
      File "/usr/lib/python2.6/pickle.py", line 1138, in load_pop
        del self.stack[-1]
    IndexError: list assignment index out of range

But it's NOT every time the same. Printing my exception i red: 
Exception: pop from empty list
Exception: list index out of range
Exception: "'"
Exception: list assignment index out of range

Another strange errors is:
File "/usr/lib/python2.6/pickle.py", line 1124, in find_class
    __import__(module)
exceptions.ImportError: No module named ond'

    for i in listaSAI:
    crm={}
        try:
    crm['uid']=i[0]
    except:
    crm['uid']=None
    try:
    crm['type_cond']=i[01]
    except:
    crm['type_cond']=None
    try:
    crm['population_id']=i[2]
    except:
    crm['population_id']=None
    try:
    crm['product_id']=i[3]
    except:
    crm['product_id']=None
    try:
    crm['a_id']=i[4]
    except:
    crm['a_id']=None
    try:
    crm['status']=i[5]
    except:
    crm['status']=None
    #time.sleep(0.001)
    serialized = pickle.dumps((count,crm))
    #print "sent num", count, crm
    s.sendall(serialized)
    count += 1


And my server:

    def dataReceived(self, data):
    try:
    self.count,account = pickle.loads(data)        
    except Exception as e:
    print "Eccezione:", e
    print self.count+1
    print  data
    print traceback.print_exc()



Printing the data in my client tells me that everything it's ok.
*If i try to slow down the process of sending using time.sleep(0.01) in my client, EVERYTHING IS FINE, and no exception are raised.*

What can i do to debug my code?

p.s. 
I suspect that exceptions.ImportError: No module named ond' refers to "type_cond" key in crm.


Invita i tuoi amici e Tiscali ti premia! Il consiglio di un amico vale più di uno spot in TV. Per ogni nuovo abbonato 30 € di premio per te e per lui! Un amico al mese e parli e navighi sempre gratis: http://freelosophy.tiscali.it/


_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python