Mailman 3 June 2007 - Twisted

[Twisted-Python] Strange SSL behaviour.
by Voznesensky Vladimir 18 Jun '07

18 Jun '07

Dear all! I have a bug demonstration code. Blocking SSL works where Twisted one does not. It is below. How should I handle this case? Can anybody tell me what to do with this bug? I can dig into twisted sources, but I'm not a twisted specialist, so it could take weeks to boil them to get the dirt. Help me, please. Thank you. VV ---8<--- from twisted.python.util import println from twisted.protocols.basic import LineReceiver from twisted.internet import protocol, reactor, defer, ssl from twisted.python.failure import Failure from OpenSSL import crypto, SSL _CMD="""VERSION=MYPROXYv2 COMMAND=%d USERNAME=%s PASSPHRASE=%s LIFETIME=%d\0""" % (0, "LOGIN", 'PASSPHRASE', 100) WHERETO = ("myproxy.cern.ch",7512) ## Myproxy client protocol. class MyproxyClient(protocol.Protocol): def connectionMade(self): self.transport.write('0') # GT compat. stuff. self.transport.write(_CMD) def dataReceived(self, data): self.transport.loseConnection() def connectionLost(self, reason=protocol.connectionDone): d = self.factory.deferred if reason != protocol.connectionDone: d.errback(reason) else: d.callback(data) ## Myproxy client factory. class MyproxyClientFactory(protocol.ClientFactory): protocol = MyproxyClient def __init__(self): self.deferred = defer.Deferred() def clientConnectionFailed(self, connector, reason): self.deferred.errback(reason) ## Context factory suitable for local needs. class CF: def getContext(self): ctx = SSL.Context(SSL.SSLv3_METHOD) # disable for compatibility with myproxy server (er, globus) # globus doesn't handle this case, apparently, and instead # chokes in proxy delegation code ctx.set_options(0x00000800L) return ctx ctx = CF().getContext() import socket conn = SSL.Connection(ctx,socket.socket()) conn.connect(WHERETO) conn.write('0') conn.write(_CMD) dat = conn.recv(8192) print 'data received by blocking call\n', dat conn.close() del ctx f = MyproxyClientFactory() contextFactory = CF() reactor.connectSSL(WHERETO[0], WHERETO[1], f, contextFactory) f.deferred.addCallbacks( callback=lambda data:(println("data received", data),reactor.stop()), errback=lambda error:(println("an error occurred", error),reactor.stop())) reactor.run() ---8<--- Yours, VV

3 3

[twisted-python] Communication between GUI thread and twisted thread.
by 甜瓜 18 Jun '07

18 Jun '07

Hi, my program has two parts running in two threads. One is a GUI layer, the other is a twisted layer for communication with a remote server. Sometimes, a button is clicked in GUI to notice the twisted thread to send out some data. (reactor.callFromThread should be used, right?) In twisted part, I use reactor.connectTCP('x.x.x.x', port, MyFactory()) to connect with remote server. MyFactory will create an instance of MyProtocol, and MyProtocol has a method 'sendSpecialMsg' for doing real jobs. However, in GUI part, how can I invoke 'sendSpecialMsg' function? Even I cannot get the MyProtocol instance! Because connectTCP and MyFactory hide the protocol object. I'm sure it is a common problem for beginners, Could you give me some advice? Thank you. --- ShenLei

8 11

[Twisted-Python] C++ port of Twisted
by Amaury Forgeot d'Arc 17 Jun '07

17 Jun '07

Hello, I love python and I find twisted fantastic, but here I have to write a C++ application, for performance pseudo-reasons :-( I was looking for a Twisted-like framework in C++. Do you know if such a library exists? ACE seems to provide powerful tools for networking, but I could not find something like Protocols and Deferred. Or is the twisted approach not recommended for C++? -- Amaury Forgeot d'Arc

8 9

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 17 Jun '07

17 Jun '07

Bug summary ______________________ Summary for 2007-06-10 through 2007-06-17 Bugs opened: 4 Bugs closed: 6 Total open bugs: 972 (-2) |== Type Changes |== Priority Changes |== Component Changes |Defect: -4 |Highest: -1 |Core: +1 |Enhancement: +3 |High: -1 |Mail: -2 |Normal: +1 |Web2: -1 |Website: +0 |Words: +1 New / Reopened Bugs ______________________ ===== Highest ===== [#2703] assertWarns has some restrictions (opened by therve) enhancement core http://twistedmatrix.com/trac/ticket/2703 [#2502] 'and' and 'or' operator suport for xpath (opened by jack) enhancement words http://twistedmatrix.com/trac/ticket/2502 ===== Normal ===== [#2704] Link to Twisted coding standard is broken (opened by mesozoic) (CLOSED, fixed) defect website http://twistedmatrix.com/trac/ticket/2704 [#2705] t.web2.http Request.writeResponse should set content-length to 0 on NO_CONTENT responses (opened by terrycojones) defect web2 http://twistedmatrix.com/trac/ticket/2705 Closed Bugs ______________________ ===== Highest ===== [#2702] twisted.words.xish.yappsrt.Scanner.__init__ is not proper epytext (opened by therve, closed by ralphm, duplicate) defect words http://twistedmatrix.com/trac/ticket/2702 [#2700] IMAP4Client.lineReceived() fails to parse literals (opened by jenner, closed by exarkun, fixed) defect mail http://twistedmatrix.com/trac/ticket/2700 [#2695] IMAP4Client breaks if server uses CONDSTORE (opened by slamb, closed by exarkun, fixed) defect mail http://twistedmatrix.com/trac/ticket/2695 [#2075] set SCRIPT_NAME (properly?) in CGI environment (opened by martinpaljak, closed by dialtone, fixed) defect web2 http://twistedmatrix.com/trac/ticket/2075 ===== High ===== [#1979] FileSaver replaces LF with CRLF (opened by agreif, closed by dialtone, fixed) defect web2 http://twistedmatrix.com/trac/ticket/1979 ===== Normal ===== [#2704] Link to Twisted coding standard is broken (opened by mesozoic, closed by glyph, fixed) defect website http://twistedmatrix.com/trac/ticket/2704

1 0

[Twisted-Python] twisted and performance
by weiwei weiwei 16 Jun '07

16 Jun '07

Hi,all! Thanks for the python twisted,it is a great software. I am from china. This days we use twisted to develop a middleware. Requirement is: 1) Receive: receive 100~200 different client¡¯s message(they all connect to our listen socket) and insert it into database(oracle) 2) Send: application insert message into db,then our middleware select message from database and send this message to its destination through socket. Our design is : reactor + adbapi.ConnectionPool 1) reactor is used in receiveing and sending message from and to socket. 2) when we received a message,we use adbapi to insert it to db 3) we use adbapi to select message from db,and use a Deferred to notify us when we got a message from db,and send it to peer,after receiving the peer¡¯s acknowledge message ,we update the databse. After testing many times,we found when the connection number is less than 20,its performance is very good.But if the connection number is more than 30 or 50, it is performance is worse and worse. DO you give me some advice about the design? Thanks in advance. Wei wei _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

5 4

[Twisted-Python] SSL: wrong version number
by Voznesensky Vladimir 15 Jun '07

15 Jun '07

Hello. I'm trying to write a twisted client for MyProxy server. This grid service uses TLS. When using blocking calls, everything is ok. When trying to estabilish a twisted connection using reactor.connectSSL(host, port, f, contextFactory) with _the_same_ context in contextFactory, the error occures: [Failure instance: Traceback (failure with no frames): <class 'OpenSSL.SSL.Error'>: [('SSL routines', 'SSL3_GET_RECORD', 'wrong version number')] You can do it yourself by simply connecting to myproxy.cern.ch:7512 Is there anybody aware of such problem? Thanks, VV

2 3

[Twisted-Python] caching suggestions
by Jonathan Vanasco 15 Jun '07

15 Jun '07

note: this is less about twisted than a project i've built on twisted that is having some issues. everyone here is smart, so i thought this would be a good list to ask on. i've got a small twisted daemon that is working as a proxy for hosting content that we store on amazon-s3 long story short: archive to s3 (redundancy, cheap storage), twisted daemon on our network fetches to cache & serve. we have cheaper bandwidth, plus do a lot of file monitoring / name abstraction that amazon's network won't support. currently there are 10k documents on s3, which are accessible via 150k+ 'keys'. the cache basiscally proxies the right doc for each key I'm running into 2 issues with it: short term- keys are mapped to s3 files via a ton of 'hints' that i store in bdb after fetching from postgresql. the hints need to be refreshed every 1-5 minutes or so. does anyone have a good suggestion on how to do that? i could do this really easily with sqlite, but i need to use bdb -- as sqlite isn't nearly fast enough; while bdb is. long term- everything is fine for now -- we only have about 3gb of data. but thats going to be growing to about 20gb soon, and we want to limit the cache to an active 10gb using an Adaptive Replacement Cache algorithm with a bdb datastore. has anyone done something similar in twisted or python in general? any input would be appreciated. thanks! // Jonathan Vanasco | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | CEO/Founder SyndiClick Networks | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | FindMeOn.com - The cure for Multiple Web Personality Disorder | Web Identity Management and 3D Social Networking | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | RoadSound.com - Tools For Bands, Stuff For Fans | Collaborative Online Management And Syndication Tools | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

3 2

[Twisted-Python] About transport.
by Francisco Herrero 14 Jun '07

14 Jun '07

Hello community. At first excuse me for my poor english, I'm from Argentina. I am building a abstraction layer for different protocols of communication for microcontrolled components simulation on a PC. That is CAN, Mil-1553 and RS422. First I m building a model width serial port. The idea is to connect via tcp to this channel. I choose as a model the class internet.serialport. This works fine but I do not know how to redirect the protocol of serialport to the protocol of TCP port: ----------------------- from rs232 import RS232Connector from twisted.internet import reactor, protocol from twisted.application import service, internet class RS232Channel(protocol.Protocol): def dataReceived(self, data): print "CHAN:" + data self.factory.port.transport.write(data) class RS232Port(protocol.Protocol): def dataReceived(self, data): print "PORT:" + data self.factory.channel.protocol.transport.write(data) class RS232Factory(protocol.Factory): protocol = RS232Port def __init__(self): conf = {} self.port = RS232Port() self.channel = RS232Connector(self.port, conf, reactor) if __name__ == '__main__': factory = RS232Factory() reactor.listenTCP(5001, factory) reactor.run() -------------- this works fine when data is received from the tcp port, but when come data from the serial port the program can't find the factory for RS232Channel protocol instance. That is logical. I try to switch the transport but I can't go so far... Anyone can tell me how to connect this 2 protocols Thanks Francisco -- "El hombre se descubre cuando se mide con un obstáculo" (Saint Exupéry)

1 0

[Twisted-Python] What is the best way to implement "need more data" semantics?
by 甜瓜 14 Jun '07

14 Jun '07

My goal is writing a specific protocol. The data transfered between client and server are mixed with text and binary. I override the method dataReceived. When it get some new data, it will call a certain parseFunc according to the current state of the protocol, such as: Step 1: dataReceived will call parseGreetingMessage(), and then call ReplyGreetingMessage() Step 2: dataReceived will call parseHandShaking(), and then call ReplyHandShaking() Step 3: dataReceived will call parseSecurity(), and then call ReplySecurity() .... However, in some cases, the data received from the server is not completely fetched by the client, so the data passed to parseFunc will be treated as incomplete, and parseFunc will try to return and wait for more data. Currently, my code for doing this seems very ulgy ^_^. Could you please give me some suggestions to make the implementation of the "need more data" semantics more clearly and more easily? The problem is that the message boundary in my protocol is not as simple as LineReceiver. Each step, I should use different parse to determine whether I get the whole data. For example: parseErrorString: The data compose an leading 4byte integer size plus following string characters. If the length of collected data is shorter than (4 + size of string), it will be treated as incomplete. parseBitmap: The data compose a bitmap header and following bitmap data. If the length of collected data is shorter than (sizeof(header) + headerwidth*header.height*bits_per_pixel), it will be treated as incomplete. My method is: parseFunc parses the data while testing the completeness. If incomplete occrred, parseFunct return. When new data arrived, the dataReceived method will call parseFunc again, and "re-parse" the previous data as well as new data. However, sometimes, re-parse is time consuming. I want to find a way to hold the result for already-parsed data. Maybe it will be similar to "generator": # In class MyProtocol(protocol.Protocol): pseudo-code def parseFunc(...): # parse header ... if not completeness: yield # need more data, go into data waiting state. # If new data arrived, dataReceived will be called, and then # call parseFunc again to continue parsing. # parse following data... if not completeness: yield # need more data .... def dataReceived(s): self.data += s self.parseFunc() --- ShenLei

1 0

[Twisted-Python] reactor and thread
by Michele Petrazzo - Unipex srl 14 Jun '07

14 Jun '07

Hi all, I have a project where I writing the "server" (on win xp) and I want to communicate with a client, so twisted it's all I need. Into the "server" I have also a python bind to a dll (with ctypes) and I see strange problems (windows crashes) when I call a dll function inside a thread. My thread it's a twisted "Protocol". My question is: it is possible to call the function from/to a specific thread? So somethink like: class MyDll: def funct(self, callBack): #do the work reactor.call_in_main_thrad(callBack) tid = create_a_thread_and_load_MyDll() # where I make the dll instance and all the work need into a thread #into a someone function into the main thread def callBack(): # a simple callback reactor.callIntoThread(tid, funct, callBack) Is this possible with twisted? (Does I'm dreaming?) P.s. If I try with the wx "reactor" (so wx.MainLoop), I see no problem! In that case, I instance the dll into the class __init__ (inherit from wx.Frame) and into a someone event (i.e. Button click) I call a dll funct and all work without crash... Thanks, Michele

3 2