Mailman 3 January 2009 - Twisted

RE: [Twisted-Python] RE: Write into a persistent connection befor estopping
by Boeuf, Jean-Francois 26 Jan '09

26 Jan '09

> -----Message d'origine----- > De : twisted-python-bounces(a)twistedmatrix.com > [mailto:twisted-python-bounces@twistedmatrix.com] De la part > de Jean-Paul Calderone > Envoyé : lundi 26 janvier 2009 16:23 > À : Twisted general discussion > Objet : Re: [Twisted-Python] RE: Write into a persistent > connection beforestopping > > On Mon, 26 Jan 2009 16:12:57 +0100, "Boeuf, Jean-Francois" > <jean-francois.boeuf(a)eads.com> wrote: > >Hi, > > > >As i didn't get any answer to my problem i writed a little > test code to > >reproduce it (I can't divulgate the production code, and > then it would > >be to much complicated to expose the matter). You can just > run it using > >twistd -y test.py and then connect to the 8080 tcp port with > a browser. > >As we can see in the browser the connection is active and > periodicaly > >writed. The logs indicate that the conections are notifyied > before port > >and connection close, but the browser never receives the > notification. > > > >Thanks for your help > > > >##################test.py begins here####################### from > >twisted.application import internet, service from twisted.internet > >import reactor from twisted.web import resource, server, > static import > >threading > > > >application = service.Application("test") > > > >LISTENING = list() > >CHECK = threading.Event() > > > >def writeToActiveConnections(): > > while not CHECK.isSet(): > > for request in LISTENING: > > request.write("active<br />") > > CHECK.wait(2) > > > >def onStop(): > > print "ON STOP CALLED" > > CHECK.set() > > for request in LISTENING: > > request.write("<strong>Connection closed because of server > >shutdown</strong>") > > print "ALL CONNECTIONS HAVE BEEN NOTIFIED" > > > >def onError(_error, _request): > > print "Connection Closed %s" % _error > > LISTENING.remove(_request) > > > >class ListenRessource(resource.Resource): > > def render_GET(self, _request): > > _request.write("listen connection opened<br />") > > LISTENING.append(_request) > > d = _request.notifyFinish() > > d.addCallback(LISTENING.remove, _request) > > d.addErrback(onError, _request) > > return server.NOT_DONE_YET > > > >class MainRessource(resource.Resource): > > def render_GET(self, _request): > > return """ > ><html> > > <head></head> > > <body> > > <h1>Test page</h1> > > <iframe src=/listen></iframe> > > </body> > ></html> > > """ > > > >class TestSite(server.Site): > > def __init__(self): > > root = static.File("/tmp") > > root.putChild("test", MainRessource()) > > root.putChild("listen", ListenRessource() ) > > server.Site.__init__(self, root) > > > >class TestService(internet.TCPServer): > > def __init__(self): > > internet.TCPServer.__init__( > > self, > > 8080, > > TestSite() > > ) > > reactor.callInThread(writeToActiveConnections) > > reactor.addSystemEventTrigger('before', 'shutdown', onStop) > > > >TestService().setServiceParent(application) > > Twisted APIs are generally not safe to be invoked from any > thread except for the thread in which the reactor is running. > > If you want to run some code periodically, don't do it with > threads and events, do it with reactor.callLater or > twisted.internet.task.LoopingCall. > > Jean-Paul > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python(a)twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > I'm Ok with that but it is no more than test sample code. The server does write periodically the connection but doesn't send "<strong>Connection closed because of server shutdown</strong>" whereas the onStop method is correctly called! Jean-François

2 1

[Twisted-Python] Truncated exception message for reactor instance
by Kangkook Jee 26 Jan '09

26 Jan '09

Hi all I'm a Twisted newbie and I have a question about exception handling of Twisted framework. Whenever I got some exception raised from somewhere, I got truncated exception message like this. >>>Traceback (most recent call last): File "./fcli.py", line 68, in <module> reactor.run() File "/opt/local/lib/python2.5/site-packages/twisted/internet/ posixbase.py", line 223, in run self.mainLoop() File "/opt/local/lib/python2.5/site-packages/twisted/internet/ posixbase.py", line 234, in mainLoop self.doIteration(t) File "/opt/local/lib/python2.5/site-packages/twisted/internet/ selectreactor.py", line 140, in doSelect _logrun(selectable, _drdw, selectable, method, dict) --- <exception caught here> --- File "/opt/local/lib/python2.5/site-packages/twisted/python/ log.py", line 51, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/opt/local/lib/python2.5/site-packages/twisted/python/ log.py", line 36, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/opt/local/lib/python2.5/site-packages/twisted/python/ context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx and this make me so hard to find exact cause of the problem. Can anyone tell me how can I solve this problem? Thanks a lot for your help in advance! /Kangkook

2 1

[Twisted-Python] RE: Write into a persistent connection before stopping
by Boeuf, Jean-Francois 26 Jan '09

26 Jan '09

Hi, As i didn't get any answer to my problem i writed a little test code to reproduce it (I can't divulgate the production code, and then it would be to much complicated to expose the matter). You can just run it using twistd -y test.py and then connect to the 8080 tcp port with a browser. As we can see in the browser the connection is active and periodicaly writed. The logs indicate that the conections are notifyied before port and connection close, but the browser never receives the notification. Thanks for your help ##################test.py begins here####################### from twisted.application import internet, service from twisted.internet import reactor from twisted.web import resource, server, static import threading application = service.Application("test") LISTENING = list() CHECK = threading.Event() def writeToActiveConnections(): while not CHECK.isSet(): for request in LISTENING: request.write("active<br />") CHECK.wait(2) def onStop(): print "ON STOP CALLED" CHECK.set() for request in LISTENING: request.write("<strong>Connection closed because of server shutdown</strong>") print "ALL CONNECTIONS HAVE BEEN NOTIFIED" def onError(_error, _request): print "Connection Closed %s" % _error LISTENING.remove(_request) class ListenRessource(resource.Resource): def render_GET(self, _request): _request.write("listen connection opened<br />") LISTENING.append(_request) d = _request.notifyFinish() d.addCallback(LISTENING.remove, _request) d.addErrback(onError, _request) return server.NOT_DONE_YET class MainRessource(resource.Resource): def render_GET(self, _request): return """ <html> <head></head> <body> <h1>Test page</h1> <iframe src=/listen></iframe> </body> </html> """ class TestSite(server.Site): def __init__(self): root = static.File("/tmp") root.putChild("test", MainRessource()) root.putChild("listen", ListenRessource() ) server.Site.__init__(self, root) class TestService(internet.TCPServer): def __init__(self): internet.TCPServer.__init__( self, 8080, TestSite() ) reactor.callInThread(writeToActiveConnections) reactor.addSystemEventTrigger('before', 'shutdown', onStop) TestService().setServiceParent(application) ##################test.py ends here####################### > -----Message d'origine----- > De : Boeuf, Jean-Francois > Envoyé : vendredi 16 janvier 2009 13:26 > À : 'twisted-python(a)twistedmatrix.com' > Objet : Write into a persistent connection before stopping > > > Hello, > > I have an application using twisted in which the web browser > opens a persistent connection when a user logs in (COMET > model). I want to write data into active persistent > connections at server stop to notify client to logout properly. > > To do that, I add a system event trigger to the reactor > calling the method that closes session before shutdown when I > start the webService. This method is called when twisted > receives a SIGTERM and I can see in twistd.log traces of > writing to the persistent connexion before services stops and > connexions are closed. But the web browser never receives > this content and listening the traffic between server and > client with wireshark, I can't see anything else than the > [FIN, ACK] packets that are sent when server ends all > connections (delayed for maximum clarity in the logs). > > The same method is called to logout client properly when > session expires and does work fully. > > You can see below the twistd.log file when the servers stops. > > Thank you for your help. > > Regards > > Jean-François > > > twistd.log > ---------- > 2009-01-16 13:26:16+0100 [-] Received SIGTERM, shutting down. > 2009-01-16 13:26:16+0100 [-] 10.37.68.212 - - > [16/Jan/2009:12:26:16 +0000] "GET > /sim/listen?sourceScreen=216 HTTP/1.1" 200 166 > "http://peyo:8080/sim?id=216" "Mozilla/5.0 (Windows; U; > Windows NT 5.1; fr; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16" > 2009-01-16 13:26:16+0100 [-] 10.37.68.212 - - > [16/Jan/2009:12:26:16 +0000] "GET > /sim/listen?sourceScreen=217 HTTP/1.1" 200 166 > "http://peyo:8080/sim?id=217" "Mozilla/5.0 (Windows; U; > Windows NT 5.1; fr; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16" > 2009-01-16 13:26:16+0100 [-] 10.37.68.212 - - > [16/Jan/2009:12:26:16 +0000] "GET > /sim/listen?sourceScreen=218 HTTP/1.1" 200 166 > "http://peyo:8080/sim?id=218&screen0=216&screen1=217&nbScreen= > 2" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1.16) > Gecko/20080702 Firefox/2.0.0.16" > 2009-01-16 13:26:25+0100 [-] (Port 8080 Closed) > 2009-01-16 13:26:25+0100 [-] Stopping factory > <cockpitServer.userSessionManagement.cockpitSite.CockpitSite > object at 0xb6db9e6c> > 2009-01-16 13:26:25+0100 [-] (Port 8082 Closed) > 2009-01-16 13:26:25+0100 [-] Stopping factory > <cockpitServer.soapService.SoapSite object at 0xb7092d0c> > 2009-01-16 13:26:25+0100 [-] Main loop terminated. > 2009-01-16 13:26:25+0100 [-] Server Shut Down >

2 1

[Twisted-Python] About twistedmatrix.com design license
by Pierre Bourdon 26 Jan '09

26 Jan '09

Hello, I would like to reuse parts of the twistedmatrix.com design for my own Trac powered project, and would like to know about its license. I asked huwshimi as he is the design creator, and he answered me the Twisted team was the design copyright owner. Would it be possible to release the design under a free license to make other people able to reuse [parts of] it, as it is one of the most complete Trac design I've ever seen while browsing the web ? :) Thanks in advance, Delroth.

1 0

[Twisted-Python] Seeking OSS 'point to point' data transfer solution
by Worsham, Michael 25 Jan '09

25 Jan '09

I was recommended to look at the Twisted application framework for replacing a legacy application that relies on RPC portmapper, so here's my issue and question... Currently, we use an in-house designed application called 'Fileman', which acts as a combined bi-directional shared library (client) and server system to transfer data files between other 'Fileman' server points. The problem is that the Fileman application requires portmapper under Linux which exposes all TCP ports (i.e. a network security nightmare). Fileman also has the ability to send data in either ASCII or binary format and at receiver end -- change directories, modify user id (uid) and group id (gid) of file once data is received. --- Here is what Fileman can perform via following functions from client side code: struct StatResult *pFileMan_FileStat(char *host, char *file, int userid, int groupid); struct StatResult *pFileMan_GetDirectoryListing(char *host, char *dir, int userid, int groupid, int *entries); int iFileMan_PutAsciiFile(char *host, char *file, char *cpBuf, int bMakeDir, int userid, int groupid); char *cpFileMan_GetAsciiFile(char *host, char *file, int userid, int groupid, int *result); int iFileMan_PutBinaryFile(char *host, char *file, char *cpBuf, int bMakeDir, int userid, int groupid, int length); char *cpFileMan_GetBinaryFile(char *host, char *file, int userid, int groupid, int *result, int *length); int iFileMan_DeleteFile(char *host, char *file, int userid, int groupid); int bFileMan_Ping(char *cpHost); void vFileMan_ResetStats(void); int iFileMan_GetNumOps(); double dFileMan_GetSecsSpent(); void vFileMan_SetTimeout(int iSecs); Above functions translates to 'over the wire' execution of: stat opendir, readdir fopen, fwrite with any necessary mkdir, chown/chgrp to given group fopen, read, subject to uid/gid restrictions fopen, fwrite with any necessary mkdir, chown/chgrp to given group fopen, read, subject to uid/gid restrictions unlink subject to uid/gid restrictions noop response from server Internal stat change Internal stat info Internal stat info Set network timeout parameters for operation Usage/Syntax from 'Fileman' client: void printUsage() { printf("Usage: \n"); printf(" \n"); printf(" fileman Get [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman GetAscii [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman GetBinary [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman Put [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman PutAscii [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman PutBinary [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman Stat [remote_host] [remote_file] [remote_uid] [remote_gid] \n"); printf(" fileman Ping [remote_host] \n"); printf(" \n"); } --- My question is can the Twisted framework be utilized in a way that can do actions of sending and receiving data in either ASCII or binary format, change uid or gid permissions of files once delivered, run as a client/server client & listener and utilize one (or two) TCP ports for both sending and listening upon? Known application replacement requirements: 1) Application must be able to run upon RHEL/CentOS 5.x Linux platform. 2) Application must be able to be locked down to one or two TCP ports for listening and sending data upon. 3) Application must be able to take command line (CLI) instructions 4) Application must be able to transfer data in ASCII or binary format (i.e. graphic images) 5) Application must be able to [at receiver end]: change directories, modify user id (uid) and group id (gid) of file once data is received. Current solutions we are actively looking at now: -- Apache's WebDAV: Uses port 80; Needs apache instance on each server; cannot change uid or gid permissions of files -- Apache's ActiveMQ: Uses channel/queue/broker system; requires trigger queue and external program for dumping queue data; Needs Java platform -- JBoss Messaging: Suffers same issues as Apache's ActiveMQ; Requires costly licensing -- IBM Websphere MQ: Suffers same issues as Apache's ActiveMQ; Requires costly licensing -- XML-RPC Library for C & C++ -- http://xmlrpc-c.sourceforge.net/ Anyone have any ideas or other plausible solutions for solving this problem? -- Michael --- Michael Worsham Sr. System Engineer, IT Architect Morris Technology Services, IHS Division

2 1

[Twisted-Python] confused on combining different py moduels to make a twisted app
by Krishnakant 25 Jan '09

25 Jan '09

hello hackers, I am a new comer to twisted and find it very interesting. By the way I got my previous problem solved regarding returning lists. Now I have another confusion. I have a lot of modules (.py) which I wish to combine and create a server application in twisted. for example I have organisation.py which gets and sets organisation related data in the application. there is a vendor.py for getting and setting vendor related data and such other modules related to inventory. Now, I understand that we can publish one object at a time and listen to the incoming requests on the given port with the reactor running. But I want to use these other modules together with the main module. How can i create a package for such a situation? I have the __init__.py file in the folder to make a python package, What I can't understand is how can I register all the classes in the different files to listen for function calls from the client? I am confused on managing the entry point so to say. happy hacking. Krishnakant.

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 25 Jan '09

25 Jan '09

1 0

[Twisted-Python] Possible error on key authentication of twisted conch
by Mustafa Sakalsiz 23 Jan '09

23 Jan '09

Hello, I have a python script that executes a python code on a remote host using SSH. It was working perfectly with Twisted 2.5. The method of returning private and public has changed in userauth.SSHUserAuthClient (Twisted 8.1). I was using (Twisted 2.5) ------------------------------------------------ def getPublicKey(self): return keys.getPublicKeyString(self.keyfile+'.pub') def getPrivateKey(self): return defer.succeed(keys.getPrivateKeyObject(self.keyfile)) Now I am using (Twisted 8.1) ---------------------------------------------------- def getPublicKey(self): return keys.Key.fromFile(self.keyfile+'.pub') def getPrivateKey(self): return defer.succeed(keys.Key.fromFile(self.keyfile)) and it doesn't work any more. I have attached the simplified script. Here is the scenario I am using and the exception I got. Any help is greatly appreciated. Scenario ---------------------- I am creating a DSA key using ssh-keygen (in Debian and Ubuntu) with an empty password $ ssh-keygen -t dsa and copy it to my localhost $ ssh-copy-id -i ~/.ssh/id_dsa localhost then run the attached script, which tries to execute a python script remotely using SSH and get the following error Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/lib/python2.5/site-packages/twisted/internet/tcp.py", line 362, in doRead return self.protocol.dataReceived(data) File "/usr/lib/python2.5/site-packages/twisted/conch/ssh/transport.py", line 314, in dataReceived self.dispatchMessage(messageNum, packet[1:]) File "/usr/lib/python2.5/site-packages/twisted/conch/ssh/transport.py", line 336, in dispatchMessage messageNum, payload) --- <exception caught here> --- File "/usr/lib/python2.5/site-packages/twisted/python/log.py", line 51, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/lib/python2.5/site-packages/twisted/python/log.py", line 36, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib/python2.5/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.5/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/usr/lib/python2.5/site-packages/twisted/conch/ssh/service.py", line 44, in packetReceived return f(packet) File "/usr/lib/python2.5/site-packages/twisted/conch/ssh/userauth.py", line 262, in ssh_USERAUTH_FAILURE if method not in self.authenticatedWith and self.tryAuth(method): File "/usr/lib/python2.5/site-packages/twisted/conch/ssh/userauth.py", line 234, in tryAuth return f() File "/usr/lib/python2.5/site-packages/twisted/conch/ssh/userauth.py", line 338, in auth_publickey keyType = getNS(publicKey)[0] File "/usr/lib/python2.5/site-packages/twisted/conch/ssh/common.py", line 51, in getNS l, = struct.unpack('!L',s[c:c+4]) exceptions.TypeError: 'Key' object is unsubscriptable Sincerely Saki

1 1

[Twisted-Python] Seeking OSS 'point to point' data transfer solution
by Worsham, Michael 21 Jan '09

21 Jan '09

I was recommended to look at the Twisted application framework for replacing a legacy application that relies on RPC portmapper, so here's my issue and question... Currently, we use an in-house designed application called 'Fileman', which acts as a combined bi-directional shared library (client) and server system to transfer data files between other 'Fileman' server points. The problem is that the Fileman application requires portmapper under Linux which exposes all TCP ports (i.e. a network security nightmare). Fileman also has the ability to send data in either ASCII or binary format and at receiver end -- change directories, modify user id (uid) and group id (gid) of file once data is received. --- Here is what Fileman can perform via following functions from client side code: struct StatResult *pFileMan_FileStat(char *host, char *file, int userid, int groupid); struct StatResult *pFileMan_GetDirectoryListing(char *host, char *dir, int userid, int groupid, int *entries); int iFileMan_PutAsciiFile(char *host, char *file, char *cpBuf, int bMakeDir, int userid, int groupid); char *cpFileMan_GetAsciiFile(char *host, char *file, int userid, int groupid, int *result); int iFileMan_PutBinaryFile(char *host, char *file, char *cpBuf, int bMakeDir, int userid, int groupid, int length); char *cpFileMan_GetBinaryFile(char *host, char *file, int userid, int groupid, int *result, int *length); int iFileMan_DeleteFile(char *host, char *file, int userid, int groupid); int bFileMan_Ping(char *cpHost); void vFileMan_ResetStats(void); int iFileMan_GetNumOps(); double dFileMan_GetSecsSpent(); void vFileMan_SetTimeout(int iSecs); Above functions translates to 'over the wire' execution of: stat opendir, readdir fopen, fwrite with any necessary mkdir, chown/chgrp to given group fopen, read, subject to uid/gid restrictions fopen, fwrite with any necessary mkdir, chown/chgrp to given group fopen, read, subject to uid/gid restrictions unlink subject to uid/gid restrictions noop response from server Internal stat change Internal stat info Internal stat info Set network timeout parameters for operation Usage/Syntax from 'Fileman' client: void printUsage() { printf("Usage: \n"); printf(" \n"); printf(" fileman Get [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman GetAscii [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman GetBinary [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman Put [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman PutAscii [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman PutBinary [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman Stat [remote_host] [remote_file] [remote_uid] [remote_gid] \n"); printf(" fileman Ping [remote_host] \n"); printf(" \n"); } --- My question is can the Twisted framework be utilized in a way that can do actions of sending and receiving data in either ASCII or binary format, change uid or gid permissions of files once delivered, run as a client/server client & listener and utilize one (or two) TCP ports for both sending and listening upon? Known application replacement requirements: 1) Application must be able to run upon RHEL/CentOS 5.x Linux platform. 2) Application must be able to be locked down to one or two TCP ports for listening and sending data upon. 3) Application must be able to take command line (CLI) instructions 4) Application must be able to transfer data in ASCII or binary format (i.e. graphic images) 5) Application must be able to [at receiver end]: change directories, modify user id (uid) and group id (gid) of file once data is received. Current solutions we are actively looking at now: -- Apache's WebDAV: Uses port 80; Needs apache instance on each server; cannot change uid or gid permissions of files -- Apache's ActiveMQ: Uses channel/queue/broker system; requires trigger queue and external program for dumping queue data; Needs Java platform -- JBoss Messaging: Suffers same issues as Apache's ActiveMQ; Requires costly licensing -- IBM Websphere MQ: Suffers same issues as Apache's ActiveMQ; Requires costly licensing -- XML-RPC Library for C & C++ -- http://xmlrpc-c.sourceforge.net/ Anyone have any ideas or other plausible solutions for solving this problem? -- Michael --- Michael Worsham Sr. System Engineer, IT Architect Morris Technology Services, IHS Division

1 0

[Twisted-Python] Plugins in files with hyphenated names found in 2.5.0 but not in 8.1.0
by Terry Jones 20 Jan '09

20 Jan '09

We just upgraded to Twisted 8.1 from 2.5 and discovered that twisted.plugin.getPlugins no longer finds our plugins. After much head scratching, Esteve noticed that the plugins were not being returned because the are in files that have hyphens in their names. There's no mention of this in the release notes, at least that I can see. Perhaps the change crept in unnoticed. I guess this restriction on plugin-containing filenames should either be documented, or the code should be made to work as it did. I'm happy to open a ticket - though the text of the ticket would depend on which solution is preferred. Terry

3 2