Mailman 3 January 2009 - Twisted

[Twisted-Python] confused on combining different py moduels to make a twisted app
by Krishnakant Jan. 25, 2009

Jan. 25, 2009

hello hackers, I am a new comer to twisted and find it very interesting. By the way I got my previous problem solved regarding returning lists. Now I have another confusion. I have a lot of modules (.py) which I wish to combine and create a server application in twisted. for example I have organisation.py which gets and sets organisation related data in the application. there is a vendor.py for getting and setting vendor related data and such other modules related to inventory. Now, I … [View More]

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Jan. 25, 2009

Jan. 25, 2009

1 0

[Twisted-Python] Possible error on key authentication of twisted conch
by Mustafa Sakalsiz Jan. 23, 2009

Jan. 23, 2009

Hello, I have a python script that executes a python code on a remote host using SSH. It was working perfectly with Twisted 2.5. The method of returning private and public has changed in userauth.SSHUserAuthClient (Twisted 8.1). I was using (Twisted 2.5) ------------------------------------------------ def getPublicKey(self): return keys.getPublicKeyString(self.keyfile+'.pub') def getPrivateKey(self): return defer.succeed(keys.getPrivateKeyObject(self.keyfile)) Now I am using (Twisted … [View More]8.1) ---------------------------------------------------- def getPublicKey(self): return keys.Key.fromFile(self.keyfile+'.pub') def getPrivateKey(self): return defer.succeed(keys.Key.fromFile(self.keyfile)) and it doesn't work any more. I have attached the simplified script. Here is the scenario I am using and the exception I got. Any help is greatly appreciated. Scenario ---------------------- I am creating a DSA key using ssh-keygen (in Debian and Ubuntu) with an empty password $ ssh-keygen -t dsa and copy it to my localhost $ ssh-copy-id -i ~/.ssh/id_dsa localhost then run the attached script, which tries to execute a python script remotely using SSH and get the following error Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/lib/python2.5/site-packages/twisted/internet/tcp.py", line 362, in doRead return self.protocol.dataReceived(data) File "/usr/lib/python2.5/site-packages/twisted/conch/ssh/transport.py", line 314, in dataReceived self.dispatchMessage(messageNum, packet[1:]) File "/usr/lib/python2.5/site-packages/twisted/conch/ssh/transport.py", line 336, in dispatchMessage messageNum, payload) --- <exception caught here> --- File "/usr/lib/python2.5/site-packages/twisted/python/log.py", line 51, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/lib/python2.5/site-packages/twisted/python/log.py", line 36, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib/python2.5/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.5/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/usr/lib/python2.5/site-packages/twisted/conch/ssh/service.py", line 44, in packetReceived return f(packet) File "/usr/lib/python2.5/site-packages/twisted/conch/ssh/userauth.py", line 262, in ssh_USERAUTH_FAILURE if method not in self.authenticatedWith and self.tryAuth(method): File "/usr/lib/python2.5/site-packages/twisted/conch/ssh/userauth.py", line 234, in tryAuth return f() File "/usr/lib/python2.5/site-packages/twisted/conch/ssh/userauth.py", line 338, in auth_publickey keyType = getNS(publicKey)[0] File "/usr/lib/python2.5/site-packages/twisted/conch/ssh/common.py", line 51, in getNS l, = struct.unpack('!L',s[c:c+4]) exceptions.TypeError: 'Key' object is unsubscriptable Sincerely Saki [View Less]

1 1

[Twisted-Python] Seeking OSS 'point to point' data transfer solution
by Worsham, Michael Jan. 21, 2009

Jan. 21, 2009

I was recommended to look at the Twisted application framework for replacing a legacy application that relies on RPC portmapper, so here's my issue and question... Currently, we use an in-house designed application called 'Fileman', which acts as a combined bi-directional shared library (client) and server system to transfer data files between other 'Fileman' server points. The problem is that the Fileman application requires portmapper under Linux which exposes all TCP ports (i.e. a … [View More]network security nightmare). Fileman also has the ability to send data in either ASCII or binary format and at receiver end -- change directories, modify user id (uid) and group id (gid) of file once data is received. --- Here is what Fileman can perform via following functions from client side code: struct StatResult *pFileMan_FileStat(char *host, char *file, int userid, int groupid); struct StatResult *pFileMan_GetDirectoryListing(char *host, char *dir, int userid, int groupid, int *entries); int iFileMan_PutAsciiFile(char *host, char *file, char *cpBuf, int bMakeDir, int userid, int groupid); char *cpFileMan_GetAsciiFile(char *host, char *file, int userid, int groupid, int *result); int iFileMan_PutBinaryFile(char *host, char *file, char *cpBuf, int bMakeDir, int userid, int groupid, int length); char *cpFileMan_GetBinaryFile(char *host, char *file, int userid, int groupid, int *result, int *length); int iFileMan_DeleteFile(char *host, char *file, int userid, int groupid); int bFileMan_Ping(char *cpHost); void vFileMan_ResetStats(void); int iFileMan_GetNumOps(); double dFileMan_GetSecsSpent(); void vFileMan_SetTimeout(int iSecs); Above functions translates to 'over the wire' execution of: stat opendir, readdir fopen, fwrite with any necessary mkdir, chown/chgrp to given group fopen, read, subject to uid/gid restrictions fopen, fwrite with any necessary mkdir, chown/chgrp to given group fopen, read, subject to uid/gid restrictions unlink subject to uid/gid restrictions noop response from server Internal stat change Internal stat info Internal stat info Set network timeout parameters for operation Usage/Syntax from 'Fileman' client: void printUsage() { printf("Usage: \n"); printf(" \n"); printf(" fileman Get [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman GetAscii [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman GetBinary [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman Put [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman PutAscii [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman PutBinary [remote_host] [remote_file] [remote_uid] [remote_gid] [local_file] \n"); printf(" fileman Stat [remote_host] [remote_file] [remote_uid] [remote_gid] \n"); printf(" fileman Ping [remote_host] \n"); printf(" \n"); } --- My question is can the Twisted framework be utilized in a way that can do actions of sending and receiving data in either ASCII or binary format, change uid or gid permissions of files once delivered, run as a client/server client & listener and utilize one (or two) TCP ports for both sending and listening upon? Known application replacement requirements: 1) Application must be able to run upon RHEL/CentOS 5.x Linux platform. 2) Application must be able to be locked down to one or two TCP ports for listening and sending data upon. 3) Application must be able to take command line (CLI) instructions 4) Application must be able to transfer data in ASCII or binary format (i.e. graphic images) 5) Application must be able to [at receiver end]: change directories, modify user id (uid) and group id (gid) of file once data is received. Current solutions we are actively looking at now: -- Apache's WebDAV: Uses port 80; Needs apache instance on each server; cannot change uid or gid permissions of files -- Apache's ActiveMQ: Uses channel/queue/broker system; requires trigger queue and external program for dumping queue data; Needs Java platform -- JBoss Messaging: Suffers same issues as Apache's ActiveMQ; Requires costly licensing -- IBM Websphere MQ: Suffers same issues as Apache's ActiveMQ; Requires costly licensing -- XML-RPC Library for C & C++ -- http://xmlrpc-c.sourceforge.net/ Anyone have any ideas or other plausible solutions for solving this problem? -- Michael --- Michael Worsham Sr. System Engineer, IT Architect Morris Technology Services, IHS Division [View Less]

1 0

[Twisted-Python] Plugins in files with hyphenated names found in 2.5.0 but not in 8.1.0
by Terry Jones Jan. 20, 2009

Jan. 20, 2009

We just upgraded to Twisted 8.1 from 2.5 and discovered that twisted.plugin.getPlugins no longer finds our plugins. After much head scratching, Esteve noticed that the plugins were not being returned because the are in files that have hyphens in their names. There's no mention of this in the release notes, at least that I can see. Perhaps the change crept in unnoticed. I guess this restriction on plugin-containing filenames should either be documented, or the code should be made to work as it … [View More]

3 2

Re: [Twisted-Python] high server loads on FreeBSD - does kqueue reactor help?
by Maarten ter Huurne Jan. 20, 2009

Jan. 20, 2009

On Tuesday 20 January 2009, Werner Thie wrote: > With a four core, four twisted http server processes (nevow/athena on > top), dual 100 MBps interfaces, all four cores showing CPU loads of 50% > plus, I observe the following degradation in the network traffic with > tcpdump: > > - 'TCP checksum incorrect' raising to almost 25% of all packets observed > from one client IP > - TCP Ack'ed lost segment raising to 0.5% of all packets observed Incorrect checksums sounds … [View More]

1 0

[Twisted-Python] Newbee searching test script for web-servers
by Sven-Erik Tiberg Jan. 20, 2009

Jan. 20, 2009

Hi Been lurcking this list for some time w/o contributing with only some questions. Mostly working with Disrtibuted Interactive 3D simulations and have used twisted as one solution transfere states between 3D browser nodes. As I'm using both apache and www.plone.org to store files that builds up 3D applications, I'm curious on how fast a file are delivered from a web-server. What I would like to use are a test scripts that accesses files by url and log access-times. If I can define multipel … [View More]

1 1

[Twisted-Python] pb: callRemotes execute serially on server-side
by David Karnowski Jan. 20, 2009

Jan. 20, 2009

Hi all, I'm having a client-side pb app call the same server-side function 10 times with this client-side code: def process_test(rootObject): for i in range(10): _logger.debug("Calling remote with arg: %d" % (i)) deferred = rootObject.callRemote("test_it", i) deferred.addCallbacks(success, problem, callbackKeywords={'i': i}, errbackKeywords={'i': i}) The server-side "remote_test_it" function takes 5 seconds to complete (per call). I find the following. The … [View More]

3 2

[Twisted-Python] Mailing list archive search?
by David Karnowski Jan. 19, 2009

Jan. 19, 2009

Is there a search function available for the mailing list archives? I don't see a search field anywhere<http://twistedmatrix.com/pipermail/twisted-python/>. Am I missing something obvious? The search field on your home page appears to just search the Tickets, Changesets, & the Wiki, and not the mailing list archives. I ended just doing it manually through a custom Google search<http://www.google.com/advanced_search?q=+site:http://twistedmatrix.com/pipe…>(adding a "site: http://… [View More]

3 2

[Twisted-Python] spawnProcess() works after reactor.stop()
by Paul Thomas Jan. 19, 2009

Jan. 19, 2009

I'm using SelectReactor on Linux for a system that spawns a lot of sub- processes. I'm finding that there are some cases where a signal to the twistd process doesn't reap the children and I think it's because the SIGINT handler defers the shutdown but doesn't prevent spawnProcess() from working in the meantime. What's happening is the signal gets handled by calling stop(). This sets a flag and schedules the shutdown event for later. - Then Linux signals all the sub-processes. - … [View More]

1 0

[Twisted-Python] strangely getting 8002 can't serialise output when returning some list objects
by Krishnakant Jan. 19, 2009

Jan. 19, 2009

hello hackers, I am a new user so perhaps I might be bothering with some question already answered. I am getting a strange response from a published object with twisted. I find no problem returning lists from a function. but when that same function returns records from a postgresql database using psycopg2 (again converted to list ) I get the dredded 8002 can't serialise output error. I have confirmed that i indeed casted the … [View More]

1 0

Re: [Twisted-Python] Tracebacks being dropped from exceptions when using inlineCallbacks
by Terry Jones Jan. 18, 2009

Jan. 18, 2009

>>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> Just to drive that point home, Terry, I found an interesting error glyph> in your initial example. Your example does this: glyph> info = sys.exc_info() glyph> f = failure.Failure(*info) glyph> but sys.exc_info() is a tuple of (type, value, traceback), whereas glyph> Failure takes a tuple of (value, type, traceback). OMG! I guess I should learn to read. glyph> So, we still have some … [View More]

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Jan. 18, 2009

Jan. 18, 2009

1 0

Re: [Twisted-Python] Tracebacks being dropped from exceptions when using inlineCallbacks
by Terry Jones Jan. 17, 2009

Jan. 17, 2009

Hi all. >>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> 1. More of this discussion should be on a ticket. See http://twistedmatrix.com/trac/ticket/3622 Terry

1 0

Re: [Twisted-Python] Tracebacks being dropped from exceptions when using inlineCallbacks
by Terry Jones Jan. 17, 2009

Jan. 17, 2009

Hi Glyph >>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> It sounds like the real issue here is that we don't have a channel glyph> to communicate a "cleaned traceback" (i.e., failure with stringified glyph> frames, but no traceback) from one bit of code to another? Yes, now that I better understand what's going on with cleaning failures, I think that's completely accurate. glyph> 1. More of this discussion should be on a ticket. The mailing list … [View More]glyph> is good for getting discussions started and figuring out what needs glyph> to be done, but now we're talking about the technical specifics of glyph> resolving a specific problem. Someone (terry?) should take the glyph> specifics in the email that started this thread and file an glyph> appropriate ticket. OK, I will do that, because I do still think there is a problem. Code below. glyph> 2. When filing that ticket, I'd really like a full working example glyph> of inlineCallbacks not showing a traceback to play with, not just glyph> examples of portions of the problem. The obvious example I tried, glyph> from twisted.internet.defer import inlineCallbacks glyph> def buggy2(): glyph> raise RuntimeError("Whoops!") glyph> @inlineCallbacks glyph> def buggy(): glyph> yield buggy2() glyph> buggy() glyph> actually prints a nice descriptive traceback. That's because buggy2 is raising an exception, there's no failure coming back via a deferred into buggy (where its traceback would ultimately be replaced with None in cleanFailure). Here's a example: from twisted.internet import reactor, defer def ok(r): print 'OK', r def nok(r): print 'FAILED', r def two(): try: 1 / 0 except Exception: return defer.fail() @defer.inlineCallbacks def one(): yield two() def zero(): one().addCallbacks(ok, nok).addBoth(lambda _: reactor.stop()) reactor.callLater(0, zero) reactor.run() Because two returns a deferred whose errback has been called, one gets a result that's a failure and that failure gets cleaned. The output of running this example is: FAILED [Failure instance: Traceback: <type 'exceptions.ZeroDivisionError'>: integer division or modulo by zero /usr/lib/python2.5/site-packages/twisted/internet/posixbase.py:228:mainLoop /usr/lib/python2.5/site-packages/twisted/internet/base.py:561:runUntilCurrent five.py:17:zero /usr/lib/python2.5/site-packages/twisted/internet/defer.py:813:unwindGenerator --- <exception caught here> --- /usr/lib/python2.5/site-packages/twisted/internet/defer.py:724:_inlineCallbacks five.py:14:one ] Showing the exception caught on line 14 (in one) not line 8. If you look at the deferred that comes back from calling two, its failure result has None as its tb attribute (via the process described in my original mail). However if we replace one with a version that doesn't use inlineCallbacks: def one(): return two() Then the failure has the original traceback: FAILED [Failure instance: Traceback: <type 'exceptions.ZeroDivisionError'>: integer division or modulo by zero /usr/lib/python2.5/site-packages/twisted/internet/posixbase.py:228:mainLoop /usr/lib/python2.5/site-packages/twisted/internet/base.py:561:runUntilCurrent five.py:17:zero five.py:14:one --- <exception caught here> --- five.py:8:two ] I'll file a ticket and we can take discussion about what to do into the tracker. Thanks! Terry [View Less]

1 0

[Twisted-Python] Tracebacks being dropped from exceptions when using inlineCallbacks
by Terry Jones Jan. 17, 2009

Jan. 17, 2009

I think I've finally gotten to the bottom of why exceptions sometimes lose their tracebacks when using inlineCallbacks. I've spent many hours trying to track down problems that result from this. I find the code handling failures, deferreds, and inlineCallbacks non-trivial even in isolation, let alone when those things start interacting. There are a lot of things that are subtle and some that I still haven't gotten to the bottom of. The following ignores some of these subtleties in the interest … [View More]of showing how things can go wrong for the innocent user of inlineCallbacks. I'm happy to go into detail later. First, consider this: import sys from twisted.internet import defer from twisted.python import failure d = defer.Deferred() try: 1 / 0 except Exception: info = sys.exc_info() f = failure.Failure(*info) print "f.tb is %r" % f.tb d.errback(f) print "f.tb is %r" % f.tb Which prints: f.tb is <traceback object at 0x9a19e0> f.tb is None Unhandled error in Deferred: That's the heart of the issue. We make a failure object, it has a traceback, but after passing it to the errback method on a deferred the traceback is gone. This happens because _runCallbacks in defer.py finds no call/errback functions to call on the deferred, drops into this code: if isinstance(self.result, failure.Failure): self.result.cleanFailure() which sets the __dict__ on the failure object to the result of the failure's __getstate__ method, which sets the traceback to None: # added 2003-06-23. See comment above in __init__ c['tb'] = None But the comment in __init__ seems to have been deleted. So in summary, passing the failure to errback() results in its traceback disappearing. (Note that, as mentioned, there are subtleties here. E.g., try calling failure() with no args. The above is just a very simple example.) So what does this have to do with inlineCallbacks? Briefly: when you decorate a function F1 with inlineCallbacks, a new function F2 is created. When you call F2, you get back a deferred. But before you get that deferred, F2 calls F1. F1 returns a generator G (it yields, right?). F2 passes that generator G to _inlineCallbacks. _inlineCallbacks initially starts G by calling its send (passing None). If G raises an exception (other than StopIteration or returning a final value via _DefGen_Return), _inlineCallbacks catches it via: except: deferred.errback() return deferred I.e., it calls the errback on the deferred that F2 is about to hand back to your original code (which called the inlineCallbacks decorated F1 function). Unfortunately, the result of this is as above. A failure is created in errback, it correctly has a traceback on it, but the traceback is then immediately removed in _runCallbacks! By the time your code gets its deferred back from calling the inlineCallbacks decorated F1, the exception has fired, the errback has been called, the deferred you get has its result sitting there waiting for you, but the traceback is gone, gone, gone. Without the original traceback, you end up cursing inlineCallbacks and digging around in what your function called and what that called, and so on. Not fun :-( I'll stop for now. I have some suggestions for fixes, but I'm already in over my head. BTW, I get the impression that the Twisted core developers don't really use inlineCallbacks. Is that correct? Terry [View Less]

4 6

[Twisted-Python] getProcessOutput on windows
by Shawn Church Jan. 17, 2009

Jan. 17, 2009

I am developing an application on Linux for deployment on Windows. I want to use the getProcessOutput function but seem to remember problems with the windows reactor and processes last time I tried this (a couple years ago). I've gone through the active bugs and did not see anything critical to my usage except maybe #2789, which is a year old and apparently not going to be addressed due to Windows limitations. Is getProcessOutput usable on Windows? If not any suggestions for an alternative? … [View More]

1 0

Re: [Twisted-Python] Tracebacks being dropped from exceptions when using inlineCallbacks
by Terry Jones Jan. 17, 2009

Jan. 17, 2009

Hi Phil >>>>> "Phil" == Phil Christensen <phil(a)bubblehouse.org> writes: Phil> On Jan 12, 2009, at 8:36 PM, Terry Jones wrote: >> # added 2003-06-23. See comment above in __init__ >> c['tb'] = None >> >> But the comment in __init__ seems to have been deleted. Phil> The comment was this: Phil> # added 2003-06-23 by Chris Armstrong. Yes, I actually have a Phil> # use case where I need this traceback object, and I've made Phil> # … [View More]

1 0

[Twisted-Python] django with twisted 8.2.x: 2009
by V S P Jan. 16, 2009

Jan. 16, 2009

Hi, wanted to ask for some help in getting django running with twisted webserver for my experiments. Using Python 2.6 Twisted 8.2.0 Django 1.0.2 on windows xp for now. I searched the documentation , most of the blog entries/etc were in '07 about this setup. And most refer to web2 plugin for twisted as it support for wsgi. I looked at the web2 page on twisted matrix and it said that its functionality will be ported back to 'web' module as they become stable. I then browsed 'web' module … [View More]

3 5

[Twisted-Python] reactor.stop() and thread and wxPython problem
by Gabriel Rossetti Jan. 16, 2009

Jan. 16, 2009

Hello everyone, I have written some code using Twisted that spawns threads and another using wxPython integration. For some reason when reactor.stop() is called they don't always exit. I suspect that the reactor is waiting on the threads to finish, which they apparently don't do, and as for the wx integration, I don't know, maybe wx's main loop is in a separate thread and it would be the same problem. Does anyone know/have an idea or has anyone else experienced this? Thank you, Gabriel

3 4

[Twisted-Python] Write into a persistent connection before stopping
by Boeuf, Jean-Francois Jan. 16, 2009

Jan. 16, 2009

Hello, I have an application using twisted in which the web browser opens a persistent connection when a user logs in (COMET model). I want to write data into active persistent connections at server stop to notify client to logout properly. To do that, I add a system event trigger to the reactor calling the method that closes session before shutdown when I start the webService. This method is called when twisted receives a SIGTERM and I can see in twistd.log traces of writing to the … [View More]

1 0

[Twisted-Python] SIP support
by Phil Mayers Jan. 15, 2009

Jan. 15, 2009

What's the current status of SIP support in twisted? I know the support in trunk is older, and there's some newer stuff (Sine) on divmod.org (see below) but that's seems heavily dependent on mantissa, and crucially doesn't support TCP. Also - I'm having problems resolving divmod.org from here, as I think "ohm.divmod.org" is not responding to DNS queries.

2 1

[Twisted-Python] adbapi and ConnectionLost.
by Sébastien HEITZMANN Jan. 15, 2009

Jan. 15, 2009

Hi, for a couple of week i search how i should handle mysql disconnection on my twisted web service. When I restart the database without restarting my service i got some ConnectionLost fired. I have used the cp_reconnect on the connection pool and it seem to work ( the lost connection is restarted ) but the current query isn't executed. So my question is how should I handle this case. Should I intercept the connectionLost and restart the query ? I have several queries in a single deffer so … [View More]

2 1

[Twisted-Python] TCP: can't redo connectTCP() from same local port if server did not explicitly close connection ??
by Alessio Pace Jan. 15, 2009

Jan. 15, 2009

Hi, I would like to have a client connect via TCP to a server (say, server "A"), then the client close the connection and connect via TCP to another server (say, server "B") using the same TCP port that was bound locally while connecting to "A". And then this done over and over again switching from B to A to B etc.. The fact is that it all works if both the client and the server do transport.loseConnection() before the client tries to connect to the other server from the same local port, … [View More]

2 2

[Twisted-Python] Software that uses Twisted
by rodrigoarancibiapla＠winctp.com Jan. 15, 2009

Jan. 15, 2009

5 5