Mailman 3 November 2008 - Twisted

[Twisted-Python] ssh and python shell with twisted
by Turi 13 Nov '08

13 Nov '08

Let me preface by saying, I am new to python and of course Twisted. I am trying to play with the examples already available http://www.devshed.com/c/a/Python/SSH-with-Twisted/. I believe that is the example from Oreilly book on Twisted. I looked at the custom ssh server script and then the providing python shell over ssh script http://www.devshed.com/c/a/Python/SSH-with-Twisted/3/. What I am curious about is how to combine these two? So I want to be able to set up the custom ssh server, which the first example explains, but have support for a command in the SSHDemoProtocol Class (like do_python_shell) which will invoke python shell over the existing ssh session. I am assuming you have to somehow do that over the existing channel, but how? Does my question make sense? --Turi

1 0

[Twisted-Python] Event notification and general RPC in Twisted...
by Blair Bethwaite 12 Nov '08

12 Nov '08

Hi all, I'm thinking about using Twisted to rewrite some communications in a Grid computing application. E.g.: [JobServer]1 ---------- *[Proxy]1 ----------- *[Agents] Agents get jobs, report status, results, etc through Proxy to JobServer. Agents are often distributed across a private network that has no external interface, hence the proxy, which is run on a machine between the private network and internet. Also, the proxy may do some caching/queuing of particular messages - especially where the number of agents behind it is large. What we have at present is a TCP socket server however we're starting to hit scalability issues and on top of that our future plans necessitate a notification framework. I've been digging around on twistedmatrix.com and in the book but so far am having trouble finding any guidance or the level of detail that tells me whether Twisted has what we need, some bits of which are: - we'd like to use a persistent stream/connection, at least between the JobServer and Proxy (traffic frequency will be reasonably high) - it needs to be interoperable with java (is there PB for java?) - each end of the connection should be able to register for and get notifications from the other (e.g. Agent gives a heartbeat, JobServer tells Agent to stop) - sometimes the Proxy might be behind a firewall and only able to connect out, we need to be able to use that connection to go back as above - we want to dynamically configure streaming connections between Proxies Hopefully that's enough context for you to point at modules and tell whether these are appropriate questions: Can I re-use a TCP stream for multiple XML-RPC or PB operations? If so, would it make sense to have the client (e.g. Proxy or Agent) initiate the connection and then make a rpc (e.g. "notify me of anything relevant") to the server which would eventually return when something relevant came along - triggering a callback in the client... wash, rinse, repeat? Sorry if I'm overlooking something that already answers this, it doesn't seem like something too unique to have not been asked before... Regards, -Blair -- In science one tries to tell people, in such a way as to be understood by everyone, something that no one ever knew before. But in poetry, it's the exact opposite. - Paul Dirac

9 11

[Twisted-Python] client send twice but server reads once
by Aydın ŞEN 12 Nov '08

12 Nov '08

Hi, In my code i'm sending sequential data from client like; self.transport.write('aydin') self.transport.write('adige') but server reads data like *'aydinadige' *How can i solve this, actually it is not clear how does dataReceived work for me best regards..

2 2

[Twisted-Python] checking connection
by Benny 12 Nov '08

12 Nov '08

Hi, I have been trying to find out how to keep checking the connection once the connection is made. Says I have successfully connected to the server, and i want to maintain and check that connection. How can i do that?

3 5

[Twisted-Python] Import TypeError
by James Joplin 09 Nov '08

09 Nov '08

I've been doing some embedded python work and everything has been working like a charm thusfar! Twisted has saved us tons of work, hats off to the development team. I have been having issues with shutting down twisted completely (I think). Even upon executing Py_Finalize or Py_EndInterpreter as necessary upon re-entering our browser plugin we hit an exception during twisted import. Here is my traceback : Traceback (most recent call last): File "<string>", line 1, in <module> File "ourCode.py", line 8, in <module> from twisted.spread import pb File "C:\Python25\lib\site-packages\twisted\spread\pb.py", line 69, in <module> from zope.interface import implements, Interface File "C:\Python25\lib\site-packages\zope\interface\__init__.py", line 58, in <module> _wire() File "C:\Python25\lib\site-packages\zope\interface\interface.py", line 809, in _wire classImplements(Attribute, IAttribute) File "C:\Python25\lib\site-packages\zope\interface\declarations.py", line 461, in classImplements spec = implementedBy(cls) File "C:\Python25\lib\site-packages\zope\interface\declarations.py", line 332, in implementedByFallback if isinstance(spec, Implements): TypeError: 'NoneType' object is not callable Changing the order of imports hasn't seemed to matter, looks to me like something in zope init blows up if it's already loaded. Is there a safe way to make sure the threads are dead? To clean imports out? Anything? I haven't been able to find much documentation to help me overcome the problem and I CANNOT forcibly unload our app from browser memory. If the C++ code we are using for startup / shutdown let me know and I will post it as well. Regards, James

2 2

[Twisted-Python] Running a client within a thread... do I need to rewrite?
by Andrew McClain 09 Nov '08

09 Nov '08

I have an existing twisted client connecting to a jabber server in order to send notifications for our web app. I'd like to fold in a STOMP message client into the my existing code, so I can do something like this: web app --> STOMP queue --> STOMP client --> jabber client --> jabber server Problem is, the STOMP client library I'm using creates a socket, spawns a thread, and listens, calling a callback method when it gets information. In order to integrate this with my twisted jabber client, do I need to rewrite the whole client using deferreds? Or could I use something like defertothread to encapsulate the STOMP client? Best, Andrew -- -------- Tap the collective. http://www.fluther.com

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 09 Nov '08

09 Nov '08

1 0

[Twisted-Python] freeing the reactor to do other jobs
by Jeff Dyke 07 Nov '08

07 Nov '08

I'm using the XMLRPC server in twisted and a few methods call other, sometimes long running, functions/methods. I'm trying to get my brain around how to free the reactor to respond to other requests while this is happening. A scenario. A call is made to the server, which selects say 10K rows from a db and needs to check each row against a table and if they do not exist, insert them. """ Oversimplified version of the process """ def getData(self,user_id): rows = self.getUserData(user_id) for row in rows: if self.existsInQueue(row['some_id']): continue else: self.insertQueue(row) I want the caller to wait on a result from this process, but I also want the reactor to be able to handle other requests as they come in. This function is not directly registered in the xmlrpc server with xmlrpc_getData, but is called by that type of method after validation that it is allowed to run in this context. What i've seen when this has thousands of rows to process is that the reactor is tied up and can not respond to requests until complete. Which obviously leads to me believe that I'm not using twisted correctly/to its potential. I have read the deferred/asynchronous doc pages...but am having a hard time getting my head around it and would appreciate any advice. When i don't specifically need the caller to get the final result, i've been suing deferToThread, but feel in some of those instances i could possibly write better code, rather then sending to a thread. Thanks jd

5 6

[Twisted-Python] making twisted provide good feedback (was Re: Having reactor run at the main thread...)
by glyph＠divmod.com 07 Nov '08

07 Nov '08

(Moving this off the twisted-web list because it is of more general interest and contains nothing web-specific.) On 09:48 am, jml(a)mumak.net wrote: >On the other hand, there's a related invalid argument that gets used a >lot by library and framework authors: "if we provide X, people might >misuse it, so we should not provide X". This argument is also bogus >(that way lies Java). Provide safe, well-documented alternatives for >the abusers and let those who want to shoot themselves in the foot do >so: sometimes they might actually know better than you. I find it peculiar that you think this is Java's disease. My perspective is that Java is unpleasant because it is remarkably tedious, redundant, and inconsistent, not because it lacks functionality. If anything, it's a mess of functionality, and there are too many ways to do the same thing. If you want to shoot yourself in the foot (or face, as the case may be) there's always the com.sun hierarchy, which despite being full of warnings signs and sparsely documented, is all public (at the language level) and accessible from Java programs which wish to go that way. In that sense it's relatively hacker friendly. For more fun with foot- shooting there's also AccessibleObject, which lets you call private methods and access private fields. I think the whole issue here is very complex, and so I'd like to repeat and perhaps more clearly explain my own views. I work at the bleeding edge of lots of software and am frequently frustrated when the authors of libraries that I use don't bother with "edge" cases that are necessary to the applications that I write. Twisted should provide excellent support for valid use-cases, no matter how obscure. On the other hand, it's equally frustrating to use a library where the incredibly obscure "you probably don't need this" and the 99%-of-the- time positive-path APIs are listed, without any particular ordering, in one giant flat list. It's particularly frustrating to google for "How do I (X)" and get a solution which uses an apparently reasonable API to do (X), only to discover months after using it that it's actually unsupported and you're not supposed to touch it and now your program is going to break horribly with the next minor version of the library. This isn't just a problem with documentation, because in reality nobody reads documentation. If you're writing a program, you're going to read _just_ enough of the API doc and do just enough groveling around with dir() and pydoc to get yourself to the point where something works. When it works, you will probably consider that sufficient proof that you used the APIs correctly, unless you had to type something that _looked_ particularly gross in the course of doing it, or you got a bunch of warnings out of your program when you ran it. This is the whole point of DeprecationWarning: somehow, your library code itself has to communicate with the the developer who is invoking it and tell them useful things about their use or misuse. I feel like Twisted has more of the latter problem than the former. There are a lot of methods which aren't really private (addReader/removeReader) that don't differentiate themselves as more or less private or refer you to the more portable and general APIs that you probably want (pauseProducing/resumeProducing). However, (modulo global reactors, of course) Twisted does support a lot of edge cases very nicely. Speaking from personal experience, there are a lot of people who show up in #twisted looking at <http://twistedmatrix.com/documents/8.1.0/api/twisted.internet._sslverify.ht…> when they should be looking at <http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.ssl.html>, or <http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.iocpreactor.t…> when they should be looking at <http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.interfaces.IR…>. In the long term, we need to write both better documentation and more code that gets these people back onto the right track. I know I've used a few libraries which provided exactly this kind of feedback and it was very satisfying and educational. It's really an art, one that's difficult to master, and I don't know that I really know the right way to do it in the general case. The danger that comes along with doing it wrong is very nicely explained here: http://blogs.msdn.com/oldnewthing/archive/2008/10/06/8969399.aspx - if you make an API which doesn't work except for the "special" cases, then everyone just starts using the "special" version.

2 1

[Twisted-Python] Silencing PotentialZombieWarning
by Reza Lotun 06 Nov '08

06 Nov '08

Hi All, I'm working on a multi-platform (win32 and os x) wx + Twisted app, where I've found the best solution is to run both in a separate thread - the main thread is wx, and the other is Twisted. It basically works like a charm (though I'd be interested in hearing about other approaches. I've also played with wxreactor way back, but didn't find it as stable). Because I spawn a few processes (using spawnProcess), and must run my reactor with installSignalHandlers=0, I get PotentialZombieWarning errors every time I spawn. Now, I handle reaping of my processes manually for various reasons, and also install my own signal handlers (in the main thread; I also pass clearSigInt=False to wx). Is there any way I can silence these warnings? Thanks, Reza -- Reza Lotun Senior Software Engineer GetPeer Limited reza(a)getpeer.com

3 4