Mailman 3 August 2009 - Twisted

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Aug. 23, 2009

Aug. 23, 2009

1 0

[Twisted-Python] twisted and qtreactor
by Christian Scharkus Aug. 21, 2009

Aug. 21, 2009

Hi :) I'm at extending our existing twisted application (http://github.com/jem777/rattlekekz) with an qtview using qtreactor (http://www.tarbox.org/qtreactor) and pyqt. I wrote this two examples to demonstrate the problem: http://dpaste.com/83246/ The Applications are supposed to print "fu" or "stfu" then the last window is closed. This works with pyqt just fine but don't work then twisted is involved. The signal connection is set up successful so I think twisted swallows or blocks the … [View More]

1 0

[Twisted-Python] Twistd .tac + wxpython
by Gabriel Rossetti Aug. 21, 2009

Aug. 21, 2009

Hello everyone, I have some services written that I'd like to run in a ta.c w/ twistd. One of them uses wxpython. I originally used the wxreactor but it seams to have some problems on Mac OS X and Windows, so with help from some people on this list I ended up using the wx mainloop as the mainloop and putting the reactor in a separate thread. This seams to work well, but now that I want to create a .tac I don't think this is possible. The problem that prompted me to create a .tac is that … [View More]

5 6

Re: [Twisted-Python] Twistd .tac + wxpython
by John Crawford Aug. 20, 2009

Aug. 20, 2009

>I have some services written that I'd like to run in a ta.c w/ twistd. >One of them uses wxpython. I originally used the wxreactor but it seams >to have some problems on Mac OS X and Windows, so with help from some >people on this list I ended up using the wx mainloop as the mainloop and >putting the reactor in a separate thread. This seams to work well, but >now that I want to create a .tac I don't think this is possible. I have been working through the Finger example … [View More]

1 0

[Twisted-Python] Twisted Trac keeps on bugging
by Gabriel Rossetti Aug. 20, 2009

Aug. 20, 2009

Hello everyone, I must be the onlyone having problems with this, sin no one has said anything, but the Twisted Trac keeps on bugging out on me, I get : Traceback (most recent call last): File "/srv/trac/.local/lib/python2.5/site-packages/trac/web/api.py", line 382, in send_error 'text/html') File "/srv/trac/.local/lib/python2.5/site-packages/trac/web/chrome.py", line 475, in render_template return stream.render(method, doctype=doctype) File "/srv/trac/.local/lib/python2.5/site-… [View More]packages/genshi/core.py", line 146, in render output = u''.join(list(generator)) File "/srv/trac/.local/lib/python2.5/site-packages/genshi/output.py", line 200, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/output.py", line 486, in __call__ for kind, data, pos in chain(stream, [(None, None, None)]): File "/srv/trac/.local/lib/python2.5/site-packages/genshi/output.py", line 436, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/core.py", line 207, in _ensure for event in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/core.py", line 207, in _ensure for event in stream: File "/srv/trac/.local/lib/python2.5/site-packages/trac/web/chrome.py", line 478, in _strip_accesskeys for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/filters.py", line 313, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/template.py", line 1145, in _match content = list(content) File "/srv/trac/.local/lib/python2.5/site-packages/genshi/filters.py", line 313, in __call__ for kind, data, pos in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/template.py", line 1120, in _match for event in stream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/template.py", line 1109, in _strip event = stream.next() File "/srv/trac/.local/lib/python2.5/site-packages/genshi/template.py", line 941, in _eval for event in substream: File "/srv/trac/.local/lib/python2.5/site-packages/genshi/template.py", line 920, in _eval ctxt): File "/srv/trac/.local/lib/python2.5/site-packages/genshi/template.py", line 930, in _eval result = data.evaluate(ctxt) File "/srv/trac/.local/lib/python2.5/site-packages/genshi/eval.py", line 101, in evaluate {'data': data}) File "/home/trac/.local/share/trac/templates/error.html", line 36, in <Expression u"get_reporter_id(req)"> <input type="hidden" name="reporter" value="${get_reporter_id(req)}" /> File "/srv/trac/.local/lib/python2.5/site-packages/trac/util/__init__.py", line 50, in get_reporter_id name = req.session.get('name', None) AttributeError: 'NoneType' object has no attribute 'get' when I try to access it, it has been doing this for several days now. Cheers, Gabriel [View Less]

2 1

[Twisted-Python] Happy Twisted User
by Reza Lotun Aug. 20, 2009

Aug. 20, 2009

Hi All, Forgive this random spammy shoutout - but I work for TweetDeck (a real-time web client offering), and we're happy users of Twisted for our backend. More specifically we use it to handle our account signups and syncing of client information. So...if you're a twitter or facebook user and ever thought about organizing tweets maybe check it out? Just to let you know - it's available in two client favours, one for iPhone and one a cross-platform Adobe AIR client. http://www.tweetdeck.com/… [View More]

3 5

[Twisted-Python] weird spammy trac emails?
by Phil Christensen Aug. 19, 2009

Aug. 19, 2009

Hey all, Have any of you gotten these weird spammy messages from trac? I'm getting change notifications on tickets I opened years ago, with no apparent info in it about what the change is except a spam URL. I've attached the ones I've received. I can't figure out how this would even be possible for a spammer to do; there must be some kind of exploit in trac.... -phil Begin forwarded message: > From: trac(a)twistedmatrix.com > Date: August 19, 2009 10:28:44 AM EDT > To: … [View More]

2 1

[Twisted-Python] Clarification on returning exceptions via Perspective Broker
by Matt Bennett Aug. 19, 2009

Aug. 19, 2009

Hi all, I'm using the Perspective Broker to run functions on various remote servers, and trying to pass exceptions raised in the remote functions back to the client. My exceptions subclass pb.Error, which means tracebacks are suppressed on the server and errbacks triggered on the client, exactly as I'd hoped. What I'm confused about though, is why the serialization/deserialization process reduces my nice structured exception into a mere string. Isn't PB's raison d'être to allow you to send … [View More]

2 2

[Twisted-Python] self.socket.accept() in doRead() in tcp.py has (11, 'Resource temporarily unavailable') error
by Gabriel Rossetti Aug. 18, 2009

Aug. 18, 2009

Hello everyone, I have been trying to debug a Twisted app of mine and I noticed that in tcp.py in doRead() self.socket.accept() raises the following exception : error: (11, 'Resource temporarily unavailable'). It seems to just smother the exception though. I read in the python doc that sockets raise that exception if the call would block, when the socket is configuered in non-blocking mode. It seems to do that because the data it receives is too big, how am I supposed to resolve this? Thank you, Gabriel

2 2

[Twisted-Python] AMQP Twisted Application and authentication
by Jonathan Stoppani Aug. 18, 2009

Aug. 18, 2009

Hi All, I'm new to twisted but already fall in love with it... ;-) I'm using the txAMQP library to connect and consume messages from an AMQP server. The examples of the txamqp library shown how to connect to the message broker using the ClientCreator class. What I'm trying to do is to use the twisted application framework. Until now I succeeded to connect by creating a factory to use with twisted.application.internet.TCPClient. My two questions are: 1) Is this the right way to do this? Is it … [View More]

1 0

[Twisted-Python] TCP startTLS() negotiation
by Kyle George Aug. 18, 2009

Aug. 18, 2009

I've noticed some behavior when negotiating TLS using startTLS(). I'm starting TLS on both sides of the connection at the same time to validate peer certificates using a custom callback. If I call startTLS() without any subsequent writes to the socket the negotiation never completes. However, if I write data immediately post startTLS() the negotiation (and the callback to the OpenSSL verify callback) is successful. But I don't want to do this because I want to wait until verification … [View More]

2 1

[Twisted-Python] twisted.spread PB over a ProcessProtocol
by Thomas Vander Stichele Aug. 18, 2009

Aug. 18, 2009

Hi everyone, for a project I'm working on I wanted to create a helper program which can spawn an instance of itself as a separate process, slaving itself to the spawner, and speaking PB to it. The idea is that the master can spawn processes which will run code that might segfault or otherwise behave badly (like consume too much memory). I'm planning to use this as a job server for a django site, where the spawned jobs then do all kind of media analysis (acoustical fingerprinting, tag reading,… [View More] ...) The jobs themselves however I simply want to write in python, defining them as methods on a slave-side pb object that the master process can call on. I prefer to go with spawnProcess and ProcessProtocol because that makes cleanup easier - no sockets or anything to deal with. I made a first implementation here: https://thomas.apestaart.org/thomas/trac/browser/tests/twisted/daemon/daemo… Basically, I tried to respect Twisted's setup for protocols and factories and such, providing a listenProcess and connectProcess method much like the usual ones. That would mean, I think, that this master/slave abstraction and the ProcessProtocol could be used by any kind of server/client factory, not just PB. Currently, I have the processprotocol creates process pipes instead of using stdin/stdout. I'm guessing that would make it windows-incompatible, correct ? I might have to fall back to using stdin/stdout, which would make the debugging harder. Questions: a) anyone want to take a look at the code and comment ? Does it make sense to do it like this ? Any obvious problems ? b) as the comment at the bottom states, uncommenting that line gives me a ProtocolError that I cannot figure out. It is repeatable; however, shuffling code around can make it go away (in fact, I had it in some other location as well at some point). I'm not sure what would be triggering that, any ideas ? I am probably doing something dirty in some part of the code. c) on hitting ctrl-c, tracebacks are generated, due to this block of code: # At least on OS X 10.4, exiting while stdout is non-blocking can # result in data loss. For some reason putting the file descriptor # back into blocking mode seems to resolve this issue. fdesc.setBlocking(self.fd) The exception is EBADF. That code looks suspicious to me - shouldn't it be at least checking for exceptions there ? the fd could already be closed IMO. Comments welcome! Thomas -- I love the way you love but I hate the way I'm supposed to love you back -- URGent, best radio on the net - 24/7 ! http://urgent.fm/ [View Less]

3 2

[Twisted-Python] Ldaptor patch for Interfaces
by Colin Alston Aug. 17, 2009

Aug. 17, 2009

TV! Please fix Ldaptor. Index: interfaces.py =================================================================== --- interfaces.py (revision 1506) +++ interfaces.py (working copy) @@ -1,6 +1,6 @@ -from twisted.python import components +from zope.interface import Interface -class ILDAPEntry(components.Interface): +class ILDAPEntry(Interface): """ Pythonic API for LDAP object access and modification. @@ -126,7 +126,7 @@ incorrect. """ -class … [View More]

2 1

[Twisted-Python] How to do basic authentication on twisted web
by Chris Adams Aug. 17, 2009

Aug. 17, 2009

Hi there, I'm new to twisted, so please excuse the noobish question, but: I'm using twisted-web on an openwrt computer, using busybox, and after looking through the docs, I'm a little unclear about how to implement something like apache's htpasswd style authentication. Using apache, I'd do something like: htpasswd -c passwrods admin topSecretPassword What's the simplest way to add a simple check for a username and password like this using twisted web? Where possible, I'd like to keep all … [View More]

3 7

[Twisted-Python] txAMQP 0.3 released, better late than never
by Esteve Fernandez Aug. 16, 2009

Aug. 16, 2009

Hi all, with great joy and a little sweat, we're happy to announce txAMQP 0.3 (https://launchpad.net/txamqp), a Python library for communicating with AMQP peers and brokers using Twisted. This marks an important milestone, as it's the first release made with the help of two new committers not directly involved with Fluidinfo (http://fluidinfo.com): Dan Di Spaltro from Cloudkick (https://www.cloudkick.com) and Thomas Hervé from Canonical (http://www.canonical.com) On the technical side, … [View More]

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Aug. 16, 2009

Aug. 16, 2009

1 0

[Twisted-Python] twisted.positioning: progress report + Factory question
by Laurens Van Houtven Aug. 15, 2009

Aug. 15, 2009

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi! Short question up front: what exactly should be in the proposed factory class? What does it take? What does it return? Second question: what should NMEASentence attributes look like? Properties? There are some proposed things in t.p.nmea in trunk. Also, progress report: - NMEAAdapters now correctly fire callbacks on receivers. There are tests that show this working for multiple sentences. - NMEA date support. You now get callbacks … [View More]

2 5

[Twisted-Python] Documentation
by John Aherne Aug. 14, 2009

Aug. 14, 2009

Prompted by exarkun, I have put together some simple documentation for beginners starting with Twisted. The points are made up of things I wish I had known from the start as a complete beginner. It still needs lots of work. The layout needs to change, there are duplications and the points need to be made more succinctly. But before I spend more time on honing the document, I thought it would be a good idea to get some feedback. The information in the document may already exist and I have … [View More]just overlooked it. People may feel it is not appropriate for the documentation in Twisted and should go elsewhere. Some of the information needs to be checked for accuracy and to avoid misleading readers. Anyway I would appreciate any feedback positive or negative. John Aherne Here ie the text: Basic Information for anyone starting with Twisted If you don't know mauch about TCP, then bear this in mind. BASIC TCP TCP is a stream of data. Once the connection is open, it stays open until closed. It does not have a beginning or an end. It does not know about your messages. You cannot wait until it sends your message since it will not tell you the message has been sent. If you want to know if your message reached the other end, you need to have in place a protocol for each end to respond that it received the data. You will then need to implement a timeout for when there are problems otherwise you may wait a long time for a response. TCP will wait forever. But intermediate routers may time you out after 2 - 30 minutes if there is no traffic on the port. TCP is just a stream of data. You have to process the stream of data looking for a marker your application has placed that signifies the end of message to your end of the application program. Twisted provide the linereceiver and sendline functions to help in the common case of using CR/LF as a terminator of messages, expecially for chat type protocols and http. The reactor and the select command will process the outgoing and incoming buffers without blocking. The reactor uses the select command. Each time the reactor cycles around, it will use select to check the read and write buffers to see if any buffer is ready to read or write. It will process those that are ready. Ignoring any not yet ready. Anyone familiar with networking and select will probably already understand this. Anyone not familiar will not realise it and needs to become familiar with how select works. If you want to know how Twisted processes network traffic, you should read up on the select command. TWISTED - DIRECT SEND DATA CALLS For simple network activity you do not need to use deferreds. They are not necessary. And you can get a lot done without deferreds just by using the transport.write or sendline functions. This is shown in the simple Chat Server example following. Provided you are dealing in small amounts of data you will not block the reactor. If you are sending megabytes of data in a file, that is a different matter. Using sendline directly is faster than using a deferred. John Goerzen in his Apress book Python Network Fundamentals has a very simple chat server example. WHAT IS BLOCKING CODE Blocking code is code that will block or may potentially block the continued execution of the main reactor thread. Think for the most part of long running processes or operations that may be long running, doing file or network i/o, calculating cpu intensive work, operations that may timeout like doing a remote call to another process or host machine, database operations are usually a culprit, that may be flooded with work or crashed, the examples go on but are mainly about i/o and cpu intensive operations. When these things happen on the reactor / main thread they block the server from doing anything else, it can't accept new connections, it can't do anything else until this blocking activity has completed and returned control to the reactor thread. WHAT ARE DEFERREDS By and large they seem very similar to callbacks. They aren't, but seem to perform the same sort of function. Please refer to other documentation on defers for more detailed explanation. As everyone hears interminably on the twisted list, deferreds do not make blocking code non-blocking. We all try it - but you shouldn't. If you have blocking code, then first think about putting it into deferToThread which will run the code in its own thread. It's not the only thing you can do but it is a good start. Return a deferred when setting up this threaded function and add appropriate callbacks and errbacks. This will run the blocking code in its own thread. You should not call transport.write or sendline functions directly from the thread since this is not thread-safe. In the thread you must call the callback or errback to return processing to the reactor thread and then send any data from the reactor thread. You can handle this without deferToThread by breaking the blocking code up into smaller pieces. Sometimes you need to transfer a large file to a socket, instead of trying to send it all at once send 10KB at a time and yield back to the reactor and reschedule the next 10KB until finished. This will work, it might not be the fastest way and still may block for an unacceptable amount of time on just 10KB, depending on how heavily taxed the i/o system is at the moment. Usually deferToThread is just easier to implement. DATABASE PROCESSING TENDS TO BE BLOCKING The adbapi module seems to be a good example of using deferreds and threads. The adbapi module returns a deferred it has created, you add your callbacks to it. The thread then calls your callback when ready. It does seem like the examplar for doing deferreds. The db stuff will normally block so put it in a thread and use deferreds to wait the result or failure. THREADS twisted is meant to avoid the problems of using threads for network processing. So why are we using threads. It's a way of moving potentially blocking code out of the way so it avoids hanging the reactor. THREADS WON'T NECESSARILY PREVENT BLOCKING A point about the db calls is that they can be very intensive. If you need to run some db function every 30 secs or 60 secs and the db takes 50% or more of the time to generate the results, you won't have much time to service any incoming requests that want to get results. The remote connections will be failing bigtime. So then I suppose you should break the code into 2 programs. One that does the db stuff, the other to handle the remote connections. The db code when it has a result will then connect to the other program and pass across its results. There may be better ways of doing this of course depending on circumstances. WHEN TO USE DEFERREDS If you have a cpu intensive process, then in all probability it will block the reactor since it will take 100% cpu time while running - whether in the main thread or in a separate thread. These are not good for running in twisted. If you have I/O activity, such as reading lines of text from a disk file, this seems a good candidate for deferreds. This is what the dbapi module does. So it seems like a good example to follow. As a general rule, it is simplest to use deferreds with threads. This is not always true so circumstances may indicate a better way of running a deferred. You still need to make sure that the bulk of the time is available for handling connections. Otherwise you will start to have failing connections Using sendline directly is faster than putting a deferred in between. BEWARE WHEN USING DEFERREDS IN THREADS Since deferToThread runs the function you pass to it in a non-reactor thread, you may not use any non-thread-safe Twisted APIs in the function you pass to it. Beware of using shared data when running in the thread such as lists and dictionaries. [View Less]

3 4

Re: [Twisted-Python] newb help needed on syslog
by Terry Jones Aug. 14, 2009

Aug. 14, 2009

>>>>> "Rebs" == Rebs Guarina <fatblowfish(a)gmail.com> writes: Rebs> I'm currently looking at writing a remote syslog client (logs go to a Rebs> central log server running syslog-ng) using twisted. Do you guys have Rebs> any tips on this? If this somewhat appears to be a repost, please Rebs> kindly redirect me to the appropriate link. Hi Rebs This on only a suggestion of a direction, one which we (Fluidinfo) are very likely to take too. The ingredients are … [View More]

2 1

Re: [Twisted-Python] Twisted, PushProducer, Words & big msgs, any limit?
by Terry Jones Aug. 13, 2009

Aug. 13, 2009

>>>>> "Gabriel" == Gabriel Rossetti <gabriel.rossetti(a)arimaz.com> writes: Gabriel> I have a sort of XMPP router I wrote using Twisted and there seems Gabriel> to be a size limit of some sort. I have been using it for over a Gabriel> year now without any problems, but today I had a larger than usual Gabriel> msg (175213 bytes) try to go through and it never did. What protocol are you using? You may be running into one of the MAX_LENGTH limits in twisted.protocol.basic Terry

2 1

[Twisted-Python] Twisted, PushProducer, Words & big msgs, any limit?
by Gabriel Rossetti Aug. 13, 2009

Aug. 13, 2009

Hello everyone, I have a sort of XMPP router I wrote using Twisted and there seems to be a size limit of some sort. I have been using it for over a year now without any problems, but today I had a larger than usual msg (175213 bytes) try to go through and it never did. I haven't pinpointed the problem yet, but I wanted to make sure there isn't some known problem. Also, I get no exception or error msg during this. I use Twisted 8.1.0 on Linux. I ran my code in debug mode and I did notice … [View More]

1 0

[Twisted-Python] Python/twisted BGP code
by Phil Mayers Aug. 13, 2009

Aug. 13, 2009

All, People might be interested in: https://launchpad.net/pybgp It is a VERY SIMPLE implementation of BGP4 in python. It's sufficient to advertise IPv4 prefixes, which is of use e.g. for doing DNS anycast, and withdrawing the routes if the local nameservers stops answering queries. There is a small amount of test coverage; I intend to write more. There is *NO* code for building a RIB or modifying the kernel; that is, this is *not* a replacement for Zebra/Quagga.

1 0

[Twisted-Python] unicode for pyPgSQL
by Pet Aug. 13, 2009

Aug. 13, 2009

Hello, does pyPgSQL doesn't support unicode? execute(query, params) #all items in params list are of <type 'unicode'> I'm getting 'ascii' codec can't encode character u'\xfc' in position 1: ordinal not in range(128) Do I need encode my params before sending a query and then decode all results back to unicode? Pet

3 4

[Twisted-Python] newb help needed on syslog
by Rebs Guarina Aug. 13, 2009

Aug. 13, 2009

Hello, I'm currently looking at writing a remote syslog client (logs go to a central log server running syslog-ng) using twisted. Do you guys have any tips on this? If this somewhat appears to be a repost, please kindly redirect me to the appropriate link. Thanks!

1 0

[Twisted-Python] Tunneling using conch (with application.service)
by Sean Hollingsworth Aug. 12, 2009

Aug. 12, 2009

I’ve got an app written that runs as a service using twisted.application.service that I need to tunnel through SSH. Right now, I use a script that makes use of Paramiko (and runs separate from my app) to set up a tunnel. That more or less works, but I’ve had some problems with the tunnel just going away and would like to integrate the tunneling into the app, using conch. I’ve found an example of tunneling via conch, at http://twistedmatrix.com/pipermail/twisted-python/2009-February/019196.html, … [View More]

2 2