Mailman 3 July 2005 - Twisted

[Twisted-Python] exceptions in adbapi
by Henning.Ramm＠mediapro-gmbh.de July 11, 2005

July 11, 2005

Hi there! I've a twisted app as database proxy server that handles all database queries for my other apps (via a pb connection). Now I get thousands of exceptions that I can't trace back, two kind: 2005-07-11 15:57:36,707 defer.328 [ERROR] [Failure instance: Traceback: _mysql_exceptions.Warning: Datensätze gefunden: 1 Geändert: 0 Warnungen: 1 D:\Python\lib\threading.py:422:run D:\Python\lib\site-packages\twisted\python\threadpool.py:145:_worker D:\Python\lib\site-packages\twisted\python\context.py:52:callWithContext D:\Python\lib\site-packages\twisted\python\context.py:31:callWithContext --- <exception caught here> --- D:\Python\lib\site-packages\twisted\internet\threads.py:25:_putResultInDeferred D:\Python\lib\site-packages\twisted\enterprise\adbapi.py:302:_runInteraction D:\Python\lib\site-packages\twisted\enterprise\adbapi.py:311:_runQuery D:\Python\lib\site-packages\MySQLdb\cursors.py:95:execute D:\Python\lib\site-packages\MySQLdb\cursors.py:114:_execute D:\Python\lib\site-packages\MySQLdb\connections.py:33:defaulterrorhandler ] 2005-07-11 15:57:36,707 defer.328 [ERROR] [Failure instance: Traceback: _mysql_exceptions.IntegrityError: (1062, "Doppelter Eintrag 'TA_V_23_130705' f\xfcr Schl\xfcssel 1.") D:\Python\lib\threading.py:422:run D:\Python\lib\site-packages\twisted\python\threadpool.py:145:_worker D:\Python\lib\site-packages\twisted\python\context.py:52:callWithContext D:\Python\lib\site-packages\twisted\python\context.py:31:callWithContext --- <exception caught here> --- D:\Python\lib\site-packages\twisted\internet\threads.py:25:_putResultInDeferred D:\Python\lib\site-packages\twisted\enterprise\adbapi.py:302:_runInteraction D:\Python\lib\site-packages\twisted\enterprise\adbapi.py:311:_runQuery D:\Python\lib\site-packages\MySQLdb\cursors.py:95:execute D:\Python\lib\site-packages\MySQLdb\cursors.py:114:_execute D:\Python\lib\site-packages\MySQLdb\connections.py:33:defaulterrorhandler ] The first one is absolutely useless - MySQL tells me only that it could process the query ("datasets found: 1, changed 0, warning: 1"); I'd like to block that completely. The second one means that I tried an insert with an existing key ("Double entry '...' for key 1"); also no problem, if I could handle it and do an update query instead. But as you can see, I can't trace the messages back to one of my methods (and I have already every function call in a try..except clause). Any hints? Best regards, Henning Hraban Ramm Südkurier Medienhaus / MediaPro Support/Admin/Development Dept.

2 1

[Twisted-Python] Understanding the IOCP reactor and adding spawnProcess
by Justin Johnson July 11, 2005

July 11, 2005

I am attempting to add spawnProcess to iocpreactor. In order to begin this task I've had to do a lot of reading on Windows network programming, specifically the various Windows I/O methods, to attempt to understand what win32eventreactor and iocpreactor are doing, and also just increase my understanding of how reactors work in general. To understand the various Winsock 2 methods that both of these reactors rely upon, I read chapters 1-5 of Network Programming for Microsoft Windows[1]. Before actually attempting to add spawnProcess, I would like to present how I think iocpreactor works and how I think I should add spawnProcess, and hopefully be corrected or confirmed in my understanding. If I'm too vague there's a good chance it's because I don't understand it very well. Please feel free to point out things that you might think are obvious but aren't sure I understand. How iocpreactor works --------------------------------- 1. Create an IO Completion Port. 2. Create a socket and associate it with the IOCP. This is the socket we will call AcceptEx (a non-blocking accept) on. The association with the IOCP is made via CreateIoCompletionPort. 3. Setup any scheduled tasks on the reactor. 4. Call AcceptEx (which doesn't block) on the socket. AcceptEx takes an overlapped structure as a parameter. Before making the call, we set two attributes of the struct: the callback and callback_args which will be called when an accept event completes on the socket. The Winsock 2 methods don't actually call the callback. The Winsock 2 methods handle copying data related to the network event that occurred on the socket into the overlapped structure and making that overlapped structure available to GetQueuedCompletionStatus. So when we handle events on sockets via GetQueuedCompletionStatus from within doIteration, we have access to the data related to the event as well as the callback and callback_args we call to handle that event. The callbacks are setup in the xxxOp classes in ops.py and always result in some transport method getting called (such as readDone, connectionDone, etc). 5. From within doIteration, call GetQueuedCompletionStatus (which does block) with a timeout of the time until the next scheduled task needs to be run. If any event occurs on the sockets currently associated with the IOCP before that time expires, GetQueuedCompletionStatus will return (stop blocking). Now we have access to the overlapped structure containing data associated with the event which was copied into the overlapped structure's buffer, such as data received from WSARecv calls, as well as the callback and callback_args. From within doIteration we call the callbacks passing in the data related to the event. Depending on the events we are handling, we may create new sockets (e.g. end point sockets in TCP connections) and associate them with the IOCP as well. All Winsock 2 API calls made are non-blocking accept for GetQueuedCompletionStatus. 6. Step 5 continues until the reactor stops. How to add spawnProcess --------------------------------------- 1. Create the processes via Windows APIs and associate their stdout/err with with the IOCP via CreateIoCompletionPort calls. 2. Close stdin. 3. Notify the ProcessProtocol via protocol.makeConnection (not sure why, looking at win32eventreactor) 4. Receive data from stdout/err via the completion port by calling GetQueuedCompletionStatus from within doIteration. Is this really possible? ProcessProtocol's methods won't get called appropriately by letting the existing callbacks in ops.py make calls to the transport (e.g. connectionDone, readDone)? Thanks for your help. Justin [1] http://www.amazon.com/exec/obidos/ASIN/0735615799

3 7

[Twisted-Python] Receiving large packets with TCP
by Stefan Näwe July 11, 2005

July 11, 2005

Hi. I'm using Twisted (2.0.1) for a TCP-based receiver. That receiver receives different telegrams sent by another process. Those telegrams are sent on different ports (i.e. one telegram type is sent on one port). The sizes of the telegram vary from ~16 bytes to ~8 kB. Everything works perfectly. I've really never been *that* productive. BUT: I'm having trouble receiving the whole contents of large telegrams. For example one telegram is exactly 7452 bytes long but I only get 2920 bytes. My twisted receiver uses the reactor to setup the different ports to listen on and creates 'message handler' objects on connect. Those message handler objects 'dissect' the received telegram and print some of its contents to STDOUT. Some parts of my code: class MyReceiver(Protocol): def dataReceived(self, data): if __debug__: log.msg("Received %d bytes" % len(data)) try: # here the message handler gets created m = self.factory.msg[self.port](data) log.msg(m) except KeyError: # can this really happen ?? log.msg("Unhandled connection from port %d" % self.port) def connectionMade(self): addr = self.transport.getHost() self.proto = addr.type self.peer = addr.host self.port = addr.port log.msg("Connection from %s" % self.transport.getPeer()) My 'main' function simply does 'reactor.listenTCP(port, MyFactory)' So what do I need to do to get the whole telegram 'in one shot' ? Do I need to re-assemble the telegrams myself ? (I hope this Information is enough...Tell me, if not.) TIA Stefan -- ---------------------------------------------------------------------- Stefan Naewe ATLAS Elektronik GmbH Dept.: NUS T4 phone: +49-(0)421-457-3969 Sebaldsbruecker Heerstr. 235 fax: +49-(0)421-457-3913 28305 Bremen

4 5

[Twisted-Python] twisted.names and multicast DNS
by Tim Allen July 11, 2005

July 11, 2005

So after two years of leaving it alone, I decided to have another look at the multicast DNS module I was writing for Twisted, and lo and behold I discover a serious design flaw in my response cache. Here's the basic situation I'm dealing with: Suppose I have two hosts on my network, called ford and arthur. I'm sitting at arthur, and I'm looking for SSH servers advertised via DNS-based Service Discovery and Multicast DNS. At the present moment, ford is switched off. I turn on ford, and when the SSH server starts up, ford broadcasts an announcement DNS datagram that says, "In the zone '_ssh._tcp.local', there is a PTR record with the data 'Ford._ssh._tcp.local'. This information expires in 7200 seconds." (it mentions the class too, of course, but the class doesn't affect the discussion below so we'll ignore it from here on). arthur receives the announcement, records the transmission time and the TTL, and caches the PTR record. Thereafter, any program on arthur that requests PTR records in '_ssh._tcp.local' will be given the PTR record about 'Ford._ssh._tcp.local'. Now I turn off Ford, and when the SSH server shuts down, ford broadcasts a goodbye datagram that says, "In the zone '_ssh._tcp.local', there is a PTR record with the data 'Ford._ssh._tcp.local'. This information expires in 0 seconds." - that is to say, exactly the same as the previous announcement but with a TTL of 0. The cache on arthur needs to compare this new record to its cache, see that it matches the definition of an existing record, and update the TTL of the existing record so that it will expire immediately. I'm trying to implement the record cache on arthur at the moment. It needs to be queried by (name, type, class) tuples, so I use such a tuple as the key of a dictionary. Since there can be multiple records available for a particular (name, type, class), the value associated with this key is a list of resource records. Because the t.names.dns.DNSDatagramProtocol gives me instances of the various t.names.dns.Record_* classes, I figured that those instances would be a good representation of the resource records I need to store. In psuedo-python, then, the ideal data structure would look like this: cache = { ('_ssh._tcp.local', PTR, IN): [ (<Record_PTR name=Ford._ssh._tcp.local>, 7200), ] } ...then, when I get a new record from the network, I can look up its (name, type, class) tuple and get the list of records. Then I compare the new record to each of the records in the list: if it matches a record I update the TTL, otherwise I add the record to the list. The problem is this: the Record_PTR class (just like all the other Record_* classes) in t.names.dns stores the TTL as an instance variable! Therefore, when I have an existing record and an updated record, they never match, and I wind up with two identical records in my data structure that differ only by TTL. As far as I can tell, I have three options from here: 1. Write my own versions of all the Record_* classes that don't maintain a TTL value, and translate backwards and forwards whenever I need to talk to Twisted's DNS layer. I'd need to update this code whenever a new DNS record type was added. 2. Write a comparison function for comparing resource records that compares instance variables that aren't named 'ttl'. This would probably need to be updated less frequently, but is horribly hacky. 3. Complain on the twisted-list and hope that somebody has a better suggestion, or changes the code to be more mDNS-friendly. As you can see, I'm trying Option 3 at the moment, but I'm open to suggestions. :)

3 3

[Twisted-Python] Questions regarding serviceCollection and Interfaces.
by kgi July 11, 2005

July 11, 2005

Hi all. I'm in the process of moving a skeleton asyncore application to Twisted, and need some help. I'm hoping that someone with much Twisted Enlightenment can point me gently in the direction of understanding. (Amusingly enough, another poster recently summarised my exact experiences: start with asyncore, find its limitations, do a bit of shoehorning and end up trying with Twisted). Because this is new code I want to do The Right Thing vis-a-vis the latest Twisted coding techniques. Everything was peachy until I reached a certain step in the "Evolution of Finger" Tutorial. Until a certain point I had completely grokked the approach that looks like this: class Foo ( Protocol ): .... factory = ServerFactory() factory.protocol = Foo application = service.Application ( "fooserver" ) internet.TCPServer ( port, factory ).setServiceParent ( application ) However, then I reached: http://twistedmatrix.com/projects/core/documentation/howto/tutorial/compone… and the approach morphed into: application = service.Application ( "fooserver" ) f = FooService() serviceCollection = service.IServiceCollection ( application ) internet.TCPServer ( port, IFooFactory ( f ) ).setServiceParent ( serviceCollection ) Here are my questions: 1. I simply don't understand the third line of the new approach! :-) In twisted.application.service.py, I see: class IServiceCollection(components.Interface): .... So, "IServiceCollection" is an Interface; my (possibly outdated) understanding is that an interface is simply a specification that you can check against when something claims to implement it, hence all the "implements(Yadda)" calls everywhere (presumably, an exception is raised when something tells porkies about what it implements). But in this case, it looks like we're actually instantiating an interface class. I can't grok what this "means", especially in light of the fact that there is "MultiService" below it that implements IServiceCollection. 2. Similarly, in the last line, "IFooFactory(f)" is taking a Service (which isn't a factory) and passing it as a construction argument to IFooFactory (which also isn't a factory). The result is used as an argument to internet.TCPServer(), which expects a factory! 3. What do I gain by using a serviceCollection over an application as the serviceParent of a TCPServer? Come to think of it, what did I gain in the first place with setServiceParent()? 4. Lastly, there are lots of examples floating around that describe how each Protocol example can reference its own Factory in order for connections to be able to talk to other connections (for example, in the chat server example). However, what approach should I take when I have a single server with multiple Protocols? For example, consider a simple cache: I have a server which accepts TCP connections from lots of clients, and handles queries from these clients. The server actually gets information from an upstream server, but caches the results that it has already seen. In this case in the server I need a Protocol/TCPServer server to handle the multiple clients, and a single Protocol/connectTCP client for the upstream connection. However, it's not clear to me how I would transfer the data from the lineReceived() method of the server Protocol to the sendLine() method of the client Protocol, since each Protocol can see its own Factory, but not the Factory of the other Protocol. Many thanks for your time, and any assistance afforded! Ricky

4 5

[Twisted-Python] XML RPC
by Md Moin July 9, 2005

July 9, 2005

I would like to setup a twisted XML RPC server with authentication functionality based on username and password.Is there any simple example available for XML RPC server/client that uses username/password authentication? I am a newbie thus if any of you guys can post a simple example of xml rpc server and client that uses authentication that will help me a lot. Thanx ____________________________________________________ Sell on Yahoo! Auctions no fees. Bid on great items. http://auctions.yahoo.com/

1 0

Re: [Twisted-Python] Could not create and run a Windows Service with Twisted 2.0.1 & Python 2.4.1
by Cory Dodt July 9, 2005

July 9, 2005

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Fixed in trunk. I recommend you delete ntsvc.cfg and your dist directory, then rerun setup.py twistedservice with svn trunk code. Jp was correct. C > It's not being imported by user code, it's being imported by t.service.app. > Py2exe doesn't know that, so it loses it. > > Luc, probably ntsvc should deal with this automatically. Let me think > about it > before you put it in the HOWTO. > > C > > > Jp Calderone wrote: > | On Fri, 8 Jul 2005 18:59:36 +0200, Luc Stepniewski > | <luc.stepniewski(a)adelux.fr> wrote: > | > |> On Friday 08 July 2005 16:53, Cory Dodt wrote: > |> > |>> Luc, it's very common to get these near code that uses namedAny or > |>> named* > |>> from reflect. Using these calls guarantees that py2exe's module finding > |>> algorithm won't know to bundle the named module. > |>> > |>> You just need an "includes" line in your setup for > |>> twisted.internet.default. > |> > |> > |> Wow, you're right! It works. It's strange I didn't need this > |> "includes" with > |> Twisted 1.3. Anyway. I'll add this to the HOWTO, as soon as I can. > | > | > | twisted.internet.default was never really public anyway, and now it's > | deprecated. The ideal solution would be to stop using it altogether. > | What's it being imported for? > | > | Jp > | > | _______________________________________________ > | Twisted-Python mailing list > | Twisted-Python(a)twistedmatrix.com > | http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFC0A+p3A5SrXAiHQcRAmo9AJ47RcWy5VZcAlBfcVEsz7bHOkbTNQCeO+nh 0n/Sl+TenS4gbyPSG8+G+Xw= =5hWe -----END PGP SIGNATURE-----

1 0

[Twisted-Python] Race condition in callInThread()
by Jp Calderone July 8, 2005

July 8, 2005

James kindly added a failing test for the nasty behavior of twisted.python.threadpool.ThreadPool._startSomeWorkers(). It seems this will be difficult to fix completely without making API-incompatible changes all the way up to reactor.callInThread, though. Fixing it for *most* cases is straightforward enough: Index: twisted/python/threadpool.py =================================================================== --- twisted/python/threadpool.py (revision 14112) +++ twisted/python/threadpool.py (working copy) @@ -99,10 +99,10 @@ return state def _startSomeWorkers(self): - if self.workers >= self.max: - return - # FIXME: Wait for any waiters to eat of the queue. - while self.workers < self.max and self.q.qsize() > 0: + while ( + self.workers < self.max and # Don't create too many + len(self.waiters) < self.q.qsize() # but create enough + ): self.startAWorker() def dispatch(self, owner, func, *args, **kw): But this isn't a complete fix. Annoyingly, the area it doesn't cover very well is exactly what the unit test covers most aggressively. The problem is the implementation of _worker: it calls back into user code before adjusting the ThreadPool's state to indicate that it is no longer busy. Since the concept of result callback/errbacks isn't present in _worker, fixing this would involve expanding the API to pass callback/errback functions all the way down so that _worker can invoke them at the right time. At least, as far as I can tell. Another option is to loosen up the test, so that it passes even if the ThreadPool creates one extra thread. Unfortunately, even though the unit test will only ever create one extra thread, user code could create as many as the ThreadPool allows to exist total, by just adding multiple work units to the queue in a single callback on the Deferred returned by deferToThread. Thoughts? Jp

1 0

[Twisted-Python] Could not create and run a Windows Service with Twisted 2.0.1 & Python 2.4.1
by Luc Stepniewski July 8, 2005

July 8, 2005

I've written a Twisted Windows Service which works perfectly with Twisted 1.3 and Python 2.3. I've used the sandboxed ntsvc from moonfallen's svn. I upgraded everything to Twisted 2.0.1 and Python 2.4.1 (even tried ActiveState's), and followed the information provided in moonfallen's README-service (that is, patch pywin32's boot_service.py). It compiles correctly to a .exe, without any supplementary error and I can install it correctly as a service (using 'xxxxxx.exe -install'). But When I try to launch it (in the GUI Service), it dies immediately, putting the following message in the Windows Event Observer: The instance's SvcRun() method failed File "win32serviceutil.pyc", line 742, in SvcRun File "ntsvc\runner.pyc", line 79, in SvcDoRun File "ntsvc\runner.pyc", line 48, in run File "twisted\application\app.pyc", line 31, in installReactor File "twisted\python\reflect.pyc", line 344, in namedModule exceptions.ImportError: No module named default I even cleaned my code to have just a .tac with 2 lines, and it does the same error. So I'm pretty sure it's not a problem in my code. Has anyone managed to create and run a service with Python 2.4.1 & Twisted 2.0.1? I'm using py2exe 0.5.4 and pywin32 204 both for python 2.4. Thanks, Luc -- Luc Stepniewski <luc.stepniewski(a)adelux.fr> Adelux - Securite, Linux Public key: <http://lstep.free.fr/pubkey.txt> Key BC0E3C2A fingerprint = A4FA466C68D27E46B427 07D083ED6340BC0E3C2A

3 4

[Twisted-Python] File Transfer/BitTorrent
by Thomas HERVE July 8, 2005

July 8, 2005

Hi all, I've seen a lot of threads talking around my problem but haven't found what I was looking for. My need : I want to have a file transfer application, with the following functionnalities (for now) : - Transfer from 1 to N - Integrity control - Failover/Resume - Handle big files (up to 5Gb) - Strong authentication/encryption (with client certificates) - Automatic actions when upload/download finished FTP is not really a possibility (mainly because of the nasty network configuration it needs). My first thought was : hey, it looks like BitTorrent could fit my needs, just a little SSL could make it. Unfortunately, after some tests, I realized that the reference implementation of BitTorrent is really messy : plugging a SSL part is a challenge, resulting in something not so great. BitTornado is cleaner, but also lacks documentation (yes, more than twisted !). So why not implement BitTorrent with Twisted ? It could be something great for the Twisted visibility. But as I don't care about BitTorrent compatibility, it might be better to start a new protocol from scratch. So : 1) Is there some people interested in BitTorrent client/server with Twisted ? 2) If not, does a draft/standard exist of good file transfer protocol, that could be done with Twisted ? Thanks for your opinions, -- Thomas

3 3