Mailman 3 November 2003 - Twisted

[Twisted-Python] Re: Mixmaster Architecture
by Mark Evans 20 Nov '03

20 Nov '03

Thank you Moshe Zadka. What you're saying is that TCP connections can trigger ephemeral UDP pseudo-connections. The server communicates over just one UDP port, and data flow is bidirectional at all points in the chain. I'm not sure whether these facts affect the suggestion here. The mixmaster opens its own assigned UDP port to receive data from the server, as well as sending data to the server over another UDP port. > Make up your mind, telnet or raw? a Line-based protocol? You're right, I should. Here is the scoop. For manual device tests I'm using PuTTY in its "raw" mode. That is the origin of "raw." Basically the telnet sessions are line-based command/response activity. The devices don't utilize the full telnet protocol, just a small subset. The terms "client" and "server" may be mixed up, from a networking standpoint. These are just conventions in the legacy software, right or wrong. The original authors called the Java app a server, and the devices clients. However the devices supply information that the Java app requests of them. So the actual roles are reversed. Normally, when talking to devices that know its protocol, the system is very simple. Here's how it works without a mixmaster script. The server initiates communications. The server broadcats a public query packet, and devices respond with their names and technical features. The server constructs a list of known devices. From there, the server invokes various device capabilities. When finished, the server shuts down the devices. Why the mixmaster? Many devices offer identical capabilites, but don't know the server's protocol. The mixmaster allows the server to use them. My apologies for confusing terminology and my thanks for your answers. Mark

1 0

[Twisted-Python] Woven View factories: defered rendering?
by Michal Pasternak 20 Nov '03

20 Nov '03

Hi, I have a webpage, that is rendered via twisted.web.woven.page.Page class. In one of it's wvfactory_ methods, I need to send an e-mail. I would also like to check if sending of the e-mail was successful (eg. mailserver is up, running, and reachable via network). I thought about something like this: def wvfactory_render(self, req, node, model): # do some stuff sendEMail().addCallbacks(self.emailOkay, self.emailFailure) return # what should I return here? def emailOkay(self, *args): # render "thank you, e-mail sent" message def emailFailure(self, *args): # render "error" message My questions: 1) is it a good, Twisted Python way to do this? If not, a) what would be the correct one? b) can you point me to some docs? 2) if it is a good way: a) is it implemented? if yes: 1) what should I return from wvfactory_render? 2) what parameters would be passed to callbacks then? b) if not, do you think it would be worth implementing? I have seen something similar like the thing I want to code in some example sources: def render(...): [ launch some callbacks ] return server.NOT_DONE_YET ... but they were about Resource class, and I'm using Page here. Anyone? Regards, -- Michal Pasternak :: http://pasternak.w.lub.pl

4 6

[Twisted-Python] Mixmaster Architecture
by Mark Evans 20 Nov '03

20 Nov '03

Here's an architecture question for Twisted Matrix gurus. I need to implement a "mixmaster" scheme, as follows: |--> Remote client A UDP TCP | Server <---> Mixmaster <---|--> Remote client B | |--> Remote client C | ... (etc.) Here is the background data: Server <---> mixmaster is a custom binary protocol over UDP transport. Mixmaster <---> clients is custom protocol(s) over TCP transport. The most important client type uses Telnet (raw). Server and mixmaster script live on the same PC. Remote clients have individual IPs and MACs. Server is closed-source Java; I cannot change it. Server's UDP communications is stock Java from Sun. Clients are different closed-source systems (not PCs). Mixmaster script is my Python. How to architect it? The Mixmaster script needs to multiplex communications between clients and server. The server will think it's talking to N clients, all of which know its peculiar protocol. It's OK for the server to believe they have the same IP address 127.0.0.1. There is other identifying information in the server protocol to distinguish the clients (including MACs and device name strings). I have an idea how to proceed, but still would like to hear your advice, and thank you for it. Mark

2 1

[Twisted-Python] another probably very simple question
by Phil Christensen 20 Nov '03

20 Nov '03

[whoops, sorry about the previous incomplete message] this is actually the first time i've ever written a PB client in python. since my java client was calling the remote methods out of the awt/swing event thread, i never had to deal with this particular issue. while i'm waiting for the update to the java pb packages (*cough*itamar*cough* ;-) ), i figured it would be worthwhile to do up a text-based client in python. i know this relates to my poor understanding of deferreds, but i just can't figure out exactly how to go about doing what i want to do. i want to be able to continually prompt the user for a string that gets sent to a "parse" method on the remote object. i'm not exactly sure the right way to do this. do I just add a callback to the deferred returned by call remote? here's my code so far: def main(): config = Options() if(config['login'] == None): config['login'] = getpass.getuser() passwd = getpass.getpass("%s@%s's password: " % (config['login'], config['host'])) client = TextClient() factory = pb.PBClientFactory() reactor.connectTCP(config['host'], int(config['port']), factory) defer = factory.login(credentials.UsernamePassword(config['login'], passwd), client) defer.addCallback(connected) reactor.run() def connected(perspective): command = raw_input("> "); defer = perspective.callRemote("parse", command) defer.addCallback(connected)

5 7

[Twisted-Python] Unable to get the UDP multicasting examples to work
by Alan Rushforth 20 Nov '03

20 Nov '03

Hi, This may well be a very trivial question but i have been trying to get the UDP multicasting example in the "how to" section of the twisted documentation to work and have had no success. I can do TCP message and normal UDP but cant multicast it. It does not report any errors but nothing every seems to be received by the echo server. Don't I need to explicitly join the multicast group at any point? The code i have been using is as follows: The multicast client: from twisted.internet.protocol import DatagramProtocol from twisted.internet import reactor class Client(DatagramProtocol): def datagramReceived(self, data, host): print 'Response Recieved:', data.strip() print 'From Host:', host reactor.listenUDP(0, Client()).write('broadcast\n', ('224.0.0.1', 9999)) reactor.run() The original Echo server: from twisted.internet.protocol import DatagramProtocol from twisted.internet import reactor class Echo(DatagramProtocol): def datagramReceived(self, data, (host, port)): print "received %r from %s:%d" % (data, host, port) self.transport.write(data, (host, port)) reactor.listenUDP(9999, Echo()) reactor.run() Thanks for any help, Alan.

2 1

Re: [Twisted-Python] FTPClient storeFile help.
by Oisin Mulvihill 20 Nov '03

20 Nov '03

Hi, Glyph Lefkowitz wrote: > Oisin Mulvihill wrote: > >> I confess I really don't know how to handle the two deferred >> object that get returned by storeFile. I just add callback/ >> errorback for each, this could be the problem. > > > If you want to wait for both, use twisted.internet.defer.DeferredList. > Why are there two deferred objects returned by storeFile? The documentation says: storeFile (self ,path ): Store a file at the given path. This method issues the 'STOR' FTP command. Returns: A tuple of two Deferreds: * Deferred IFinishableConsumer . You must call the finish method on the IFinishableConsumer when the file is completely transferred. * Deferred list of control-connection responses. This really doesn't help me much. I get the impression that storeFile isn't sending the contents of file, because its waiting for me to stream the file contents somehow. Maybe via IFinishableConsumer? In my current program, using ftplib, I do the following to upload a file: : ftp.storbinary('STOR %s' % (filename), fp) : This has the filename and a file descriptor passed to it. Is something like this needed with storeFile? Please can someone give me an answer to this, as looking at the code for twisted.protocols.ftp isn't helping much either. This function seems to be a mystery to me and I want to understand how to use it properly. Thanks, om >> Can anyone enlighten me as to what I'm doing wrong. I would >> really like to use proper async behavior for doing the uploads, >> instead of using threads + ftplib. > > > The FTP implementation in Twisted is sub-optimal (although that's > mostly the server). If you don't get any satisfactory responses I > recommend assigning a bug to spiv in the tracker :). > -- Oisin Mulvihill Engines Of Creation Email: oisin(a)enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540

3 7

[Twisted-Python] home for ftp goodies?
by Jonathan Simms 20 Nov '03

20 Nov '03

I was talking with jp and itamar about the ftp extras. I've implemented an anonymous avatar, a realm, and have begun working on a method or two for filepath permissions checking. I wanted some opinions on how we can separate these extra 'goodies' from the protocol implementation itself. any thoughts?

1 0

[Twisted-Python] looking for twisted Tkinter documentation
by John Benson 20 Nov '03

20 Nov '03

Hi, I found the following at http://twistedmatrix.com/documents/howto/choosing-reactor#auto16: ----- Tkinter The support for Tkinter doesn't use a specialized reactor. Instead, there is some specialized support code: from Tkinter import * from twisted.internet import tksupport root = Tk() root.withdraw() # Install the Reactor support tksupport.install(root) An example Twisted application that uses Tk can be found in twisted/words/ui/tkim.py. ----- But I didn't find it in either the twisted or TwistedDocs directories in my Python 2.3 site-packages directory. The copyright,.py on my twisted installation shows twisted version 1.1.0. I also had a look at the examples index in http://twistedmatrix.com/documents/examples/#auto8 and found links to uses of qt, wx, gtk2 and curses, but nothing about Tkinter. Having loaded up on Tkinter books on the strength of the assertion that Tkinter is the standard, most cross-platform GUI for Python, I'm understandably anxious to use it with twisted. I'm writing a socket server that needs to both respond to incoming client requests and occasionally send unsolicited messages to the client based on Tkinter button presses and timer expirations. I was originally going use some lower-level Python modules, but I was intrigued by the stylistic and software engineering claims of twisted, especially the use of aspect-oriented programming as a tool for managing unrelated complexities, so I'm going to try to dive into twisted a little deeper before giving up and heading back to the shallow end of the pool. Thanks in advance for any information that will help me integrate Tkinter with twisted. P.S. I found the itamarst.org tutorials (formidavel!) and am working my way through them.

3 2

[Twisted-Python] Re: [Twisted-commits] Today is screw with twisted's guts day.
by Andrew Bennetts 20 Nov '03

20 Nov '03

On Wed, Nov 19, 2003 at 02:07:39PM -0700, exarkun CVS wrote: > Index: Twisted/twisted/internet/base.py > diff -u Twisted/twisted/internet/base.py:1.67 Twisted/twisted/internet/base.py:1.68 > --- Twisted/twisted/internet/base.py:1.67 Fri Oct 24 12:45:58 2003 > +++ Twisted/twisted/internet/base.py Wed Nov 19 14:07:38 2003 > @@ -400,12 +400,12 @@ > """Run all pending timed calls. > """ > if self.threadCallQueue: > - for i in range(len(self.threadCallQueue)): > - callable, args, kw = self.threadCallQueue.pop(0) > + for (f, a, kw) in self.threadCallQueue: > try: > - callable(*args, **kw) > + f(*a, **kw) > except: > - log.deferr() > + log.err() > + del self.threadCallQueue[:] > now = time() > while self._pendingTimedCalls and (self._pendingTimedCalls[-1].time <= now): > call = self._pendingTimedCalls.pop() I think this change introduces a race condition. It's possible for a thread to call callFromThread between the end of the for loop and the del statement. (callFromThread is defined as: def callFromThread(self, f, *args, **kw): """See twisted.internet.interfaces.IReactorThreads.callFromThread. """ assert callable(f), "%s is not callable" % f if threadable.isInIOThread(): self.callLater(0, f, *args, **kw) else: # lists are thread-safe in CPython, but not in Jython # this is probably a bug in Jython, but until fixed this code # won't work in Jython. self.threadCallQueue.append((f, args, kw)) self.wakeUp() ) So I think it's possible to lose thread calls. I *think* this patch will fix it, but more eyeballs are always welcome with threaded code: diff -u -r1.68 base.py --- twisted/internet/base.py 19 Nov 2003 21:07:38 -0000 1.68 +++ twisted/internet/base.py 20 Nov 2003 00:52:08 -0000 @@ -400,12 +400,14 @@ """Run all pending timed calls. """ if self.threadCallQueue: + count = 0 for (f, a, kw) in self.threadCallQueue: try: f(*a, **kw) except: log.err() - del self.threadCallQueue[:] + count += 1 + del self.threadCallQueue[:count] now = time() while self._pendingTimedCalls and (self._pendingTimedCalls[-1].time <= now): call = self._pendingTimedCalls.pop() -Andrew.

2 3

[Twisted-Python] Re: Browser oddities and Woven... (Thomas Raschbacher)
by Stuart Hungerford 20 Nov '03

20 Nov '03

Quoth twisted-python-request(a)twistedmatrix.com: >| I've developed a Woven application that generates three >| HTML pages, let's call them A, B, and C. >| >| A's render() method does a redirect to B. B's view methods >| generate an XHTML meta tag with the content attribute set >| to a refresh and new URL for C: >| >| <meta content="10;http://example.org/C" http-equiv="refresh" /> >| >AFAIK this would be the correct syntax: ><META name="refresh" content="3,http://www.acme.com/intro.html" /> >~ ^ >(ie ',' instead of ';') >http://www.w3.org/TR/WD-html40-970708/struct/global.html#h-7.1.3.2 Many thanks for pointing that out. Unfortunately it makes no difference to the behaviour with IE or Mozilla. I've since tried with Opera 7 and it goes into the endless loop too, which makes me think there's a problem with the (X)HTML my Woven pages are generating and Mozilla is being very forgiving. I've tried adding the equivalent HTTP headers to the request headers in Woven and also run the generated XHTML through the W3C validator. It reported a couple of errors with some <tr> attributes but nothing about the <meta> elements. This is the generated XHTML <head>: <?xml version="1.0"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"> <head> <meta content="en-us" http-equiv="Content-Language" /> <meta content="text/html; charset=iso-8859-1" http-equiv="Content-Type" /> <meta content="Pacific And Regional Archive for DIgital Sources in Endangered Cultures " name="description" /> <meta content="digital, digitisation, digitization, pacific, language, music, endangered, university, sydney, melbourne, ANU, australian, national, culture, metadata" name="keywords" /> <meta content="language resource, language resources, language archive, linguistic archive, open archives, metadata, dublin core, controlled vocabulary, linguistics, linguistic data, language data, Pacific, endangered culture, endangered language, musicology, indigneous song" name="keywords" /> <meta http-equiv="refresh" content="5,http://store.apac.edu.au:8181/paradisec/archive/request/download/238/AM3-008-008A.mp3" /> <link href="http://store.apac.edu.au:8181/PARADISEC/resources/css/paradisec.css" type="text/css" rel="stylesheet" /> <title>Downloading your file AM3/008/AM3-008-008A.mp3</title> </head> Cheers, Stu -- :: Stuart Hungerford (stuart.hungerford(a)anu.edu.au) :: ANU Internet Futures Group

1 0