Mailman 3 November 2006 - Twisted

[Twisted-Python] Re: Callback blocked in event loop
by Ed Suominen 11 Nov '06

11 Nov '06

> On Fri, 10 Nov 2006 15:24:51 +0000 (GMT), Matt P wrote: >>> Events may be raised in the server, as a twisted.pb call to >>> fireEvent on the server and also come in on a separate thread (from >>> a JMS message queue) and get posted onto the DefferedQueue for >>> events. >>> >>> If events are raised in the server, remote listeners get their >>> pb.Referenceable callback immediately. It works as I would like. >>> >>> [snip] >>> >>> So, only in the case where teh DeferredQueue.put(...) is called >>> from another thread do I have the problem. I can't see why the >>> blocking. Does this behaviour make sense to anyone!? > > DeferredQueue isn't threadsafe. Instead of calling put() directly from > a non-reactor thread, call it with reactor.callFromThread: > > reactor.callFromThread(q.put, value) > > Jean-Paul Matt, this is exactly the sort of thing for which I wrote the taskqueue module that is patiently awaiting review in ticket #1768 (http://twistedmatrix.com/trac/ticket/1768). It is a critical part of my sAsync project ("SQLAlchemy done asynchronously"). You can check out the latest version as used in that project at: http://foss.eepatents.com/sAsync/browser/trunk/sasync/taskqueue.py. The part that would appear to be of particular interested to you is the ThreadQueue subclass of TaskQueue. Best regards, Ed Suominen

1 0

[Twisted-Python] Newbie to Twisted. Converting Asyncore App to Twisted App
by Sean Roark 10 Nov '06

10 Nov '06

I am new to this Twisted Architecture so if my plan makes no sense, please guide me down a smoother path. I have a program using Asyncore as the communications interface but I want to switch to Twisted. Server1 Server 2 <-------TCP------| | <---TCP------- | My App | ----TCP--------->| |----TCP-----> Basically MyApp takes stuff from Server 1, modifies it in some way, then sends it to server 2 and vice versa for Server 2 to Server 1. It is a translator between two ICD's languages spoken by two different systems. All data in and out is raw. I have modules written that can parse the raw data into objects, translate the objects into the other ICD, then repack the objects into raw data. Currently, My App uses asyncore to manage the socket between the servers as pictured above. Asyncore sticks the raw data on a Queue for the next object to retrieve. Each object is a thread that blocks on the Queue until data arrives. When a object receives data on the Queue, it performs its operation and sticks the modified data on another queue for the next thread to handle. Eventually, data ends up on a Queue being monitored by an Asyncore dispatcher that grabs the data and sends it to a server. Server1 ----TCP----->Asyncore.recv() --put--> Queue --get-->Parser--put--> Queue --get--> Translator --put--> Queue --get--> Packer --put---> Queue --get-->Asyncore.send() I would like to use Twisted for the following reasons: 1. Add a gui, stand alone or web based, allowing users to view statistics being generated by modules and to inject messages to one of the servers. Other services may be necessary in the future 2. We have a problem with asyncore behaving differently on Windows servers then Linux servers with regards to reconnecting after network drops. It is greatly beneficial if we can run on either platform. 3. Twistd is a great way to run this program as a daemon on Linux but still works on Windows 4. My own personal education 5. Twisted seems so cool Questions that have arisen as i try to code this in Twisted. 1. What is the proper method for gluing my processing modules to Twisted? I have followed the finger tutorial and it seems I should call them from a Service class. Finger tutorial part 3<http://twistedmatrix.com/projects/core/documentation/howto/tutorial/style.h…> This leads to question 2 I have toyed with using DeferredQueue's to pass data between the modules. I would pretty much follow the pattern above, but the DeferredQueue's should allow my modules to NOT run as threads. Instead they just fire events when data arrives on their Queue. 2. How do I have My App send data to the servers via events? In the finger tutorial Part 3 there are getUser and getUsers methods that are called by the FingerProtocol class via the FingerClientFactory. This all makes sense for processing data from the network and responding to it, but suppose I want some other object, say my Packer object, to fire an event that will make FingerProtocol send some data from Packer to the server? How do I expose a method to do this if Packer only knows about the Service, but the Service has no access to methods in FingerProtocol? I have come up with two solutions to this problem, neither of which is impressive to me. First solution, use DeferredQueue's to pass all the data around. This does work as a prototype, but I have no Service class wrapping it. class QueueProtocol(Protocol): "Handles the network receive and send work for a send and a receive queue" def dataReceived(self, data): print "Received Data:",data self.factory.putData(data) def connectionMade(self): print "Conn Made" self.sendData() def sendData(self): print "sendData" self.factory.getData().addCallback(self.sendHelp) def sendHelp(self,data): print "SendHelp",data self.transport.write(data+"\r\n") self.sendData() class QueueClientFactory(ClientFactory): "Handle the Connection" protocol = QueueProtocol def __init__(self,recQueue, sendQueue): self.recQueue = recQueue #both deferredQueue's self.sendQueue = sendQueue def putData(self,data): self.recQueue.put(data) def getData(self): "Returns a defered because sendQueue is a DeferredQueue" return self.sendQueue.get() Second solution, that I have not managed to code. I have the feeling I'm way of the mark here. Each component (yes I have been reading the Zope docs) in the application knows about interfaces. All components will implement a receive(data) method and a process(data) method. Each of these objects can pass data to other components by adding another components receive(data) method as a callback to its own process(data) method. Parse def receive(data): self.process(data).addCallback( Translate.receive(data) ) The Parse object receives data from somewhere. The process method takes this raw data and parses it into an object. When process has completed the callback method, defined in the Translate object, is called with the new object as a parameter. Parse's Factory class will take an instance of Translate in __init__. Some guidance would make my brain hurt less. Thanks, -- Sean Roark

1 0

[Twisted-Python] I want to know whether Twisted can run in Python2.5 .
by 张沈鹏 10 Nov '06

10 Nov '06

I want to know whether Twisted can run in Python2.5 .

3 3

[Twisted-Python] Callback blocked in event loop
by Matt P 10 Nov '06

10 Nov '06

Hello I've got a bit of an issue that someone may recognise and bea ble to set me straight! I have a twisted.spread server which, amongst other things, provides an event pub/sub engine. The subscription manager and event publisher runs in the twisted event loop: an event is posted onto a DeferredQueue and the callback on that Queue iterates through listeners calling eventFired(...). Some event listeners run in-server, but others are seperate processes which register by calling the register function on the server and providing a pb.Referenceable that the event publisher can call eventFired(...) on. Events may be raised in the server, as a twisted.pb call to fireEvent on the server and also come in on a separate thread (from a JMS message queue) and get posted onto the DefferedQueue for events. If events are raised in the server, remote listeners get their pb.Referenceable callback immediately. It works as I would like. However, if a message comes in off the JMS bus, the DeferredQueue callback gets called, it in turns calls the referenceable callback and nothing happens. Only when the server handles *any* other Twisted event do the event callbacks execute and my remote listeners get the message. So, only in the case where teh DeferredQueue.put(...) is called from another thread do I have the problem. I can't see why the blocking. Does this behaviour make sense to anyone!? Many thanks in advance for any help!! Matthew

3 2

[Twisted-Python] SMTP server
by Jan Bakuwel 10 Nov '06

10 Nov '06

Hoi all, This is my first message to the list so I think its appropriate to express my appreciation for all the excellent work done by the people who created & contributed to the twisted libraries. I have a small but interesting problem that I'd like to ask your advise on, please see the description below. If you notice that I haven't understood something properly, I hope you'll correct me :-) I've implemented a SMTP server (I've called it "xsmtp") using the twisted & python libraries. A MTA (Postfix) delivers email messages to the xsmtp server: the MTA opens a single SMTP session to the xsmpt server, then multiple "RCPT TO"s follow, one for each recipient. As a result The "validateFrom" procedure is called by the twisted library once, followed by multiple calls to the "validateTo" procedure. This is creating a small problem for me. Since I must process the email in the validateTo procedure, I will now receive multiple copies of the same email (albeit for different recipients). One of the tasks of the xsmtp server is to transmit these emails to another xsmtp server (SMTP cannot be used due to the inherently high latency connection between the two). Ideally I do not transfer the same email twice but would like to leave the delivery to multiple recipients up to the MTA on the other end. In a schema: -> MTA -> xsmtp -- high latency link --> xsmtp -> MTA -> local delivery To implement smtp.IMessageDelivery I have to provide for the following procedures: validateFrom: called once shortly after SMTP session is initiated receivedHeader: called multiple times for each recipient validateTo: called multiple tiems for each recipient One could say that "validateFrom" is the start of the process (of receiving an email) "receivedHeader/validateTo" is the core of the process. What I am missing is something that indicates the end of the process. It would help me tremendously if smtp.IMessageDelivery would also provide an abstract method that is called once the message delivery is completed. Any suggestions, thoughts are very welcome; perhaps this is already possible with the twisted libraries in a different way as I'm suggesting here? best regards, Jan -- "It is an illusion to think that we are free to apply our increasing knowledge in any way we choose. In reality we are struggling to deal with the consequences of it." -- John Gray ---- Electronic Technology Coordinator Information/Communication/Technology Greenpeace International Ottho Heldringstraat 5 1066 AZ AMSTERDAM Netherlands (MET) direct +31 (0)20 7182084 fax +31 (0)20 5148151 reception +31 (0)20 5148150 email jan.bakuwel&int.greenpeace.org private jan.bakuwel&hccnet.nl (replace & by @ in the emailaddress)

2 3

[Twisted-Python] article
by Katarzyna Winiarz 09 Nov '06

09 Nov '06

Hello, We need of an article about project Twisted- Python for the magazine for programers and developers of Linux (www.linuxdevjournal.com). This magazine is published bimonthly in french. The article must be strictly technic and pratically oriented. Longeur it's 10-15 pages in text editor(20 000 - 30 000 caracters). We will pay 25 euros for the page of the texte past the print, without listings and figures. Kasia -- Katarzyna Winiarz katarzyna.winiarz(a)linuxdevjournal.com www.linuxdevjournal.com tél. 0048 22 887 13 44 fax. 0048 22 887 10 11

1 0

[Twisted-Python] Installation of Twisted (Sumo) failing in web, words (I think)
by James Kanze 09 Nov '06

09 Nov '06

I think I'm having a problem with the installation of the twisted Sumo package (TwistedSumo-2006-02-12, but Twisted-2.4.0 doesn't seem to work either). Basically, I'm trying to get PyAIM-t working, under Linux (on a machine with a very stripped down configuration, and on which I don't have root access). PyAIM-t requires Twisted (core, web and words, at least), and recommends the 2.2.0 version, which is in the TwistedSumo package above. So I installed and built the package, to the best of my ability (this is my first time dealing with Python). But PyAIM-t still doesn't seem to find what it is looking for. I added a few print statements where the problem was occuring, so the code looks something like this: try: print sys.path print "trying domish" from twisted.words.xish.domish import SuxElementStream, \ Element, unescapedFromXml print "got domish" # ... except ImportError: print sys.exc_info()[1] # ... The output here is: [... , '/home/team02/jakan/gnu/lib/python2.4/site-packages', ...] trying domish No module named words.xish.domish What makes me thing that the problem is in my installation of twisted: when I invoke "python setup.py" from the root, I get a lot of messages along the lines of: package init filie 'twisted/plugins/__init__.py' not found (or not a regular file) In addition, to get anything in the library directory from the path above, I have to add the --force flag, so that my install command looks like: python setup.py --prefix=/home/team02/jakan/gnu \ --exec-prefix=/home/team02/jakan/gnu/i686 --force I'm sure that I'm doing something stupid and obvious that I'm missing, but I don't quite no where to turn. (This is really my first contact with Python, starting yesterday. I want to learn it, but management wants a proof of concept that what we want to do will work before approving the project.) -- James Kanze (GABI Software) mailto:james.kanze@gmail.com Conseils en informatique orientée objet/ Beratung in objektorientierter Datenverarbeitung 9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

1 0

[Twisted-Python] Twisted article
by Katarzyna Winiarz 09 Nov '06

09 Nov '06

Hello, I am the editor of the magazines Linux Developer's Journal and MSCoder. Those magazines are designed for programers and developer's of plateforms Linux (ldj) and of Microsoft (MSCoder).They are published bimonthly in french, spanish and german. I would like to ask you if someone of you could write for us the article about Twisted in french linguistic version. I would be very content. Best regards Kasia -- Katarzyna Winiarz katarzyna.winiarz(a)linuxdevjournal.com www.linuxdevjournal.com www.mscoder.org tél. 0048 22 887 13 44 fax. 0048 22 887 10 11

2 1

[Twisted-Python] Understanding Deferreds/callback further
by Yi Qiang 09 Nov '06

09 Nov '06

Hi guys, I am still having trouble writing programs that will make the most use of deferreds. I will describe the problem I am trying to solve and the solution I've tried. The problem: I am trying to write a client that gets a job from a PB server, processes it, submits the result and gets another job. I need to be able to do this for multiple jobs (i.e. I need to be able to get 5 jobs, and start 5 similar chains: get job --> work on job --> submit result --> get another job). This looks like a perfectly solvable problem using deferreds since working on the job might take a while and using callbacks seems to be the most appropriate. So I have something like this in my main method: monitor = Monitor(server, port) monitor.connect() def connect(self): factory = pb.PBClientFactory() reactor.connectTCP(self.hostname, self.port, factory) return factory.login(self.credentials).addCallback( self._connected).addErrback( self._catch_failure) Then in self._connected I would like to start the chain of getting jobs/processing them/submitting jobs: def _connected(self, remoteobj): self.remoteobj = remoteobj for worker in self.workers: self.get_jobs(worker).addCallback( self._got_job, worker).addCallback( self.get_jobs, worker).addErrback( self._catch_failure) _got_job takes both a job and a worker and starts to process the job. Which for now returns a defer.succeed(). This is the part where I stumble on though, the exact implementation of how to make it so that one calls back the other. Any help would be appreciated. Please let me know also if you need more specific information. Thanks, Yi

2 6

Re: [Twisted-Python] Understanding Deferreds/callback further
by Samuel Reynolds 09 Nov '06

09 Nov '06

>Ok. The thing I am trying to design/code is a job distribution >system, for a distributed computation model. There is a server >which interacts with the job database (which is just a bsddb). The >server accepts connections from clients and distributes jobs to >them. The server also accepts connections from a 2nd type of >client, which submits jobs to the server. Sounds like a distributed task queue. Users/user processes submit tasks to the master queue manager (the "server") via a defined interface (HTTP post, SOAP, XML-RPC, PB, or ???). The master queue manager adds the requested tasks to the master job queue. A secondary queue manager on each "client" machine (the "monitor") requests tasks from the master queue when its local queue drops below a defined threshold number of jobs (or drops to zero). When the local queue is empty, the secondary queue manager periodically requests tasks from the master. It adds any tasks received to its local queue. For each received task, the secondary queue manager starts a worker task, immediately receiving a deferred. It registers success and error callbacks on the deferred. When it gets the success or error callback from a worker, the secondary queue manager records the result and reports it to the master queue controller (the "server"). When it has successfully reported to the master, it deletes the task from the local queue. Additional wrinkles: (1) Intertask dependencies may require that some tasks be performed sequentially, on the same slave machine, or both. (2) Some tasks may be more important than others, and so must be processed (or at least started) first. (3) Task cancellation at the master may require that the master be able to send asynchronous task-cancel instructions to the slaves. >The workers of course have to submit their results back to the >server, but in the system I envisioned the monitor takes care of >that by continuously polling the status of the workers. If the worker returns a deferred, the monitor doesn't need to poll; it'll get a callback when the worker completes (or fails). HTH. - Sam

1 0