Mailman 3 November 2006 - Twisted

[Twisted-Python] Re: Callback blocked in event loop
by Ed Suominen Nov. 11, 2006

Nov. 11, 2006

> On Fri, 10 Nov 2006 15:24:51 +0000 (GMT), Matt P wrote: >>> Events may be raised in the server, as a twisted.pb call to >>> fireEvent on the server and also come in on a separate thread (from >>> a JMS message queue) and get posted onto the DefferedQueue for >>> events. >>> >>> If events are raised in the server, remote listeners get their >>> pb.Referenceable callback immediately. It works as I would like. >>> &… [View More]

1 0

[Twisted-Python] Newbie to Twisted. Converting Asyncore App to Twisted App
by Sean Roark Nov. 10, 2006

Nov. 10, 2006

I am new to this Twisted Architecture so if my plan makes no sense, please guide me down a smoother path. I have a program using Asyncore as the communications interface but I want to switch to Twisted. Server1 Server 2 <-------TCP------| | <---TCP------- | My App | ----TCP--------->| |----TCP-----> Basically MyApp takes stuff from Server 1, modifies it in some way, then sends it to server 2 and vice … [View More]versa for Server 2 to Server 1. It is a translator between two ICD's languages spoken by two different systems. All data in and out is raw. I have modules written that can parse the raw data into objects, translate the objects into the other ICD, then repack the objects into raw data. Currently, My App uses asyncore to manage the socket between the servers as pictured above. Asyncore sticks the raw data on a Queue for the next object to retrieve. Each object is a thread that blocks on the Queue until data arrives. When a object receives data on the Queue, it performs its operation and sticks the modified data on another queue for the next thread to handle. Eventually, data ends up on a Queue being monitored by an Asyncore dispatcher that grabs the data and sends it to a server. Server1 ----TCP----->Asyncore.recv() --put--> Queue --get-->Parser--put--> Queue --get--> Translator --put--> Queue --get--> Packer --put---> Queue --get-->Asyncore.send() I would like to use Twisted for the following reasons: 1. Add a gui, stand alone or web based, allowing users to view statistics being generated by modules and to inject messages to one of the servers. Other services may be necessary in the future 2. We have a problem with asyncore behaving differently on Windows servers then Linux servers with regards to reconnecting after network drops. It is greatly beneficial if we can run on either platform. 3. Twistd is a great way to run this program as a daemon on Linux but still works on Windows 4. My own personal education 5. Twisted seems so cool Questions that have arisen as i try to code this in Twisted. 1. What is the proper method for gluing my processing modules to Twisted? I have followed the finger tutorial and it seems I should call them from a Service class. Finger tutorial part 3<http://twistedmatrix.com/projects/core/documentation/howto/tutorial/style.h…> This leads to question 2 I have toyed with using DeferredQueue's to pass data between the modules. I would pretty much follow the pattern above, but the DeferredQueue's should allow my modules to NOT run as threads. Instead they just fire events when data arrives on their Queue. 2. How do I have My App send data to the servers via events? In the finger tutorial Part 3 there are getUser and getUsers methods that are called by the FingerProtocol class via the FingerClientFactory. This all makes sense for processing data from the network and responding to it, but suppose I want some other object, say my Packer object, to fire an event that will make FingerProtocol send some data from Packer to the server? How do I expose a method to do this if Packer only knows about the Service, but the Service has no access to methods in FingerProtocol? I have come up with two solutions to this problem, neither of which is impressive to me. First solution, use DeferredQueue's to pass all the data around. This does work as a prototype, but I have no Service class wrapping it. class QueueProtocol(Protocol): "Handles the network receive and send work for a send and a receive queue" def dataReceived(self, data): print "Received Data:",data self.factory.putData(data) def connectionMade(self): print "Conn Made" self.sendData() def sendData(self): print "sendData" self.factory.getData().addCallback(self.sendHelp) def sendHelp(self,data): print "SendHelp",data self.transport.write(data+"\r\n") self.sendData() class QueueClientFactory(ClientFactory): "Handle the Connection" protocol = QueueProtocol def __init__(self,recQueue, sendQueue): self.recQueue = recQueue #both deferredQueue's self.sendQueue = sendQueue def putData(self,data): self.recQueue.put(data) def getData(self): "Returns a defered because sendQueue is a DeferredQueue" return self.sendQueue.get() Second solution, that I have not managed to code. I have the feeling I'm way of the mark here. Each component (yes I have been reading the Zope docs) in the application knows about interfaces. All components will implement a receive(data) method and a process(data) method. Each of these objects can pass data to other components by adding another components receive(data) method as a callback to its own process(data) method. Parse def receive(data): self.process(data).addCallback( Translate.receive(data) ) The Parse object receives data from somewhere. The process method takes this raw data and parses it into an object. When process has completed the callback method, defined in the Translate object, is called with the new object as a parameter. Parse's Factory class will take an instance of Translate in __init__. Some guidance would make my brain hurt less. Thanks, -- Sean Roark [View Less]

1 0

[Twisted-Python] I want to know whether Twisted can run in Python2.5 .
by 张沈鹏 Nov. 10, 2006

Nov. 10, 2006

I want to know whether Twisted can run in Python2.5 .

3 3

[Twisted-Python] Callback blocked in event loop
by Matt P Nov. 10, 2006

Nov. 10, 2006

Hello I've got a bit of an issue that someone may recognise and bea ble to set me straight! I have a twisted.spread server which, amongst other things, provides an event pub/sub engine. The subscription manager and event publisher runs in the twisted event loop: an event is posted onto a DeferredQueue and the callback on that Queue iterates through listeners calling eventFired(...). Some event listeners run in-server, but others are seperate processes which register by calling the … [View More]

3 2

[Twisted-Python] SMTP server
by Jan Bakuwel Nov. 10, 2006

Nov. 10, 2006

Hoi all, This is my first message to the list so I think its appropriate to express my appreciation for all the excellent work done by the people who created & contributed to the twisted libraries. I have a small but interesting problem that I'd like to ask your advise on, please see the description below. If you notice that I haven't understood something properly, I hope you'll correct me :-) I've implemented a SMTP server (I've called it "xsmtp") using the twisted & python … [View More]libraries. A MTA (Postfix) delivers email messages to the xsmtp server: the MTA opens a single SMTP session to the xsmpt server, then multiple "RCPT TO"s follow, one for each recipient. As a result The "validateFrom" procedure is called by the twisted library once, followed by multiple calls to the "validateTo" procedure. This is creating a small problem for me. Since I must process the email in the validateTo procedure, I will now receive multiple copies of the same email (albeit for different recipients). One of the tasks of the xsmtp server is to transmit these emails to another xsmtp server (SMTP cannot be used due to the inherently high latency connection between the two). Ideally I do not transfer the same email twice but would like to leave the delivery to multiple recipients up to the MTA on the other end. In a schema: -> MTA -> xsmtp -- high latency link --> xsmtp -> MTA -> local delivery To implement smtp.IMessageDelivery I have to provide for the following procedures: validateFrom: called once shortly after SMTP session is initiated receivedHeader: called multiple times for each recipient validateTo: called multiple tiems for each recipient One could say that "validateFrom" is the start of the process (of receiving an email) "receivedHeader/validateTo" is the core of the process. What I am missing is something that indicates the end of the process. It would help me tremendously if smtp.IMessageDelivery would also provide an abstract method that is called once the message delivery is completed. Any suggestions, thoughts are very welcome; perhaps this is already possible with the twisted libraries in a different way as I'm suggesting here? best regards, Jan -- "It is an illusion to think that we are free to apply our increasing knowledge in any way we choose. In reality we are struggling to deal with the consequences of it." -- John Gray ---- Electronic Technology Coordinator Information/Communication/Technology Greenpeace International Ottho Heldringstraat 5 1066 AZ AMSTERDAM Netherlands (MET) direct +31 (0)20 7182084 fax +31 (0)20 5148151 reception +31 (0)20 5148150 email jan.bakuwel&int.greenpeace.org private jan.bakuwel&hccnet.nl (replace & by @ in the emailaddress) [View Less]

2 3

[Twisted-Python] article
by Katarzyna Winiarz Nov. 9, 2006

Nov. 9, 2006

Hello, We need of an article about project Twisted- Python for the magazine for programers and developers of Linux (www.linuxdevjournal.com). This magazine is published bimonthly in french. The article must be strictly technic and pratically oriented. Longeur it's 10-15 pages in text editor(20 000 - 30 000 caracters). We will pay 25 euros for the page of the texte past the print, without listings and figures. Kasia -- Katarzyna Winiarz katarzyna.winiarz(a)linuxdevjournal.com www.… [View More]

1 0

[Twisted-Python] Installation of Twisted (Sumo) failing in web, words (I think)
by James Kanze Nov. 9, 2006

Nov. 9, 2006

I think I'm having a problem with the installation of the twisted Sumo package (TwistedSumo-2006-02-12, but Twisted-2.4.0 doesn't seem to work either). Basically, I'm trying to get PyAIM-t working, under Linux (on a machine with a very stripped down configuration, and on which I don't have root access). PyAIM-t requires Twisted (core, web and words, at least), and recommends the 2.2.0 version, which is in the TwistedSumo package above. So I installed and built the package, to the best of my … [View More]

1 0

[Twisted-Python] Twisted article
by Katarzyna Winiarz Nov. 9, 2006

Nov. 9, 2006

Hello, I am the editor of the magazines Linux Developer's Journal and MSCoder. Those magazines are designed for programers and developer's of plateforms Linux (ldj) and of Microsoft (MSCoder).They are published bimonthly in french, spanish and german. I would like to ask you if someone of you could write for us the article about Twisted in french linguistic version. I would be very content. Best regards Kasia -- Katarzyna Winiarz katarzyna.winiarz(a)linuxdevjournal.com www.… [View More]

2 1

[Twisted-Python] Understanding Deferreds/callback further
by Yi Qiang Nov. 9, 2006

Nov. 9, 2006

Hi guys, I am still having trouble writing programs that will make the most use of deferreds. I will describe the problem I am trying to solve and the solution I've tried. The problem: I am trying to write a client that gets a job from a PB server, processes it, submits the result and gets another job. I need to be able to do this for multiple jobs (i.e. I need to be able to get 5 jobs, and start 5 similar chains: get job --> work on job --> submit result --> get another job). This … [View More]

2 6

Re: [Twisted-Python] Understanding Deferreds/callback further
by Samuel Reynolds Nov. 9, 2006

Nov. 9, 2006

>Ok. The thing I am trying to design/code is a job distribution >system, for a distributed computation model. There is a server >which interacts with the job database (which is just a bsddb). The >server accepts connections from clients and distributes jobs to >them. The server also accepts connections from a 2nd type of >client, which submits jobs to the server. Sounds like a distributed task queue. Users/user processes submit tasks to the master queue manager (the … [View More]

1 0