Mailman 3 March 2009 - Twisted

[Twisted-Python] So how does everyone use an ORM in twisted?
by Ryan Lepidi March 23, 2009

March 23, 2009

I have searched a bit, and it seems as though you can't use SQL Alchemy with twisted due to SA being thread unsafe. The only solution I found was sAsync, but this is old and unmaintained. What is everyone else doing for database access for a multiuser server? Is everyone really using raw sql queries or is there some easy solution that I'm missing? I'm not even stuck on SQL Alchemy; if there is some other ORM that works in this case, I'd be willing to learn it.

3 2

[Twisted-Python] Stackless Twisted shocking performance
by Simon Pickles March 23, 2009

March 23, 2009

Hi, Now I am first to admit my problem lies within my app design somewhere, but I thought I'd raise it here for advice. I have a server made from a central stackless python hub, with a boost python c++ interface, to which c++ clients connect. This hub is 'surrounded' by stackless python zones which are passed work by the hub. The zones send the updates to the hub which passes them onto the clients. The hub and zones communicate using twisted.perspectivebroker. This all seemed to work … [View More]

3 3

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com March 22, 2009

March 22, 2009

1 0

[Twisted-Python] Re: [Stackless] Stackless Twisted shocking
by Andrew Francis March 21, 2009

March 21, 2009

Hi Simon: --- On Sat, 3/21/09, stackless-request(a)stackless.com <stackless-request(a)stackless.com> wrote: > As more updates are sent zone->hub, per second, the > incoming client->hub->zone messages take SECONDS to get thru! > This is with approx 50 messages per second between a single zone and the >hub. > Is this the sort of performance I can expect? I am not an expert in performance tuning... and I have to do some performance testing soon. That said, I have … [View More]

1 0

[Twisted-Python] LineReceiver MAX_LENGTH limitation
by Pet March 18, 2009

March 18, 2009

Hi, in LineReceiver there is MAX_LENGTH = 16384 limitation. Is it possible to set it bigger at start of an application? Regards, Pet

2 2

[Twisted-Python] Re: So how does everyone use an ORM in twisted?
by Maciej Szumocki March 18, 2009

March 18, 2009

> From: Ryan Lepidi <ryeguy1(a)gmail.com> > Subject: [Twisted-Python] So how does everyone use an ORM in twisted? > To: twisted-python(a)twistedmatrix.com > Message-ID: > <bb39cd350903170805p2fca87casbc3cd6c932bb9439(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > I have searched a bit, and it seems as though you can't use SQL Alchemy with > twisted due to SA being thread unsafe. The only solution I found was sAsync, > but this … [View More]

6 7

[Twisted-Python] Ticket 1009
by Andrew Francis March 17, 2009

March 17, 2009

Hi Colleagues: I guess I am making today "Look at Twisted Documentation Day." I am looking at ticket 1009. For inline generator examples, here is the example I use in "Adventures in Stackless Python Twisted Integration." In turn, this example comes from the discussions in I believe January 2008 with JP Calerone. I can modify the example (and perhaps add illustrations) to show situations where inline generators are handy. """ Example of using an inline generator to make a Twisted call from … [View More]

1 0

[Twisted-Python] Ticket 1138 Re: Twisted-Python Digest, Vol 60, Issue 12
by Andrew Francis March 17, 2009

March 17, 2009

Hello Colleagues: >For ticket 1138 (cool movie), when I access the URL: >http://twistedmatrix.com/projects/core/documentation/howto/tutorial/deferre… >I get, URL not found. Which deferred tutorial are you referring to? >"Deferreds are Beautiful (A Tutorial) at : >http://twistedmatrix.com/projects/core/documentation/howto/deferredindepth.…. >I see the reference to python.util there. Looking at Ticket 1138. Again, I believe the ticket should be referring to the … [View More]

1 0

[Twisted-Python] Novice question on twisted framework
by Sury Soni March 17, 2009

March 17, 2009

Hi, I have an existing application, written using Ice Middleware (www.zeroc.com) This application is my source of infinite queue (server) Following is my application (processing client to my infinite queue server) idea to be implemented using Twisted Framework. I can loop through my infinite queue and receive email data (Using .next() kind of function) Using this email data object, I need to trigger event to send email using ESMTPSenderFactory and ESMTPSender. As, later I … [View More]would be extending ESMTP* classes to do my little funky stuff. What is the best code design I can use to implement this stuff. Following is basic algorithm kind of code I am doing right now, but I feel I am missing something. #================ #Code Begins Here: .tac file #================ import StringIO from twisted.application import service from twisted.application import internet from twisted.internet import protocol, defer from twisted.mail import smtp, relaymanager from twisted.internet import reactor Def process_queue: While True: Try: Data = my_queue.next() getMailExchange('localhost).addCallback(cbMailExchange, data) Except: Break Reactor.stop() reactor.callLater(0, process_queue) application = service.Application("SMTP Client Tutorial") class SMTPTutorialClient(smtp.ESMTPClient): mailFrom = "test(a)test.com" mailTo = "receiver(a)test.com" mailData = '''\ Date: Fri, 6 Feb 2004 10:14:39 -0800 From: Test <test(a)test.com> To: Receiver <receiver(a)test.com> Subject: Test Mail! Hello, how are you, goodbye. ''' def __init__(self, secret="", identity="", data = ""): smtp.ESMTPClient.__init__(self, secret=secret, identity=identity) self.secret = secret self.identity = identity self.data = data def getMailFrom(self): result = self.mailFrom self.mailFrom = None return result def getMailTo(self): return [self.mailTo] def getMailData(self): return StringIO.StringIO(self.mailData+self.data) def sentMail(self, code, resp, numOk, addresses, log): print 'Sent', numOk, 'messages' #reactor.stop() def getMailExchange(host): def cbMX(mxRecord): return str(mxRecord.name) return relaymanager.MXCalculator().getMX(host).addCallback(cbMX) class SMTPClientFactory(protocol.ClientFactory): protocol = SMTPTutorialClient def __init__(self, data): self.data = data def buildProtocol(self, addr): return self.protocol(secret=None, identity='example.com', data=self.data) def cbMailExchange(exchange, data): smtpClientService = internet.TCPClient(exchange, 2500, SMTPClientFactory(data)) smtpClientService.setServiceParent(application) #reactor.run() reactor.run() #================ #Code Ends Here #================ I have couple of #reactor.run() commented code, I feel like lost between application and reactor based code. My understanding is that function process_queue drives the control flow of my application. Please correct me. What is happening is, when my loop is finished in function process_queue, then only actual events of SMTP factory and protocol objects are created. Therefore, I am not getting event driven code, which is the beauty of Twisted Framework. Please advice the best document to look and fix this code. Thanks. ____________________________________ Sury Prakash Soni Developer ____________________________________ Next Digital Level 8, 15 William St, Melbourne VIC 3000 Australia p +61 3 8612 6888 f +61 3 8612 6899 m 0433 661 327 ssoni(a)nextdigital.com <mailto:ssoni@nextdigital.com> www.nextdigital.com <http://www.nextdigital.com/> ____________________________________ This email and any attachments are intended only for the use of the recipient and may be confidential and/or legally privileged. Next Digital Group Pty Ltd ("Next Digital") disclaims liability for any errors, omissions, viruses, loss and/or damage arising from using, opening or transmitting this email. If you are not the intended recipient you must not use, interfere with, disclose, copy or retain this email and you should notify the sender immediately by return email or by contacting Next Digital by telephone on +61 3 8612 6888. [View Less]

2 1

[Twisted-Python] Noob Question
by Shelby Ramsey March 15, 2009

March 15, 2009

Hello: I've been playing with twisted (to take advantage of deferreds) and trying to accomplish this: -- log into a machine as a client -- send auth -- receive "events" from the machine The events have a Content-Length: \d+ as the protocol for determining the length of the message. What I'm struggling with is how to parse that and then receive just that length as an individual message ... Thanks for any help that can be provided. SDR

4 12

[Twisted-Python] Twisted.mail SMTP server question
by Kalman KHIRALY March 15, 2009

March 15, 2009

Hi! I would like to use twisted mail for the following purpose: Redirect all incoming myname(a)mydomain.com mails to a specified gmail address: myname(a)mydomain.com -> myname.mydomain(a)gmail.com Is it possible to use Twisted.Mail for this? Any pointers would be highly appreciated. If not can somebody recommend me a nice python solution for this? Best regards, Khiraly ps: 4 year ago I did the above feature using postfix and mysql. It took me more the 2 months to correctly set up all … [View More]

4 3

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com March 15, 2009

March 15, 2009

1 0

[Twisted-Python] Using sqlalchemy in twisted.
by Peter Cai March 13, 2009

March 13, 2009

Hi, all I am using sqlalchemy in twisted in my project in the way below. Defer any database operation so the twisted's main thread won't be blocked. And I use scoped_session, so that sessions won't have to be created again and again. ====================================== class Database() def __init__(self, conn_str): self.conn_str = conn_str self.engine = create_engine(self.conn_str, echo=False) self.Session = scoped_session(sessionmaker(bind = self.engine, … [View More]

4 8

[Twisted-Python] Gsoc 2009
by kartik rustagi March 11, 2009

March 11, 2009

Hello everyone, Is twisted applying as an organization in this year's Gsoc? I as a student will be interested in working for it. Regards Kartik Rustagi

2 1

[Twisted-Python] getting sshsimpleserver.py to work within a wrapper
by jkv March 10, 2009

March 10, 2009

Hi, I just recently got aware of twisted and im currently playing around with it. Is there any way to make reactor.listen listen on stdio instead of a real network connection? In one of the examples (sshsimpleserver.py) reactor.listenTCP is used to listen on the network socket, as far as i can read from the twisted core documentation there are no builtin way to make reactor.listen* listen on stdin instead... Any hints? Regards, jkv

3 3

[Twisted-Python] twistd tap file are not created
by Pet March 9, 2009

March 9, 2009

Hello, I'm trying an example from twisted book and when I run this: twistd -n reverse_app.py application works, but no .tap file is created Another question, how are twisted daemons actually stopped? Pet reverse_app.py from twisted.application import service import reverse application = service.Application("Reverser") reverserService = reverse.ReverserService() reverserService.setServiceParent(application) reverse.py: from twisted.application import service, internet from twisted.… [View More]

2 2

[Twisted-Python] sending large files from web2 http server
by Markus Wanner March 9, 2009

March 9, 2009

Hi, I'm trying to stream longish data via web2, but experience sudden stalls in data transfer, followed by a connection abort after a certain timeout. I can't completely reproduce the issue, yet, but figured that the size of the blocks I'm returning via the stream's read method affects the failure rate (an stream.IByteStream implementation). I've tried with chunk sizes between 1 MiB down to 256 bytes. At that rate, I suddenly get the following unhandled errors, which don't seem to have to do … [View More]

3 4

[Twisted-Python] Error when trying to import reactor
by Ian Fountain March 8, 2009

March 8, 2009

I am new to all this so you will have to forgive me. I have a vista machine with python 2.5 on it and I just installed twisted (py2.5), pyOpenSSL (py2.5), and pycrypto (py2.5) but when I try to do the following: "from twisted.internet import reactor" I get the following error: "This application has failed to start because libeay32.dll was not found. Re-installing the application may fix this problem." I then found a copy of libeay32.dll and put it in my System32 folder and then get the … [View More]

3 5

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com March 8, 2009

March 8, 2009

1 0

[Twisted-Python] URL not Found Re: Twisted developers: please donate your time!
by Andrew Francis March 7, 2009

March 7, 2009

Hi Jean-Paul: I am not sure if I should take this off-line.... >http://twistedmatrix.com/trac/query?>status=new&status=assigned&status=reopened&keywords=~documentation&order=p>riority shows all of the open tickets with the "documentation" keyword. >So there's a lot to choose from, many are about the reactor. Okay I am looking through the list. For ticket 1138 (cool movie), when I access the URL: http://twistedmatrix.com/projects/core/documentation/howto/… [View More]

1 0

[Twisted-Python] Reentrant reactor iteration
by Marcel Keller March 7, 2009

March 7, 2009

Hi, I am working on the VIFF project (viff.dk) which uses Twisted. I found out that our code is sometimes inefficient because we are generating many deferreds (maybe about 10000) in a callback. While doing that, no network communication is performed. Therefore, I investigated the possibility of adding a function to the reactor which is called every iteration and from which the iteration could be called safely. Then, we could generate all deferreds in that function and activate the … [View More]reactor from to time. See the attached patch for details. Some of our code runs twice as fast when using that hack. Are there any chances something similar might be included in Twisted? Or does anyone have a better solution for the described problem? Furthermore, I have a question, which is probably related. The documentation of IReactorCore says about the iterate() function: "The reactor must have been started (via the run() method) prior to any invocations of this method. It must also be stopped manually after the last call to this method (via the stop() method). This method is not re-entrant: you must not call it recursively; in particular, you must not call it while the reactor is running." This looks to me as if the reactor needs to be running and not running at the same time so that iterate() can be called. Is there an error in my reasoning? Best regards, Marcel Keller --- /usr/lib/python2.5/site-packages/twisted/internet/base.py 2008-07-29 22:13:54.000000000 +0200 +++ internet/base.py 2009-02-20 12:27:42.000000000 +0100 @@ -1127,17 +1127,32 @@ self.startRunning(installSignalHandlers=installSignalHandlers) self.mainLoop() + + def setLoopCall(self, f, *args, **kw): + self.loopCall = (f, args, kw) + + + def myIteration(self, t): + # Advance simulation time in delayed event + # processors. + self.runUntilCurrent() + + if (t is None): + t2 = self.timeout() + t = self.running and t2 + + self.doIteration(t) + + if ("loopCall" in dir(self)): + f, args, kw = self.loopCall + f(*args, **kw) + def mainLoop(self): while self._started: try: while self._started: - # Advance simulation time in delayed event - # processors. - self.runUntilCurrent() - t2 = self.timeout() - t = self.running and t2 - self.doIteration(t) + self.myIteration(None) except: log.msg("Unexpected error in main loop.") log.err() [View Less]

3 4

[Twisted-Python] StandardIO flush'n
by Ben Hyde March 7, 2009

March 7, 2009

I'm new to this, but can somebody explain this behavior to me? Thanks! bash-3.2$ (echo a; echo b; sleep 1; echo c) | /tmp/bar.py line: a line: b a b line: c c bash-3.2$ (echo a; echo b; echo c) | /tmp/bar.py line: a line: b line: c bash-3.2$ cat /tmp/bar.py #!/usr/bin/env python from twisted.internet import reactor from twisted.protocols.basic import LineOnlyReceiver from twisted.internet.stdio import StandardIO class MyLineOnlyReceiver(LineOnlyReceiver): delimiter = '\n' def … [View More]

1 0

[Twisted-Python] newbie problem with SMTPClient
by Dave Britton March 7, 2009

March 7, 2009

I love the idea of twisted but I think I must have a twisted learning disability, as I have gotten nowhere in what ought to be a simple matter. I need to send out emails to small groups from my apache server running a python cgi using mod_python, but my hosting service doesn't have a MTA. Instead of learning how to install and configure exim I thought I would use twisted to make a simple mail client. I started with the tutorial example that appears at: http://twistedmatrix.com/projects/mail/… [View More]documentation/tutorial/smtpclient/sm… (.... is indenting) ============ tutorial code ================ import StringIO from twisted.application import service application = service.Application("SMTP Client Tutorial") from twisted.application import internet from twisted.internet import protocol from twisted.internet import defer from twisted.mail import smtp, relaymanager class SMTPTutorialClient(smtp.ESMTPClient): ....mailFrom = "tutorial_sender(a)example.com" ....mailTo = "tutorial_recipient(a)example.net" ....mailData = '''\ Date: Fri, 6 Feb 2004 10:14:39 -0800 From: Tutorial Guy <tutorial_sender(a)example.com> To: Tutorial Gal <tutorial_recipient(a)example.net> Subject: Tutorate! Hello, how are you, goodbye. ''' ....def getMailFrom(self): ........result = self.mailFrom ........self.mailFrom = None ........return result ....def getMailTo(self): ........return [self.mailTo] ....def getMailData(self): ........return StringIO.StringIO(self.mailData) ....def sentMail(self, code, resp, numOk, addresses, log): ........print 'Sent', numOk, 'messages' ........from twisted.internet import reactor ........reactor.stop() class SMTPClientFactory(protocol.ClientFactory): ....protocol = SMTPTutorialClient ....def buildProtocol(self, addr): ........return self.protocol(secret=None, identity='example.com') def getMailExchange(host): ....def cbMX(mxRecord): ........return str(mxRecord.exchange) ....return relaymanager.MXCalculator().getMX(host).addCallback(cbMX) def cbMailExchange(exchange): ....smtpClientFactory = SMTPClientFactory() ....smtpClientService = internet.TCPClient(exchange, 25, smtpClientFactory) ....smtpClientService.setServiceParent(application) getMailExchange('example.net').addCallback(cbMailExchange) ============ end tutorial code ============ This nicely looks up the right MX record and sends out an email, just what I need. Now I want to expand it to allow me to give it a list of email addresses to send the message to (not just call this same routine multple times, which seems wasteful and slow and doesn't use twisted's power to process the multiple emails in multiple threads), but I'm having terrible trouble figuring out how to do that, which tells me I'm missing a paradigm somewhere, there's something I'm not getting. Trouble 1 is figuring out the right way to pass additional parameters to callbacks. Is this right: Dosomething(that-returns-a-deferred).addCallback(Then-do-the-next-thing, extra-parameter1, extraparameter2) The function Then-do-the-next-thing() will receive the deferred returned results from Dosomething() as its first argument, and extra-parameter1 and extraparameter2 as the next two. That is as if calling: Then-do-the-next-thing(Result-returned-by-Dosomething(),extra-parameter1, extraparameter2). Have I got this correct? So, if this is right, then where do I want to put the additional argument that contains the next email address to send, if I iterate through the list and hand each one to the email sending process like this: elist=['addr1(a)domain.com','addr2(a)nextdomain.com'...] for e in elist: .... e2={'mxhost':'','toaddr':e} .... getMailExchange(e2).addCallback(cbMailExchange) In the tutorial, getMailExchange() is passed just the domain of the addressee, and the sending out of the email happens when the callback returns the MX exchange. I changed that to split the email address, so now it returns both the full address and the mx: def getMailExchange(addr): .... host=addr.split('@')[1] .... def cbMX(mxRecord): .... .... return [addr,str(mxRecord.name)] .... return relaymanager.MXCalculator().getMX(host).addCallback(cbMX) At this point I can't figure out how to get the email address passed to wherever it needs to go. And I don't know really where it needs to go... yikes. In the tutorial the actual email address is hard coded into the class SMTPTutorialClient(smtp.ESMTPClient) as a class attribute, mailTo. I need to change that to be variable. How do I get this value (of mailTo) changed for each of the instances created by smtpClientFactory = SMTPClientFactory()? I think I must be confused about the roles of Factories and Protocols. I can't seem to figure out a way that works to get the email address passed into the system as a variable. Rather than waste people's time by describing my various failures, I thought I'd just ask for suggestions about the right twisted way to do it. Thanks for any suggestions and directions! -Dave [View Less]

2 2

[Twisted-Python] How to invoke the remote method based on twisted in django web app?
by Boern March 7, 2009

March 7, 2009

hi,all: I developed a web app in diango and a remote service in twisted, and I want to invoke the twisted remote method in django web.example: the remote service code : class Echoer(pb.Root): def remote_echo(self, task): print 'echoing:', task return task if __name__ == '__main__': reactor.listenTCP(8789, pb.PBServerFactory(Echoer())) reactor.run() ------------------------------------------------------------------------------------------------------------- and … [View More]the djiango views.py code : def register_task(requst): """register the task""" ................. factory = pb.PBClientFactory() reactor.connectTCP("localhost", 8789, factory) d = factory.getRootObject() d.addCallback(lambda object: object.callRemote("echo", task)) d.addCallback(lambda echo: 'server echoed: '+echo[0]+str(echo[1])) d.addErrback(lambda reason: 'error: '+str(reason.value)) d.addCallback(util.println) d.addCallback(lambda _: reactor.stop()) reactor.run() .......................... return HttpResponseRedirect('/push_task/') # Redirect after POST ------------------------------------------------------------------------------------------------------------- but when I post a request to django web app ,a exception display: Traceback (most recent call last): File "C:\Python25\Lib\site-packages\twisted\internet\base.py", line 374, in fi reEvent DeferredList(beforeResults).addCallback(self._continueFiring) File "C:\Python25\Lib\site-packages\twisted\internet\defer.py", line 195, in a ddCallback callbackKeywords=kw) File "C:\Python25\Lib\site-packages\twisted\internet\defer.py", line 186, in a ddCallbacks self._runCallbacks() File "C:\Python25\Lib\site-packages\twisted\internet\defer.py", line 328, in _ runCallbacks self.result = callback(self.result, *args, **kw) --- <exception caught here> --- File "C:\Python25\Lib\site-packages\twisted\internet\base.py", line 387, in _c ontinueFiring callable(*args, **kwargs) File "C:\Python25\Lib\site-packages\twisted\internet\base.py", line 1123, in _ reallyStartRunning self._handleSignals() File "C:\Python25\Lib\site-packages\twisted\internet\base.py", line 1068, in _ handleSignals signal.signal(signal.SIGINT, self.sigInt) exceptions.ValueError: signal only works in main thread -------------------------------------------------------------------------------------------------------------- so ,How I correctly invoke the remote method? -- Boern Parx [View Less]

3 3

Re: [Twisted-Python] A few comments on deferreds
by Terry Jones March 6, 2009

March 6, 2009

Hi JP >>>>> "JP" == Jean-Paul Calderone <exarkun(a)divmod.com> writes: JP> On Fri, 6 Mar 2009 12:10:55 +0100, Terry Jones <terry(a)jon.es> wrote: >> 1. If you have a deferred d1, and you call d1.chainDeferred(d2) I think >> it makes sense for there to be a flag inside the deferred class that >> warns you if you later call d1.addCallbacks. That's because adding more >> functions to d1's callback/errback chain is probably not useful. (It … [View More]>> might conceivably be useful, e.g., to let you know that d1 has been >> called). The result of d1 has been passed to d2, and because d2.callback >> and d2.errback don't return it (see point 2 below), it is not passed >> further down d1's chain. So it's most likely that someone adding a >> callback to d1 after calling d1.chainDeferred doesn't really know what >> they're doing. JP> Indeed. I can't think of any case where I've wanted to add more JP> callbacks to a Deferred after chaining it to (from? with?) another JP> Deferred. Right. >> 2. Deferred.callback and Deferred.errback currently (implicitly) return >> None. They could be changed to return their passed argument. That >> would make a call to d1.chainDeferred act like a deferred fork - the >> result would go down d2's callback chain, but also continue down the >> callback/errback chain of d1. JP> But it would be catastrophic for error handling. Suddenly all of your JP> errors on d1 which are handled perfectly correctly with an errback on JP> d2 would become unhandled failures. Ah yes :-) Oops. If you really wanted to do this, you could use another helper function (e.g., Deferred.forkDeferred) which created and added a call/errback functions to d1 that called d2 call/errback and then returned its arg). I don't have a use case for that, so I'm not suggesting adding it to Twisted. Anyone who wanted that kind of deferred forking functionality could write it for themselves in a few lines. >> 3. There's also the possibility of adding a function to the Deferred class: >> >> def incorporateDeferred(self, d2): >> self.callbacks.extend(d2.callbacks) >> >> which might be what a naive reader might expect chainDeferred to do. >> This raises the question of what you might later do with d2 (probably >> nothing), but that's just like what happens with d1: if you call >> d1.chainDeferred you're probably not going to return d1 to anyone. All >> you can usefully do is fire it. JP> If this accounted for future changes to d2, it would make more sense. Yes. But as I mentioned, this mirrors the situation with d1.chainDeferred in which future changes to d1 are also probably useless. It feels to me like extending callbacks might be what people are expecting, as d1 then remains useful. (This leads to thoughts of a Deferred class that can't be fired, just used to accumulate callbacks that are solely used to extend other deferreds, but I guess deferreds are already abstract enough without going there...) JP> I think the correct direction to go in is to prevent people from trying JP> to add more callbacks to d1. Agreed. It's a simple change and it almost certainly flags a misunderstanding. I can open an issue and submit a patch if you like. Terry [View Less]

1 0