[Twisted-Python] newbie problem with SMTPClient
I love the idea of twisted but I think I must have a twisted learning disability, as I have gotten nowhere in what ought to be a simple matter. I need to send out emails to small groups from my apache server running a python cgi using mod_python, but my hosting service doesn't have a MTA. Instead of learning how to install and configure exim I thought I would use twisted to make a simple mail client. I started with the tutorial example that appears at: http://twistedmatrix.com/projects/mail/documentation/tutorial/smtpclient/smt... (.... is indenting) ============ tutorial code ================ import StringIO from twisted.application import service application = service.Application("SMTP Client Tutorial") from twisted.application import internet from twisted.internet import protocol from twisted.internet import defer from twisted.mail import smtp, relaymanager class SMTPTutorialClient(smtp.ESMTPClient): ....mailFrom = "tutorial_sender@example.com" ....mailTo = "tutorial_recipient@example.net" ....mailData = '''\ Date: Fri, 6 Feb 2004 10:14:39 -0800 From: Tutorial Guy <tutorial_sender@example.com> To: Tutorial Gal <tutorial_recipient@example.net> Subject: Tutorate! Hello, how are you, goodbye. ''' ....def getMailFrom(self): ........result = self.mailFrom ........self.mailFrom = None ........return result ....def getMailTo(self): ........return [self.mailTo] ....def getMailData(self): ........return StringIO.StringIO(self.mailData) ....def sentMail(self, code, resp, numOk, addresses, log): ........print 'Sent', numOk, 'messages' ........from twisted.internet import reactor ........reactor.stop() class SMTPClientFactory(protocol.ClientFactory): ....protocol = SMTPTutorialClient ....def buildProtocol(self, addr): ........return self.protocol(secret=None, identity='example.com') def getMailExchange(host): ....def cbMX(mxRecord): ........return str(mxRecord.exchange) ....return relaymanager.MXCalculator().getMX(host).addCallback(cbMX) def cbMailExchange(exchange): ....smtpClientFactory = SMTPClientFactory() ....smtpClientService = internet.TCPClient(exchange, 25, smtpClientFactory) ....smtpClientService.setServiceParent(application) getMailExchange('example.net').addCallback(cbMailExchange) ============ end tutorial code ============ This nicely looks up the right MX record and sends out an email, just what I need. Now I want to expand it to allow me to give it a list of email addresses to send the message to (not just call this same routine multple times, which seems wasteful and slow and doesn't use twisted's power to process the multiple emails in multiple threads), but I'm having terrible trouble figuring out how to do that, which tells me I'm missing a paradigm somewhere, there's something I'm not getting. Trouble 1 is figuring out the right way to pass additional parameters to callbacks. Is this right: Dosomething(that-returns-a-deferred).addCallback(Then-do-the-next-thing, extra-parameter1, extraparameter2) The function Then-do-the-next-thing() will receive the deferred returned results from Dosomething() as its first argument, and extra-parameter1 and extraparameter2 as the next two. That is as if calling: Then-do-the-next-thing(Result-returned-by-Dosomething(),extra-parameter1, extraparameter2). Have I got this correct? So, if this is right, then where do I want to put the additional argument that contains the next email address to send, if I iterate through the list and hand each one to the email sending process like this: elist=['addr1@domain.com','addr2@nextdomain.com'...] for e in elist: .... e2={'mxhost':'','toaddr':e} .... getMailExchange(e2).addCallback(cbMailExchange) In the tutorial, getMailExchange() is passed just the domain of the addressee, and the sending out of the email happens when the callback returns the MX exchange. I changed that to split the email address, so now it returns both the full address and the mx: def getMailExchange(addr): .... host=addr.split('@')[1] .... def cbMX(mxRecord): .... .... return [addr,str(mxRecord.name)] .... return relaymanager.MXCalculator().getMX(host).addCallback(cbMX) At this point I can't figure out how to get the email address passed to wherever it needs to go. And I don't know really where it needs to go... yikes. In the tutorial the actual email address is hard coded into the class SMTPTutorialClient(smtp.ESMTPClient) as a class attribute, mailTo. I need to change that to be variable. How do I get this value (of mailTo) changed for each of the instances created by smtpClientFactory = SMTPClientFactory()? I think I must be confused about the roles of Factories and Protocols. I can't seem to figure out a way that works to get the email address passed into the system as a variable. Rather than waste people's time by describing my various failures, I thought I'd just ask for suggestions about the right twisted way to do it. Thanks for any suggestions and directions! -Dave
On Mon, 2 Mar 2009 15:57:17 -0500, Dave Britton <dave@davebritton.com> wrote:
I love the idea of twisted but I think I must have a twisted learning disability, as I have gotten nowhere in what ought to be a simple matter.
I need to send out emails to small groups from my apache server running a python cgi using mod_python, but my hosting service doesn't have a MTA. Instead of learning how to install and configure exim I thought I would use twisted to make a simple mail client. I started with the tutorial example that appears at:
It may be that installing exim is actually a better idea. In order to get reliable message deliver, you'll need to handle transient failures. That means persisting state over time (as long as several days) and performing redelivery attempts. However, if you don't mind losing outgoing messages when there is a transient failure...
[snip]
This nicely looks up the right MX record and sends out an email, just what I need. Now I want to expand it to allow me to give it a list of email addresses to send the message to (not just call this same routine multple times, which seems wasteful and slow and doesn't use twisted's power to process the multiple emails in multiple threads), but I'm having terrible trouble figuring out how to do that, which tells me I'm missing a paradigm somewhere, there's something I'm not getting.
There is a reason that just calling the top-level function you've written once for each email might be wasteful, but I don't think it's the reason you're thinking. If you just naively call getMailExchange(host).addCallback(cbMailExchange) once for each email, then you will get parallel processing. Almost as soon as you call most Twisted APIs, they'll start an operation (to be precise, many of them start it immediately - before they even return - and others start it when you allow program execution flow to return to the reactor). So if you call getMailExchange in a loop, each iteration of the loop will start a new operation and they'll all run in parallel. That seems like just what you're after. The reason this isn't the most efficient solution is that your list of email addresses might contain two addresses which have the same mail exchange host. In this case, you could connect to that mail exchange once and address your message to both addresses and then deliver the message body to the mail exchange just once. This is why getMailTo returns a list. So I think your other questions aren't directly relevant to this problem, but I'll answer them anyway, since they're good questions.
Yes, this is right.
I'm not sure why you switched to a dictionary here. Ignoring that, the biggest potential problem with this code snippet is that it creates a Deferred (the one returned by getMailExchange) and then drops it on the floor (albeit after adding a callback). You'll almost certainly want to build up a list of Deferreds in cases like this, and then use something like twisted.internet.defer.gatherResults to find out when they've all fired. Otherwise, you don't really know when your list of operations has completed. As to where to put extra arguments for tracking which email addresses to send the message to next, I don't see why you'd want that in this case. Your for loop iterates over all the addresses, so none of your callbacks should need to know about any more addresses. That is to say, by the end of the for loop, there are no more addresses to which the message needs to be sent - you've started sending to all of them already.
Likely nothing in this code cares. However, the address is going to be passed to whatever callback is added to the Deferred this function returns - because you've included it in the list returned by cbMX.
In the tutorial the actual email address is hard coded into the class SMTPTutorialClient(smtp.ESMTPClient) as a class attribute, mailTo. I need to change that to be variable.
How do I get this value (of mailTo) changed for each of the instances created by smtpClientFactory = SMTPClientFactory()? I think I must be confused about the roles of Factories and Protocols.
This is probably simpler than you expected - just override __init__ to accept any extra data you need, and then save that data as an instance attribute.
I can't seem to figure out a way that works to get the email address passed into the system as a variable. Rather than waste people's time by describing my various failures, I thought I'd just ask for suggestions about the right twisted way to do it.
Another API you might want to look at is twisted.mail.smtp.sendmail. It's probably rather unfortunate that SMTP client tutorial doesn't cover that function. Hope this helps, Jean-Paul
Thanks! sendmail is just what I needed, and its source should help me figure out the bigger picture. -Dave ----- Original Message ----- From: "Jean-Paul Calderone" <exarkun@divmod.com> To: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Monday, March 02, 2009 4:22 PM Subject: Re: [Twisted-Python] newbie problem with SMTPClient python cgi using mod_python, but my hosting service doesn't have a MTA. Instead of learning how to install and configure exim I thought I would use twisted to make a simple mail client. I started with the tutorial example that appears at:
what I need. Now I want to expand it to allow me to give it a list of email addresses to send the message to (not just call this same routine multple times, which seems wasteful and slow and doesn't use twisted's power to process the multiple emails in multiple threads), but I'm having terrible trouble figuring out how to do that, which tells me I'm missing a paradigm somewhere, there's something I'm not getting.
soon that contains the next email address to send, if I iterate through the list and hand each one to the email sending process like this: passed into the system as a variable. Rather than waste people's time by describing my various failures, I thought I'd just ask for suggestions about the right twisted way to do it.
On Mon, 2 Mar 2009 15:57:17 -0500, Dave Britton <dave@davebritton.com> wrote:
I love the idea of twisted but I think I must have a twisted learning disability, as I have gotten nowhere in what ought to be a simple matter.
I need to send out emails to small groups from my apache server running a python cgi using mod_python, but my hosting service doesn't have a MTA. Instead of learning how to install and configure exim I thought I would use twisted to make a simple mail client. I started with the tutorial example that appears at:
It may be that installing exim is actually a better idea. In order to get reliable message deliver, you'll need to handle transient failures. That means persisting state over time (as long as several days) and performing redelivery attempts. However, if you don't mind losing outgoing messages when there is a transient failure...
[snip]
This nicely looks up the right MX record and sends out an email, just what I need. Now I want to expand it to allow me to give it a list of email addresses to send the message to (not just call this same routine multple times, which seems wasteful and slow and doesn't use twisted's power to process the multiple emails in multiple threads), but I'm having terrible trouble figuring out how to do that, which tells me I'm missing a paradigm somewhere, there's something I'm not getting.
There is a reason that just calling the top-level function you've written once for each email might be wasteful, but I don't think it's the reason you're thinking. If you just naively call getMailExchange(host).addCallback(cbMailExchange) once for each email, then you will get parallel processing. Almost as soon as you call most Twisted APIs, they'll start an operation (to be precise, many of them start it immediately - before they even return - and others start it when you allow program execution flow to return to the reactor). So if you call getMailExchange in a loop, each iteration of the loop will start a new operation and they'll all run in parallel. That seems like just what you're after. The reason this isn't the most efficient solution is that your list of email addresses might contain two addresses which have the same mail exchange host. In this case, you could connect to that mail exchange once and address your message to both addresses and then deliver the message body to the mail exchange just once. This is why getMailTo returns a list. So I think your other questions aren't directly relevant to this problem, but I'll answer them anyway, since they're good questions.
Yes, this is right.
I'm not sure why you switched to a dictionary here. Ignoring that, the biggest potential problem with this code snippet is that it creates a Deferred (the one returned by getMailExchange) and then drops it on the floor (albeit after adding a callback). You'll almost certainly want to build up a list of Deferreds in cases like this, and then use something like twisted.internet.defer.gatherResults to find out when they've all fired. Otherwise, you don't really know when your list of operations has completed. As to where to put extra arguments for tracking which email addresses to send the message to next, I don't see why you'd want that in this case. Your for loop iterates over all the addresses, so none of your callbacks should need to know about any more addresses. That is to say, by the end of the for loop, there are no more addresses to which the message needs to be sent - you've started sending to all of them already.
Likely nothing in this code cares. However, the address is going to be passed to whatever callback is added to the Deferred this function returns - because you've included it in the list returned by cbMX.
In the tutorial the actual email address is hard coded into the class SMTPTutorialClient(smtp.ESMTPClient) as a class attribute, mailTo. I need to change that to be variable.
How do I get this value (of mailTo) changed for each of the instances created by smtpClientFactory = SMTPClientFactory()? I think I must be confused about the roles of Factories and Protocols.
This is probably simpler than you expected - just override __init__ to accept any extra data you need, and then save that data as an instance attribute.
I can't seem to figure out a way that works to get the email address passed into the system as a variable. Rather than waste people's time by describing my various failures, I thought I'd just ask for suggestions about the right twisted way to do it.
Another API you might want to look at is twisted.mail.smtp.sendmail. It's probably rather unfortunate that SMTP client tutorial doesn't cover that function. Hope this helps, Jean-Paul
Thanks! sendmail is just what I needed, and its source should help me figure out the bigger picture. -Dave ----- Original Message ----- From: "Jean-Paul Calderone" <exarkun@divmod.com> To: "Twisted general discussion" <twisted-python@twistedmatrix.com> Sent: Monday, March 02, 2009 4:22 PM Subject: Re: [Twisted-Python] newbie problem with SMTPClient python cgi using mod_python, but my hosting service doesn't have a MTA. Instead of learning how to install and configure exim I thought I would use twisted to make a simple mail client. I started with the tutorial example that appears at:
what I need. Now I want to expand it to allow me to give it a list of email addresses to send the message to (not just call this same routine multple times, which seems wasteful and slow and doesn't use twisted's power to process the multiple emails in multiple threads), but I'm having terrible trouble figuring out how to do that, which tells me I'm missing a paradigm somewhere, there's something I'm not getting.
soon that contains the next email address to send, if I iterate through the list and hand each one to the email sending process like this: passed into the system as a variable. Rather than waste people's time by describing my various failures, I thought I'd just ask for suggestions about the right twisted way to do it.
participants (2)
-
Dave Britton -
Jean-Paul Calderone