[Twisted-Python] Simple multiplex-relayer with twisted.protocols.smtp?
Good day, hackers.
I'm trying to implement a rather simple, localhost-bound mail relay with Twisted. The setup is follows:
Internet --- MTA -> multiplexer (*) -> parallel filters -> MTA -> dest.
Multiplexer marked with (*) is my doing. Filters are independent systems which accept mail according to destination domain and do their magic according to domain-wide settings. Different domains can have completely separate configs and even backends.
I have read and tried to use existing sample from http://twistedmatrix.com/pipermail/twisted-python/2003-October/005885.html and the surrounding thread, but unfortunately have not been so far able to get the system working.
Basically the system should be rather trivial: accept everything, regardless of sender or recipient (frontal MTA has already bounced incorrect mails). Check recipient and forward to matching filter set. Use smtp.ESMTP for inbound, smtp.SMTPClient for outbound. However, the code snippet below freaks out. It now accepts unconditionally both sender and recipient but issuing DATA causes an immediate stacktrace with the following last error:
File "/usr/lib/python2.3/site-packages/twisted/protocols/smtp.py", line 625, in do_DATA assert self.delivery exceptions.AssertionError:
Tracing the problem, it is apparent that there is no delivery method in use and incidentally, this was apparently the bugging reason in the original (a year old now) problem. Call me stupid but I couldn't fix this by following the example and responses.
Version of Twisted in use: 1.3, as packaged for Debian. (In time we'll get 2.0 as well.) Any help will be appreaciated. The leap from very simple Finger examples to smtp parts is rather long and steep...
Code:
[--snip--] #!/usr/bin/python
from twisted.internet import reactor, protocol, defer from twisted.protocols import smtp from twisted.python import log import sys
class RelayUtility: """Utility class for holding runtime values"""
def __init__(self): self.maxconns = 20 self.active = 0
class RelayMessage(smtp.IMessage): def __init__(self): smtp.IMessage.__init__(self) self.msg = []
class RelayProtocol(smtp.ESMTP): """Relayer; sucks the mail in"""
def __init__(self): self.util = util # Normal operations smtp.ESMTP.__init__(self) self.host = "nowhere.dot.invalid"
def connectionLost(self, reason): self.util.active -= 1
def connectionMade(self): # The easiest way. Increments upon connection, decrements # upon disconnection; In case of full queue, just kick the client self.util.active += 1 if (self.util.active <= self.util.maxconns): smtp.ESMTP.connectionMade(self) else: self.sendCode(430, "Queue full. Try again later.") self.transport.loseConnection()
# This can't be right def validateFrom(self, helo, origin): return smtp.Address(origin, None)
# This is certainly not right, DATA barks def validateTo(self, user): return RelayMessage
class RelayFactory(smtp.SMTPFactory): protocol = RelayProtocol
util = RelayUtility() log.startLogging(sys.stdout) reactor.listenTCP(10025, RelayFactory()) reactor.run()
[--snap--]
On Fri, 26 Nov 2004 11:04:44 +0200, bostik@stinghorn.com (Mika Bostrom) wrote:
Good day, hackers.
I'm trying to implement a rather simple, localhost-bound mail relay with Twisted. The setup is follows:
[snip]
Code:
[--snip--] #!/usr/bin/python
from twisted.internet import reactor, protocol, defer from twisted.protocols import smtp from twisted.python import log import sys
class RelayUtility: """Utility class for holding runtime values"""
def __init__(self): self.maxconns =3D 20 self.active =3D 0
class RelayMessage(smtp.IMessage): def __init__(self): smtp.IMessage.__init__(self) self.msg =3D []
The above class is the most obvious problem I see here. Interfaces are not meant to be subclassed in this manner. What you really want is something more like:
class RelayMessage: __implements__ = smtp.IMessage
def lineReceived(self, line): # Do something with the line; perhaps buffer it in memory, # perhaps try and send it to another connection.
def eomReceived(self): # The message has been fully received; flush the buffer or take # whatever other action is appropriate to ensure message delivery. # Return a Deferred that fires when the message has been successfully # delivered.
def connectionLost(self): # Discard message content, delivery is a failure
class RelayProtocol(smtp.ESMTP): """Relayer; sucks the mail in"""
def __init__(self): self.util =3D util # Normal operations smtp.ESMTP.__init__(self) self.host =3D "nowhere.dot.invalid"
def connectionLost(self, reason): self.util.active -=3D 1
def connectionMade(self): # The easiest way. Increments upon connection, decrements # upon disconnection; In case of full queue, just kick the client self.util.active +=3D 1 if (self.util.active <=3D self.util.maxconns): smtp.ESMTP.connectionMade(self) else: self.sendCode(430, "Queue full. Try again later.") self.transport.loseConnection()
# This can't be right def validateFrom(self, helo, origin): return smtp.Address(origin, None)
# This is certainly not right, DATA barks def validateTo(self, user): return RelayMessage
You _could_ do things this way, but a preferable way is probably:
class RelayDeliveryFactory: __implements__ = smtp.IMessageDeliveryFactory
def getMessageDelivery(self): return RelayDelivery()
class RelayDelivery: __implements__ = smtp.IMessageDelivery
def receivedHeader(self, helo, origin, recipients): return "Received: something"
def validateFrom(self, helo, origin): return origin
def validateTo(self, user): return RelayMessage
class RelayFactory(smtp.SMTPFactory): protocol =3D RelayProtocol
Then add this buildProtocol method:
def buildProtocol(self, addr): p = smtp.SMTPFactory.buildProtocol(self, addr) p.deliveryFactory = RelayDeliveryFactory() return p
ESMTP will call getMessageDelivery on its deliveryFactory attribute, now that it isn't None. On the object it returns, it will call receivedHeader, validateFrom, and validateTo. And on the object returned by calling the object returned by validateTo, it will pass the contents of the message being delivered, letting you relay it wherever is appropriate.
Hope this helps,
Jp
On Fri, 26 Nov 2004, Jp Calderone wrote:
On Fri, 26 Nov 2004 11:04:44 +0200, bostik@stinghorn.com (Mika Bostrom) wrote:
class RelayMessage(smtp.IMessage): def __init__(self): smtp.IMessage.__init__(self) self.msg =3D []
The above class is the most obvious problem I see here. Interfaces are not meant to be subclassed in this manner. What you really want is something more like:
[codelet snipped]
Thank you, this was indeed the case. You also gave a a very solid rule of thumb which will come in handy later on. "Interfaces are not meant to be subclassed." (Yes, cut short; on purpose, as that will be easier to remember.)
Your advice also makes a nice example of how to use smtp module as a base.
ESMTP will call getMessageDelivery on its deliveryFactory attribute, now that it isn't None. On the object it returns, it will call receivedHeader, validateFrom, and validateTo. And on the object returned by calling the object returned by validateTo, it will pass the contents of the message being delivered, letting you relay it wherever is appropriate.
This part felt a little like black magic, and my first reaction was that of a scarred Delphi programmer: smelled a bit like excessive nesting. Until I get a better grasp of what actually happens beneath the hood, I'll just consider this a necessary evil.
However, at least by your example and fixes the actual reception works. I can carry on experimenting with what/how to do with the mail from thereon.
Hope this helps,
It certainly did. Thank you so much. I may return to list with additional questions if your examples and hints have failed to enlighten me after all.
Take care,
On Mon, 29 Nov 2004, Mika Bostrom wrote:
On Fri, 26 Nov 2004, Jp Calderone wrote:
ESMTP will call getMessageDelivery on its deliveryFactory attribute, now that it isn't None. On the object it returns, it will call receivedHeader, validateFrom, and validateTo. And on the object returned by calling the object returned by validateTo, it will pass the contents of the message being delivered, letting you relay it wherever is appropriate.
This part felt a little like black magic, [...]
This part still feels a little voodooish, but I have figured out at least something. I don't need to use SMTPClient at all; just use smtp.sendmail() to forward the mail to next point of entry.
However, since my intention was to multiplex outgoing mails to different scanning engines, just forwarding blindly is not exactly productive. Let's assume I have 5 or 6 different scanning engines running in localhost, all of them accepting inbound mail via SMTP in irregular ports. smtp.sendmail() uses the standard port, so I came up with this:
[--snip--] diff -uN twisted/protocols/smtp.py twisted/protocols/smtp.py.new --- twisted/protocols/smtp.py 2004-04-09 06:06:54.000000000 +0300 +++ twisted/protocols/smtp.py.new 2004-11-29 14:16:01.000000000 +0200 @@ -1318,7 +1318,7 @@ p.factory = self return p
-def sendmail(smtphost, from_addr, to_addrs, msg): +def sendmail(smtphost, from_addr, to_addrs, msg, port=25): """Send an email
This interface is intended to be a direct replacement for @@ -1338,6 +1338,9 @@ to pass an email.Message directly, but doing the conversion with email.Generator manually will give you more control over the process). + @param port: The port to connect to on smtphost. If none is + provided, standard 25 is used. +
@rtype: L{Deferred} @returns: A L{Deferred}, its callback will be called if a message is sent @@ -1354,7 +1357,7 @@
d = defer.Deferred() factory = SMTPSenderFactory(from_addr, to_addrs, msg, d) - reactor.connectTCP(smtphost, 25, factory) + reactor.connectTCP(smtphost, port, factory)
return d
[--snap--]
It should allow me to inject the mail to any destination, host:port not being a hindrance. It should also retain compatibility with all existing scripts that use smtp.sendmail()
Now, to _use_ that, I came up with something like this: (referring to previous mails with complete code)
[--schnippel--] class RelayMessage: __implements__ = smtp.IMessage
def __init__(self): # Clear before adding util.connup() self.msg = []
def lineReceived(self, line): self.msg.append(line)
def eomReceived(self): # Add message to outgoing queue # XXX Does not work yet, address information needs to be dug out dd = smtp.sendmail('localhost', self._from, self._to, self.msg, 10050) dd.addCallBack(util.conndown) # When mail has been passed, update counter
# Return a success return defer.succeed('Tally ho.')
def connectionLost(self): # If connection is ever lost in this stage, it is a delivery error self.conndown() self.msg = []
[--schnappel--]
The problem here is naturally that to use smtp.sendmail() I need to extract addresses for sender and recipient(s). I *know* they are there somewhere, but after experimentation and minor excavation I feel utterly lost.
smtp.SMTP.{_cbFromValidate,_cbToValidate} routines place these strings in SMTP._from and SMTP._to, but I haven't been able to dig them out so I could use them. This feels like the most straightforward way of doing this and as such, one I would like to implement.
There must be some clean path from smtp.IMessage interfaces to access items in a smtp.STMP instance. If anyone feels generous and knowledgeable, I for one would gladly welcome the hint as to how this is done.
For what it's worth, if the patch really works I'll naturally try to submit it for inclusion.
On Tue, 2004-11-30 at 16:18 +0200, Mika Bostrom wrote:
However, since my intention was to multiplex outgoing mails to different scanning engines, just forwarding blindly is not exactly productive. Let's assume I have 5 or 6 different scanning engines running in localhost, all of them accepting inbound mail via SMTP in irregular ports. smtp.sendmail() uses the standard port, so I came up with this:
Could you add this patch to http://twistedmatrix.com/bugs/?
On Tue, 30 Nov 2004, Itamar Shtull-Trauring wrote:
On Tue, 2004-11-30 at 16:18 +0200, Mika Bostrom wrote:
However, since my intention was to multiplex outgoing mails to different scanning engines, just forwarding blindly is not exactly productive. Let's assume I have 5 or 6 different scanning engines running in localhost, all of them accepting inbound mail via SMTP in irregular ports. smtp.sendmail() uses the standard port, so I came up with this:
Could you add this patch to http://twistedmatrix.com/bugs/?
Certainly, and I intend to. I just want to test it in rather heterogenic use cases first, to be sure that it does in fact do what I want it to do.
participants (3)
-
bostik＠stinghorn.com -
Itamar Shtull-Trauring -
Jp Calderone