It seems that entering one's email address to edit subscription
options dosent work (It was prolly broken by the cgi 'driver' setup)
Here's a patch to subscribe.py that seems to fix it.
-The Dragon De Monsyne
*** /usr/src/mailman-1.0b5/Mailman/Cgi/subscribe.py Mon Jul 27 17:48:31 1998
--- Mailman/Cgi/subscribe.py Mon Aug 31 07:22:17 1998
***************
*** 61,69 ****
def call_script(which, pathinfo):
"A little bit of a hack to call one of the scripts..."
os.environ['PATH_INFO'] = string.join(pathinfo, '/')
! file = os.path.join(mm_cfg.SCRIPTS_DIR, which)
! list.Unlock()
! execfile(file)
sys.exit(0)
#######
--- 61,81 ----
def call_script(which, pathinfo):
"A little bit of a hack to call one of the scripts..."
os.environ['PATH_INFO'] = string.join(pathinfo, '/')
!
! # See the reference manual for why we have to do things this way.
! # Note that importing should have no side-effects!
! pkg = __import__('Mailman.Cgi', globals(), locals(), [which])
! module = getattr(pkg, which)
! main = getattr(module, 'main')
! try:
! main()
! except SystemExit:
! # this is a valid way for the function to exit
! pass
!
! #file = os.path.join(mm_cfg.SCRIPTS_DIR, which)
! #list.Unlock()
! #execfile(file)
sys.exit(0)
#######
***************
*** 92,97 ****
--- 104,110 ----
print doc.Format()
list.Unlock()
sys.exit(0)
+ list.Unlock()
call_script('options', [list._internal_name, member])
if not form.has_key("email"):
error = 1
Diff against yesterday's CVS checkout.
Fixes / extends
- decentralized CGI names
- CGI extensions
- Ack mail bug
- qmail posting (set QMAIL to 1 in mm_cfg not to use inner spool)
- other mailers will be handled soon
- distclean dependencies
- 'nofiles' group checking in configure (for qmail)
--
hacker: /n./ One who enjoys the intellectual challenge of creatively
overcoming or circumventing limitations.
PGP 0x1DE3631D / A8 B4 92 EE 1F 55 27 C8 86 64 9C 42 41 A4 BD B8
I just have a question.... Peering through the queueing system, I
note that there does not seem to be any locking of the queuefiles...
Correct? The way it appears to me that mailman is working is thus:
every outgoing message is queued, then an attempt to feed it to
the MTA is made. After that it tries to run the queue to catch anything
waiting there.
What keeps another process running the queue from grabbing &
delivering the queued message whilst the first proccess is trying to send
the same message to the MTA? Especially whence you have several proccesses
rapidly forked all doing delivery at once?
Here's the situation I'm worried about:
Proccess 1 queues the message, and attempts to feed it to the MTA
Proccess 2 running queue reads the queued message, and also attempts to
deliver it to the MTA.
Proccess 1 succeeds in in delivering the message and deletes it from the
queue.
Proccess 2 ALSO succeeds in delivering the message, creating a duplicate.
Proccess 2 tries to delete the message from the queue, and can't since
proccess 1 already deleted it, generating this traceback, which
I _have_ seen in my error log:
Aug 29 04:10:59 1998 contact_transport: Traceback (innermost last):
contact_transport: File
"/usr/services/mailman/scripts/contact_transport", line 52, in ?
contact_transport: OutgoingQueue.processQueue()
contact_transport: File
"/usr/services/mailman/Mailman/OutgoingQueue.py", line 38, in
processQueue
contact_transport: Utils.TrySMTPDelivery(recip,sender,text,full_fname)
contact_transport: File "/usr/services/mailman/Mailman/Utils.py", line
230, in TrySMTPDelivery
contact_transport: OutgoingQueue.dequeueMessage(queue_entry)
contact_transport: File "/usr/services/mailman/Mailman/OutgoingQueue.py",
line 25, in dequeueMessage
contact_transport: os.unlink(msg)
contact_transport: os . error : (2, 'No such file or directory')
Having pointed out a possible problem, perhaps I can suggest
a possible solution? Howabout this:
Whenever Mailman goes to deliver mail, it dosen't actually deliver
it. Rather, it just queues the message. Instead, there is a single,
separate proccess that is kept running, and all it does is dequeue
messsages. It would keep a PID file and touch it periodically, so you
could run a cron job to make sure it is still running. Since it would hold
no locks on any lists, it wouldn't have to worry about forking to avoid
deadlocks, and since it would be run under the mailman uid it wouldn't be
affected by any possible setgid weirdness (i.e. w/ linux)).
If anyone is interested in this, let me know, & I'll put
something together. It should be really quick.
-The Dragon De Monsyne
Patch patch patch patch patch.... :>
This fixes a nasty (mis)feature of listinfo that if the hostname
you use for your lists' email addresses != the hostname for your base
mailman url, the list won't show up on the listinfo page , even if
advertized. :P
(for example, all of my lists have host_name= 'lists.integral.org' and
web_page_url = 'www.integral.org', so I got nothing on my listinfo page
:/ )
-The Dragon De Monsyne
*** /usr/src/mailman-1.0b5/Mailman/Cgi/listinfo.py Mon Jul 27 17:48:31 1998
--- listinfo.py Sat Aug 29 03:25:48 1998
***************
*** 57,62 ****
--- 57,63 ----
def FormatListinfoOverview(error=None):
"Present a general welcome and itemize the (public) lists for this host."
+ import urlparse
doc = Document()
legend = "%s mailing lists" % mm_cfg.DEFAULT_HOST_NAME
doc.SetTitle(legend)
***************
*** 74,88 ****
# visited! An absolute URL would do...
if os.environ.has_key('HTTP_HOST'):
http_host = os.environ['HTTP_HOST']
else:
! http_host = None
!
for n in names:
l = MailList.MailList(n, lock = 0)
if l.advertised:
if (http_host
! and (string.find(http_host, l.host_name) == -1
! and string.find(l.host_name, http_host) == -1)):
# List is for different identity for this host - skip it.
continue
else:
--- 75,93 ----
# visited! An absolute URL would do...
if os.environ.has_key('HTTP_HOST'):
http_host = os.environ['HTTP_HOST']
+ http_host = string.split(string.lower(http_host),':')[0]
else:
! http_host = None
for n in names:
l = MailList.MailList(n, lock = 0)
if l.advertised:
+ if http_host:
+ list_host = string.split(string.lower(urlparse.urlparse(
+ l.web_page_url)[1]),':')[0]
+
if (http_host
! and (string.find(http_host, list_host) == -1
! and string.find(list_host,http_host) == -1)):
# List is for different identity for this host - skip it.
continue
else:
Ok, this patch changes mailman to use the smtplib.py included
here, instead of the current one.
The smtplib.py included here is the most recent version of the
smtplib included in the Python 1.5.1 distribution. This exact version will
likely be included in the python 1.5.2 distribution
Having Mailman use this smtplib is good b'cause any
improvements/bugfixes made to it because of mailman can then be folded
back into tyhe Python distribution. Plus this lib has esmtp support, which
could be usefull for DSN tweaking.
To aply this:
-Remove the smtplib in the Mailman package
-Put this smplib.py in your PYTHONPATH. (If you have 1.5.1,
replace the smtplib that came with the python distribution with
this one, it has a few bugfixes)
-apply the two included patches. Ones for the Mailman package,
ones a very minor one for the contact_transport script.
-The Dragon De Monsyne
#!/usr/bin/python
"""SMTP/ESMTP client class.
Author: The Dragon De Monsyne <dragondm(a)integral.org>
ESMTP support, test code and doc fixes added by
Eric S. Raymond <esr(a)thyrsus.com>
Better RFC 821 compliance (MAIL and RCPT, and CRLF in data)
by Carey Evans <c.evans(a)clear.net.nz>, for picky mail servers.
(This was modified from the Python 1.5 library HTTP lib.)
This should follow RFC 821 (SMTP) and RFC 1869 (ESMTP).
Notes:
Please remember, when doing ESMTP, that the names of the SMTP service
extentions are NOT the same thing as the option keyords for the RCPT and MAIL
commands!
Example:
>>> import smtplib
>>> s=smtplib.SMTP("localhost")
>>> print s.help()
This is Sendmail version 8.8.4
Topics:
HELO EHLO MAIL RCPT DATA
RSET NOOP QUIT HELP VRFY
EXPN VERB ETRN DSN
For more info use "HELP <topic>".
To report bugs in the implementation send email to
sendmail-bugs(a)sendmail.org.
For local information send email to Postmaster at your site.
End of HELP info
>>> s.putcmd("vrfy","someone@here")
>>> s.getreply()
(250, "Somebody OverHere <somebody(a)here.my.org>")
>>> s.quit()
"""
import socket
import string,re,rfc822
SMTP_PORT = 25
CRLF="\r\n"
# used for exceptions
SMTPServerDisconnected="Server not connected"
SMTPSenderRefused="Sender address refused"
SMTPRecipientsRefused="All Recipients refused"
SMTPDataError="Error transmitting message data"
def quoteaddr(addr):
"""Quote a subset of the email addresses defined by RFC 821.
Should be able to handle anything rfc822.parseaddr can handle."""
m=None
try:
m=rfc822.parseaddr(addr)[1]
except AttributeError:
pass
if not m:
#something weird here.. punt -ddm
return addr
else:
return "<%s>" % m
def quotedata(data):
"""Quote data for email.
Double leading '.', and change Unix newline '\n', or Mac '\r' into
Internet CRLF end-of-line."""
return re.sub(r'(?m)^\.', '..',
re.sub(r'(?:\r\n|\n|\r(?!\n))', CRLF, data))
class SMTP:
"""This class manages a connection to an SMTP or ESMTP server.
SMTP Objects:
SMTP objects have the following attributes:
helo_resp
This is the message given by the server in responce to the
most recent HELO command.
ehlo_resp
This is the message given by the server in responce to the
most recent EHLO command. This is usually multiline.
does_esmtp
This is a True value _after you do an EHLO command_, if the
server supports ESMTP.
esmtp_features
This is a dictionary, which, if the server supports ESMTP,
will _after you do an EHLO command_, contain the names of the
SMTP service extentions this server supports, and their
parameters (if any).
Note, all extention names are mapped to lower case in the
dictionary.
For method docs, see each method's docstrings. In general, there is
a method of the same name to preform each SMTP comand, and there
is a method called 'sendmail' that will do an entiere mail
transaction."""
debuglevel = 0
file = None
helo_resp = None
ehlo_resp = None
does_esmtp = 0
def __init__(self, host = '', port = 0):
"""Initialize a new instance.
If specified, `host' is the name of the remote host to which
to connect. If specified, `port' specifies the port to which
to connect. By default, smtplib.SMTP_PORT is used.
"""
self.esmtp_features = {}
if host: self.connect(host, port)
def set_debuglevel(self, debuglevel):
"""Set the debug output level.
A non-false value results in debug messages for connection and
for all messages sent to and received from the server.
"""
self.debuglevel = debuglevel
def connect(self, host='localhost', port = 0):
"""Connect to a host on a given port.
If the hostname ends with a colon (`:') followed by a number,
and there is no port specified, that suffix will be stripped
off and the number interpreted as the port number to use.
Note: This method is automatically invoked by __init__,
if a host is specified during instantiation.
"""
if not port:
i = string.find(host, ':')
if i >= 0:
host, port = host[:i], host[i+1:]
try: port = string.atoi(port)
except string.atoi_error:
raise socket.error, "nonnumeric port"
if not port: port = SMTP_PORT
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
if self.debuglevel > 0: print 'connect:', (host, port)
self.sock.connect(host, port)
(code,msg)=self.getreply()
if self.debuglevel >0 : print "connect:", msg
return msg
def send(self, str):
"""Send `str' to the server."""
if self.debuglevel > 0: print 'send:', `str`
if self.sock:
try:
self.sock.send(str)
except socket.error:
raise SMTPServerDisconnected
else:
raise SMTPServerDisconnected
def putcmd(self, cmd, args=""):
"""Send a command to the server.
"""
str = '%s %s%s' % (cmd, args, CRLF)
self.send(str)
def getreply(self):
"""Get a reply from the server.
Returns a tuple consisting of:
- server response code (e.g. '250', or such, if all goes well)
Note: returns -1 if it can't read response code.
- server response string corresponding to response code
(note : multiline responses converted to a single,
multiline string)
"""
resp=[]
self.file = self.sock.makefile('rb')
while 1:
line = self.file.readline()
if self.debuglevel > 0: print 'reply:', `line`
resp.append(string.strip(line[4:]))
code=line[:3]
#check if multiline resp
if line[3:4]!="-":
break
try:
errcode = string.atoi(code)
except(ValueError):
errcode = -1
errmsg = string.join(resp,"\n")
if self.debuglevel > 0:
print 'reply: retcode (%s); Msg: %s' % (errcode,errmsg)
return errcode, errmsg
def docmd(self, cmd, args=""):
""" Send a command, and return its response code """
self.putcmd(cmd,args)
(code,msg)=self.getreply()
return code
# std smtp commands
def helo(self, name=''):
""" SMTP 'helo' command. Hostname to send for this command
defaults to the FQDN of the local host """
name=string.strip(name)
if len(name)==0:
name=socket.gethostbyaddr(socket.gethostname())[0]
self.putcmd("helo",name)
(code,msg)=self.getreply()
self.helo_resp=msg
return code
def ehlo(self, name=''):
""" SMTP 'ehlo' command. Hostname to send for this command
defaults to the FQDN of the local host. """
name=string.strip(name)
if len(name)==0:
name=socket.gethostbyaddr(socket.gethostname())[0]
self.putcmd("ehlo",name)
(code,msg)=self.getreply()
# According to RFC1869 some (badly written)
# MTA's will disconnect on an ehlo. Toss an exception if
# that happens -ddm
if code == -1 and len(msg) == 0:
raise SMTPServerDisconnected
self.ehlo_resp=msg
if code<>250:
return code
self.does_esmtp=1
#parse the ehlo responce -ddm
resp=string.split(self.ehlo_resp,'\n')
del resp[0]
for each in resp:
m=re.match(r'(?P<feature>[A-Za-z0-9][A-Za-z0-9\-]*)',each)
if m:
feature=string.lower(m.group("feature"))
params=string.strip(m.string[m.end("feature"):])
self.esmtp_features[feature]=params
return code
def has_extn(self, opt):
"""Does the server support a given SMTP service extension?"""
return self.esmtp_features.has_key(string.lower(opt))
def help(self, args=''):
""" SMTP 'help' command. Returns help text from server """
self.putcmd("help", args)
(code,msg)=self.getreply()
return msg
def rset(self):
""" SMTP 'rset' command. Resets session. """
code=self.docmd("rset")
return code
def noop(self):
""" SMTP 'noop' command. Doesn't do anything :> """
code=self.docmd("noop")
return code
def mail(self,sender,options=[]):
""" SMTP 'mail' command. Begins mail xfer session. """
optionlist = ''
if options and self.does_esmtp:
optionlist = string.join(options, ' ')
self.putcmd("mail", "FROM:%s %s" % (quoteaddr(sender) ,optionlist))
return self.getreply()
def rcpt(self,recip,options=[]):
""" SMTP 'rcpt' command. Indicates 1 recipient for this mail. """
optionlist = ''
if options and self.does_esmtp:
optionlist = string.join(options, ' ')
self.putcmd("rcpt","TO:%s %s" % (quoteaddr(recip),optionlist))
return self.getreply()
def data(self,msg):
""" SMTP 'DATA' command. Sends message data to server.
Automatically quotes lines beginning with a period per rfc821. """
self.putcmd("data")
(code,repl)=self.getreply()
if self.debuglevel >0 : print "data:", (code,repl)
if code <> 354:
return -1
else:
self.send(quotedata(msg))
self.send("%s.%s" % (CRLF, CRLF))
(code,msg)=self.getreply()
if self.debuglevel >0 : print "data:", (code,msg)
return code
def vrfy(self, address):
return self.verify(address)
def verify(self, address):
""" SMTP 'verify' command. Checks for address validity. """
self.putcmd("vrfy", quoteaddr(address))
return self.getreply()
def expn(self, address):
""" SMTP 'verify' command. Checks for address validity. """
self.putcmd("expn", quoteaddr(address))
return self.getreply()
#some useful methods
def sendmail(self,from_addr,to_addrs,msg,mail_options=[],rcpt_options=[]):
""" This command performs an entire mail transaction.
The arguments are:
- from_addr : The address sending this mail.
- to_addrs : a list of addresses to send this mail to
- msg : the message to send.
- mail_options : list of ESMTP options (such as 8bitmime)
for the mail command
- rcpt_options : List of ESMTP options (such as DSN commands)
for all the rcpt commands
If there has been no previous EHLO or HELO command this session,
this method tries ESMTP EHLO first. If the server does ESMTP, message
size and each of the specified options will be passed to it.
If EHLO fails, HELO will be tried and ESMTP options suppressed.
This method will return normally if the mail is accepted for at least
one recipient. Otherwise it will throw an exception (either
SMTPSenderRefused, SMTPRecipientsRefused, or SMTPDataError)
That is, if this method does not throw an exception, then someone
should get your mail. If this method does not throw an exception,
it returns a dictionary, with one entry for each recipient that was
refused.
Example:
>>> import smtplib
>>> s=smtplib.SMTP("localhost")
>>> tolist=["one(a)one.org","two(a)two.org","three(a)three.org","four(a)four.org"]
>>> msg = '''
... From: Me(a)my.org
... Subject: testin'...
...
... This is a test '''
>>> s.sendmail("me(a)my.org",tolist,msg)
{ "three(a)three.org" : ( 550 ,"User unknown" ) }
>>> s.quit()
In the above example, the message was accepted for delivery to
three of the four addresses, and one was rejected, with the error
code 550. If all addresses are accepted, then the method
will return an empty dictionary.
"""
if not self.helo_resp and not self.ehlo_resp:
if self.ehlo() >= 400:
self.helo()
esmtp_opts = []
if self.does_esmtp:
# Hmmm? what's this? -ddm
# self.esmtp_features['7bit']=""
if self.has_extn('size'):
esmtp_opts.append("size=" + `len(msg)`)
for option in mail_options:
esmtp_opts.append(option)
(code,resp) = self.mail(from_addr, esmtp_opts)
if code <> 250:
self.rset()
raise SMTPSenderRefused
senderrs={}
for each in to_addrs:
(code,resp)=self.rcpt(each, rcpt_options)
if (code <> 250) and (code <> 251):
senderrs[each]=(code,resp)
if len(senderrs)==len(to_addrs):
# the server refused all our recipients
self.rset()
raise SMTPRecipientsRefused
code=self.data(msg)
if code <>250 :
self.rset()
raise SMTPDataError
#if we got here then somebody got our mail
return senderrs
def close(self):
"""Close the connection to the SMTP server."""
if self.file:
self.file.close()
self.file = None
if self.sock:
self.sock.close()
self.sock = None
def quit(self):
"""Terminate the SMTP session."""
self.docmd("quit")
self.close()
# Test the sendmail method, which tests most of the others.
# Note: This always sends to localhost.
if __name__ == '__main__':
import sys, rfc822
def prompt(prompt):
sys.stdout.write(prompt + ": ")
return string.strip(sys.stdin.readline())
fromaddr = prompt("From")
toaddrs = string.splitfields(prompt("To"), ',')
print "Enter message, end with ^D:"
msg = ''
while 1:
line = sys.stdin.readline()
if not line:
break
msg = msg + line
print "Message length is " + `len(msg)`
server = SMTP('localhost')
server.set_debuglevel(1)
server.sendmail(fromaddr, toaddrs, msg)
server.quit()
diff -c /usr/src/mailman-1.0b5/Mailman/Bouncer.py Mailman/Bouncer.py
*** /usr/src/mailman-1.0b5/Mailman/Bouncer.py Mon Jul 27 17:48:31 1998
--- Mailman/Bouncer.py Sat Aug 29 00:11:10 1998
***************
*** 202,208 ****
# 'bounce.txt' has a trailing newline
text = text + \
string.join(msg.headers, '') + '\n' + \
! Utils.QuotePeriods(msg.body) + '\n' + \
'--' + boundary + '--'
if negative:
--- 202,208 ----
# 'bounce.txt' has a trailing newline
text = text + \
string.join(msg.headers, '') + '\n' + \
! msg.body + '\n' + \
'--' + boundary + '--'
if negative:
diff -c /usr/src/mailman-1.0b5/Mailman/Deliverer.py Mailman/Deliverer.py
*** /usr/src/mailman-1.0b5/Mailman/Deliverer.py Mon Jul 27 17:48:31 1998
--- Mailman/Deliverer.py Sat Aug 29 00:12:09 1998
***************
*** 43,50 ****
add_headers=['Errors-To: %s\n'
% Self.GetAdminEmail()])
! def QuotePeriods(self, text):
! return string.join(string.split(text, '\n.\n'), '\n .\n')
def DeliverToList(self, msg, recipients,
header="", footer="", remove_to=0, tmpfile_prefix = ""):
if not(len(recipients)):
--- 43,50 ----
add_headers=['Errors-To: %s\n'
% Self.GetAdminEmail()])
! # def QuotePeriods(self, text):
! # return string.join(string.split(text, '\n.\n'), '\n .\n')
def DeliverToList(self, msg, recipients,
header="", footer="", remove_to=0, tmpfile_prefix = ""):
if not(len(recipients)):
***************
*** 75,81 ****
cmdproc.write(string.join(msg.headers, '') + "\n")
if header: # The *body* header:
cmdproc.write(header + "\n")
! cmdproc.write(self.QuotePeriods(msg.body))
if footer:
cmdproc.write(footer)
--- 75,81 ----
cmdproc.write(string.join(msg.headers, '') + "\n")
if header: # The *body* header:
cmdproc.write(header + "\n")
! cmdproc.write(msg.body)
if footer:
cmdproc.write(footer)
diff -c /usr/src/mailman-1.0b5/Mailman/OutgoingQueue.py Mailman/OutgoingQueue.py
*** /usr/src/mailman-1.0b5/Mailman/OutgoingQueue.py Mon Jul 27 17:48:31 1998
--- Mailman/OutgoingQueue.py Fri Aug 28 20:50:51 1998
***************
*** 25,31 ****
os.unlink(msg)
def processQueue():
! import os, smtplib
files = os.listdir(mm_cfg.DATA_DIR)
for file in files:
if TEMPLATE <> file[:len(TEMPLATE)]:
--- 25,31 ----
os.unlink(msg)
def processQueue():
! import os
files = os.listdir(mm_cfg.DATA_DIR)
for file in files:
if TEMPLATE <> file[:len(TEMPLATE)]:
diff -c /usr/src/mailman-1.0b5/Mailman/Utils.py Mailman/Utils.py
*** /usr/src/mailman-1.0b5/Mailman/Utils.py Mon Jul 27 17:48:31 1998
--- Mailman/Utils.py Sat Aug 29 00:24:56 1998
***************
*** 154,160 ****
msg = Message.OutgoingMessage()
msg.SetSender(sender)
msg.SetHeader('Subject', subject, 1)
! msg.SetBody(QuotePeriods(text))
DeliverToUser(msg, recipient, add_headers=add_headers)
def DeliverToUser(msg, recipient, add_headers=[]):
--- 154,160 ----
msg = Message.OutgoingMessage()
msg.SetSender(sender)
msg.SetHeader('Subject', subject, 1)
! msg.SetBody(text)
DeliverToUser(msg, recipient, add_headers=add_headers)
def DeliverToUser(msg, recipient, add_headers=[]):
***************
*** 184,212 ****
i = i + '\n'
msg.headers.append(i)
! text = string.join(msg.headers, '')+ '\n'+ QuotePeriods(msg.body)
import OutgoingQueue
! queue_id = OutgoingQueue.enqueueMessage(sender, recipient, text)
! TrySMTPDelivery(recipient,sender,text,queue_id)
# Just in case there's still something waiting to be sent...
OutgoingQueue.processQueue()
finally:
os._exit(0)
def TrySMTPDelivery(recipient, sender, text, queue_entry):
! import smtplib
try:
! con = smtplib.SmtpConnection(mm_cfg.SMTPHOST)
! con.helo(mm_cfg.DEFAULT_HOST_NAME)
! con.send(to=recipient,frm=sender,text=text)
con.quit()
import OutgoingQueue
OutgoingQueue.dequeueMessage(queue_entry)
finally:
# except: # Todo: This might want to handle special cases.
pass # Just try again later.
! def QuotePeriods(text):
! return string.join(string.split(text, '\n.\n'), '\n .\n')
def ValidEmail(str):
"""Verify that the an email address isn't grossly invalid."""
--- 184,240 ----
i = i + '\n'
msg.headers.append(i)
! text = string.join(msg.headers, '')+ '\n'+ msg.body
import OutgoingQueue
! queue_id = OutgoingQueue.enqueueMessage(sender, [recipient], text)
! TrySMTPDelivery([recipient],sender,text,queue_id)
# Just in case there's still something waiting to be sent...
OutgoingQueue.processQueue()
finally:
os._exit(0)
+ # DO make sure recipient is a list! -ddm
def TrySMTPDelivery(recipient, sender, text, queue_entry):
! import smtplib,sys
! #from Mailman.Logging.StampedLogger import StampedLogger
! #stdout_local=sys.stdout
! bad_addrs={}
! ######################Remove this!! ################
! # This is a temporary hack, to make old q'd messages work!!!!!!!!! -ddm
! if type(recipient) == type("string"):
! recipient = [recipient]
! ####################################################
try:
! # con = smtplib.SmtpConnection(mm_cfg.SMTPHOST)
! # con.helo(mm_cfg.DEFAULT_HOST_NAME)
! # con.send(to=recipient,frm=sender,text=text)
! # con.quit()
!
! # sys.stdout=StampedLogger('smtplib_debug',
! # label=sys.argv[0],
! # manual_reprime=1,
! # nofail=0)
!
! con = smtplib.SMTP(mm_cfg.SMTPHOST)
! # con.set_debuglevel(1)
! if con.ehlo(mm_cfg.DEFAULT_HOST_NAME) >= 400:
! con.helo(mm_cfg.DEFAULT_HOST_NAME)
! try:
! bad_addrs=con.sendmail(to_addrs = recipient, from_addr = sender, msg = text)
! except smtplib.SMTPRecipientsRefused:
! for each in recipient:
! bad_addrs[each]=(500,"Refused [smtplib]")
con.quit()
+
import OutgoingQueue
OutgoingQueue.dequeueMessage(queue_entry)
finally:
+ # sys.stdout=stdout_local
# except: # Todo: This might want to handle special cases.
pass # Just try again later.
!
! #def QuotePeriods(text):
! # return string.join(string.split(text, '\n.\n'), '\n .\n')
def ValidEmail(str):
"""Verify that the an email address isn't grossly invalid."""
*** /usr/src/mailman-1.0b5/mail/contact_transport Mon Jul 27 17:48:33 1998
--- contact_transport Fri Aug 28 20:37:07 1998
***************
*** 31,37 ****
# XXX: this really should be merged with Python's standard smtplib library
from Mailman import mm_cfg
! from Mailman import smtplib
from Mailman import Utils
from Mailman import OutgoingQueue
--- 31,37 ----
# XXX: this really should be merged with Python's standard smtplib library
from Mailman import mm_cfg
! #from Mailman import smtplib
from Mailman import Utils
from Mailman import OutgoingQueue
Hmm.. I found a bug in the post script whilst I was doing soething
else.. Dunno what the effect of this is, but it's a trivial fix.
Have a patch...
-The Dragon De Monsyne
*** /usr/src/mailman-1.0b5/scripts/post Sat Aug 29 00:38:50 1998
--- post Fri Aug 28 20:47:53 1998
***************
*** 100,107 ****
current_list.SendTextToUser( subject = 'Mail sent to %s' %
current_list.real_name,
recipient = the_sender,
! text = body,
! raw = 1)
# Let another process run.
finally:
current_list.Unlock()
--- 100,109 ----
current_list.SendTextToUser( subject = 'Mail sent to %s' %
current_list.real_name,
recipient = the_sender,
! text = body)
! # What's this? -ddm
! # raw = 1)
!
# Let another process run.
finally:
current_list.Unlock()
I have a couple minor problems with mailman:
1) If a message exceeds the configured size (40k), I get a bounce notice
from sendmail as well as the normal approval notice from mailman. The
error is unknown mailer error 1
This behavior is new in b5.
Aug 27 14:07:03 1998 post: Traceback (innermost last):
post: File "/home/mailman/scripts/post", line 100, in ?
post: current_list.SendTextToUser( subject = 'Mail sent to %s' %
post: TypeError : unexpected keyword argument: raw
Aug 27 14:07:09 1998 contact_transport: Traceback (innermost last):
contact_transport: File "/home/mailman/scripts/contact_transport", line
52, in
?
contact_transport: OutgoingQueue.processQueue()
contact_transport: File "/home/mailman/Mailman/OutgoingQueue.py", line
38, in
processQueue
contact_transport: Utils.TrySMTPDelivery(recip,sender,text,full_fname)
contact_transport: File "../Mailman/Utils.py", line 204, in TrySMTPDelivery
contact_transport: File "/home/mailman/Mailman/OutgoingQueue.py", line
25, in
dequeueMessage
contact_transport: os.unlink(msg)
contact_transport: OSError : [Errno 2] No such file or directory:
'/home/mailma
n/data/mm_q.2'
I'm not sure what raw=1 is intended to do, but it is causing the error when
the attempt is made to notify the user their mail is being held. The
definition of SendTextToUser in
Mailman/Deliverer.py does not have raw as an argument.
scripts/post:
current_list.SendTextToUser( subject = 'Mail sent to %s' %
current_list.real_name,
recipient = the_sender,
text = body,
raw = 1)
2) After performing some admin functions, e.g. mass subscribing or
attending to administrative requests, the list info page shows up with all
the fields blank and I have to re-enter data. There are no apparent errors
and I'm not sure how to debug this. Does anyone else have this problem? I
have had it in every version since the original I found at list.org before
the crash, however, prior to b5 it only happened when mass subscribing.
Hi.
I checked out the latest sources in order to try a translation to Swedish. I quickly found out that this was to become a rather complicated task. Presentation texts are practically shattered all over the sources.
Is there currently any work going on in this area?
What can I do to help improving the situation?
Note that I have limited time to allocate in this matter (well, who doesn't :), but want to do my part if others are interested to make an effort as well. The Mailman software is a really good peice of software in concept and design, part from the localization aspects that is.
I also want to mention that I earlier made a try to localize Majordomo. Same problem. Since I'm a python fan, I was delighted to find out about Mailman. Anyway, localization is an important issue, IMHO.
Cheers,
Tomas
Well, it's a bit rough, and there's a few more things i'd like to
do with it, But my free time may be scarse lately & I'm going on vacation
in a week, so I figure I'd better post this & let some folks play with it.
Anyway, this is the intgrated web-archiver module for Mailman.
For the moment, it still needs bsddb (mebbe after I get back I can try
tackling replacing that)
To get this working, here is what ye need to do:
1) Make sure you have the bsddb module compiled :P
2) Make sure you have somewhere in your PYTHONPATH
the following python modules:
a) The latest pipermail (0.0.5)
b) Digicool's DocumentTemplate package
<http://www.digicool.com/releases/bobo/DocumentTemplate-rn.html>
3) Make sure you get rid of the old pipermail 0.0.2 sitting in
the Mailman package.
4) Put HyperArch.py in the Mailman package directory.
5) Put the 'arch' script in the ${prefix}/cron directory
6) Apply the included patch.
7) add an entry like this to your crontab:
# Periodically update the webarchive.
0 3,9,17,23 * * * /usr/local/bin/python /usr/services/mailman/cron/arch
Notes:
You can run the arch script as often as you like, depending on how
up-to-date you want your webarchives to be.
You also can run the script from the command line to manually add
articles to the archive. (I just ran three years worth of traffic from a
reasonabally high-volume majordomo list I'm moving over thru this system,
and it went thru fine.)
You can set archives to use yearly, quarterly, or monthly archives
with this.
You will probably want to use the 'Set date to when re-sent' option
on your lists with this module.
All the HTML generated by this module is generated from a template.
This makes customizing the output trivial, just edit the template.
(Right now the templates are static strings. This should be changed soon.)
If you do not run the arch cronjob, then this module will do nothing,
and will not interfere with an external archiver.
-The Dragon De Monsyne
diff -c /usr/src/mailman-1.0b5/Mailman/Archiver.py Mailman/Archiver.py
*** /usr/src/mailman-1.0b5/Mailman/Archiver.py Mon Jul 27 17:48:31 1998
--- Mailman/Archiver.py Sun Aug 16 11:11:52 1998
***************
*** 42,49 ****
self.archive_private = mm_cfg.DEFAULT_ARCHIVE_PRIVATE
## self.archive_update_frequency = \
## mm_cfg.DEFAULT_ARCHIVE_UPDATE_FREQUENCY
! ## self.archive_volume_frequency = \
! ## mm_cfg.DEFAULT_ARCHIVE_VOLUME_FREQUENCY
## self.archive_retain_text_copy = \
## mm_cfg.DEFAULT_ARCHIVE_RETAIN_TEXT_COPY
--- 42,49 ----
self.archive_private = mm_cfg.DEFAULT_ARCHIVE_PRIVATE
## self.archive_update_frequency = \
## mm_cfg.DEFAULT_ARCHIVE_UPDATE_FREQUENCY
! self.archive_volume_frequency = \
! mm_cfg.DEFAULT_ARCHIVE_VOLUME_FREQUENCY
## self.archive_retain_text_copy = \
## mm_cfg.DEFAULT_ARCHIVE_RETAIN_TEXT_COPY
***************
*** 62,71 ****
def GetBaseArchiveURL(self):
if self.archive_private:
return os.path.join(mm_cfg.PRIVATE_ARCHIVE_URL,
! self._internal_name + ".html")
else:
return os.path.join(mm_cfg.PUBLIC_ARCHIVE_URL,
! self._internal_name + ".html")
def GetConfigInfo(self):
return [
--- 62,71 ----
def GetBaseArchiveURL(self):
if self.archive_private:
return os.path.join(mm_cfg.PRIVATE_ARCHIVE_URL,
! self._internal_name + mm_cfg.PRIVATE_ARCHIVE_URL_EXT)
else:
return os.path.join(mm_cfg.PUBLIC_ARCHIVE_URL,
! self._internal_name + mm_cfg.PRIVATE_ARCHIVE_URL_EXT)
def GetConfigInfo(self):
return [
***************
*** 85,93 ****
## "How often should new messages be incorporated? "
## "0 for no archival, 1 for daily, 2 for hourly"),
! ## ('archive_volume_frequency', mm_cfg.Radio, ('Yearly', 'Monthly'),
! ## 0,
! ## 'How often should a new archive volume be started?'),
## ('archive_retain_text_copy', mm_cfg.Toggle, ('No', 'Yes'),
## 0,
--- 85,93 ----
## "How often should new messages be incorporated? "
## "0 for no archival, 1 for daily, 2 for hourly"),
! ('archive_volume_frequency', mm_cfg.Radio,
! ('Yearly', 'Monthly','Quarterly'), 0,
! 'How often should a new archive volume be started?'),
## ('archive_retain_text_copy', mm_cfg.Toggle, ('No', 'Yes'),
## 0,
diff -c /usr/src/mailman-1.0b5/Mailman/Defaults.py Mailman/Defaults.py
*** /usr/src/mailman-1.0b5/Mailman/Defaults.py Wed Aug 19 01:46:37 1998
--- Mailman/Defaults.py Thu Aug 27 00:22:46 1998
***************
*** 37,42 ****
--- 37,48 ----
PUBLIC_ARCHIVE_URL = 'http://www.OVERRIDE.WITH.YOUR.PUBLIC.ARCHIVE.URL/'
PRIVATE_ARCHIVE_URL = 'http://www.OVERRIDE.WITH.YOUR.PRIVATE.ARCHIVE.URL/'
+ DEFAULT_ARCHIVE_VOLUME_FREQUENCY = 1
+
+
+ PUBLIC_ARCHIVE_URL_EXT = ''
+ PRIVATE_ARCHIVE_URL_EXT = ''
+
DEFAULT_ARCHIVE_PRIVATE = 0 # 0=public, 1=private
HOME_PAGE = 'index.html'
MAILMAN_OWNER = 'mailman-owner@%s' % DEFAULT_HOST_NAME
***************
*** 216,219 ****
VERSION = '1.0b5'
# Data file version number
! DATA_FILE_VERSION = 3
--- 222,225 ----
VERSION = '1.0b5'
# Data file version number
! DATA_FILE_VERSION = 4
diff -c /usr/src/mailman-1.0b5/Mailman/htmlformat.py Mailman/htmlformat.py
*** /usr/src/mailman-1.0b5/Mailman/htmlformat.py Mon Jul 27 17:48:31 1998
--- Mailman/htmlformat.py Wed Aug 19 02:24:43 1998
***************
*** 383,389 ****
InputObj.__init__(self, name, "TEXT", value, checked=0, size=size)
class TextArea:
! def __init__(self, name, text='', rows=None, cols=None, wrap='soft'):
self.name = name
self.text = text
self.rows = rows
--- 383,389 ----
InputObj.__init__(self, name, "TEXT", value, checked=0, size=size)
class TextArea:
! def __init__(self, name, text='', rows=None, cols=None, wrap='off'):
self.name = name
self.text = text
self.rows = rows
diff -c /usr/src/mailman-1.0b5/Mailman/versions.py Mailman/versions.py
*** /usr/src/mailman-1.0b5/Mailman/versions.py Mon Jul 27 17:48:31 1998
--- Mailman/versions.py Wed Aug 19 02:31:06 1998
***************
*** 63,70 ****
PreferStored('automatically_remove', 'automatic_bounce_action')
# - dropped vars:
for a in ['archive_retain_text_copy',
! 'archive_update_frequency',
! 'archive_volume_frequency']:
if hasattr(l, a): delattr(l, a)
def UpdateOldUsers(l):
--- 63,69 ----
PreferStored('automatically_remove', 'automatic_bounce_action')
# - dropped vars:
for a in ['archive_retain_text_copy',
! 'archive_update_frequency']:
if hasattr(l, a): delattr(l, a)
def UpdateOldUsers(l):
#!/usr/local/bin/python
import sys, string, getopt, os
import paths
import Mailman.HyperArch
import Mailman.Utils
from Mailman.HyperArch import HyperArchive
from Mailman.MailList import MailList
def ArchList(list):
l=MailList(list)
h=HyperArchive(l)
h.processListArch()
h.close()
def PrintUsage():
print (' usage: arch [-v | --verbose] [-l "list" | --list "list"] '
'[mailboxfiles]')
import sys
sys.exit(1)
def main():
if '-h' in sys.argv or '-?' in sys.argv:
PrintUsage()
return
config={'VERBOSE':0}
# Now we parse the command line options
opts, params = getopt.getopt(sys.argv[1:], 'vl:', ['verbose','list='])
for option, arg in opts:
if option=='-l': config['LIST']=arg
if option=='-v': config['VERBOSE']=1
if option=='--list': config['LIST']=arg
if option=='--verbose': config['VERBOSE']=1
if config.has_key('LIST'):
if params:
l=MailList(config['LIST'],lock=0)
h=HyperArchive(l)
if config['VERBOSE']:
h.VERBOSE=1
for each in params:
try:
f=open(each,'r')
except IOError:
sys.stderr.write("Cannot open %s\n" % each )
continue
h.processUnixMailbox(f,Mailman.HyperArch.Article)
f.close()
h.close()
else:
l=MailList(config['LIST'])
h=HyperArchive(l)
if config['VERBOSE']:
h.VERBOSE=1
h.processListArch()
h.close()
else:
#loop thru all the lists, doing archiving -ddm.
for each in Mailman.Utils.list_names():
if config['VERBOSE']:
sys.stderr.write("Processing list %s\n" % each)
l=MailList(each)
if not l.archive:
l.Unlock()
del l
else:
h=HyperArchive(l)
if config['VERBOSE']:
h.VERBOSE=1
h.processListArch()
h.close()
if __name__ == '__main__':
main()
"""HyperArch: Pipermail archiving for MailMan, using DocumentTemplate
templates
- The Dragon De Monsyne <dragondm(a)integral.org>
See <http://www.digicool.com/site/Principia/DTML.html> for some explanation
of DTML (DocumentTemplate) Syntax.
(Note: the URL listed acually documents a slightly extended DTML syntax
used by a product called Principia. Bug the folks at Digicool fer 'plain'
DTML documentation :> )
TODO:
- The templates should be be files in Mailman's Template dir, instead
of static strings.
- Each list should be able to have it's own templates.
Also, it should automatically fall back to default template in case
of error in list specific template.
- Should be able to force all HTML to be regenerated next time the archive
is run, incase a template is changed.
- Replace pipermail.BSDDBdatabase with something that dosen't require a C
extention (i.e. bsddb).
(Perhaps BoboPOS <http://www.digicool.com/releases/bobo/BoboPOS-rn.html>
could be used here?)
- Run a command to generate tarball of html archives for downloading
(prolly in the 'update_dirty_archives' method )
"""
import re, cgi, urllib, string
import time, pickle, os, posixfile
import DocumentTemplate
import pipermail
import mm_cfg
article_text_template="""From <!--#var email --> <!--#var datestr -->
Date: <!--#var datestr -->
From: <!--#var author --> <<!--#var email -->>
Subject: <!--#var subject -->
<!--#in body --><!--#var sequence-item --><!--#/in -->
"""
article_template="""<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<TITLE><!--#var subject html_quote --></TITLE>
<LINK REL="Index" HREF="index.html" >
<LINK REL="made" HREF="mailto:<!--#var email url_quote -->">
<!--linkthreads-->
<!--#if prev -->
<LINK REL="Previous" HREF="<!--#var expr="prev.filename" url_quote -->">
<!--#/if -->
<!--#if next -->
<LINK REL="Next" HREF="<!--#var expr="next.filename" url_quote -->">
<!--#/if -->
<!--endlinkthreads-->
</HEAD>
<BODY BGCOLOR="#ffffff">
<H1><!--#var subject html_quote --></H1>
<B><!--#var author html_quote --></B>
<A HREF="mailto:<!--#var email url_quote -->" TITLE="<!--#var subject html_quote -->"><!--#var email html_quote --></A><BR>
<I><!--#var datestr html_quote --></I>
<P><UL>
<!--threads-->
<!--#if prev -->
<LI> Previous message: <A HREF="<!--#var expr="prev.filename" url_quote -->"><!--#var expr="prev.subject" html_quote--></A></li>
<!--#/if -->
<!--#if next -->
<LI> Next message: <A HREF="<!--#var expr="next.filename" url_quote -->"><!--#var expr="next.subject" html_quote --></A></li>
<!--#/if -->
<!--endthreads-->
<LI> <B>Messages sorted by:</B>
<a href="date.html#<!--#var sequence -->">[ date ]</a>
<a href="thread.html#<!--#var sequence -->">[ thread ]</a>
<a href="subject.html#<!--#var sequence -->">[ subject ]</a>
<a href="author.html#<!--#var sequence -->">[ author ]</a>
</LI>
</UL>
<HR>
<!--beginarticle-->
<!--#in body --><!--#var sequence-item --><!--#/in -->
<!--endarticle-->
<HR>
<P><UL>
<!--threads-->
<!--#if prev -->
<LI> Previous message: <A HREF="<!--#var expr="prev.filename" url_quote -->"><!--#var expr="prev.subject" html_quote--></A></li>
<!--#/if -->
<!--#if next -->
<LI> Next message: <A HREF="<!--#var expr="next.filename" url_quote -->"><!--#var expr="next.subject" html_quote --></A></li>
<!--#/if -->
<!--endthreads-->
<LI> <B>Messages sorted by:</B>
<a href="date.html#<!--#var sequence -->">[ date ]</a>
<a href="thread.html#<!--#var sequence -->">[ thread ]</a>
<a href="subject.html#<!--#var sequence -->">[ subject ]</a>
<a href="author.html#<!--#var sequence -->">[ author ]</a>
</LI>
</UL>
</body></html>
"""
index_header_template="""<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<title>The <!--#var expr="maillist.real_name" --> <!--#var archive --> Archive by <!--#var type --></title>
</HEAD>
<BODY BGCOLOR="#ffffff">
<a name="start"></A>
<h1><!--#var archive --> Archives by <!--#var type --></h1>
<ul>
<li> <b>Messages sorted by:</b>
<!--#if expr="type<>'Thread' " -->
<a href="thread.html#start">[ thread ]</a>
<!--#/if -->
<!--#if expr="type<>'Subject' " -->
<a href="subject.html#start">[ subject ]</a>
<!--#/if -->
<!--#if expr="type<>'Author' " -->
<a href="author.html#start">[ author ]</a>
<!--#/if -->
<!--#if expr="type<> 'Date' " -->
<a href="date.html#start">[ date ]</a>
<!--#/if --> </li>
<li><b><a href="<!--#var expr="maillist.GetAbsoluteScriptURL('listinfo')" -->">More info on this list...</a></b></li>
<!--#if ARCHIVES -->
<li> <b><a href="<!--#var ARCHIVES -->">Other mail archives</a></b> </li>
<!--#/if -->
</ul>
<p><b>Starting:</b> <i><!--#var firstdate --></i><br>
<b>Ending:</b> <i><!--#var lastdate --></i><br>
<b>Messages:</b> <!--#var size --><p>
<ul>
"""
index_footer_template="""
</ul>
<p>
<a name="end"><b>Last message date:</b></a>
<i><!--#var lastdate --></i><br>
<b>Archived on:</b> <i><!--#var archivedate --></i>
<p>
<ul>
<li> <b>Messages sorted by:</b>
<!--#if expr="type<>'Thread' " -->
<a href="thread.html#start">[ thread ]</a>
<!--#/if -->
<!--#if expr="type<>'Subject' " -->
<a href="subject.html#start">[ subject ]</a>
<!--#/if -->
<!--#if expr="type<>'Author' " -->
<a href="author.html#start">[ author ]</a>
<!--#/if -->
<!--#if expr="type<> 'Date' " -->
<a href="date.html#start">[ date ]</a>
<!--#/if --> </li>
<li><b><a href="<!--#var expr="maillist.GetAbsoluteScriptURL('listinfo')" -->">More info on this list...</a></b></li>
<!--#if ARCHIVES -->
<li> <b><a href="<!--#var ARCHIVES -->">Other mail archives</a></b> </li>
<!--#/if -->
</ul>
<p>
<hr>
<i>This archive was generated by <a href="http://starship.skyport.net/crew/amk/maintained/pipermail.html">Pipermail <!--#var version --></a>.</i>
</BODY>
</HTML>"""
TOC_template="""<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<title>The <!--#var expr="maillist.real_name" --> Archives</title>
</HEAD>
<BODY BGCOLOR="#ffffff">
<h1>The <!--#var expr="maillist.real_name" --> Archives </h1>
<p>
<a href="<!--#var expr="maillist.GetAbsoluteScriptURL('listinfo')" -->">More info on this list...</a>
</p>
<!--#if archives -->
<!--#in archives -->
<!--#if sequence-start -->
<table border=3>
<tr><td>Archive</td> <td>View by:</td> <td>Downloadable version</td></tr>
<!--#/if -->
<tr>
<td><!--#var sequence-item -->:</td>
<td>
<A href="<!--#var sequence-item -->/thread.html">[ Thread ]</a>
<A href="<!--#var sequence-item -->/subject.html">[ Subject ]</a>
<A href="<!--#var sequence-item -->/author.html">[ Author ]</a>
<A href="<!--#var sequence-item -->/date.html">[ Date ]</a>
</td>
<td><A href="<!--#var sequence-item -->.txt.gz">[ Text ]</a></td>
</tr>
<!--#if sequence-end -->
</table>
<!--#/if -->
<!--#/in -->
<!--#else -->
<P>Currently, there are no archives. </P>
<!--#/if -->
</BODY>
</HTML>"""
def CGIescape(arg):
s=cgi.escape(str(arg))
s=re.sub('"', '"', s)
return s
# Parenthesized human name
paren_name_pat=re.compile(r'([(].*[)])')
# Subject lines preceded with 'Re:'
REpat=re.compile( r"\s*RE\s*:\s*",
re.IGNORECASE)
# E-mail addresses and URLs in text
emailpat=re.compile(r'([-+,.\w]+(a)[-+.\w]+)')
# Argh! This pattern is buggy, and will choke on URLs with GET parameters.
urlpat=re.compile(r'(\w+://[^>)\s]+)') # URLs in text
# Blank lines
blankpat=re.compile(r'^\s*$')
#
# Starting <html> directive
htmlpat=re.compile(r'^\s*<HTML>\s*$', re.IGNORECASE)
# Ending </html> directive
nohtmlpat=re.compile(r'^\s*</HTML>\s*$', re.IGNORECASE)
# Match quoted text
quotedpat=re.compile(r'^([>|:]|>)+')
# Note: I'm overriding most, if not all of the pipermail Article class here -ddm
# The Article class encapsulates a single posting. The attributes
# are:
#
# sequence : Sequence number, unique for each article in a set of archives
# subject : Subject
# datestr : The posting date, in human-readable format
# date : The posting date, in purely numeric format
# headers : Any other headers of interest
# author : The author's name (and possibly organization)
# email : The author's e-mail address
# msgid : A unique message ID
# in_reply_to : If !="", this is the msgid of the article being replied to
# references: A (possibly empty) list of msgid's of earlier articles in the thread
# body : A list of strings making up the message body
class Article(pipermail.Article):
__last_article_time=time.time()
html_tmpl=DocumentTemplate.HTML(article_template)
text_tmpl=DocumentTemplate.HTML(article_text_template)
def as_html(self):
return self.html_tmpl(self)
def as_text(self):
return self.text_tmpl(self)
def __init__(self, message=None, sequence=0, keepHeaders=[]):
import time
if message==None: return
self.sequence=sequence
self.parentID = None
self.threadKey = None
self.prev=None
self.next=None
# otherwise the current sequence number is used.
id=pipermail.strip_separators(message.getheader('Message-Id'))
if id=="": self.msgid=str(self.sequence)
else: self.msgid=id
if message.has_key('Subject'): self.subject=str(message['Subject'])
else: self.subject='No subject'
i=0
while (i!=-1):
result=REpat.match(self.subject)
if result:
i = result.end(0)
self.subject=self.subject[i:]
else: i=-1
if self.subject=="": self.subject='No subject'
if message.has_key('Date'):
self.datestr=str(message['Date'])
date=message.getdate_tz('Date')
else:
self.datestr='None'
date=None
if date!=None:
date, tzoffset=date[:9], date[-1]
if not tzoffset:
tzoffset = 0
date=time.mktime(date)-tzoffset
else:
date=self.__last_article_time+1
# print 'Article without date:', self.msgid
self.__last_article_time=date
self.date='%011i' % (date,)
# Figure out the e-mail address and poster's name
self.author, self.email=message.getaddr('From')
# e=message.getheader('Reply-To')
# if e!=None: self.email=e
self.email=pipermail.strip_separators(self.email)
self.author=pipermail.strip_separators(self.author)
if self.author=="": self.author=self.email
# Save the 'In-Reply-To:' and 'References:' lines
i_r_t=message.getheader('In-Reply-To')
if i_r_t==None: self.in_reply_to=''
else:
match=pipermail.msgid_pat.search(i_r_t)
if match==None: self.in_reply_to=''
else: self.in_reply_to=pipermail.strip_separators(match.group(1))
references=message.getheader('References')
if references==None: self.references=[]
else: self.references=map(pipermail.strip_separators, string.split(references))
# Save any other interesting headers
self.headers={}
for i in keepHeaders:
if message.has_key(i): self.headers[i]=message[i]
# Read the message body
self.body=[]
message.rewindbody()
while (1):
line=message.fp.readline()
if line=="": break
self.body.append(line)
def loadbody_fromHTML(self,fileobj):
self.body=[]
begin=0
while(1):
line=fileobj.readline()
if not line:
break
if (not begin) and string.strip(line)=='<!--beginarticle-->':
begin=1
continue
if string.strip(line)=='<!--endarticle-->':
break
if begin:
self.body.append(line)
def __getstate__(self):
d={}
for each in self.__dict__.keys():
if each in ['maillist','prev','next','body']:
d[each] = None
else:
d[each] = self.__dict__[each]
d['body']=[]
return d
class HyperArchive(pipermail.T):
# some defaults
DIRMODE=0755
FILEMODE=0644
VERBOSE=0
DEFAULTINDEX='thread'
ARCHIVE_PERIOD='month'
THREADLAZY=0
THREADLEVELS=3
ALLOWHTML=1
SHOWHTML=1
IQUOTES=1
SHOWBR=1
html_hdr_tmpl=DocumentTemplate.HTML(index_header_template)
html_foot_tmpl=DocumentTemplate.HTML(index_footer_template)
html_TOC_tmpl=DocumentTemplate.HTML(TOC_template)
def html_foot(self):
return self.html_foot_tmpl(self)
def html_head(self):
return self.html_hdr_tmpl(self)
def html_TOC(self):
return self.html_TOC_tmpl(self)
def __init__(self, maillist,unlock=1):
self.maillist=maillist
self._unlocklist=unlock
self._lock_file=None
if hasattr(self.maillist,'archive_volume_frequency'):
if self.maillist.archive_volume_frequency == 0:
self.ARCHIVE_PERIOD='year'
elif self.maillist.archive_volume_frequency == 2:
self.ARCHIVE_PERIOD='quarter'
else:
self.ARCHIVE_PERIOD='month'
pipermail.T.__init__(self, maillist.archive_directory, reload=1, database=pipermail.BSDDBdatabase(maillist.archive_directory))
def GetArchLock(self):
if self._lock_file:
return 1
ou = os.umask(0)
try:
self._lock_file = posixfile.open(
os.path.join(mm_cfg.LOCK_DIR, '%s(a)arch.lock' %
self.maillist._internal_name), 'a+')
finally:
os.umask(ou)
# minor race condition here, there is no way to atomicly
# check & get a lock. That shouldn't matter here tho' -ddm
if not self._lock_file.lock('w?', 1):
self._lock_file.lock('w|', 1)
else:
return 0
return 1
def DropArchLock(self):
if self._lock_file:
self._lock_file.lock('u')
self._lock_file.close()
self._lock_file = None
def processListArch(self):
name = self.maillist.ArchiveFileName()
wname= name+'.working'
ename= name+'.err_unarchived'
try:
os.stat(name)
except (IOError,os.error):
#no archive file, nothin to do -ddm
return
#see if arch is locked here -ddm
if not self.GetArchLock():
#another archiver is running, nothing to do. -ddm
return
#if the working file is still here, the archiver may have
# crashed during archiving. Save it, log an error, and move on.
try:
wf=open(wname,'r')
self.maillist.LogMsg("error","Archive working file %s present. "
"Check %s for possibly unarchived msgs" %
(wname,ename) )
ef=open(ename, 'a+')
ef.seek(1,2)
if ef.read(1) <> '\n':
ef.write('\n')
ef.write(wf.read())
ef.close()
wf.close()
os.unlink(wname)
except IOError:
pass
os.rename(name,wname)
if self._unlocklist:
self.maillist.Unlock()
archfile=open(wname,'r')
self.processUnixMailbox(archfile, Article)
archfile.close()
os.unlink(wname)
self.DropArchLock()
def get_filename(self, article):
return '%06i.html' % (article.sequence,)
def get_archives(self, article):
"""Return a list of indexes where the article should be filed.
A string can be returned if the list only contains one entry,
and the empty list is legal."""
if article.subject in ['subscribe', 'unsubscribe']: return None
return self.dateToVolName(string.atof(article.date))
# The following two methods should be inverses of each other. -ddm
def dateToVolName(self,date):
datetuple=time.gmtime(date)
if self.ARCHIVE_PERIOD=='year':
return time.strftime("%Y",datetuple)
elif self.ARCHIVE_PERIOD=='quarter':
if adate[1] in [1,2,3]:
return time.strftime("%Yq1",datetuple)
elif adate[1] in [4,5,6]:
return time.strftime("%Yq2",datetuple)
elif adate[1] in [7,8,9]:
return time.strftime("%Yq3",datetuple)
else:
return time.strftime("%Yq4",datetuple)
# month. -ddm
else:
return time.strftime("%Y-%B",datetuple)
def volNameToDate(self,volname):
volname=string.strip(volname)
volre= { 'year' : r'^(?P<year>[0-9]{4,4})$',
'quarter' : r'^(?P<year>[0-9]{4,4})q(?P<quarter>[1234])$',
'month' : r'^(?P<year>[0-9]{4,4})-(?P<month>[a-zA-Z]+)$' }
for each in volre.keys():
match=re.match(volre[each],volname)
if match:
year=string.atoi(match.group('year'))
month=1
if each == 'quarter':
q=string.atoi(match.group('quarter'))
month=(q*3)-2
elif each == 'month':
monthstr=string.lower(match.group('month'))
m=[]
for i in range(1,13):
m.append(string.lower(
time.strftime("%B",(1999,i,1,0,0,0,0,1,0))))
try:
month=m.index(monthstr)+1
except ValueError:
pass
return time.mktime((year,month,1,0,0,0,0,1,-1))
return 0.0
def sortarchives(self):
def sf(a,b,s=self):
al=s.volNameToDate(a)
bl=s.volNameToDate(b)
if al>bl:
return 1
elif al<bl:
return -1
else:
return 0
if self.ARCHIVE_PERIOD in ('month','year','quarter'):
self.archives.sort(sf)
else:
self.archives.sort()
def message(self, msg):
if self.VERBOSE:
import sys
sys.stderr.write(msg)
if msg[-1:]!='\n': sys.stderr.write('\n')
def open_new_archive(self, archive, archivedir):
import os
index_html=os.path.join(archivedir, 'index.html')
try: os.unlink(index_html)
except: pass
os.symlink(self.DEFAULTINDEX+'.html',index_html)
def write_index_header(self):
self.depth=0
print self.html_head()
if not self.THREADLAZY and self.type=='Thread':
# Update the threaded index
self.message("Computing threaded index\n")
self.updateThreadedIndex()
def write_index_footer(self):
import string
for i in range(self.depth): print '</UL>'
print self.html_foot()
def write_index_entry(self, article):
print '<LI> <A HREF="%s">%s</A> <A NAME="%i"></A><I>%s</I>' % (urllib.quote(article.filename),
CGIescape(article.subject), article.sequence,
CGIescape(article.author))
def write_threadindex_entry(self, article, depth):
if depth<0:
sys.stderr.write('depth<0') ; depth=0
if depth>self.THREADLEVELS: depth=self.THREADLEVELS
if depth<self.depth:
for i in range(self.depth-depth): print '</UL>'
elif depth>self.depth:
for i in range(depth-self.depth): print '<UL>'
print '<!--%i %s -->' % (depth, article.threadKey)
self.depth=depth
print '<LI> <A HREF="%s">%s</A> <A NAME="%i"></A><I>%s</I>' % (CGIescape(urllib.quote(article.filename)),
CGIescape(article.subject), article.sequence+910,
CGIescape(article.author))
def write_TOC(self):
self.sortarchives()
toc=open(os.path.join(self.basedir, 'index.html'), 'w')
toc.write(self.html_TOC())
toc.close()
# Archive an Article object.
def add_article(self, article):
# Determine into what archives the article should be placed
archives=self.get_archives(article)
if archives==None: archives=[] # If no value was returned, ignore it
if type(archives)==type(''): archives=[archives] # If a string was returned, convert to a list
if archives==[]: return # Ignore the article
# Add the article to each archive in turn
article.filename=filename=self.get_filename(article)
article_text=article.as_text()
temp=self.format_article(article) # Reformat the article
self.message("Processing article #"+str(article.sequence)+' into archives '+str(archives))
for i in archives:
self.archive=i
archivedir=os.path.join(self.basedir, i)
# If it's a new archive, create it
if i not in self.archives:
self.archives.append(i) ; self.update_TOC=1
self.database.newArchive(i)
# If the archive directory doesn't exist, create it
try: os.stat(archivedir)
except os.error, errdata:
errno, errmsg=errdata
if errno==2:
os.mkdir(archivedir, self.DIRMODE)
else: raise os.error, errdata
self.open_new_archive(i, archivedir)
# Write the HTML-ized article to the html archive.
f=open(os.path.join(archivedir, filename), 'w')
os.chmod(os.path.join(archivedir, filename), self.FILEMODE)
f.write(temp.as_html())
f.close()
# Write the text article to the text archive.
archivetextfile=os.path.join(self.basedir,"%s.txt" % i)
f=open(archivetextfile, 'a+')
os.chmod(archivetextfile, self.FILEMODE)
f.write(article_text)
f.close()
authorkey=pipermail.fixAuthor(article.author)+'\000'+article.date
subjectkey=string.lower(article.subject)+'\000'+article.date
# Update parenting info
parentID=None
if article.in_reply_to!='': parentID=article.in_reply_to
elif article.references!=[]:
# Remove article IDs that aren't in the archive
refs=filter(lambda x, self=self: self.database.hasArticle(self.archive, x),
article.references)
if len(refs):
refs=map(lambda x, s=self: s.database.getArticle(s.archive, x), refs)
maxdate=refs[0]
for ref in refs[1:]:
if ref.date>maxdate.date: maxdate=ref
parentID=maxdate.msgid
else:
# Get the oldest article with a matching subject, and assume this is
# a follow-up to that article
parentID=self.database.getOldestArticle(self.archive, article.subject)
if parentID!=None and not self.database.hasArticle(self.archive, parentID):
parentID=None
article.parentID=parentID
if parentID!=None:
parent=self.database.getArticle(self.archive, parentID)
article.threadKey=parent.threadKey+article.date+'-'
else: article.threadKey=article.date+'-'
self.database.setThreadKey(self.archive, article.threadKey+'\000'+article.msgid, article.msgid)
self.database.addArticle(i, temp, subjectkey, authorkey)
if i not in self._dirty_archives:
self._dirty_archives.append(i)
del temp
# Update only archives that have been marked as "changed".
def update_dirty_archives(self):
for i in self._dirty_archives:
self.update_archive(i)
archz=None
archt=None
try:
import gzip
try:
archt=open(os.path.join(self.basedir,"%s.txt" % i),"r")
try:
os.rename(os.path.join(self.basedir,"%s.txt.gz" % i),
os.path.join(self.basedir,"%s.old.txt.gz" % i))
archz=gzip.open(os.path.join(self.basedir,"%s.old.txt.gz" % i),"r")
except (IOError, RuntimeError, os.error):
pass
newz=gzip.open(os.path.join(self.basedir,"%s.txt.gz" % i),"w")
if archz :
newz.write(archz.read())
archz.close()
os.unlink(os.path.join(self.basedir,"%s.old.txt.gz" % i))
newz.write(archt.read())
newz.close()
archt.close()
os.unlink(os.path.join(self.basedir,"%s.txt" % i))
except IOError:
pass
except ImportError:
pass
self._dirty_archives=[]
def close(self):
"Close an archive, saving its state and updating any changed archives."
self.update_dirty_archives()# Update all changed archives
# If required, update the table of contents
if self.update_TOC or 1:
self.update_TOC=0
self.write_TOC()
# Save the collective state
self.message('Pickling archive state into '+os.path.join(self.basedir, 'pipermail.pck'))
self.database.close()
del self.database
f=open(os.path.join(self.basedir, 'pipermail.pck'), 'w')
pickle.dump(self.__getstate__(), f)
f.close()
def __getstate__(self):
d={}
for each in self.__dict__.keys():
if not (each in ['maillist','_lock_file','_unlocklist']):
d[each] = self.__dict__[each]
return d
# Add <A HREF="..."> tags around URLs and e-mail addresses.
def __processbody_URLquote(self, source, dest):
body2=[]
last_line_was_quoted=0
for i in xrange(0, len(source)):
Lorig=L=source[i] ; prefix=suffix=""
if L==None: continue
# Italicise quoted text
if self.IQUOTES:
quoted=quotedpat.match(L)
if quoted==None: last_line_was_quoted=0
else:
quoted = quoted.end(0)
prefix=CGIescape(L[:quoted]) + '<i>'
suffix='</I>'
if self.SHOWHTML: suffix=suffix+'<BR>'
if not last_line_was_quoted: prefix='<BR>'+prefix
L= L[quoted:]
last_line_was_quoted=1
# Check for an e-mail address
L2="" ; jr=emailpat.search(L) ; kr=urlpat.search(L)
while jr!=None or kr!=None:
if jr==None: j=-1
else: j = jr.start(0)
if kr==None: k=-1
else: k = kr.start(0)
if j!=-1 and (j<k or k==-1): text=jr.group(1) ; URL='mailto:'+text ; pos=j
elif k!=-1 and (j>k or j==-1): text=URL=kr.group(1) ; pos=k
else: # j==k
raise ValueError, "j==k: This can't happen!"
length=len(text)
# sys.stderr.write("URL: %s %s %s \n" % (CGIescape(L[:pos]), URL, CGIescape(text)))
L2=L2+'%s<A HREF="%s">%s</A>' % (CGIescape(L[:pos]), URL, CGIescape(text))
L=L[pos+length:]
jr=emailpat.search(L) ; kr=urlpat.search(L)
if jr==None and kr==None: L=CGIescape(L)
L=prefix+L2+L+suffix
if L!=Lorig: source[i], dest[i]=None, L
# Escape all special characters
def __processbody_CGIescape(self, source, dest):
import cgi
for i in xrange(0, len(source)):
if source[i]!=None:
dest[i]=cgi.escape(source[i]) ; source[i]=None
# Perform Hypermail-style processing of <HTML></HTML> directives
# in message bodies. Lines between <HTML> and </HTML> will be written
# out precisely as they are; other lines will be passed to func2
# for further processing .
def __processbody_HTML(self, source, dest):
l=len(source) ; i=0
while i<l:
while i<l and htmlpat.match(source[i])==None: i=i+1
if i<l: source[i]=None ; i=i+1
while i<l and nohtmlpat.match(source[i])==None:
dest[i], source[i] = source[i], None
i=i+1
if i<l: source[i]=None ; i=i+1
def format_article(self, article):
source=article.body ; dest=[None]*len(source)
# Handle <HTML> </HTML> directives
if self.ALLOWHTML:
self.__processbody_HTML(source, dest)
self.__processbody_URLquote(source, dest)
if not self.SHOWHTML:
# Do simple formatting here: <PRE>..</PRE>
for i in range(0, len(source)):
s=source[i]
if s==None: continue
dest[i]=CGIescape(s) ; source[i]=None
if len(dest) > 0:
dest[0]='<PRE>'+dest[0] ; dest[-1]=dest[-1]+'</PRE>'
else:
# Do fancy formatting here
if self.SHOWBR:
# Add <BR> onto every line
for i in range(0, len(source)):
s=source[i]
if s==None: continue
s=CGIescape(s) +'<BR>'
dest[i]=s ; source[i]=None
else:
for i in range(0, len(source)):
s=source[i]
if s==None: continue
s=CGIescape(s)
if s[0:1] in ' \t\n': s='<P>'+s
dest[i]=s ; source[i]=None
article.body=filter(lambda x: x!=None, dest)
return article
def update_article(self, arcdir, article, prev, next):
import os
self.message('Updating HTML for article '+str(article.sequence))
try:
f=open(os.path.join(arcdir, article.filename), 'r')
article.loadbody_fromHTML(f)
f.close()
except IOError:
self.message("article file %s is missing!" % os.path.join(arcdir, article.filename))
article.prev=prev
article.next=next
f=open(os.path.join(arcdir, article.filename), 'w')
f.write(article.as_html())
f.close()
I seem to remember at least one person posting some replacement code to
try to clear up the duplicates problem...anyone know if it works yet? I
need to go dig that code out (assuming I'm right in remembering it)
-------------------------------------------------------------------------------
Corbett J. Klempay Quote of the Week:
http://www2.acm.jhu.edu/~cklempay "There are two things a real man likes -
danger and play. And he likes women
because she is the most dangerous of
playthings."
PGP Fingerprint: 7DA2 DB6E 7F5E 8973 A8E7 347B 2429 7728 76C2 BEA1
-------------------------------------------------------------------------------
