Re: [Mailman-Developers] [Mailman-Users] handler to auto detach attachment and link it to a website keeping html
Coding question.
A bit of context:
Goal is to develop a custom handler to perform:
- detach all attachments
- post detached content somewhere available on http
- modify the content of the original email, keeping original html, adding an html link to the moved document
- adding a small clip image as an attachment symbol as an embed object
Le 16/04/2014 16:01, Mark Sapiro a écrit :
<https://docs.python.org/2/library/email.mime.html> to create new sub-parts and use the Message.attach() method to attach them.
How can I replace the part which contain the html part, by a new one which will become :
Content-Type: multipart/related
Content-Type: text/html; Content-Type: image/png; name="clip-24.png"
The text/html is the old one.
Adding a part is ok, but replacing it with a new one which embed the previous one… I dont see.
part of the code I've produced:
if msg.is_multipart():
syslog('debug', 'multipart message rewriting')
related = None
for part in msg.walk():
ctype = part.get_content_type()
if ctype == 'text/plain' and not part['X-Mailman-Part']:
new_footer = txt_attacht_replace + footer_attach
part.set_payload(part.get_payload() + new_footer)
continue
elif ctype == 'text/html':
related = MIMEMultipart('related')
clip = MIMEText(attach_clip)
clip.add_header('Content-Disposition', 'inline')
clip.add_header('Content-ID', '<part1.%d>' % 123412)
clip['Content-Transfer-Encoding'] = 'base64'
del clip['Content-type']
del clip['Content-transfer-encoding']
clip['Content-type'] = 'image/png; name="attachment-24.png"' clip['Content-transfer-encoding'] = 'base64'
html_footer = html_attachment_holder % {'html_attachment_clip_tpl': html_footer_attach} html_footer += '</body>' old_content = part.get_payload() new_content = re.sub(r'</body>', html_footer, old_content) part.set_payload(new_content)
related.attach(part)
related.attach(clip)
# WRONG
del part
msg.attach(related)
continue
On 04/18/2014 09:48 AM, Sylvain Viart wrote:
Coding question.
A bit of context:
Goal is to develop a custom handler to perform:
- detach all attachments
- post detached content somewhere available on http
- modify the content of the original email, keeping original html, adding an html link to the moved document
- adding a small clip image as an attachment symbol as an embed object
Le 16/04/2014 16:01, Mark Sapiro a écrit :
<https://docs.python.org/2/library/email.mime.html> to create new sub-parts and use the Message.attach() method to attach them.
How can I replace the part which contain the html part, by a new one which will become :
Content-Type: multipart/related
Content-Type: text/html; Content-Type: image/png; name="clip-24.png"
The text/html is the old one.
Adding a part is ok, but replacing it with a new one which embed the previous one… I dont see.
part of the code I've produced:
if msg.is_multipart(): syslog('debug', 'multipart message rewriting') related = None for part in msg.walk(): ctype = part.get_content_type() if ctype == 'text/plain' and not part['X-Mailman-Part']: new_footer = txt_attacht_replace + footer_attach part.set_payload(part.get_payload() + new_footer) continue elif ctype == 'text/html': related = MIMEMultipart('related')
This stuff ----------------
clip = MIMEText(attach_clip) clip.add_header('Content-Disposition', 'inline') clip.add_header('Content-ID', '<part1.%d>' % 123412) clip['Content-Transfer-Encoding'] = 'base64' del clip['Content-type'] del clip['Content-transfer-encoding'] clip['Content-type'] = 'image/png;name="attachment-24.png"' clip['Content-transfer-encoding'] = 'base64'
should be ---------------
clip = MIMEImage(xxx, 'png')where xxx is the actual png data, e.g. xxx = open('some_file.png').read()
html_footer = html_attachment_holder %{'html_attachment_clip_tpl': html_footer_attach} html_footer += '</body>' old_content = part.get_payload() new_content = re.sub(r'</body>', html_footer, old_content) part.set_payload(new_content)
related.attach(part) related.attach(clip)
Now related is the part you want. all that remains is to replace the original part with related
# WRONG del part msg.attach(related)
Correct. That's wrong. You need to do this differently. You need to walk the message, but with your own code rather than msg.walk so you can build your new message as you go.
You need some variant of
from email.mime.multipart import MIMEMultipart from email.mime.image import MIMEImage
def fix_msg(msg): if msg.is_multipart(): parts = msg.get_payload() # remove the next level parts, then process and reattach them msg.set_payload(None) for p in parts: msg.attach(fix_msg(p)) return msg else: # process the 'leaf' parts ctype = msg.get_content_type() if ctype == 'text/plain' and not part['X-Mailman-Part']: # add footer to plain text new_footer = txt_attacht_replace + footer_attach msg.set_payload(part.get_payload() + new_footer) return msg elif ctype == 'text/html': # build multipart/related for HTML related = MIMEMultipart('related') clip = MIMEImage(xxx, 'png') html_footer = html_attachment_holder % {'html_attachment_clip_tpl': html_footer_attach} html_footer += '</body>' old_content = msg.get_payload() new_content = re.sub(r'</body>', html_footer, old_content) msg.set_payload(new_content) related.attach(msg) related.attach(clip) return related
def process(mlist, msg, msgdata) ... msg = fix_msg(msg) ...
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi,
Thanks Mark, exactly what I need to replace with the multipart/related in python!
Le 18/04/2014 22:30, Mark Sapiro a écrit :
This stuff ----------------
clip = MIMEText(attach_clip) clip.add_header('Content-Disposition', 'inline') clip.add_header('Content-ID', '<part1.%d>' % 123412) clip['Content-Transfer-Encoding'] = 'base64' del clip['Content-type'] del clip['Content-transfer-encoding'] clip['Content-type'] = 'image/png;name="attachment-24.png"' clip['Content-transfer-encoding'] = 'base64'
should be ---------------
clip = MIMEImage(xxx, 'png')
Yeah, I know. But I've embedded the base64 txt data of the clip image in the handler.py's code. So I don't need to manage an external file for the clip. Could MIMEImage handle internal data as well?
It seems extra effort, to mime64 decode and let MIMEImage reencode it, right?
Oh, great! That exactly the following code I was fighting with. I've seen some code like that in Handlers/MimeDel.py, thanks a lot. I wasn't able to adapt this recursive behavior.
def fix_msg(msg): if msg.is_multipart(): parts = msg.get_payload() # remove the next level parts, then process and reattach them msg.set_payload(None) for p in parts: msg.attach(fix_msg(p)) return msg else: # process the 'leaf' parts [truncated …] ctype = msg.get_content_type() if ctype == 'text/plain' and not part['X-Mailman-Part']: return msg elif ctype == 'text/html': return related
On 04/18/2014 11:57 PM, Sylvain Viart wrote:
Le 18/04/2014 22:30, Mark Sapiro a écrit :
should be ---------------
clip = MIMEImage(xxx, 'png')Yeah, I know. But I've embedded the base64 txt data of the clip image in the handler.py's code. So I don't need to manage an external file for the clip. Could MIMEImage handle internal data as well?
It seems extra effort, to mime64 decode and let MIMEImage reencode it, right?
You don't have to do that. If xxx is the base64 encoded image data, all you need is
clip = MIMEImage(xxx, 'png', _encoder=email.encoders.encode_noop)
clip['Content-Transfer-Encoding'] = 'base64'-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi,
Seems working… Great.
Le 19/04/2014 20:24, Mark Sapiro a écrit :
You don't have to do that. If xxx is the base64 encoded image data, all you need is
clip = MIMEImage(xxx, 'png', _encoder=email.encoders.encode_noop) clip['Content-Transfer-Encoding'] = 'base64'
Cool!
The working code I produced is:
from email.mime.image import MIMEImage from email import encoders #[…] clip = MIMEImage(ATTACH_CLIP, 'png', _encoder=encoders.encode_noop) clip['Content-Transfer-Encoding'] = 'base64' clip.add_header('Content-ID', '<part1.%s>' % clip_cid)
Thank you. :-)
List configuration question.
I need to pass some configuration plugin to this handler. Like remote ftp parameters. I've seen code like this:
if mlist.convert_html_to_plaintext and mm_cfg.HTML_TO_PLAIN_TEXT_COMMAND:
I suppose that I configure it that way:
/etc/mailman/mm_cfg.py HTML_TO_PLAIN_TEXT_COMMAND = 'path/to/converter'
and in my extend()
mlist.convert_html_to_plaintext = 1
is in /var/lib/mailman/lists/mytestlist/extend.py [debian base path]
I took it here http://wiki.list.org/pages/viewpage.action?pageId=4030615. I used extend.py to install my custom handler that way:
import copy from Mailman import mm_cfg def extend(mlist): mlist.pipeline = copy.copy(mm_cfg.GLOBAL_PIPELINE) # The next line inserts MyHandler ahead of Moderate. mlist.pipeline.insert(mlist.pipeline.index('Moderate'), 'MyHandler')
Also, it's indicated that there's a kind of caching involved with the Handler's code…:
Note however, that the first time Mailman saves the list, the pipeline attribute will be saved along with it, so simply removing extend.py from lists/test-list/ won't remove the special pipeline.
I do perform a mailman restart to load new handler's code:
/etc/init.d/mailman restart
Is there some documentation about list configuration override order?
I found a topic giving some information for personalizing the list:
4.48 How can I change the HTML (or .txt) templates used by my mailing lists? http://wiki.list.org/pages/viewpage.action?pageId=4030605
Regards, Sylvain.
On 04/22/2014 03:46 AM, Sylvain Viart wrote:
List configuration question.
I need to pass some configuration plugin to this handler. Like remote ftp parameters. I've seen code like this:
if mlist.convert_html_to_plaintext andmm_cfg.HTML_TO_PLAIN_TEXT_COMMAND:
I suppose that I configure it that way:
/etc/mailman/mm_cfg.py HTML_TO_PLAIN_TEXT_COMMAND = 'path/to/converter'
and in my extend()
mlist.convert_html_to_plaintext = 1
is in /var/lib/mailman/lists/mytestlist/extend.py [debian base path]
If you are doing this as a part of content filtering you just set the list's Content filtering -> convert_html_to_plaintext to Yes, but I suspect you are not.
In that case, don't hijack content filtering settings for your own purpose. It won't work.
If you are going to hard code some setting for your handler to use, just hard code it in the handler.
I took it here http://wiki.list.org/pages/viewpage.action?pageId=4030615. I used extend.py to install my custom handler that way:
import copy from Mailman import mm_cfg def extend(mlist): mlist.pipeline = copy.copy(mm_cfg.GLOBAL_PIPELINE) # The next line inserts MyHandler ahead of Moderate. mlist.pipeline.insert(mlist.pipeline.index('Moderate'), 'MyHandler')
Also, it's indicated that there's a kind of caching involved with the Handler's code…:
Note however, that the first time Mailman saves the list, the pipeline attribute will be saved along with it, so simply removing extend.py from lists/test-list/ won't remove the special pipeline.
It is not caching per se. Once you access the list once and save it, your modified pipeline is saved as a list attribute just like any other setting. Removing extend.py will not remove the pipeline attribute from the list. That is what the above is trying to tell you.
I do perform a mailman restart to load new handler's code:
/etc/init.d/mailman restart
Yes. That is correct.
Is there some documentation about list configuration override order?
Not outside the source code. See the __init__ method of the MailList class in Mailman/MailList.py. The list's extend.py if any is executed before the list's configuration is loaded from disk. Thus any list attribute set in extend.py that also exists in the list's saved configuration will be overridden by the list config.
I found a topic giving some information for personalizing the list:
4.48 How can I change the HTML (or .txt) templates used by my mailing lists? http://wiki.list.org/pages/viewpage.action?pageId=4030605
Which only talks about search rules for the built in templates and has nothing to do with anything else.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi,
Thanks for your helpful answer. :-)
Sorry it seems I don't have explained my needs correctly:
On 04/22/2014 03:46 AM, Sylvain Viart wrote:
List configuration question.
I need to pass some configuration plugin to this handler. Like remote ftp parameters.
I'm looking for a way to handle my own list's parameter, and I quoted some /equivalent/ parameter. Just some chuck of code I would like to implement for my own handler.
I need to set :
mlist.ftp_remote_host = 'ftp.example.com' mlist.ftp_remote_login = 'somelogin' mlist.ftp_remote_pass = 'secretstuff'
I can set site global vars in /etc/mailman/mm_cfg.py but how do I set local list parameter exactly? I wont code web interface to handle this, text file config will be great enough.
In that case, don't hijack content filtering settings for your own purpose. It won't work. If you are going to hard code some setting for your handler to use, just hard code it in the handler.
Sorry about this quoting introducing a confusion. No hijack here. No hard code neither because if I want to open and share the code, I have to remove sensible information from it. Code can embed default information, but not the config.
Is there some documentation about list configuration override order?
Not outside the source code. See the __init__ method of the MailList class in Mailman/MailList.py. The list's extend.py if any is executed before the list's configuration is loaded from disk. Thus any list attribute set in extend.py that also exists in the list's saved configuration will be overridden by the list config.
And where do I set list configuration? I'm not familiar enough with this code to /see/ where it comes from.
Mailman/MailList.py […] mailman 2.1.15
filename = os.path.join(self.fullpath(), 'extend.py')
dict = {}
try:
execfile(filename, dict)
except IOError, e:
# Ignore missing files, but log other errors
if e.errno == errno.ENOENT:
pass
else:
syslog('error', 'IOError reading list extension: %s', e)
else:
func = dict.get('extend')
if func:
func(self) <============== run the extend() function found in mylists/extend.py if lock: # This will load the database. self.Lock() <============== do some more stuff possibly overwriting what was done in extend() else: self.Load()
There should be something about bin/config_list I think. Web config seems detailed here: http://terri.zone12.com/doc/mailman/mailman-admin/node9.html
Is there a room for free parameter as well?
Regards, Sylvain.
On 04/22/2014 10:13 PM, Sylvain Viart wrote:
I need to set :
mlist.ftp_remote_host = 'ftp.example.com' mlist.ftp_remote_login = 'somelogin' mlist.ftp_remote_pass = 'secretstuff'
I can set site global vars in /etc/mailman/mm_cfg.py but how do I set local list parameter exactly?
bin/withlist or bin/config_list.
E.g., put the above three lines in a file and run
bin/config_list -i /path/to/file listname.
Note, I just removed the information about using extend.py from the FAQ at <http://wiki.list.org/x/l4A9> because the only reason it was there was for the situation where GLOBAL_PIPELINE would change in the future, and it won't work for that anyway because the list's existing pipeline attribute will override it.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I'm a bit confused, but it works… Thanks. :-)
Le 23/04/2014 15:30, Mark Sapiro a écrit :
I need to set :
mlist.ftp_remote_host = 'ftp.example.com' mlist.ftp_remote_login = 'somelogin' mlist.ftp_remote_pass = 'secretstuff'
I can set site global vars in /etc/mailman/mm_cfg.py but how do I set local list parameter exactly?
bin/withlist or bin/config_list.
E.g., put the above three lines in a file and run
bin/config_list -i /path/to/file listname.
Works great as you said! :-) I wasn't sure how to use this tool. I supposed you have to dump the whole config first with -o, add my stuff and reload the whole… But it's state's keeping between call of bin/config_list… Cool.
# config_list -v -i t.py testlist # withlist -i testlist Loading list testlist (unlocked) The variable `m' is the testlist MailList instance
m.ftp_remote_host 'ftp.example.com'
this one gives nothing… # config_list -o - testlist | grep ftp
Can I guess that this tool don't exactly dump the full list state? It can't be used for duplicating or full backup for instance?
My test this morning was failing because of wrong usage… See bellow. I tested your script set_attributes <http://fog.ccsf.edu/%7Emsapiro/scripts/set_attributes>
Mark Sapiro's page : scripts that automate certain mailing list
management tasks <http://fog.ccsf.edu/~msapiro/scripts/
<http://fog.ccsf.edu/%7Emsapiro/scripts/>>Which gives:
# /usr/lib/mailman/bin/list_set_attributes "ftp_remote_host = 'ftp.example.org'" testlist attribute "ftp_remote_host" changed Non-standard property restored: ftp_remote_host
If the extend.py exists with variable defined as follow:
to be set in lists/yourlistname/extend.py
import copy from Mailman import mm_cfg def extend(mlist): mlist.pipeline = copy.copy(mm_cfg.GLOBAL_PIPELINE) # The next line inserts MyHandler ahead of Moderate. mlist.pipeline.insert(mlist.pipeline.index('Moderate'), 'MyHandler') # Alternatively, the next line replaces Moderate with MyHandler #mlist.pipeline[mlist.pipeline.index('Moderate')] = 'MyHandler' # Pick one of the two above example alternatives
mlist.ftp_remote_host = 'ftp.example.com'
mlist.ftp_remote_login = 'username'
mlist.ftp_remote_pass = 'secr3te'
# put the ending slash /
mlist.remote_http_base = 'http://example.com/root/for/username/'If the value are not defined first I got:
/usr/lib/mailman/bin/list_set_attributes "dumy_remote_host = 'ftp.example.org'" testlist attribute "dumy_remote_host" ignored
works with "mlist.dumy_remote_host = 'ftp.example.org'" as you explained at the top, of course…
Note, I just removed the information about using extend.py from the FAQ at <http://wiki.list.org/x/l4A9> because the only reason it was there was for the situation where GLOBAL_PIPELINE would change in the future, and it won't work for that anyway because the list's existing pipeline attribute will override it.
Hum… I'm using it. How I'm supposed to modify the pipeline, so?… /usr/lib/mailman/bin/version Using Mailman version: 2.1.15
Using config_list ok… Hum probably not…
# cat t.py import copy from Mailman import mm_cfg mlist.pipeline = copy.copy(mm_cfg.GLOBAL_PIPELINE) # The next line inserts MyHandler ahead of Moderate. mlist.pipeline.insert(mlist.pipeline.index('Moderate'), 'MyHandler')
# config_list -v -i t.py testlist attribute "mm_cfg" ignored attribute "copy" ignored
Regards, Sylvain.
On 04/23/2014 08:31 AM, Sylvain Viart wrote:
Le 23/04/2014 15:30, Mark Sapiro a écrit :
E.g., put the above three lines in a file and run
bin/config_list -i /path/to/file listname.
Works great as you said! :-) I wasn't sure how to use this tool. I supposed you have to dump the whole config first with -o, add my stuff and reload the whole… But it's state's keeping between call of bin/config_list… Cool.
No. config_list will only set/change those things that are in it's input. So for what you want to do, an input file with just those three lines is what you want.
# config_list -v -i t.py testlist # withlist -i testlist Loading list testlist (unlocked) The variable `m' is the testlist MailList instance
m.ftp_remote_host 'ftp.example.com'
this one gives nothing… # config_list -o - testlist | grep ftp
Can I guess that this tool don't exactly dump the full list state? It can't be used for duplicating or full backup for instance?
Config_list is a very powerful tool. It's input file is actually Python and is exectuted, so you can do almost anything with it. See the -i option description in 'config_list -h'.
You can set things in two ways. The first paragraph refers to putting stuff like
ftp_remote_host = 'ftp.example.com'
in the input. This works to set attributes that already exist in the list object. It will warn you if the attribute is 'non-standard', i.e. doesn't appear in the web admin UI, but it will set it if it is already an attribute of the list. It won't work in your case (at least the first time) because ftp_remote_host is a "variable that isn't already an attribute of the list object (and) is ignored"
The second paragraph says you have access to the list object through the variable 'mlist', so you can set anything, existing or not, via syntax like
mlist.ftp_remote_host = 'ftp.example.com'
On the other hand, 'config_list -o' writes only those standard attributes that appear in the web admin UI.
My test this morning was failing because of wrong usage… See bellow. I tested your script set_attributes <http://fog.ccsf.edu/%7Emsapiro/scripts/set_attributes>
Mark Sapiro's page : scripts that automate certain mailing list management tasks <http://fog.ccsf.edu/~msapiro/scripts/ <http://fog.ccsf.edu/%7Emsapiro/scripts/>>
Which gives:
# /usr/lib/mailman/bin/list_set_attributes "ftp_remote_host = 'ftp.example.org'" testlist attribute "ftp_remote_host" changed Non-standard property restored: ftp_remote_host
Which means you probably had already put it there some other way or the output would have been "attribute "ftp_remote_host" ignored"
...
Note, I just removed the information about using extend.py from the FAQ at <http://wiki.list.org/x/l4A9> because the only reason it was there was for the situation where GLOBAL_PIPELINE would change in the future, and it won't work for that anyway because the list's existing pipeline attribute will override it.
Hum… I'm using it. How I'm supposed to modify the pipeline, so?…
As I say above, the first time you accessed the list with the extend.py as above, you set the pipeline and maybe other attributes. Once those attributes exist in the list object, you can't change them via the extend.py mechanism. Well you can if you also save the list via mlist.Save() after resetting them, but I don't recommended that which is why I removed the extend.py stuff from the FAQ.
Use either withlist or config_list to set these attributes.
/usr/lib/mailman/bin/version Using Mailman version: 2.1.15
Using config_list ok… Hum probably not…
# cat t.py import copy from Mailman import mm_cfg mlist.pipeline = copy.copy(mm_cfg.GLOBAL_PIPELINE) # The next line inserts MyHandler ahead of Moderate. mlist.pipeline.insert(mlist.pipeline.index('Moderate'), 'MyHandler')
# config_list -v -i t.py testlist attribute "mm_cfg" ignored attribute "copy" ignored
Open Mailman/Defaults.py
Copy the definition of GLOBAL_PIPELINE
paste it into a new file changing the name from GLOBAL_PIPELINE to mlist.pipeline and add your handler so it becomes
mlist.pipeline = [ 'SpamDetect', 'Approve', 'Replybot', # inserting MyHandler here. 'MyHandler', 'Moderate', 'Hold', 'MimeDel', 'Scrubber', 'Emergency', 'Tagger', 'CalcRecips', 'AvoidDuplicates', 'Cleanse', 'CleanseDKIM', 'CookHeaders', 'ToDigest', 'ToArchive', 'ToUsenet', 'AfterDelivery', 'Acknowledge', 'WrapMessage', 'ToOutgoing', ]
I also removed the comments and added one. This becomes the input to config_list (t.py in your example).
Note that if I understand the purpose of your handler, it should come (not necessarily immediately) after 'Hold'. You really don't want to archive attachments for messages you might reject or discard.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Le 24/04/2014 01:27, Mark Sapiro a écrit :
Open Mailman/Defaults.py
Copy the definition of GLOBAL_PIPELINE
paste it into a new file changing the name from GLOBAL_PIPELINE to mlist.pipeline and add your handler so it becomes […]
Note that if I understand the purpose of your handler, it should come (not necessarily immediately) after 'Hold'. You really don't want to archive attachments for messages you might reject or discard.
Thanks I will explore that behavior.
First version on github:
https://github.com/Sylvain303/mailman-AttachmentMove
I'll document the installation process and correct bugs still presents…
- Detaching embedded image related part,
- nesting related if already present …
Thank you, so much for your help.
Sylvain.
participants (2)
-
Mark Sapiro -
Sylvain Viart