Feature Request - Interactive HTML Digests
Hi guys,
I'm really excited about the developments ongoing with MM3... :)
One thing I'd really like to see in MM3 with respect to list digests is what I'll call 'Interactive HTML Digests'. Yahoo 'full featured' digests go about halfway to where I'd like to see MM3 go.
This was discussed briefly on the users list in the thread 'Replying to digests' on 12/29...
Here's what I'm imagining:
Each message subject in the summary of messages at the top of the digest are embedded hyperlinks that if clicked, will scroll the digest down to that message (Yahoos do this now),
Each message displayed in the digest body would have the following in a single line at the very bottom *and* top of each message (Yahoos are only at the bottom):
a) separate button/links for: 'Reply to Sender', 'Reply to List' and maybe 'Reply to all' (that last one could be optional?), and b) little 'Back to top' links to scroll back to the top of the message summary
- When replying to a message, preserve the message subject *and* all of the individual message references so that replying to messages from the HTML digests doesn't break threading - basically so it looks as if it was replied to individually and separately, not from the list digest.
Apparently Yahoo's doesn't preserve the message header/references (bad), but it does preserve the subject.
Also - I use Thunderbird, and when I click Yahoos 'Reply to Group', it does 2 things wrong, but I don't know if MM could do anything about this or if it would be a Thunderbird issue:
it uses the email clients 'default account' instead of the account the user is in - so, if you just type something and click 'Send', it is sent from the wrong account and will bounce since the email address for that account is not a list/group member, and
Thunderbird's new 'Quote only selected text' doesn't work - in fact, *nothing* is quoted. This issue is not as big as the first one, since it can be worked around by simply copying the highlighted text (more often than not I select text to quote rather than quoting the whole thing) then 'pasting as quote'. Two more steps - not that big a deal, but hey, if it can be coded to use the Clients features, all the better.
Since I'm not a programmer, all I can do is offer to help test/debug and document this feature should you guys see any value in implementing it.
Thanks for listening... :)
Tanstaafl wrote:
Here's what I'm imagining:
[...]
- When replying to a message, preserve the message subject *and* all of the individual message references so that replying to messages from the HTML digests doesn't break threading - basically so it looks as if it was replied to individually and separately, not from the list digest.
Current Mailman MIME format digests do this if the user's MUA supports it.
Apparently Yahoo's doesn't preserve the message header/references (bad), but it does preserve the subject.
Also - I use Thunderbird, and when I click Yahoos 'Reply to Group', it does 2 things wrong, but I don't know if MM could do anything about this or if it would be a Thunderbird issue:
- it uses the email clients 'default account' instead of the account the user is in - so, if you just type something and click 'Send', it is sent from the wrong account and will bounce since the email address for that account is not a list/group member, and
This is a TBird issue.
- Thunderbird's new 'Quote only selected text' doesn't work - in fact, *nothing* is quoted. This issue is not as big as the first one, since it can be worked around by simply copying the highlighted text (more often than not I select text to quote rather than quoting the whole thing) then 'pasting as quote'. Two more steps - not that big a deal, but hey, if it can be coded to use the Clients features, all the better.
Works for me with Tbird 3.0.1 and Mailman MIME digests and "Reply" or "Reply All". "Reply List" is not offerred for a reply to a message from the MIME digest, because there is no List-Post: header in the individual message parts
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
Tanstaafl wrote:
Here's what I'm imagining:
[...]
- When replying to a message, preserve the message subject *and* all of the individual message references so that replying to messages from the HTML digests doesn't break threading - basically so it looks as if it was replied to individually and separately, not from the list digest.
Current Mailman MIME format digests do this if the user's MUA supports it.
Apparently Yahoo's doesn't preserve the message header/references (bad), but it does preserve the subject.
There's an inherent difficulty in what you propose. I suspect it is the reason why quoting doesn't work with TBird and Yahoo digests.
Your points 1 and 2 imply that the digest must be a single HTML part. Thus, a reply of some sort button is really a mailto: link which in Yahoo's case, probably has a query fragment with "Subject=..." and could additionally have query fragments for "In-Reply-To=..." and "References=...". This mechanism could also support "fixed" quoting of the entire message with a "body=..." fragment, but quoting selected text would require some kind of scripting.
My point is that MUAs aren't web browsers, and all this would depend heavily on on features that are not supported uniformly if at all in MUAs.
My personal feeling is that if things can be done in a way that works reasonably in MUAs that don't support the features, then it might be feasable, but if something works with one MUA and is a disaster in another, it is better not done.
Certainly, this kind of HTML digest would have to be optional.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2010-02-17 11:15 AM, Mark Sapiro wrote:
- When replying to a message, preserve the message subject *and* all of the individual message references so that replying to messages from the HTML digests doesn't break threading - basically so it looks as if it was replied to individually and separately, not from the list digest.
Current Mailman MIME format digests do this if the user's MUA supports it.
Ok, I switched to MIME digest for the users list for a better understanding of what is supposed to be happening.
Currently, the MIME digest is a plain text message, and I don't see any way to reply to an individual message, much less preserve the message subject I'm replying to - unless you mean I'm supposed to actually open the message I want to reply to in a separate window, then click Reply?
If so, that's not what I'm talking about... the title of this feature request is 'Interactive HTML Digests'... meaning, I'm asking for a 3rd choice for the option:
'When receiving digests, which format is default?'
[ ] Plain [ ] MIME [ ] Interactive HTML
Where the HTML digest supports (as many as possible of) the features described in my request...
Note: in general, I don't like HTML email, but for digests, if what I'm describing can be even partly accomplished, I think it would be a 'good use' of HTML email...
Also - I use Thunderbird, and when I click Yahoos 'Reply to Group', it does 2 things wrong, but I don't know if MM could do anything about this or if it would be a Thunderbird issue:
There's an inherent difficulty in what you propose. I suspect it is the reason why quoting doesn't work with TBird and Yahoo digests.
Your points 1 and 2 imply that the digest must be a single HTML part.
Correct...
Thus, a reply of some sort button is really a mailto: link which in Yahoo's case, probably has a query fragment with "Subject=..." and could additionally have query fragments for "In-Reply-To=..." and "References=...". This mechanism could also support "fixed" quoting of the entire message with a "body=..." fragment, but quoting selected text would require some kind of scripting.
My point is that MUAs aren't web browsers, and all this would depend heavily on on features that are not supported uniformly if at all in MUAs.
I'd be happy to attach one of Yahoos digests if you or anyone else was inclined to take a peek at just what it does...
Most MUAs can render HTML email messages fine. Of course, I understand they aren't 'browsers', so whatever code that was used would have to be an extremely limited subset that should work in most email clients capable of rendering HTML emails.
- it uses the email clients 'default account' instead of the account the user is in - so, if you just type something and click 'Send', it is sent from the wrong account and will bounce since the email address for that account is not a list/group member, and
This is a TBird issue.
I know, but I guess I should have said '... if it would be *strictly* a TB issue ...'...
The bigger question is, is there anything sane that MM3 could do to work around such limitations using HTML code in most/all mail clients if an interactive HTML version of the digests was implemented?
- Thunderbird's new 'Quote only selected text' doesn't work - in fact, *nothing* is quoted. This issue is not as big as the first one, since it can be worked around by simply copying the highlighted text (more often than not I select text to quote rather than quoting the whole thing) then 'pasting as quote'. Two more steps - not that big a deal, but hey, if it can be coded to use the Clients features, all the better.
Works for me with Tbird 3.0.1 and Mailman MIME digests and "Reply" or "Reply All". "Reply List" is not offerred for a reply to a message from the MIME digest, because there is no List-Post: header in the individual message parts
I'm curious why? If I had received the message individually, there would have been, so why not include them in messages in the digest?
My personal feeling is that if things can be done in a way that works reasonably in MUAs that don't support the features, then it might be feasable, but if something works with one MUA and is a disaster in another, it is better not done.
I hope you aren't suggesting that Mailman should limit its features to the 'least common denominator'... I don't see anything wrong with *optionally* supporting advanced features of modern mail clients.
Certainly, this kind of HTML digest would have to be optional.
But of course... and absolutely not the default.
--
Charles
Tanstaafl writes:
Most MUAs can render HTML email messages fine.
That depends on your definitions of "HTML", "email messages", and "fine".
All of them are pretty ambiguous, and different *senders* view them differently. That makes it very difficult for an application like Mailman which often needs to manipulate the message to handle it. For example, at one time Outlook sent "HTML" mail that had tags encoded in ASCII and content (including the occasional ALT attribute!) in little- endian Unicode. What in the world are you supposed to do with that?
True, that day is long gone, but it's still true that you really don't know what is going to come down the pipe.
they aren't 'browsers', so whatever code that was used would have to be an extremely limited subset that should work in most email clients capable of rendering HTML emails.
The problem is that you can't restrict the original senders to that extremely limited subset, and it's not Mailman's place to try to edit at the HTML level, although manipulating MIME is something that it is competent to do (and does).
Works for me with Tbird 3.0.1 and Mailman MIME digests and "Reply" or "Reply All". "Reply List" is not offerred for a reply to a message from the MIME digest, because there is no List-Post: header in the individual message parts
I'm curious why? If I had received the message individually, there would have been, so why not include them in messages in the digest?
That seems like a reasonable idea. The only problem I can see is if there was a prexisting List-Post. For example, many announce lists are subscribed to by a related discussion list. In that case, the right thing to do is to remove the announce list's List-Post and substitute the discussion list's. With an umbrella list (a list whose only members are other lists) it would be the other way around: you want the umbrella list's List-Post to remain.
I suspect this probably could be handled by the existing configuration options in almost all cases. The announce list usually doesn't need a List-Post because the group of allowed posters is typically extremely limited, and all know the address. The sublists of an umbrella list similarly have only one allowed poster -- the umbrella list -- so they don't need the List-Post field (or any RFC 2369 headers for that matter).
I hope you aren't suggesting that Mailman should limit its features to the 'least common denominator'... I don't see anything wrong with *optionally* supporting advanced features of modern mail clients.
I don't think you understand how primitive these features in fact are. Even something as simple as inserting the footer into the HTML works for some clients but not for others.
On 2010-02-20 1:23 AM, Stephen J. Turnbull wrote:
Tanstaafl writes:
Most MUAs can render HTML email messages fine.
That depends on your definitions of "HTML", "email messages", and "fine".
All of them are pretty ambiguous, and different *senders* view them differently.
<snip>
I think you misunderstand. I'm not talking about normal list traffic.
I'm talking about creating a special HTML formatted List Digest message that MM3 would generate *itself*.
Unless maybe you're talking about how to handle HTML list traffic in such an HTML Digest message.
Personally, I usually set all of my lists to plain text anyway, so this wouldn't be an issue for any lists I host/maintain, but yes, I guess I can see how that might introduce another level of complexity to this request, but maybe such messages could be encapsulated somehow, since the features I'm talking about all have to do with interacting with the *headers* of each message, not the body/content.
So, again, the main question is if it could use just enough HTML formatting to make the features I outlined work for the majority of modern email clients, while still maintaining readability for those that may not be able to make use of the interactive features. If it can, then I think this would be an awesome option for those of us who subscribe to a whole bunch of email lists. I subscribe to most as individual messages only because interacting with the list via a Digest message is very frustrating and requires lots of copying and pasting, and still breaks threading etc because of the missing headers.
"Reply List" is not offerred for a reply to a message from the MIME digest, because there is no List-Post: header in the individual message parts
I'm curious why? If I had received the message individually, there would have been, so why not include them in messages in the digest?
That seems like a reasonable idea. The only problem I can see is if there was a prexisting List-Post. For example, many announce lists are subscribed to by a related discussion list. In that case, the right thing to do is to remove the announce list's List-Post and substitute the discussion list's. With an umbrella list (a list whose only members are other lists) it would be the other way around: you want the umbrella list's List-Post to remain.
I suspect this probably could be handled by the existing configuration options in almost all cases.
Cool... so, do I need to send a new email to make this Feature Request official? ;)
--
Charles
Tanstaafl writes:
I think you misunderstand. I'm not talking about normal list traffic.
Not as far as I can tell, and I'm talking about digests, too.
I'm talking about creating a special HTML formatted List Digest message that MM3 would generate *itself*.
Exactly. "Special HTML" and "lowest common denominator HTML" don't mix. (Yes, that's mere word play, but in this case the parallel happens to work.)
So, again, the main question is if it could use just enough HTML formatting to make the features I outlined work for the majority of modern email clients,
My users expect to be able to view the messages in a digest as an email folder. That's the most important digest feature for them; they do not want to have to page through messages they don't care about to get to the ones they do care about, and they expect to use their normal MUA commands, not HTML fragment addresses in links, to navigate.
Maybe yours don't, and if they don't, it could work.
I subscribe to most as individual messages only because interacting with the list via a Digest message is very frustrating and requires lots of copying and pasting, and still breaks threading etc because of the missing headers.
Really? I haven't had a problem like that for 15 years. (But then, I've exclusively used Emacs-based MUAs for 25 years, which might have something to do with it.)
Cool... so, do I need to send a new email to make this Feature Request official? ;)
I think you should post a bug report (weirdly enough) to the Mailman project on Launchpad.net. There's also a wiki page for Mailman 3 (I forget the address exactly but it's linked from http://www.list.org/) that you could update. You don't have to do it yourself, but if that doesn't get done by somebody, experience shows that the request tends to get lost.
On 2/20/2010 10:53 AM, Stephen J. Turnbull wrote:
Tanstaafl writes: Exactly. "Special HTML" and "lowest common denominator HTML" don't mix. (Yes, that's mere word play, but in this case the parallel happens to work.)
Well... I only meant special in the sense of the use case... not the HTML code that would be needed.
My users expect to be able to view the messages in a digest as an email folder. That's the most important digest feature for them;
I wouldn't have had a clue what you're talking about here, but maybe you're talking about the Vi-in-Emacs feature Barry mentioned?
If so, that's fine - I'm not suggesting that the current digest offerings be changed in any way, shape or form, so implementation of this feature request wouldn't affect your users in the slightest.
they do not want to have to page through messages they don't care about to get to the ones they do care about,
Nor do I, but I don't use Vi-in-Emacs, so my feature request is to allow a way for people who don't use it to be able to use digests but not have to page through messages they don't care about to get to the ones they do, and easily interact with them without breaking threading for everyone else.
Oh - and if you aren't talking about Vi-in-Emacs, no offense was intended, and I'd really like to know what you meant by 'view the messages in a digest in an email folder'... :)
and they expect to use their normal MUA commands, not HTML fragment addresses in links, to navigate.
See above. No one would force anyone to choose this new digest version.
Really? I haven't had a problem like that for 15 years. (But then, I've exclusively used Emacs-based MUAs for 25 years, which might have something to do with it.)
Ya think? ;)
I think you should post a bug report (weirdly enough) to the Mailman project on Launchpad.net. There's also a wiki page for Mailman 3 (I forget the address exactly but it's linked from http://www.list.org/) that you could update.
Thanks, I'll do one of those tomorrow (gotta run now)...
You don't have to do it yourself, but if that doesn't get done by somebody, experience shows that the request tends to get lost.
I know - I've really been making a nuisance of myself on the Mozilla Bugzilla for that very reason...
--
Best regards,
Charles
On Feb 20, 2010, at 01:52 PM, Tanstaafl wrote:
I wouldn't have had a clue what you're talking about here, but maybe you're talking about the Vi-in-Emacs feature Barry mentioned?
VM (View Mail) for Emacs:
http://www.nongnu.org/viewmail/
those-were-the-days-ly y'rs, -Barry
Tanstaafl writes:
On 2/20/2010 10:53 AM, Stephen J. Turnbull wrote:
My users expect to be able to view the messages in a digest as an email folder. That's the most important digest feature for them;
I wouldn't have had a clue what you're talking about here, but maybe you're talking about the Vi-in-Emacs feature Barry mentioned?
VM. Yes. Also Gnus does this the same way. I believe pretty much all of the major Emacs MUAs do it this way.
Online manual (older version): http://www.wonderworks.com/vm/user-manual/ Current development: http://www.nongnu.org/viewmail/
they do not want to have to page through messages they don't care about to get to the ones they do care about,
Nor do I, but I don't use Vi-in-Emacs, so my feature request is to allow a way for people who don't use it to be able to use digests but not have to page through messages they don't care about to get to the ones they do, and easily interact with them without breaking threading for everyone else.
How do you propose to get that effect though? HTML is not designed to make it easy!
I'd really like to know what you meant by 'view the messages in a digest in an email folder'... :)
Here's my MUA reading a digest in my main mail folder INBOX:
http://turnbull.sk.tsukuba.ac.jp/outgoing/INBOX.png
and here's my MUA reading the messages in the digest:
http://turnbull.sk.tsukuba.ac.jp/outgoing/AUCTeX.png
If the players didn't have numbers on their jerseys, I bet it would take you a minute to figure out Who's on First.
See above. No one would force anyone to choose this new digest version.
That's not the point. The point is that in my experience these are minimum requirements for a digest view, and I don't see how you plan to implement that in portable HTML unless you make *really* draconian restrictions on what formats people are allowed to post in.
I think you should post a bug report (weirdly enough) to the Mailman project on Launchpad.net.
Thanks, I'll do one of those tomorrow (gotta run now)...
It looks like somebody has borrowed Guido's time machine and the feature (ie, List-Post in each message in digest) is already implemented. But it's not default yet, so you could ask for that. ;-)
On 2010-02-21 3:00 AM, Stephen J. Turnbull wrote:
Tanstaafl writes:
On 2/20/2010 10:53 AM, Stephen J. Turnbull wrote: Here's my MUA reading a digest in my main mail folder INBOX:
http://turnbull.sk.tsukuba.ac.jp/outgoing/INBOX.png
and here's my MUA reading the messages in the digest:
Yikes... I wouldn't be able to use Emacs then... that is horrible looking (to me - no offense intended)... ;)
See above. No one would force anyone to choose this new digest version.
That's not the point. The point is that in my experience these are minimum requirements for a digest view,
Sorry, maybe I'm dense but I don't know what you mean by 'these' when you said 'these are minimum requirements'... and reading the previous messages wasn't helpful either...
and I don't see how you plan to implement that in portable HTML unless you make *really* draconian restrictions on what formats people are allowed to post in.
I'm guessing that comment is intended more for the 'Reply' features. Apparently some of the features I'm asking for should be [relatively] easily doable (and Barry confirmed this) - like capturing the subject, and adding the anchors for scrolling down to a message from the message summary at the top, then back to the top.
Being that I'm not a programmer, maybe I just don't know enough about what can and can't be done, but my thinking was that the headers of each message could somehow be 'encapsulated' and hidden (ie, not 'rendered') in the generated 'Interactive HTML Digest' in such a way that would allow the more complex 'special features' in this request to work.
This would mean that there would be *no* restrictions, much less draconian ones, on what formats people can post in. The only issue would then be what clients can make use of the HTML code that makes the magic happen.
It looks like somebody has borrowed Guido's time machine and the feature (ie, List-Post in each message in digest) is already implemented. But it's not default yet, so you could ask for that. ;-)
Hmmmm... is there an option somewhere to enable it? I don't see anything in the Digest Options for any of the lists I manage...
--
Charles
Tanstaafl writes:
That's not the point. The point is that in my experience these are minimum requirements for a digest view,
Sorry, maybe I'm dense but I don't know what you mean by 'these' when you said 'these are minimum requirements'... and reading the previous messages wasn't helpful either...
Navigation, navigation, and navigation.
and I don't see how you plan to implement that in portable HTML unless you make *really* draconian restrictions on what formats people are allowed to post in.
I'm guessing that comment is intended more for the 'Reply' features. Apparently some of the features I'm asking for should be [relatively] easily doable (and Barry confirmed this) - like capturing the subject, and adding the anchors for scrolling down to a message from the message summary at the top, then back to the top.
Those parts are basically trivial, yes. The problem is messages that are originally HTML mail, and perhaps attachments.
The thing is that the users I'm used to don't want to change interfaces from mail reader toolbar to HTML links embedded in the message, and they wouldn't be happy with jumping back and forth between the summary/TOC and the messages; they want to see the TOC and the current message at the same time.
Being that I'm not a programmer, maybe I just don't know enough about what can and can't be done, but my thinking was that the headers of each message could somehow be 'encapsulated' and hidden (ie, not 'rendered') in the generated 'Interactive HTML Digest' in such a way that would allow the more complex 'special features' in this request to work.
Headers are not a problem; Mailman already does filter out many "uninteresting" headers in the plain text digest. The problem is if the message itself is structured, such as in HTML, or containing attachments. "Simple" HTML simply doesn't lend itself to "encapsulating" structured documents, except with devices like frames.
It looks like somebody has borrowed Guido's time machine and the feature (ie, List-Post in each message in digest) is already implemented. But it's not default yet, so you could ask for that. ;-)
Hmmmm... is there an option somewhere to enable it? I don't see anything in the Digest Options for any of the lists I manage...
No. Mark explained how to get that, you need to access the command line interface and change the list config to add the List-Post header to the "keep" list. I think that rather than have an option it might as well be the default to keep it.
On Feb 22, 2010, at 11:13 PM, Stephen J. Turnbull wrote:
No. Mark explained how to get that, you need to access the command line interface and change the list config to add the List-Post header to the "keep" list. I think that rather than have an option it might as well be the default to keep it.
I think this makes sense too. Please submit a bug report.
-Barry
On 2010-02-22 9:13 AM, Stephen J. Turnbull wrote:
Tanstaafl writes:
That's not the point. The point is that in my experience these are minimum requirements for a digest view,
Sorry, maybe I'm dense but I don't know what you mean by 'these' when you said 'these are minimum requirements'... and reading the previous messages wasn't helpful either...
Navigation, navigation, and navigation.
Ok... but again (how many times have I said this now??), the current MIME and plain text digest versions will continue to work as they currently do. Someone would have to explicitly choose this new digest, and only someone who had a compatible mail client would/should be doing so - and would find out quickly enough that their mail client isn't compatible if that was the case.
So, you can still navigate your digests the way you always have. What this feature request would do is enable those of us who use non EMACS based GUI mail clients to have an effective way to interact with digest messages too.
Those parts are basically trivial, yes. The problem is messages that are originally HTML mail, and perhaps attachments.
Ok, I can see that attachments is one thing I hadn't considered... I don't know how those could be handled...
Mark had said that this request would require that the digest be one big inline HTML message - but maybe this could be handled by the 'encapsulation' process. Again, I'm asking questions, and please remember these questions are coming from a non-programmer, so if they are self-evidently stupid from a programmers point of view - well, that's why.
The thing is that the users I'm used to don't want to change interfaces ...
<sigh> See above comment re: navigation...
Being that I'm not a programmer, maybe I just don't know enough about what can and can't be done, but my thinking was that the headers of each message could somehow be 'encapsulated' and hidden (ie, not 'rendered') in the generated 'Interactive HTML Digest' in such a way that would allow the more complex 'special features' in this request to work.
Headers are not a problem; Mailman already does filter out many "uninteresting" headers in the plain text digest. The problem is if the message itself is structured, such as in HTML, or containing attachments.
Yes, but the headers don't have are a separate MIME part that don't have any HTML formatting, right? So, any back-end code that worked its magic on the headers (which is where all of the information is that would allow Replies to not break threading is contained) would work on HTML messages too, right? Or, if not, what am I missing?
"Simple" HTML simply doesn't lend itself to "encapsulating" structured documents, except with devices like frames.
All I mean by 'simple' is simple enough that most mail clients capable of rendering HTML email won't have a problem with it, and the fact is, most can render fairly complex HTML.
It looks like somebody has borrowed Guido's time machine and the feature (ie, List-Post in each message in digest) is already implemented. But it's not default yet, so you could ask for that. ;-)
Hmmmm... is there an option somewhere to enable it? I don't see anything in the Digest Options for any of the lists I manage...
No. Mark explained how to get that, you need to access the command line interface and change the list config to add the List-Post header to the "keep" list. I think that rather than have an option it might as well be the default to keep it.
Agreed...
--
Charles
Tanstaafl wrote:
On 2010-02-22 9:13 AM, Stephen J. Turnbull wrote:
Tanstaafl writes:
Those parts are basically trivial, yes. The problem is messages that are originally HTML mail, and perhaps attachments.
Ok, I can see that attachments is one thing I hadn't considered... I don't know how those could be handled...
Mark had said that this request would require that the digest be one big inline HTML message - but maybe this could be handled by the 'encapsulation' process. Again, I'm asking questions, and please remember these questions are coming from a non-programmer, so if they are self-evidently stupid from a programmers point of view - well, that's why.
What does encapsulation mean to you?
Consider a digest containing what was originally an HTML message. Even that is difficult. You can't just insert that message's HTML body into the digest. What if that HTML were such that parts of it were rendered at the 'top' ahead of even the table of contents. Unless the process creating the digest has a complete HTML rendering engine (compatible with the one in your MUA), it can't know how the entire digest will render if it just inserts arbitrary HTML from a message into the middle of a digest.
Thus, you are reduced to something like HTMLizing the current plain text digest with links to all it's scrubbed attachments and HTML parts. This would be acceptable for lists that accept plain text only, but I'm not sure it would work well for other lists.
Yes, but the headers don't have are a separate MIME part that don't have any HTML formatting, right? So, any back-end code that worked its magic on the headers (which is where all of the information is that would allow Replies to not break threading is contained) would work on HTML messages too, right? Or, if not, what am I missing?
As I have mentioned in other posts in this thread, the 'reply*' buttons are doable with mailto: links provided you don't want any special features like quoting selected text. The mailto: could include a body= fragment that was a quoting of the entire (plain text) message being replied to, but if you want to quote selected text, you'd have to select the text and then use the MUA's 'reply-list' button to generate the reply, but then you don't have the proper subject or the in-reply-to and references for threading. Anything else requires javascript behind the button.
Hmmmm... is there an option somewhere to enable it? I don't see anything in the Digest Options for any of the lists I manage...
No. Mark explained how to get that, you need to access the command line interface and change the list config to add the List-Post header to the "keep" list. I think that rather than have an option it might as well be the default to keep it.
Agreed...
Defaulting to keep the List-Post header in the MIME digest messages is easy enough. I can do that if no one comes up with a good reason why not. Barry mentioned List-Post for other lists/umbrellas, but CookHeaders already removes those.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2010-02-22 12:15 PM, Mark Sapiro wrote:
What does encapsulation mean to you?
In the context of this discussion, a way of 'wrapping up' the individual messages in such a way as to be able to be both manipulated by the html code, and to control how they are displayed/presented.
Thus, you are reduced to something like HTMLizing the current plain text digest with links to all it's scrubbed attachments and HTML parts. This would be acceptable for lists that accept plain text only, but I'm not sure it would work well for other lists.
Honestly, I'd be fine with this new HTML digest being limited to only plain-text based lists, at least at least and until some python/html guru came up with the code to make it work reasonably well for HTML based lists too.
As I have mentioned in other posts in this thread, the 'reply*' buttons are doable with mailto: links provided you don't want any special features like quoting selected text.
The mailto: could include a body= fragment that was a quoting of the entire (plain text) message being replied to, but if you want to quote selected text, you'd have to select the text and then use the MUA's 'reply-list' button to generate the reply, but then you don't have the proper subject or the in-reply-to and references for threading. Anything else requires javascript behind the button.
Wow - I must have missed the significance of it when you said it... if you are actually saying that grabbing the in-reply-to references would be doable, then this would be more than enough to make me happy. :)
Grabbing the subject would be an even bigger bonus. ;)
But no, quoting selected text is a very distant second to the Reply buttons, and something I'd be ok with never being implemented...
Defaulting to keep the List-Post header in the MIME digest messages is easy enough. I can do that if no one comes up with a good reason why not. Barry mentioned List-Post for other lists/umbrellas, but CookHeaders already removes those.
Thanks a lot Mark, Stephen and Barry for taking the time to respond to what must be - to you - a very ignorance-based discussion from my side.
--
Charles
Tanstaafl wrote:
Wow - I must have missed the significance of it when you said it... if you are actually saying that grabbing the in-reply-to references would be doable, then this would be more than enough to make me happy. :)
Grabbing the subject would be an even bigger bonus. ;)
Go to <http://mail.python.org/pipermail/mailman-developers/2010-February/021002.html> and look at the mailto: URL that is linked from "tanstaafl at libertytrek.org ".
What you appear to be asking for is an emailed digest that looks a lot like a pipermail archive index followed by the actual messages with reply links as 'reply list' buttons instead of being hidden behind the poster's email address. (Granted, you're asking for more buttons, but ...)
Maybe you could just use the existing list archives ;)
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Tanstaafl writes:
Ok... but again (how many times have I said this now??), MIME and plain text digest versions will continue to work as they currently do.
And how many times do I have to reply that that is not the point, I am discussing the design of the new feature? I'm drawing an analogy: The fact that users of these other implementations *like* their features suggests that maybe users of the new one would too. If people aren't going to use the new feature, it's a waste of time and energy to implement it.
I personally would *not* find the HTML digest preferable to dealing with multiple messages in the design you're implicitly talking about, and I don't think I know anybody who would; they would likely prefer to handle it with threading or Google-style "conversations". If you know otherwise about the users you are representing, please say so.
"Simple" HTML simply doesn't lend itself to "encapsulating" structured documents, except with devices like frames.
All I mean by 'simple' is simple enough that most mail clients capable of rendering HTML email won't have a problem with it, and the fact is, most can render fairly complex HTML.
My turn to *sigh*. I *know* what you mean. I am telling you that in my opinion it will require quite a bit of effort to write a program (or Mailman function) to create HTML that does the job for a wide enough selection of MUAs. So far, I think Mark agrees.
I'm not going to work on it myself, because I don't think the benefit (to others)/cost (to me) ratio is anywhere near high enough. That's mostly because I see supporting this feature as an unending time sink for the next 5 years, because HTML is just not intended to be used for this purpose.
On 2010-02-23 4:52 AM, Stephen J. Turnbull wrote:
Tanstaafl writes:
Stephen J. Turnbull wrote:
"Simple" HTML simply doesn't lend itself to "encapsulating" structured documents, except with devices like frames.
All I mean by 'simple' is simple enough that most mail clients capable of rendering HTML email won't have a problem with it, and the fact is, most can render fairly complex HTML.
My turn to *sigh*. I *know* what you mean. I am telling you that in my opinion it will require quite a bit of effort to write a program (or Mailman function) to create HTML that does the job for a wide enough selection of MUAs.
You also seem to be missing the fact that I've already said I didn't expect (hoped? maybe) that *all* of these features would be implementable, and certainly not immediately.
I didn't actually come out and say it, but what I was asking for was just the framework...
So how about this...
- Create the new Digest format, but only add the very-most-basic features that can easily be added... like, for example, make the messages in the message summary hyperlinks that scroll down to the message, and add 'back to top' links at the top/bottom of each message.
If some basic Reply mailto links could be added that maybe simply grabbed the subject, that would be a bonus, but not necessary.
Maybe it would also be possible to mimic the google 'threaded' digest feature, where each thread is grouped in the digest (based on the date of the first post), *but* only the first post of the thread has an entry in the message summary at the top, with a [##] in parenthesis denoting how many messages exist for that thread. This keeps the message summary much shorter and more manageable, especially for high volume lists.
Also, maybe peeking at the message source for one of the Yahoo and Google Digests could make this easier...
- Make it template based, so as to be easily modifiable by the MM admin.
This way, if some HTML magician comes along and likes the concept, they could not only easily do so, but could also easily contribute their changes to the community once they have been confirmed to work in most MUAs that render HTML emails.
Obviously, there would also have to be a back-end function that manipulates the individual messages that the MM admin can also play with, but I don't see this as a real problem as long as that function only affects the hmtl digest, and doesn't mess with any other MM functions or does nothing more than read in the individual messages for manipulation for the digest.
I'm not going to work on it myself, because I don't think the benefit (to others)/cost (to me) ratio is anywhere near high enough. That's mostly because I see supporting this feature as an unending time sink for the next 5 years, because HTML is just not intended to be used for this purpose.
I'd argue that HTML support in mail clients will only get better over time, making adding new features easier and more manageable...
So far, I think Mark agrees.
Well, I wouldn't presume to speak for Mark, but I the more complex features, like the Reply links and not breaking threading
I'm not going to work on it myself, because I don't think the benefit (to others)/cost (to me) ratio is anywhere near high enough. That's mostly because I see supporting this feature as an unending time sink for the next 5 years, because HTML is just not intended to be used for this purpose.
Well, no offense, but by that argument, we'd all still be in flintstones vehicles, because the concept of 'the wheel' was never intended to be used in an internal combustion engine - until it was.
;)
--
Charles
Tanstaafl writes:
Also, maybe peeking at the message source for one of the Yahoo and Google Digests could make this easier...
All 250KB of Javascript?<wink/>
There is *serious* magic in there; this is *wa-a-ay* beyond the remit of Mailman. Mailman is *not* a full-featured MUA, and it never will be.
Also, I somehow suspect that the libraries that implement all the interesting stuff are *not* free software.
- Make it template based, so as to be easily modifiable by the MM admin.
Impossible. The hard work is not in outputting the page, it's in parsing the incoming messages and deciding how to insert them into the relatively inflexible HTML container. The MM admin can probably make it prettier, but I doubt that it will be possible to do this in such a way that they can change the basic functionality.
On 2010-02-23 9:27 AM, Stephen J. Turnbull wrote:
Tanstaafl writes:
Also, maybe peeking at the message source for one of the Yahoo and Google Digests could make this easier...
All 250KB of Javascript?<wink/>
There is *serious* magic in there; this is *wa-a-ay* beyond the remit of Mailman. Mailman is *not* a full-featured MUA, and it never will be.
Also, I somehow suspect that the libraries that implement all the interesting stuff are *not* free software.
Ok, granted (I closed the source window pretty quickly... it was giving me a headache)... ;)
The point I'm getting at is, I'd like to see the basic framework for a new HTML digest added to MM. Barry said he is fine with it as long as it is done right, and Mark seems to concur. Well, you have to start somewhere, so why not just start with the really simple/basic #anchor tags being the first 'feature', and just see what happens. Mark seems to believe that adding the Reply-To-List/Sender' mailto: links that can also grab the in-reply-to references will be doable (no idea how hard though), so maybe those could be version 1.1 of the new digest... and maybe those are the only features that will ever be added - or maybe some python/html guru will see the potential and step up and figure out how to make some of the more magical features happen for HTML based lists too...
- Make it template based, so as to be easily modifiable by the MM admin.
Impossible.
Ahem... 'Nothing is impossible' my Dad would say. ;)
But, sorry, I misspoke... I didn't really mean the average MM admin... I meant a developer with adequate knowledge of the tools/code.
The hard work is not in outputting the page, it's in parsing the incoming messages and deciding how to insert them into
That would be the MM 'function' I mentioned...
the relatively inflexible HTML container.
And this would be the HTML 'template' I mentioned...
But this begs the question...
How does MM generate the two digests it supports now? Does it store the individual messages in some temporary location until it is time to generate the digest, then do whatever it does to generate it? Or does it process each message as it comes in, and cumulatively add them until the trigger for sending the digest is pulled?
The reason I ask this is, a way would have to be found to provide anyone who chose to work on this a 'test bed' of messages to be used to generate the digests to test any of their modifications, otherwise testing modifications would be a royal pain...
The MM admin can probably make it prettier, but I doubt that it will be possible to do this in such a way that they can change the basic functionality.
No offense, but I don't see that at all... MM is open source, and every single function of it can be modified to change the basic functioality - all that is necessary is someone competent with the appropriate tools - in this case, python (, C?) and HTML - and the code.
--
Charles
Tanstaafl writes:
The point I'm getting at is, I'd like to see the basic framework for a new HTML digest added to MM. Barry said he is fine with it as long as it is done right, and Mark seems to concur. Well, you have to start somewhere, so why not just start with the really simple/basic #anchor tags being the first 'feature',
No reason not to, but even doing that right is non-trivial. The problem that you are not acknowledging is that generating HTML is *not* the problem (although turning it into a user-modifiable template might not be easy, especially if you want to have reasonable reply-to features). It's what happens before you even *think* about generating the digest itself, in terms of filtering out parts that can't be put into HTML, etc.
It would be possible to do what you suggest as follows:
Reject (return to sender) all posts with a top-level MIME type other than text/plain. No HTML, no attachments. It probably would be wise to reject anything that lacks a Content-Type header as well. (Now you know how to spell "draconian".) Put the rest into a file in the usual "mbox" format.
Output the HTML preamble.
Split the mbox into messages. (Mailman already needs to do this.)
For each message:
Output links to next and previous messages, first message in this thread, and digest TOC (table of contents).
Split the message into a list of header fields (From, To, Subject, etc, as well as the more esoteric stuff you normally don't see), plus the body text as a blob. (Mailman already needs to do this.)
Output a table of field name (eg, From) and content. Wrap fields with address content with mailto links for each person. The big question here is how to arrange for quoting of the body. The obvious and safe kludge would result in copying the body into the mailto link once for every reply link, which would multiply the size of the digest by 2-10 times, depending on how many of those there were. Wrap the Subject field content in an internal link to the beginning of the thread in the digest TOC (table of contents). (Straightforward.) Javascript could be used to make the burden of the mailto links reasonable, but I don't like the idea of including Javascript in mail messages much.
Output the body wrapped in an HTML PRE element.
After the last message, output another navigation section, then any postamble stuff (footer text, etc.)
But this is just a barebones start. It would really be bad if there were more than about 20 messages in the digest, or if any of them are more than a couple of windows tall. And it might need to be completely refactored to add reasonable reply functionality.
and just see what happens.
Not a chance of that unless you can find a volunteer to field the bug reports. There will be documentation to write, configuration to add to the config file and the web interface, etc, etc.
Mark seems to believe that adding the Reply-To-List/Sender' mailto: links that can also grab the in-reply-to references will be doable
Links based on header content are easy; the headers are well-defined, and their contents have a reasonably well-understood syntax. Quoting the body text is very likely a major mess, even if the message is text/plain. Even formatting the body text will be non-trivial in the presence of messages with long lines (ie, > 74 columns).
- Make it template based, so as to be easily modifiable by the MM admin.
Impossible.
Ahem... 'Nothing is impossible' my Dad would say. ;)
'There ain't no such thing as a free lunch,' my economics textbooks say. But then, you know that.
How does MM generate the two digests it supports now? Does it store the individual messages in some temporary location until it is time to generate the digest, then do whatever it does to generate it?
Yes.
Or does it process each message as it comes in, and cumulatively add them until the trigger for sending the digest is pulled?
No.
The reason I ask this is, a way would have to be found to provide anyone who chose to work on this a 'test bed' of messages to be used to generate the digests to test any of their modifications, otherwise testing modifications would be a royal pain...
This is not a problem.
The MM admin can probably make it prettier, but I doubt that it will be possible to do this in such a way that they can change the basic functionality.
No offense, but I don't see that at all...
That's because once again you've conveniently lost all the context. We're talking about the HTML template which is available to the MM admin, and not the
python (, C?) and HTML - and the code.
Of course somebody who can hack code can make it do anything. But can *you*? You're the person I'm thinking of when I write "impossible" referring to the MM admin changing the basic function. Of course, you *could* learn to hack code ("nothing is impossible"), but so far I detect no such appetite in you. OTOH, the people who can hack the code very likely don't need this feature at all.
Of course, if you have say USD10,000 to offer, that would change matters entirely I bet. :-)
On 2010-02-24 1:41 AM, Stephen J. Turnbull wrote:
The problem that you are not acknowledging is that generating HTML is *not* the problem (although turning it into a user-modifiable template might not be easy, especially if you want to have reasonable reply-to features). It's what happens before you even *think* about generating the digest itself, in terms of filtering out parts that can't be put into HTML, etc.
It's not that I'm not acknowledging it, it's more like I just don't understand even remotely what is entailed... I picture just a function, and a template - obviously overly simplistic
Ok, Mark had posted once that my request implied that this new digest would need to be one big html message, instead of having a bunch of attached messages...
A really dumb question - is there no way to (reliably, or even at all?) 'interact' with just the headers of messages that are attached? Ie, consider an HTML digest, with the individual messages as attachments, with mailto: hyperlinks in the digest *body* that interact with the headers in the *attached* messages... it seems it would eliminate the issue of having to break down and deal with the individual parts of the messages in the digest... but I'm probably just displaying more ignorance for you all to laugh about (no worries, my skin is pretty thick, and I know I'm totally ignorant about a lot of this stuff).
Obviously, for this to work, attachments would have to be displayed in-line, but I already do this, and so does everyone I know, and every MUA I've looked at can view attachments inline, so this shouldn't be an issue...
The big question here is how to arrange for quoting of the body.
Actually, since already said that it obviously heavily complicates the process, I'd just as soon live without any quoting at all for this kind of digest. But maybe it would be possible to just generate the attribution line and put it at the top of the body of the reply. That way if quoting needs to be done, the user need only select/copy the text to quote before clicking the appropriate Reply mailto link, then 'paste as quotation' under the attribution...
Links based on header content are easy; the headers are well-defined, and their contents have a reasonably well-understood syntax.
Thanks for confirming that. Now, if the answer to my first question above (about being able to reference the headers of *attached* messages), maybe this will be a little more doable, given no need to quote anything when these Reply-to mailto: links are used.
- Make it template based, so as to be easily modifiable by the MM admin.
Impossible.
Ahem... 'Nothing is impossible' my Dad would say. ;)
'There ain't no such thing as a free lunch,' my economics textbooks say. But then, you know that.
Yes - but my point was there is a huge difference between 'impossible' and 'will be a lot of work from someone'.
Of course somebody who can hack code can make it do anything. But can *you*? You're the person I'm thinking of when I write "impossible" referring to the MM admin changing the basic function.
Why would you do that? I've already explained that I'm not a programmer. This is a Feature Request, not a proposal for work that I am wanting to do myself. Everything I said in terms of hacking the code would obviously be in context of someone not only capable of doing so, but being *willing* to do so... and maybe it will be no one - it looks like I'm about the only one who really likes the idea (Mark and Barry didn't hate/reject it, but neither sound interested enough in working on it)...
Of course, you *could* learn to hack code ("nothing is impossible"), but so far I detect no such appetite in you.
Its not a matter of appetite, it's a simple matter of time. It would take me years to get to the point I'd even be able to start thinking about taking on something like this...
OTOH, the people who can hack the code very likely don't need this feature at all.
Why? Because they all use EMACS? Or because their coding skills are so powerful that they can directly manipulate the digest stream with their visual cortex?
Sorry, but the condescension in your comments is getting a little old.
Of course, if you have say USD10,000 to offer, that would change matters entirely I bet. :-)
I wish I was independently wealthy, because there are a lot of things on my personal wish list for a lot of the free software I use that I would love to be able to pay to get implemented... sadly, I am not.
Anyway, again, thanks for helping to flesh this out...
--
Charles
Tanstaafl wrote:
A really dumb question - is there no way to (reliably, or even at all?) 'interact' with just the headers of messages that are attached? Ie, consider an HTML digest, with the individual messages as attachments, with mailto: hyperlinks in the digest *body* that interact with the headers in the *attached* messages... it seems it would eliminate the issue of having to break down and deal with the individual parts of the messages in the digest... but I'm probably just displaying more ignorance for you all to laugh about (no worries, my skin is pretty thick, and I know I'm totally ignorant about a lot of this stuff).
I think you were doing better when you first proposed what you wanted, i.e. a digest like Yahoo Groups "fully featured" digest with the addition of the 'reply' and 'top' links at the top as well as at the bottom of the message and 'threading' for the reply links.
I even know how to do that, including in MM 3 at least, the link back to the message and/or thread in an archive.
The part I don't know how to do well is the dealing with HTML messages and message attachments. Yahoo deals with them by converting HTML message bodies to plain text and pretending the attachments don't exist. I don't think pretending attachments don't exist is a proper solution. Converting HTML to plain text is something I can do with lynx or elinks, and in fact, Mailman already has a setting for HTML_TO_PLAINTEXT_COMMAND used by content filtering, so it could be used for digest conversion too, but it isn't the 'nicest' solution. I'e', if a list allows HTML in the first place, the list members probably want to see all the colors and fonts, embedded images and background stationery.
I think possibly a better approach might be to take the present MIME format digest and convert the 'contents' part to an HTML part where clicking on a message subject would cause the MUA to open that message in a separate window/tab from where it can be viewd in it's full richness and replied to. I don't offhand know if such a thing can be done or if so, what MUAs might support it.
Now you are trying to ask in technical terms about things about which you are admittedly ignorant, and your questions wind up not making much sense.
I think this thread would be better off if you would concentrate on the features you want and let those who might implement them figure out how or if they can be provided.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2010-02-24 7:44 PM, Mark Sapiro wrote:
I think possibly a better approach might be to take the present MIME format digest and convert the 'contents' part to an HTML part
Sounds like you're talking about 'encapsulating' it to me... ;) sorry couldn't resist...
where clicking on a message subject would cause the MUA to open that message in a separate window/tab from where it can be viewed in it's full richness and replied to. I don't offhand know if such a thing can be done or if so, what MUAs might support it.
Hmmm... I actually really like this suggestion, especially if it might be [much?] easier to implement. Worth exploring at least.
But, instead of clicking on the subject - which wouldn't be very intuitive to me - I'd prefer just a single 'Open to Reply' link, which would open the message as you described. Then, as you said, it would be up to the MUA features to do the heavy lifting. This would be perfectly acceptable to me, especially if it is an order of magnitude easier to accomplish.
Too bad it isn't possible to keep the individual messages as attachments and work with them that way...
Hmmm... another brain-dead suggestion... keep the messages fully intact as attachments, but also convert/dump the plain-text content into the Digest body for purposes of scrolling between messages and the message summary at the top? Yes, it would increase the size of the digest - but text is small, so maybe it wouldn't be all that bad?
Then the link would be either 'Open to Reply', or, if the original message was other than text/plain, the link could change to 'Open to Reply/View in Original Format'.
I think this thread would be better off if you would concentrate on the features you want and let those who might implement them figure out how or if they can be provided.
Ok, I'll try... sorry for wasting all of you guys' time...
--
Charles
Tanstaafl wrote:
Hmmm... another brain-dead suggestion... keep the messages fully intact as attachments, but also convert/dump the plain-text content into the Digest body for purposes of scrolling between messages and the message summary at the top? Yes, it would increase the size of the digest - but text is small, so maybe it wouldn't be all that bad?
Yes, that would be possible, but how would you tell the MUA to hide the attached messages until they are explicitly opened?
In other words, this would be effectively the current MIME format digest except the front matter would be an HTML part similar to the Yahoo Digest.
Stephen has expressed this well in another reply, but instead of asking for a NEW digest format which no MUA will initially at least understand, why not ask the developers of your MUA of choice to do a better job of presenting the existing, standard MIME multipart/digest format message in a way more to your liking.
I.e., there's no inherent reason why Tbird for example couldn't present the MIME digest as it currently does and also present an "open as subfolder" button to do exactly that.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Tanstaafl writes:
A really dumb question - is there no way to (reliably, or even at all?) 'interact' with just the headers of messages that are attached?
Yes, there is. It's called a "MIME digest", and Mailman already provides that feature as a subscriber option. This gives the receiving MUA all the information it needs to do everything you have asked for (and mutt, Emacs/VM, and Emacs/Gnus all do it; I would suppose some of the more "user-friendly" MUAs do too).
In HTML, it's indeed possible to attach the messages and provide links to them as blobs of bytes (spammers do this all the time, for example), but there is no HTML facility to tell a browser or MUA that "this here blob is a mail message, do the right thing please", in the way that it is possible to say "this here blob is an image, do the right thing please". It's just a blob of bytes, and you are at the mercy of the predefined facilities in the viewing software as to what you can do with it. Dealing with images as objects embedded in HTML is nearly universally implemented in HTML viewers (including full-featured browsers and MUAs). Dealing with message/rfc822 objects *embedded in HTML* (as opposed to being received from the mail system) is not implemented by *anybody* as far as I know.
If you're viewing the message in a full browser like Firefox, you might actually be able to configure it to call out to a mail program (eg, Thunderbird or Mutt) to display message/rfc822, and tell it to do that with HTML code something like
<object content-id="some-random-looking-string" />.But then it's not clear how you would be able to set up links on parts of that object, there's no way to communicate "global" information provided by the digest (such as the TOC or the List-Post header) to the MUA in the subprocess, and one has to wonder why you're using Firefox to read mail if you're just going to call Thunderbird to display messages.
Obviously, for this to work, attachments would have to be displayed in-line, but I already do this, and so does everyone I know, and every MUA I've looked at can view attachments inline, so this shouldn't be an issue...
But that is *exactly* where we started from! It *is* a problem for you, or you wouldn't have posted here. True, it is *not* a problem for me, or for Mark, or for anybody else I talk to using an MUA implementing a "folder view" of digests or similar. We just ask for a MIME digest and we get all those features (except the List-Post header, and we're on the way to improving that situation).
But it is a problem for you, and I really don't know why when it works fine for everybody else I know. Why your MUA doesn't handle this very well I do not know. Have you tried switching your subscription to MIME digests? (ISTR discussion of MIME digests but I don't recall you mentioning whether you tried them.) If that doesn't help, it's something you should take up with with the developers of the MUA product you use. Or maybe it does handle it, but the feature is obscure and you need to find out how to use it effectively.
I don't blame you for associating the problem with mailing lists and Mailman in particular, rather than with your MUA, which I gather you otherwise find satisfactory. These days digest use is overwhelmingly dominated by mailing lists, while ordinary users rarely use them. But really, AFAICT the problem (except for propagating the List-Post header from the digest "container" down to the individual messages, which is already being dealt with) is that *the software you are using to read mail* doesn't handle the perfectly usable (with appropriate mail-reading software), standard-for-20-years-now MIME digests that Mailman already produces (and has done so for a decade or so).
Of course somebody who can hack code can make it do anything. But can *you*? You're the person I'm thinking of when I write "impossible" referring to the MM admin changing the basic function.
Why would you do that?
Because you're the kind of person people on this list generally mean by "admin". When we refer to a person as an "admin," we are talking about a role where a person at a certain level of competence (which need *not* include programming), with certain privileges, uses documented features to change the list's configuration or do other operations requiring stylized human intervention (eg, moderation).
People in the admin role don't mess with the code. The kind of person who changes code is called a "developer". Admin-cum-developer types who work on code when they've got an itch to scratch are common, it's true, but that is *not* "what admins do", it's "what developers do".
If that's not what *you* mean by admin, I don't know how we're supposed to know what you mean. I don't understand why you expect some random admin "out there" to do anything useful. Asking here first is the right thing to do for any feature request. What mailman- developers doesn't do for you is relatively unlikely to get done in general, and especially so in this case, IMO:
OTOH, the people who can hack the code very likely don't need this feature at all.
Why? Because they all use EMACS?
Or some other MIME-capable MUA, which includes all of the ones you've mentioned to the best of my knowledge (which isn't that extensive, but if MIME digests were that hard to use effectively I'd think it would be a FAQ on this list). As Mark said, it's hard to get enthusiastic about implementing this feature because it really is something that the MUA *should* be able to do. Email digests are 40 years old at this point in time, and highly capable, well-thought-out protocols are available for constructing them (on Mailman's side) and for interpreting and presenting them (on the MUA side).
Sorry, but the condescension in your comments is getting a little old.
I apologize for displaying my irritation.
On Feb 23, 2010, at 01:24 PM, Tanstaafl wrote:
The point I'm getting at is, I'd like to see the basic framework for a new HTML digest added to MM. Barry said he is fine with it as long as it is done right, and Mark seems to concur.
Note that this "basic framework" may be something as simple as a plugin architecture to allow third parties to easily integrate their own digest formats into Mailman 3. We as the core developers needn't develop - or more importantly, maintain - it.
How does MM generate the two digests it supports now? Does it store the individual messages in some temporary location until it is time to generate the digest, then do whatever it does to generate it? Or does it process each message as it comes in, and cumulatively add them until the trigger for sending the digest is pulled?
Mailman 3 stores each message in a little maildir for that mailing list. When the time comes to send a digest, a queue runner wakes up, reads the messages from the maildir and composes both the MIME and RFC 1153 digests at the same time. There is an interface describing the API that digesters must adhere to, but the current infrastructure does not allow for third party digesters. That would not be hard to add for anybody who's really interested in developing such a beast (I'm not).
-Barry
Tanstaafl wrote:
How does MM generate the two digests it supports now? Does it store the individual messages in some temporary location until it is time to generate the digest, then do whatever it does to generate it? Or does it process each message as it comes in, and cumulatively add them until the trigger for sending the digest is pulled?
As posts are delivered from a digestable list, they are accumulated in a standard *nix format mailbox (lists/LISTNAME/digest.mbox). The two digest formats are produced from this mbox, and it is emptied for the next one.
No offense, but I don't see that at all... MM is open source, and every single function of it can be modified to change the basic functioality - all that is necessary is someone competent with the appropriate tools - in this case, python (, C?) and HTML - and the code.
Who is interested enough and motivated enough to do the work in a way that would not turn into an endless bug-fixing time sink.
The examples you quote, digests from YahooGroups and GoogleGroups, basically convert all digested messages to plain text before digesting them. All fonts, colors and other HTML features are just removed as are any attachments (which just disappear without any indication that they were even there). The Yahoo digest does keep links from the original HTML, but the Google digest doesn't even do that.
Even this "least common denominator" form of digest still requires that the digest preparation process have access to an HTML rendering engine of some kind to do the plain text conversion.
Also see Stephen's reply at <http://mail.python.org/pipermail/mailman-developers/2010-February/021005.html> which I read after composing most of the above (I was waiting for a digest from Yahoo to confirm attachments were stripped).
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Just a quick follow up. I am on the daw-mac (digital audio workstation) mailing list at yahoogroups. They do a not horrible job of sending out HTML digests. I will try to put a copy of such a message up on the wiki after sanitizing it so we can look at how they do it.
-Barry
On 2010-02-24 9:12 AM, Barry Warsaw wrote:
I am on the daw-mac (digital audio workstation) mailing list at yahoogroups. They do a not horrible job of sending out HTML digests. I will try to put a copy of such a message up on the wiki after sanitizing it so we can look at how they do it.
Excellent! I looked at the source during my exchange with Stephen, but all it did was give me a headache...
--
Charles
On 2/20/2010 4:09 AM, Tanstaafl wrote:
On 2010-02-20 1:23 AM, Stephen J. Turnbull wrote:
Tanstaafl writes:
Personally, I usually set all of my lists to plain text anyway, so this wouldn't be an issue for any lists I host/maintain, but yes, I guess I can see how that might introduce another level of complexity to this request, but maybe such messages could be encapsulated somehow, since the features I'm talking about all have to do with interacting with the *headers* of each message, not the body/content.
Actually, you are not talking about interacting with message headers at all. You are talking about an HTML document which contains some rendering of message headers and contents and some buttons which may or may not do what you want them to depending on the specific MUA that you use to view this HTML document.
Further, your buttons would invoke something like
mailto:list@example.com?Subject=Re:%20The%20Subject&In-Reply-To=<message-id>&References=<message-id1>%0A%20<message-id2>%0A%20<message-id>&body=somebody%20wrote:%0A>line%20of%20text%0A>second%20line%0A
which is an RFC 2368 compliant mailto URL, at least if the 'body' contains only 7-bit characters, but it is not really intended to be used in this way, and I have no idea how many MUAs support it, particularly if the body= part were long, and what it would look like if the original message were not just plain text.
If you want things like quoting only selected text, the digest would not only need to be an HTML page, but an HTML page with javascript. Would you use an MUA that executed javascript in HTML email? Even Microsoft eventually figured out that automatically running executables in email was a really bad idea, no matter what useful things could be done with it.
So, again, the main question is if it could use just enough HTML formatting to make the features I outlined work for the majority of modern email clients, while still maintaining readability for those that may not be able to make use of the interactive features. If it can, then I think this would be an awesome option for those of us who subscribe to a whole bunch of email lists. I subscribe to most as individual messages only because interacting with the list via a Digest message is very frustrating and requires lots of copying and pasting, and still breaks threading etc because of the missing headers.
I actually find interacting with the MIME format digest where I have the ability to open any specific message of interest and reply to it (even quoting only selected text if the MUA supports it) to be fairly painless, and it doesn't break threading.
"Reply List" is not offerred for a reply to a message from the MIME digest, because there is no List-Post: header in the individual message parts
I'm curious why? If I had received the message individually, there would have been, so why not include them in messages in the digest?
I suspect this probably could be handled by the existing configuration options in almost all cases.
Cool... so, do I need to send a new email to make this Feature Request official? ;)
No. Just put
MIME_DIGEST_KEEP_HEADERS.append('List-Post')
in mm_cfg.py.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2010-02-20 11:31 AM, Mark Sapiro wrote:
On 2/20/2010 4:09 AM, Tanstaafl wrote:
On 2010-02-20 1:23 AM, Stephen J. Turnbull wrote:
Tanstaafl writes: Actually, you are not talking about interacting with message headers at all. You are talking about an HTML document which contains some rendering of message headers and contents and some buttons which may or may not do what you want them to depending on the specific MUA that you use to view this HTML document.
Actually, I was talking about (though admittedly wasn't very clear) whether or not MM3 could work some magic when building the digest to make this work.
Meaning, MM has the individual, fully intact messages, including the full headers - so I was thinking that *maybe* (I'm not saying it can be done, I'm asking the question) it could work some kind of magic through some scripting on the backend, to build an HTML digest with embedded links that could somehow invoke *the MUAs* Reply functions, but feeding the MUA what is necessary to make it *think* it is replying to the individual message rather than the digest.
But maybe its just not possible...
If you want things like quoting only selected text, the digest would not only need to be an HTML page, but an HTML page with javascript. Would you use an MUA that executed javascript in HTML email? Even Microsoft eventually figured out that automatically running executables in email was a really bad idea, no matter what useful things could be done with it.
Ok, agreed about the JS, but again I was thinking this could be left to the mail client - if *it* is capable of quoting only selected text, then it would 'just work'... I mean, it works now in TB3, so...
I actually find interacting with the MIME format digest where I have the ability to open any specific message of interest and reply to it (even quoting only selected text if the MUA supports it) to be fairly painless, and it doesn't break threading.
Ok, but this doesn't work for me right now with the mailman-users list, apparently because the List headers aren't there?
"Reply List" is not offerred for a reply to a message from the MIME digest, because there is no List-Post: header in the individual message parts
No. Just put
MIME_DIGEST_KEEP_HEADERS.append('List-Post')
in mm_cfg.py.
But I can't do that for lists that aren't under my control...
My question was about where to add a feature request to:
make this a configurable option in the GUI, and
make it enabled by default with a similar 'strongly recommended' comment next to it like it is with the same setting for individual messages.
Although I can't think of any, apparently I'm missing something else and there is one or more good reason[s] I haven't thought of to do this, otherwise it would already be the default?
Anyway, thanks for taking the time to consider this and respond as you all have...
--
Charles
On 2010-02-20 11:31 AM, Mark Sapiro wrote:
I actually find interacting with the MIME format digest where I have the ability to open any specific message of interest and reply to it (even quoting only selected text if the MUA supports it) to be fairly painless, and it doesn't break threading.
Yes, that works (except the Reply-To_List, which I think will be fixed by changing the default soon?), but I'm curious (maybe I'm missing something obvious?)...
In a digest with lots (hundreds) of messages with long threads, how do you know which individual/attached message to open so you can then Reply as desired?
Or... do you use EMACS too? :(
--
Charles
Tanstaafl wrote:
On 2010-02-20 11:31 AM, Mark Sapiro wrote:
I actually find interacting with the MIME format digest where I have the ability to open any specific message of interest and reply to it (even quoting only selected text if the MUA supports it) to be fairly painless, and it doesn't break threading.
Yes, that works (except the Reply-To_List, which I think will be fixed by changing the default soon?),
Yes, see <https://bugs.launchpad.net/mailman/+bug/526143>
but I'm curious (maybe I'm missing something obvious?)...
In a digest with lots (hundreds) of messages with long threads, how do you know which individual/attached message to open so you can then Reply as desired?
Or... do you use EMACS too? :(
One does not need Emacs VM. Even mutt can do a credible job of dealing with MIME digests. Maybe you could beat on the Thunderbird developers to do a better job of presenting digests. It seems like a lot of what you want could be handled better on the client side.
There are lots of reasons why this isn't an issue for me. Depending on what I'm doing, I may not need to know which message to open in order to reply because it's already open in order to read it in the first place.
Also, those Mailman lists from which I do receive digests have reasonable size limits on digests so I've never received a digest with hundreds of messages, probably I've never received one with as many as 20 messages.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Feb 19, 2010, at 04:42 PM, Tanstaafl wrote:
Currently, the MIME digest is a plain text message, and I don't see any way to reply to an individual message, much less preserve the message subject I'm replying to - unless you mean I'm supposed to actually open the message I want to reply to in a separate window, then click Reply?
I've only ever seen this in one MUA, VM-in-Emacs. That MUA had a very cool feature that allowed you to "burst" MIME digests (and I think RFC 1153 digests). You'd be presented with a "virtual" folder containing just the messages in the digest. This was really cool because you could save and reply to individual messages just as if they arrived individually. I'd love to have this in more mail readers.
My personal feeling is that if things can be done in a way that works reasonably in MUAs that don't support the features, then it might be feasable, but if something works with one MUA and is a disaster in another, it is better not done.
I hope you aren't suggesting that Mailman should limit its features to the 'least common denominator'... I don't see anything wrong with *optionally* supporting advanced features of modern mail clients.
As I mentioned in my previous message, I'm totally fine with an HTML digest that text-based MUA users can ignore. But it should work reasonably well across the field of HTML supporting MUAs, and shouldn't be disastrous on any of them. Or put it another way: if there are RFCs for this, follow them. If there aren't, act as if there were by writing something RFC like in the wiki that we can at least point to as a best practices document that MUA authors could follow.
-Barry
On 2/20/2010 11:12 AM, Barry Warsaw wrote:
On Feb 19, 2010, at 04:42 PM, Tanstaafl wrote: I've only ever seen this in one MUA, VM-in-Emacs. That MUA had a very cool feature that allowed you to "burst" MIME digests (and I think RFC 1153 digests). You'd be presented with a "virtual" folder containing just the messages in the digest. This was really cool because you could save and reply to individual messages just as if they arrived individually. I'd love to have this in more mail readers.
Interesting... is this feature documented anywhere online?
As I mentioned in my previous message, I'm totally fine with an HTML digest that text-based MUA users can ignore. But it should work reasonably well across the field of HTML supporting MUAs, and shouldn't be disastrous on any of them.
But really, how many different HTML supporting MUAs are there?
People who use web based mail won't be a problem. Then there's TB, Outlook/Windows Mail (I think we can forget about Outlook Express finally), Apple Mail... Pegasus? Old versions of Eudora? How many more? I really don't think we should be worrying about mail clients that have a tiny user base, just document which ones don't like the new digest...
Or put it another way: if there are RFCs for this, follow them. If there aren't, act as if there were by writing something RFC like in the wiki that we can at least point to as a best practices document that MUA authors could follow.
Agreed... :)
--
Best regards,
Charles
On 2010-02-17 11:15 AM, Mark Sapiro wrote:
"Reply List" is not offerred for a reply to a message from the MIME digest, because there is no List-Post: header in the individual message parts
By the way - why is it that TB3's new 'Reply-To-List' feature isn't working for this list? I still have to 'Reply All' then delete the senders address...
:(
--
Charles
On 2010-02-20 7:12 AM, Tanstaafl wrote:
On 2010-02-17 11:15 AM, Mark Sapiro wrote:
"Reply List" is not offerred for a reply to a message from the MIME digest, because there is no List-Post: header in the individual message parts
By the way - why is it that TB3's new 'Reply-To-List' feature isn't working for this list? I still have to 'Reply All' then delete the senders address...
And even stranger, I tried 'Reply-To-List' on my own message and it worked... !?
--
Charles
On 2/20/2010 7:30 AM, Tanstaafl wrote:
By the way - why is it that TB3's new 'Reply-To-List' feature isn't working for this list? I still have to 'Reply All' then delete the senders address...
And even stranger, I tried 'Reply-To-List' on my own message and it worked... !?
And of course, I'm not talking about digests here, I get individual messages...
--
Best regards,
Charles
On Feb 20, 2010, at 07:30 AM, Tanstaafl wrote:
On 2010-02-20 7:12 AM, Tanstaafl wrote:
On 2010-02-17 11:15 AM, Mark Sapiro wrote:
"Reply List" is not offerred for a reply to a message from the MIME digest, because there is no List-Post: header in the individual message parts
By the way - why is it that TB3's new 'Reply-To-List' feature isn't working for this list? I still have to 'Reply All' then delete the senders address...
And even stranger, I tried 'Reply-To-List' on my own message and it worked... !?
It generally works in Claws, but I've seen some cases where it doesn't work. I think Claws has a "smart" reply functionality that doesn't always get it right, so I've tried to train myself to use the explicit Reply-To-List keybinding, which so far works better. Also, I have to be careful to reply to the list copy of the message and not the one sent directly to me. ;)
-Barry
Barry Warsaw wrote:
On Feb 20, 2010, at 07:30 AM, Tanstaafl wrote:
And even stranger, I tried 'Reply-To-List' on my own message and it worked... !?
Also, I have to be careful to reply to the list copy of the message and not the one sent directly to me. ;)
I think that's the answer. Both Stephen and I generally reply-all (I do this deliberately for reasons that I won't go into here). Thus, the message Tanstaafl receives and replies to is not the one from the list
- thus, no List-*: headers.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2/20/2010 12:11 PM, Mark Sapiro wrote:
Barry Warsaw wrote:
Also, I have to be careful to reply to the list copy of the message and not the one sent directly to me. ;)
I think that's the answer. Both Stephen and I generally reply-all (I do this deliberately for reasons that I won't go into here). Thus, the message Tanstaafl receives and replies to is not the one from the list
- thus, no List-*: headers.
Ahh... and I have my list pref set to 'no dupes', which means I don't even see the list message...
I always delete the individual email address when I use 'Reply all', so you'll only get the list message...
Ok, mystery solved... :)
On 2/20/2010 4:12 AM, Tanstaafl wrote:
By the way - why is it that TB3's new 'Reply-To-List' feature isn't working for this list? I still have to 'Reply All' then delete the senders address...
Works for me. This is a Reply-list to your message with TB 3.0.1.
There was a Reply-list bug in 3.0, but that only affected messages with a List-Post: with multiple URLs which is not the case here.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Feb 17, 2010, at 08:33 AM, Tanstaafl wrote:
I'm really excited about the developments ongoing with MM3... :)
Great! If you're at Pycon, come sprint with us. If you're not here, we're going to at the very least be active on #mailman for virtual sprinters.
One thing I'd really like to see in MM3 with respect to list digests is what I'll call 'Interactive HTML Digests'. Yahoo 'full featured' digests go about halfway to where I'd like to see MM3 go.
I think it would be nice to have a rich HTML-based digest format, and I believe it should be fairly easy to put such a thing together (read: branches welcome!). If you look in mailman/interfaces/member.py you'll see that there's a DeliverMode enum with a summary_digests flag. This is really a placeholder for essentially what you're suggesting.
MM3 has a separate DigestRunner it uses to compose digests when the time is right. It does this to move the work of digest creation and sending out of the main processing runner. Right now, it's fairly hardcoded to just the two existing digests (MIME and RFC 1153), but with a little refactoring and interface definition, it could be easily generalized to include another style of digest.
The important methods are .add_toc() for adding a table of contents, .add_message() for adding a single message to the digest, and .finish() for producing the email ready message tree. There are two classes implemented currently MIMEDigester and RFC1153Digester. I could easily see an HTMLDigester or some such.
- Each message subject in the summary of messages at the top of the digest are embedded hyperlinks that if clicked, will scroll the digest down to that message (Yahoos do this now),
This should be doable with an HTML supporting mail reader. I think may be an RFC for this kind of inter-part linking (can't find it right now; multipart/related or something like that).
- Each message displayed in the digest body would have the following in a single line at the very bottom *and* top of each message (Yahoos are only at the bottom):
a) separate button/links for: 'Reply to Sender', 'Reply to List' and maybe 'Reply to all' (that last one could be optional?), and b) little 'Back to top' links to scroll back to the top of the message summary
Would these require JavaScript? Hmm, maybe they can be done with mail:to links.
- When replying to a message, preserve the message subject *and* all of the individual message references so that replying to messages from the HTML digests doesn't break threading - basically so it looks as if it was replied to individually and separately, not from the list digest.
I have no idea how you'd do this in a cross MUA way. That to me is the biggest risk with something like this. For those who choose HTML/Summary/Rich digests, I'd like to see broad MUA support, but I honestly have no idea of the state of HTML and JavaScript support across mail readers. You don't have to worry about text-only, or text-preferred mail readers though because users of those won't be selecting HTML digests.
-Barry
Thanks for chiming in Barry... :)
On 2/20/2010 9:08 AM, Barry Warsaw wrote:
On Feb 17, 2010, at 08:33 AM, Tanstaafl wrote:
I'm really excited about the developments ongoing with MM3... :)
Great! If you're at Pycon, come sprint with us. If you're not here, we're going to at the very least be active on #mailman for virtual sprinters.
I'd be happy to join in if I was capable of contributing anything of value... but I'm not a programmer - really - anyone who saw any of my shell scripts would run away screaming... ;)
So, the only way I can contribute is test things, make feature requests
- ie, make more work for you guys... ;) - and do GUI mockups...
I think it would be nice to have a rich HTML-based digest format,
:)
If you look in mailman/interfaces/member.py you'll see that there's a DeliverMode enum with a summary_digests flag. This is really a placeholder for essentially what you're suggesting.
Ok... I'm capable enough of finding the code in question, but you really, really don't want to see what I'd put there... it'd be something like:
"Do: make my pretty HTML fully interactive Digest NOW!; OnError: Scream-n-yell"
a) separate button/links for: 'Reply to Sender', 'Reply to List' and maybe 'Reply to all' (that last one could be optional?), and b) little 'Back to top' links to scroll back to the top of the message summary
Would these require JavaScript? Hmm, maybe they can be done with mail:to links.
Dunno, but even if it did (require JS), would that be all that bad? Don't most GUI mail readers that do HTML email do JS too?
- When replying to a message, preserve the message subject *and* all of the individual message references so that replying to messages from the HTML digests doesn't break threading - basically so it looks as if it was replied to individually and separately, not from the list digest.
I have no idea how you'd do this in a cross MUA way. That to me is the biggest risk with something like this. For those who choose HTML/Summary/Rich digests, I'd like to see broad MUA support, but I honestly have no idea of the state of HTML and JavaScript support across mail readers.
Maybe the best thing to do then before going any further would be to figure out what would be needed to accomplish the goal and test the most popular mail clients?
I imagine pretty much all web browsers would work fine, so that covers a lot of people who only use cloud/webmail...
So, that leaves, what... TB, Outlook/Windows Mail, Apple Mail? Any others?
I don't think this idea should be scrapped just because *some* clients won't work... just include the list of supported mail clients in the description for this Digest version, then leave it up to the users to be able to intelligently select the digest version they want/need...
Thanks again Barry... and even if all of the features in my original request can't be initially supported, apparently at least some of them can, and then maybe someday a way will be found to accomplish the rest... :)
--
Best regards,
Charles
Tanstaafl wrote:
On 2/20/2010 9:08 AM, Barry Warsaw wrote:
On Feb 17, 2010, at 08:33 AM, Tanstaafl wrote:
a) separate button/links for: 'Reply to Sender', 'Reply to List' and maybe 'Reply to all' (that last one could be optional?), and b) little 'Back to top' links to scroll back to the top of the message summary
Would these require JavaScript? Hmm, maybe they can be done with mail:to links.
Dunno, but even if it did (require JS), would that be all that bad? Don't most GUI mail readers that do HTML email do JS too?
They can be done with mailto: if quoting is fixed. If you want to be able to select text and then reply quoting selected text I think that would require JavaScript, and yes it would be bad.
See the example mailto: and the two following paragraphs in my post at <http://mail.python.org/pipermail/mailman-developers/2010-February/020969.html>.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Saturday 20 February 2010, Tanstaafl wrote:
So, that leaves, what... TB, Outlook/Windows Mail, Apple Mail? Any others?
Both Forte Agent and KMail will display HTML email. If someone has a MM3 test list I am willing to subscribe with both of these clients for testing.
William
participants (5)
-
Barry Warsaw -
Mark Sapiro -
Stephen J. Turnbull -
Tanstaafl -
William Bagwell