Patches item #657951, was opened at 2002-12-23 19:17 Message generated for change (Comment added) made by ppsys You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=657951&group_id=103 Category: Pipermail Group: Mailman 2.2 / 3.0 Status: Open Resolution: None Priority: 9 Submitted By: A.M. Kuchling (akuchling) Assigned to: Nobody/Anonymous (nobody) Summary: Generate RSS summary in archives Initial Comment: Here's a first-draft patch. Things that need fixing: * The generated RSS feed needs to be validated. (It passed the W3C's RDF validator, but RSS validators still need to be checked.) * The date should be given in YYYY-MM-DD format, which requires parsing the .fromdate attribute. * How do I get the URL for an archived message? The generated RSS currently just uses the filename, which is wrong. How do I get at the PUBLIC_ARCHIVE_URL setting? * Getting the most recent N postings is inefficient; the code loops through all of the archived messages and takes the last N of them. We could add .last() and .prev() methods to the Database class, but that's more ambitious for 2.1beta than I like. (Would be nice to get this into 2.1final...) * The list index page should have a LINK element pointing to the RSS file. Please make any comments you have, and I'll rework the patch accordingly. ---------------------------------------------------------------------- Comment By: Richard Barrett (ppsys) Date: 2004-12-06 18:01 Message: Logged In: YES user_id=75166 The following is based on the July 2003 version of the patch file posted on sourceforge. The RSS patch adds the RSS() function as member function of the HyperArchive class defined in HyperArch.py. It has been reported that the following statement in RSS(): date, msgid = self.database.dateIndex.first() may generate an AttributeError exception: AttributeError: HyperDatabase instance has no attribute 'dateIndex' The RSS patch appears to make the assumption that whenever the RSS() function is called from the write_TOC() member function of the HyperArchive class the __openIndices() function has already been called on the latest period archive associated with the list, whose TOC page is being generated by write_TOC(), and that no intervening call to __closeIndices() has been made. If the assumption were correct then whenever the RSS() function was called on a HyperArchive instance, the xxxxxIndices attributes of the HyperDatabase instance "owned" by the HyperArchive instance would be pointing to valid instance of DumbBTree. Unfortunately, this assumption is not correct. In order to do its work, write_TOC() does not itself need to perform any call to the __openIndices() function for the list/archive/database whose TOC page is to be recreated. It just happens that in some circumstances, some of the code which might call write_TOC may have called the __openIndices() function at some prior point and left the HyperDatabase instance with a valid set of xxxxxIndices attributes in place when write_TOC() is called. For the RSS patch to be work reliably the code in the RSS() function has to be changed so that it ensures that the conditions it wants prevail when it executes the statement giving the problem. The following is an untested code change but if part of the RSS() function's code definition in HyperArch.py is modified from: <quote> # Get the most recent messages. The only index operation # we can count on is traversal by increasing date, so # we end up traversing all of the entries and remembering the last # N of them. Sigh. items = [] try: date, msgid = self.database.dateIndex.first() items.append(msgid) except KeyError: pass while 1: try: </quote> to read: <quote> # Get the most recent messages. The only index operation # we can count on is traversal by increasing date, so # we end up traversing all of the entries and remembering the last # N of them. Sigh. items = [] got_first = 0 try: msgid = self.database.first(self.archives[0], 'date') if msgid: items.append(msgid) got_first = 1 except KeyError: pass while got_first and 1: try: </quote> this should fix the exception problem. ---------------------------------------------------------------------- Comment By: Roy M. Silvernail (codewhacker) Date: 2003-12-23 19:35 Message: Logged In: YES user_id=670974 I'm trying to enrich the RSS output by adding a proper [description] and a [content:encoded] module, but I am having the devil's own time locating the raw message text. Be happy to contribute a patch if you can point me to the raw content (without the italics markup for quoting). Thanks! ---------------------------------------------------------------------- Comment By: Michael Weber (wookiew) Date: 2003-09-10 16:29 Message: Logged In: YES user_id=863445 So far the patch is included (by the way: i hope that Defaults.py.in in the patch *means* Defaults.py ) and mailman get a restart. Hopefully i add the two lines in listinfo.html ( /de/ because we have german speaking lists) and take a look for the xml file). After search the whole device (only to be sure) i can say: There is no file like this. Is another patch need before? Another setup to make? I cant find any hint here... so i have to ask. But the idea is great... if it work on my lists its genious... regards, Michael running version 2.1.1 ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-09-02 04:15 Message: Logged In: YES user_id=12800 Bumping priority. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2003-07-15 18:55 Message: Logged In: YES user_id=11375 OK, done! This patch is now ready to go in: some people have looked at the RSS and haven't spotted any problems. Barry, can I please get CVS write access to check this in? ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2003-07-11 01:00 Message: Logged In: YES user_id=11375 Attaching correct version of the patch. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2003-07-11 00:52 Message: Logged In: YES user_id=11375 Here at last is an updated version of the patch that's crawling closer to being complete. There's now a RSS_NUM_ARTICLES setting in Defaults.py, the generated URLs are correct, and I modified the English template to link to the RSS file. Remaining things: check the generated RSS for correctness; edit all of the other language templates to include the RSS file (I may ask for CVS write access to do that). It would be really nice if the Mailman upgrade script could update existing general list information pages to include the LINK element; any suggestion about how to go about that? ---------------------------------------------------------------------- Comment By: Dan Brickley (danbri) Date: 2003-06-22 15:00 Message: Logged In: YES user_id=7830 OK, I've regenerated the patch with some code which works for me. http://rdfweb.org/2003/06/mailman-rss/rsspatch Health warning: * I suspect it may fail in conditions when get_archives() returns a list not a string (does this ever happen?). * See also problems mentioned below, regenerating partial archives seems tricky. Hope this is useful anyways... Dan <danbri@w3.org> ---------------------------------------------------------------------- Comment By: Dan Brickley (danbri) Date: 2003-06-22 11:48 Message: Logged In: YES user_id=7830 I thought I'd have a look at this myself, though have modest knowledge of both Python and MailMan. In the course of trying to patch the patch, I tried running the archiver over just the last couple of messages, to speed things along: "../../bin/arch -s 4390 rdfweb-dev". Traceback (most recent call last): File "../../bin/arch", line 187, in ? main() File "../../bin/arch", line 177, in main archiver.close() File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 310, in close self.write_TOC() File "/usr/local/mailman/Mailman/Archiver/HyperArch.py", line 1082, in write_TOC rss.write(self.RSS()) File "/usr/local/mailman/Mailman/Archiver/HyperArch.py", line 769, in RSS date, msgid = self.database.dateIndex.first() AttributeError: HyperDatabase instance has no attribute 'dateIndex' Not sure what's going on there, but this seemed as good a place of any to keep note of it. Investigating... ---------------------------------------------------------------------- Comment By: Dan Brickley (danbri) Date: 2003-06-22 11:04 Message: Logged In: YES user_id=7830 Does anyone have a patch to remove the hardwiring of "2002-December" and get the appropriate date from mailman somehow? ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-04-18 22:43 Message: Logged In: YES user_id=12800 Andrew, to get the url for the archived message use mlist.GetBaseArchiveURL(), which knows about private vs. public archives, the host name and the list name. From there you should be able to tack on just the part of the path under "archives/private/listname". See Mailman/Handlers/Scrubber.py for an example. Only other minor comment: NUM_ARTICLES can probably go in Defaults.py.in ---------------------------------------------------------------------- Comment By: Justin Mason (jmason) Date: 2003-03-26 21:49 Message: Logged In: YES user_id=935 big thumbs up from me too. Much better solution than http://taint.org/mmrss/ ;) ---------------------------------------------------------------------- Comment By: Uche Ogbuji (uche) Date: 2003-03-18 01:09 Message: Logged In: YES user_id=38966 I'd like to add my vote to this item. This is a fantastic idea, Andrew. Thanks. --Uche ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2002-12-23 20:42 Message: Logged In: YES user_id=11375 Updated patch: * Dates are now rendered as ISO-8601 (date only, not the time of the message) * By hard-wiring 2002-December, I got the RSS to validate using Mark Pilgrim's validator. ---------------------------------------------------------------------- Comment By: captain larry (captainlarry) Date: 2002-12-23 19:36 Message: Logged In: YES user_id=147905 Just voting for support here. This is *great* thanks for the patch and I hope the maintainers include it as soon as it's appropriate :) Adam. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2002-12-23 19:27 Message: Logged In: YES user_id=12800 Deferring until post-2.1 ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2002-12-23 19:21 Message: Logged In: YES user_id=11375 Argh; SF choked on the file upload. Attaching the patch again... ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=300103&aid=657951&group_id=103