[Mailman-Developers] Indexing pipermail archives
J C Lawrence
Fri, 23 Jun 2000 11:01:08 -0700
On Fri, 23 Jun 2000 12:43:20 -0400 (EDT)
Jeremy Hylton <email@example.com> wrote:
> Speaking as a non-expert, these changes sound good reasonable. Do
> robots other than htdig understand these meta tags?
My experience, from watching my Apache logs (I'm very well indexed),
is that ALL of the commercial search engine spiders and robots
honour both robots.txt and the robots meta tag.
J C Lawrence Home: firstname.lastname@example.org
----------(*) Other: email@example.com
--=| A man is as sane as he is dangerous to his environment |=--