[Mailman-Users] Robots.txt

[Greg Sims]

Thanks Mark and Stephen,

This is not straightforward in my case either.  I am running mailman under
Plesk.  There exists the file:

/etc/httpd/conf.d/mailman.conf

This files sets up a ScriptAlias for /mailman/ and an Alias for
/pipermail/.  I added the following to this file:

Alias /robots.txt /var/www/vhosts/raystedman.org/not-drupal/lists-robots.txt

This directory is a convenient location within the webroot of
RayStedman.org.  I created lists-robots.txt in the directory above and
restarted Apache.  The result is the existence of:

http://lists.raystedman.org/robots.txt

which hides all the files for the lists.raystedman.org subdomain from the
search engines.  I will now sit back a see how long it takes for these urls
in this subdomain to be delisted.  I will give them a push via google
webmaster tools which offers an option to delist a URL.

Thanks again for your help guys!

Happy New Year, Greg







On Sat, Dec 27, 2014 at 11:03 AM, Mark Sapiro <mark at msapiro.net> wrote:

> On 12/26/2014 09:21 AM, Greg Sims wrote:
> >
> > I read the archives of this list and found a couple of entries.  The
> answer
> > seems to be including "Disallows" in our robots.txt file.  I did this as
> > you can see here -- towards the bottom of the file:
> >
> > http://www.raystedman.org/robots.txt
> >
> > This has been in place for four weeks so I do not believe it is
> working.  I
> > believe this problem is there needs to be a robots.txt file for each
> > subdomain.  In this case, mailman lives in the lists.raystedman.org
> > subdomain.  Is there a way to have a robots.txt file for the
> > lists.raystedman.org subdomain?
>
>
> Absolutely. Just put it in the root directory that contains the page
> which is served when you go to <http://lists.raystedman.org/>. I.e., put
> it where it will be served if you go to
> <http://lists.raystedman.org/robots.txt>
>
> However, note that can take a loooong time for pages to age out of
> search results.
>
> --
> Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
> San Francisco Bay Area, California    better use your sense - B. Dylan
> ------------------------------------------------------
> Mailman-Users mailing list Mailman-Users at python.org
> https://mail.python.org/mailman/listinfo/mailman-users
> Mailman FAQ: http://wiki.list.org/x/AgA3
> Security Policy: http://wiki.list.org/x/QIA9
> Searchable Archives:
> http://www.mail-archive.com/mailman-users%40python.org/
> Unsubscribe:
> https://mail.python.org/mailman/options/mailman-users/greg%40headingup.net
>