<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">I think subdomains need there own robots.txt which <a href="http://docs.python.org">docs.python.org</a> nor <a href="http://docs.python.org/(2">docs.python.org/(2</a> or 3)/ have.</div>
<div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">and <a href="http://python.org/robots.txt">http://python.org/robots.txt</a> (below) seems a little sparse.</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">
For sure /dev/ is not blocked</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><pre style="color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">
# Directions for robots. See this URL:
# <a href="http://www.robotstxt.org/wc/norobots.html">http://www.robotstxt.org/wc/norobots.html</a>
# for a description of the file format.
User-agent: HTTrack
User-agent: puf
User-agent: MSIECrawler
Disallow: /
# The Krugle web crawler (though based on Nutch) is OK.
User-agent: Krugle
Allow: /
Disallow: /moin
Disallow: /pypi
Disallow: /~guido/orlijn/
Disallow: /wwwstats/
Disallow: /ftpstats/
# No one should be crawling us with Nutch.
User-agent: Nutch
Disallow: /
# Hide old versions of the documentation and various large sets of files.
User-agent: *
Disallow: /~guido/orlijn/
Disallow: /wwwstats/
Disallow: /webstats/
Disallow: /ftpstats/
Disallow: /moin
Disallow: /pypi
Disallow: /dev/buildbot/</pre></div></div><div class="gmail_extra"><br clear="all"><div><div>Vincent Davis</div><div>720-301-3003<span></span><span></span></div></div>
<br><br><div class="gmail_quote">On Sat, Jan 25, 2014 at 9:04 PM, Nick Coghlan <span dir="ltr"><<a href="mailto:ncoghlan@gmail.com" target="_blank">ncoghlan@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On 26 January 2014 05:05, Benjamin Peterson <<a href="mailto:benjamin@python.org">benjamin@python.org</a>> wrote:<br>
><br>
><br>
> On Sat, Jan 25, 2014, at 10:55 AM, Vincent Davis wrote:<br>
>> On Sat, Jan 25, 2014 at 10:12 AM, Benjamin Peterson<br>
>> <<a href="mailto:benjamin@python.org">benjamin@python.org</a>>wrote:<br>
>><br>
>> > Internal links with no version redirect to the Python 2 version for<br>
>> > backwards compatibility reasons.<br>
>> ><br>
>><br>
>> On Sat, Jan 25, 2014 at 10:26 AM, Georg Brandl <<a href="mailto:g.brandl@gmx.net">g.brandl@gmx.net</a>> wrote:<br>
>><br>
>> > Yep, and the URLs without version never served Python 3 docs as far as I<br>
>> > can<br>
>> ><br>
>> remember, so I don't know where Google has these <title>s from.<br>
>><br>
>> That is not consistent with<br>
>> <a href="http://docs.python.org" target="_blank">http://docs.python.org</a> (no version number) redirects to<br>
>> <a href="http://docs.python.org/3/" target="_blank">http://docs.python.org/3/</a><br>
><br>
> This is recent. It used to go to Python 2 docs.<br>
<br>
</div><a href="http://www.python.org/dev/peps/pep-0430/" target="_blank">http://www.python.org/dev/peps/pep-0430/</a> covers the rationale for the<br>
current arrangement.<br>
<br>
The main issue is the extensive use of existing deep links into the<br>
Python 2 documentation from Python 2 specific tutorials and other<br>
references. Those third party references not only include vast numbers<br>
of online resources that we don't control, but also books that can't<br>
be updated at all.<br>
<br>
So, the canonical URLs on <a href="http://docs.python.org" target="_blank">docs.python.org</a> now always include the major<br>
version number in the path so they're unambiguous, the Python 3 docs<br>
are displayed by default, and unqualified deep links redirect to<br>
Python 2 for backwards compatibility.<br>
<br>
The robots.txt on <a href="http://python.org" target="_blank">python.org</a> is *supposed* to keep the web crawlers<br>
away from the "/dev/" subtree (since most people searching for Python<br>
info aren't going to want the docs for an unreleased version), but I<br>
don't know if that's documented anywhere, or even if it's currently<br>
still configured that way.<br>
<br>
>> Maybe this is related to google search results.<br>
>> Seems wrong to me to point to 2.7 rather that 3.3 but I am sure there was<br>
>> discussion about that.<br>
><br>
> The internal links all used to go to Python 2.<br>
<br>
There's also a lot of weight given in Google to the extensive array of<br>
existing unqualified deep links, which relate to Python 2.<br>
<br>
>> I looked (googled) for an example of a google link to current version of<br>
>> python 3.3 documentation. My approach was to google "python" and<br>
>> something<br>
>> listed in<br>
>> <a href="http://docs.python.org/3/whatsnew/3.3.html" target="_blank">http://docs.python.org/3/whatsnew/3.3.html</a><br>
>> These results all seem to point to <a href="http://docs.python.org/dev/library" target="_blank">http://docs.python.org/dev/library</a><br>
>> i.e.<br>
>> 3.4.0b2<br>
<br>
Which suggests that the Google web crawler *is* spidering the dev<br>
docs, which we generally don't want :P<br>
<br>
Cheers,<br>
Nick.<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Nick Coghlan | <a href="mailto:ncoghlan@gmail.com">ncoghlan@gmail.com</a> | Brisbane, Australia<br>
</font></span></blockquote></div><br></div>