[Tutor] fetching wikipedia articles

Andre Engels andreengels at gmail.com
Fri Jan 23 09:55:03 CET 2009

On Fri, Jan 23, 2009 at 9:09 AM, amit sethi <amit.pureenergy at gmail.com> wrote:
> Well that is interesting but why should that happen in case  I am using a
> different User Agent because I tried doing
> status=rp.can_fetch('Mozilla/5.0',
> "http://en.wikipedia.org/wiki/Sachin_Tendulkar")
> but even that returns false
> Is there something wrong with the syntax , Is there a catch that i don't
> understand.

The problem is that you are using the standard Python user agent when
getting the robots.txt. Because the user agent is refused, it cannot
get the robots.txt file itself to look at.

André Engels, andreengels at gmail.com

More information about the Tutor mailing list