[Tutor] fetching wikipedia articles
amit.pureenergy at gmail.com
Fri Jan 23 10:37:09 CET 2009
so is there a way around that problem ??
On Fri, Jan 23, 2009 at 2:25 PM, Andre Engels <andreengels at gmail.com> wrote:
> On Fri, Jan 23, 2009 at 9:09 AM, amit sethi <amit.pureenergy at gmail.com>
> > Well that is interesting but why should that happen in case I am using a
> > different User Agent because I tried doing
> > status=rp.can_fetch('Mozilla/5.0',
> > "http://en.wikipedia.org/wiki/Sachin_Tendulkar")
> > but even that returns false
> > Is there something wrong with the syntax , Is there a catch that i don't
> > understand.
> The problem is that you are using the standard Python user agent when
> getting the robots.txt. Because the user agent is refused, it cannot
> get the robots.txt file itself to look at.
> André Engels, andreengels at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Tutor