so is there a way around that problem ??<br><br><div class="gmail_quote">On Fri, Jan 23, 2009 at 2:25 PM, Andre Engels <span dir="ltr"><<a href="mailto:andreengels@gmail.com">andreengels@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="Ih2E3d">On Fri, Jan 23, 2009 at 9:09 AM, amit sethi <<a href="mailto:amit.pureenergy@gmail.com">amit.pureenergy@gmail.com</a>> wrote:<br>
> Well that is interesting but why should that happen in case I am using a<br>
> different User Agent because I tried doing<br>
> status=rp.can_fetch('Mozilla/5.0',<br>
> "<a href="http://en.wikipedia.org/wiki/Sachin_Tendulkar" target="_blank">http://en.wikipedia.org/wiki/Sachin_Tendulkar</a>")<br>
> but even that returns false<br>
> Is there something wrong with the syntax , Is there a catch that i don't<br>
> understand.<br>
<br>
</div>The problem is that you are using the standard Python user agent when<br>
getting the robots.txt. Because the user agent is refused, it cannot<br>
get the robots.txt file itself to look at.<br>
<font color="#888888"><br>
--<br>
</font><div><div></div><div class="Wj3C7c">André Engels, <a href="mailto:andreengels@gmail.com">andreengels@gmail.com</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>A-M-I-T S|S<br>