webspider getting stuck

[John Nagle]

notnorwegian at yahoo.se wrote:
> i am writing a simple webspider .
> 
> how do i avoid getting stuck at something like this:
> Enter username for W3CACL at www.w3.org:
> 
> ?


   It's a silly feature of urllib.  See

http://docs.python.org/lib/module-urllib.html

where it says:

"Note: When performing basic authentication, a FancyURLopener instance calls its 
prompt_user_passwd() method. The default implementation asks the users for the 
required information on the controlling terminal. A subclass may override this 
method to support more appropriate behavior if needed."

Yes, the default behavior when faced with a site that wants authentication
is to to ask for a user name and password on standard input.  This is
seldom what you want.

So subclass and overrride.

					John Nagle