search an entire website given the homepage URL

Bell, Kevin kevin.bell at slcgov.com
Tue Apr 25 13:02:07 EDT 2006


I know I can use urllib2 to get at a website given urllib2.urlopen(url)
but I'm unsure how to then go through all pages that are linked to it,
but still in the domain.  If I want to search through the entire python
website give the homepage, how would I go about it?  I don't reinvent
the wheel if someones already written a module for parsing out the links
and drilling through them!

TIA
   KEV 



More information about the Python-list mailing list