spidering script

Nikita the Spider NikitaTheSpider at gmail.com
Sat Jan 20 21:41:18 CET 2007

In article <8N6dnSE2eO6QKDLYnZ2dnUVZ_uejnZ2d at fdn.com>,
 "David Waizer" <dwaizer at noreply.com> wrote:

> Hello..
> I'm  looking for a script (perl, python, sh...)or program (such as wget) 
> that will help me get a list of ALL the links on a website.
> For example ./magicscript.pl www.yahoo.com and outputs it to a file, it 
> would be kind of like a spidering software..

In addition to others' suggestions about Beautiful Soup, you might also 
want to look at the HTMLData module:


Whole-site HTML validation, link checking and more

More information about the Python-list mailing list