Not necessarily related to python Web Crawlers

defn noob circularfunc at
Sat Jul 5 11:07:04 CEST 2008

just crawling is supereasy. its how to index and search that is hard.
just start at, scrape out all the links and then for every
site visit every link.
i wrote a crawler in 15 lines of code. but then it all it did was
visit the sites, not indexing them or anything.

you could write a faster one in C++ probably but if you are new to it
doing it in python will let you experiment and learn faster.

some links:

More information about the Python-list mailing list