Web-crawling

Rene Pijlman reply.in.the.newsgroup at my.address.is.invalid
Sat Oct 4 08:22:53 EDT 2003


John Bradbury:
>I am trying to develop a special putpose crawler using htmllib & urllib.
>How do you tell the server application that you are a modern browser and can
>handle frames?

I don't know of any "I can handle frames" header and I don't see why the
server would care, but you could mimic the User-agent header sent by a
modern browser.

-- 
René Pijlman




More information about the Python-list mailing list