Web-crawling

[John Bradbury]

I don't know what is causing the problem, but the site I am accessing is
sending out forms for a browser that has a low resolution and does not
support frames.  Excuse my ignorance, but where do you set up the User-agent
header you suggested.

Many thanks for your prompt reply.

John Bradbury

"Rene Pijlman" <reply.in.the.newsgroup at my.address.is.invalid> wrote in
message news:bretnvcng69nqpoeug71jon4obs0moe63f at 4ax.com...
> John Bradbury:
> >I am trying to develop a special putpose crawler using htmllib & urllib.
> >How do you tell the server application that you are a modern browser and
can
> >handle frames?
>
> I don't know of any "I can handle frames" header and I don't see why the
> server would care, but you could mimic the User-agent header sent by a
> modern browser.
>
> --
> René Pijlman