Use of httplib?

Steve Holden sholden at
Thu Dec 14 00:55:23 CET 2000

David Lees <DavidL at> wrote in message
news:3A37E456.772811C9 at
> I am running the client code below to try and send and receive text from
> a server.  My problem is that I do not know how to recieve back an
> arbitrary number of lines.  In a test case, when I know the exact number
> of lines as shown below (e.g. 11) I can use readline.  However, in
> general I do not know how many lines there are and the program hangs on
> the connection when  I accidentaly read to the end.  The comment out
> pieces of code also either hang or do not give back all the text.
Hmm ... the manual page for httplib reminds us that it is used by urllib,
and although it does implement the HTTP client-side protocol it's actually
much easier to use urllib.  Here's a snippet form a crawler program I wrote
some time ago (warning: not tested under 2.0) which shows that it's very
easy to use urllib to read HTML pages from a server:

    ignored = []
    distance = 1
    while distance < MAXDIST:
    for URL in l[distance-1]:
        if URL == '-':
            f = sys.stdin
            f = urllib.urlopen(URL)
        data =
        if f is not sys.stdin:

        p = myHTMLParser(fmt, URL)
        links, rubbish = p.close()

> I am copying from the Library Reference section 11.4, but it does not
> seem to document the readline, read or readlines methods for http.
Bottom line: use urllib, that's what it's for.  Easier, too!  Good luck.


More information about the Python-list mailing list