Check URL --> Simply? (fwd) (fwd)

Dr. David Mertz mertz at
Thu Aug 16 07:42:24 CEST 2001

David Eppstein <eppstein at> wrote:
|Humans can't usually see the http response line with the actual 404 number
|in it in place of the 200 indicating an ok page, but machines can -- why
|don't you use that?

A couple reasons.  For one thing, my example was used to illustrate use
of some regular expressions.  But as an answer to the thread, that's
pretty weak :-).

Let's try again:  Because one might have previously crawled or proxied a
site, and want to figure out which downloaded pages are good (maybe not
using custom Python code for the crawling).  Not too bad.

But better still:

    #---------- ----------#
    import sys
    from httplib import HTTP
    from urlparse import urlparse

    def checkURL(url):
        p = urlparse(url)
        h = HTTP(p[1])
        h.putrequest('HEAD', p[2])
        return h.getreply()

    if __name__ == '__main__':
        for url in ('','',
            print url, checkURL(url)[:2]

    % python (200, 'OK') (302, 'Object moved') (301, 'Moved Permanently') (301, 'Moved Permanently') (301, 'Moved Permanently') (200, 'OK') (404, 'Not Found')

I tried a few sites to get these examples... but not all *that* many.
All the sites that end in 'nonsense' LOOK, to my human eyes, like broken
links... and all the others look like content (well, except,
which refuses to load--I think because I won't give it a cookie--and
wouldn't actually be other than nonsense if it would load :-)).

More information about the Python-list mailing list