Fast URL validation?

[Michael Ströder]

Erik Max Francis wrote:
> 
> I use HEAD for most of my Web validation system.  It should be noted
> that several Web servers have been configured to reject HEAD requests,
> probably out of (unwarranted) paranoia.  The most robust solution would
> be a mechanism which uses HEAD and then falls back to a full GET if the
> HEAD request gives a suspicious response.

Additionally one could do separate hostname lookups in DNS before sending 
the HTTP request while maintaining a cache of failed lookups. This could 
speed up the whole thing if a DNS name is no longer present but is used in 
many of the URLs to be tested. Well, it might not work in your network setup 
(DNS lookups solely done by HTTP proxy).

Ciao, Michael.