[Tutor] Download file via HTTP GET with progress monitoring & custom headers?

[Kent Johnson]

On Sun, Aug 10, 2008 at 6:09 PM, xbmuncher <xboxmuncher at gmail.com> wrote:
> I want to download a file via HTTP GET protocol (perhaps HTTP POST in the
> future, whatever method should allow relativly easy changes to be made to
> use POST later on)
> I need to send custom headers like in urllib2.urlrequest()
> I need to be able to monitor the download progress WHILE it downloads like
> the hook function in urllib
>
> I looked into urllib2 and it did everything except allow me to check the
> progress of the download during the download, it only downloads the file
> first with urlopen(). I also tried urllib and the reporthook function is
> great, except I can't send custom headers! I wish I could send a REQ object
> to the urlretrieve() function in urllib that way I can prepare those custom
> headers.. :(
>
> Anyways, i would appreciate if soemone could write me a quick example to do
> this, or point me to the right library/funcs to do it. I want to be able to
> do it in native python libraries.

urllib2.urlopen() doesn't read the data from the remote site, it
returns a file-like object with a read() method. You can read the data
in chunks and call a reporthook yourself.

Take a look at the source to URLopener.retrieve() in urllib.py for some hints.

Kent