httplib and large file uploads

[Eric S. Johansson]

Jesse Noller wrote:
> Hey All,
> 
> I'm working on an script that will generate a file of N size (where N is 
> 1k-1gig) in small chunks, in memory (and hash the data on the fly) and 
> pass it to an httplib object for upload. I don't want to store the file 
> on the disk, or completely in memory at any time. The problem arises 
> after getting the http connection (PUT) - and then trying to figure out 
> how to iterate/hand the chunks I am generating to the httplib 
> connection's send() call. For example (this code does not work as is):

I've been wrestling with a similar one for a while and been mostly held 
up because the Java applet developer has been... less than prompt.

The only way I've been able to successfully upload very large files is 
by performing multiple put requests and on the server-side aggregating 
them into the "real file".  obviously, world will be more difficult if 
you don't have a uniform chunk size or sequential creation of chunks.

---eric