[python-uk] Handling large file uploads

[William Mayor]

> On 11 Apr 2017, at 11:21, Hansel Dunlop <hansel at interpretthis.org> wrote:
> 
> Hello all
> 
> I'm working on an application that has to accept large uploads. Think ~ 2GB+ size files getting uploaded over slowish connections. These files are eventually going to end up in S3.
> 
> Uploading smallish files is not a problem. But things get a bit complicated when you're dealing with large files and load balanced servers. Servers that can be replaced at any time. Has anyone done something similar? 
> 
> My current plan is:
> 
> 1. Accept chunked uploads. So the app/browser sends individual POST requests with ~10mb chunks. Once that upload is complete the server responds with a chunk id and the current offset
> 2. The server stores each intermediate chunk in a temporary S3 bucket
> 3. Once the final chunk has been uploaded the server kicks off another process that stitches the pieces together and puts the whole file into it's final location. And then deletes the intermediate pieces. 
> 
> I think I have to do the file in chunks like this but maybe there is some way to stream the files somewhere? 
> 
> -- 
> 
>                                 Hansel
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk

I’ve don’t something similar in the past where I ended up having the users’ browsers upload directly to S3 and then send the file’s key back to me. That way my server didn’t have to manage any of the files and (hopefully) the process was quicker from the user’s point of view.

I used STS to create temporary AWS credentials that could only be used to upload files to a very specific bucket and “folder”. I then used the AWS JS SDK to manage the uploads client-side.

I liked this solution as it really reduced the amount of complex code that I had to write/maintain. All I had to do was glue the AWS SDKs together.

Billy