[python-uk] Handling large file uploads

[James Browne]

Hi Hansel,

Here's the documentation for browser-based POSTing using S3:
http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingHTTPPOST.html

If you want to implement chunking the file and stitching together yourself,
here's quite a nice JavaScript library to help with some of the heavy
lifting on the front-end. We've used this will some success for large files:

http://www.resumablejs.com/

Kind regards,

James Browne


On 11 April 2017 at 11:32, William Mayor <mail at williammayor.co.uk> wrote:

>
> > On 11 Apr 2017, at 11:21, Hansel Dunlop <hansel at interpretthis.org>
> wrote:
> >
> > Hello all
> >
> > I'm working on an application that has to accept large uploads. Think ~
> 2GB+ size files getting uploaded over slowish connections. These files are
> eventually going to end up in S3.
> >
> > Uploading smallish files is not a problem. But things get a bit
> complicated when you're dealing with large files and load balanced servers.
> Servers that can be replaced at any time. Has anyone done something similar?
> >
> > My current plan is:
> >
> > 1. Accept chunked uploads. So the app/browser sends individual POST
> requests with ~10mb chunks. Once that upload is complete the server
> responds with a chunk id and the current offset
> > 2. The server stores each intermediate chunk in a temporary S3 bucket
> > 3. Once the final chunk has been uploaded the server kicks off another
> process that stitches the pieces together and puts the whole file into it's
> final location. And then deletes the intermediate pieces.
> >
> > I think I have to do the file in chunks like this but maybe there is
> some way to stream the files somewhere?
> >
> > --
> >
> >                                 Hansel
> > _______________________________________________
> > python-uk mailing list
> > python-uk at python.org
> > https://mail.python.org/mailman/listinfo/python-uk
>
> I’ve don’t something similar in the past where I ended up having the
> users’ browsers upload directly to S3 and then send the file’s key back to
> me. That way my server didn’t have to manage any of the files and
> (hopefully) the process was quicker from the user’s point of view.
>
> I used STS to create temporary AWS credentials that could only be used to
> upload files to a very specific bucket and “folder”. I then used the AWS JS
> SDK to manage the uploads client-side.
>
> I liked this solution as it really reduced the amount of complex code that
> I had to write/maintain. All I had to do was glue the AWS SDKs together.
>
> Billy
>
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk
>



-- 
James Browne
jb at jamesbrowne.me
07779 804 426
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-uk/attachments/20170411/3a9e6a9a/attachment-0001.html>