[python-uk] Handling large file uploads

[Hansel Dunlop]

Thanks everyone,

Uploading directly to S3 sounds the avenue to pursue. It removes a whole
load of complications!

Cheers

Hansel

On Tue, Apr 11, 2017 at 12:18 PM, Roger Gammans <rgammans at gammascience.co.uk
> wrote:

> Indeed.
>
> Both the rsync and bittorrent protocols split files into chunks. There are
> a lot of similarities bwtween these protocols.
>
> Also it might be worth looking at are de-duping backup tools like
> duplicity.
>
> On Tue, 2017-04-11 at 11:33 +0100, Javier Llopis wrote:
>
> I haven't done anything similar, but this upload in chunks then reassemble
> idea bears some similarity to peer to peer file sharing software. I would
> look there for ideas in case I got stuck with some problem.
>
> J
>
>
> ------------------------------
> *From*: "Hansel Dunlop" <hansel at interpretthis.org>
> *Sent*: 11 April 2017 11:22
> *To*: "UK Python Users" <python-uk at python.org>
> *Subject*: [python-uk] Handling large file uploads
>
> Hello all
>
> I'm working on an application that has to accept large uploads. Think ~
> 2GB+ size files getting uploaded over slowish connections. These files are
> eventually going to end up in S3.
>
> Uploading smallish files is not a problem. But things get a bit
> complicated when you're dealing with large files and load balanced servers.
> Servers that can be replaced at any time. Has anyone done something
> similar?
>
> My current plan is:
>
> 1. Accept chunked uploads. So the app/browser sends individual POST
> requests with ~10mb chunks. Once that upload is complete the server
> responds with a chunk id and the current offset
> 2. The server stores each intermediate chunk in a temporary S3 bucket
> 3. Once the final chunk has been uploaded the server kicks off another
> process that stitches the pieces together and puts the whole file into it's
> final location. And then deletes the intermediate pieces.
>
> I think I have to do the file in chunks like this but maybe there is some
> way to stream the files somewhere?
>
> --
>
>                                 Hansel
>
> _______________________________________________
> python-uk mailing listpython-uk at python.orghttps://mail.python.org/mailman/listinfo/python-uk
>
>
>
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk
>
>


-- 

                                Hansel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-uk/attachments/20170411/cea4f2dd/attachment-0001.html>