help on file storage for split multi part download
gagsl-py2 at yahoo.com.ar
Sun Mar 9 17:55:15 CET 2008
En Sat, 08 Mar 2008 08:27:12 -0200, <coolman.guron at gmail.com> escribió:
> On Mar 7, 2:14 pm, "Gabriel Genellina" <gagsl-... at yahoo.com.ar> wrote:
>> En Fri, 07 Mar 2008 04:16:42 -0200, <coolman.gu... at gmail.com> escribi�:
>> > BUT the thing thats going in my mind is thread safety. i plan to start
>> > each part of the filedownloadin a different thread. and then when
>> > each thread had downloaded more than 100kb (or eof or boundary
>> > reached) write the buffer to the disk. can this be achieved using
>> > mutex ? i have never shared objects between threads.
>> Use a different (single) thread to write the file; the others put write
>> requests on a Queue.queue object, and the writer just gets the requests
>> and processes them.
>> > is there a way to write this without using threads at all ???
>> Using asyncore, and perhaps the Twisted framework.
> asyncore is basically a server thing right?
asyncore is usually used to build servers, because in a server you want to
handle many requests with few resources, but you can use it to write a
Here is an example: http://effbot.org/zone/asyncore-ftp-client.htm
How many files and how many simultaneous connections do you plan to
handle? Using multiple threads to download and a single thread to write,
connected thru a queue, looks like the "simplest thing that probably
works" to me unless you have other constraints.
More information about the Python-list