cgi.fieldstorage()

[Diez B. Roggisch]

gert schrieb:
> On Jul 24, 7:32 pm, "Diez B. Roggisch" <de... at nospam.web.de> wrote:
>> gert schrieb:
>>
>>> this is a non standard way to store multi part post data on disk
>>> def application(environ, response):
>>>     with open('/usr/httpd/var/wsgiTemp','w') as f:
>>>         while True:
>>>             chunk = environ['wsgi.input'].read(8192).decode('latin1')
>>>             if not chunk: break
>>>             f.write(chunk)
>>>     response('200 OK',[])
>>>     return ['complete']
>>> my question is how do i handle the file, so i can shuffle it into a db
>>> using small chunks of memorie ?
>> I don't think that's possible with the current DB-API. There is no
>> stream-based BLOB-interface (as e.g. JDBC offers).
>>
>> So the answer certainly depends on your used RDBMS. For oracle, you
>> would be lucky:
>>
>> http://cx-oracle.sourceforge.net/html/lob.html
>>
>> Other adapters I don't know about.
> 
> sqlite :) ok let say for now it would be impossible on a db level, but
> before i reach the impossible, i still need to parse the file to
> prepare the chunks. How do i do that ? How do i get the chunks without
> loading the hole file into memorie ?

It's memory - memorie might be some nice dutch girl you know :)

Apart from that, your code above does exactly that - reads the data 
chunkwise. If the WSGI-implementation works proper, this will be the 
socket's input stream, so there is no memory overhead involved.

Now of course if you want to have a multi-pass processing of the file 
without putting it into memory, then you need to save it to the harddisk 
befor.

But honestly - we are talking about a web-application here. My 
DSL-connection has 1 MBit upstream, the average server has at least 2GB 
of memory available - so we are talking 20000 seconds uploading time to 
fill that memory. Which is about 5 hours. And you are decoding the data 
to a certain decoding, so we are not talking about binary data here - 
are you really sure memory is an issue?

Diez

Diez