[Web-SIG] Shortening execution time of Python script

[Chris Withers]

David Shi wrote:
> I have a Python script that automatically downloads zip files containing 
> large datasets from another server and then unzips the files to further 
> process the data.

This smells more like it belongs on comp.lang.python that wb-sig, but 
here goes...

> The script works fine when two datasets each has several kilobytes size, 
> but the script stops half way when datasets were about 11,000KBytes.
>  
> I think that the execution time is too long and ArcGIS Server just 
> simply killed the process.

How does ArcGIS execute this script?

> What actions can I try to reduce the execution time?
>  
> ArcGIS Server only works on the basis of 32 bits and I was told that the 
> maximum memory it can utilise is 4 MBytes.

For speed analysis, run the script through cProfile:

http://docs.python.org/library/profile.html

For analysis of how much memory your script is using, use heapy:

http://guppy-pe.sourceforge.net/heapy_tutorial.html

However, you mention downloading large files. Are you using httplib, 
urllib or urllib2 to do this? If so, you could be suffering from this bug:

http://bugs.python.org/issue6838

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk