[Web-SIG] Shortening execution time of Python script
chris at simplistix.co.uk
Thu Sep 10 09:39:20 CEST 2009
David Shi wrote:
> I have a Python script that automatically downloads zip files containing
> large datasets from another server and then unzips the files to further
> process the data.
This smells more like it belongs on comp.lang.python that wb-sig, but
> The script works fine when two datasets each has several kilobytes size,
> but the script stops half way when datasets were about 11,000KBytes.
> I think that the execution time is too long and ArcGIS Server just
> simply killed the process.
How does ArcGIS execute this script?
> What actions can I try to reduce the execution time?
> ArcGIS Server only works on the basis of 32 bits and I was told that the
> maximum memory it can utilise is 4 MBytes.
For speed analysis, run the script through cProfile:
For analysis of how much memory your script is using, use heapy:
However, you mention downloading large files. Are you using httplib,
urllib or urllib2 to do this? If so, you could be suffering from this bug:
Simplistix - Content Management, Batch Processing & Python Consulting
More information about the Web-SIG