[Web-SIG] Shortening execution time of Python script

[Sean Gillies]

Chris,

Many Europeans call geographic information systems (GIS) "SIG". Web- 
SIG, to someone who doesn't read the charter, looks like "Web GIS".  
Hence the random ArcGIS question, I think.

Cheers,

On Sep 10, 2009, at 9:39 AM, Chris Withers wrote:

> David Shi wrote:
>> I have a Python script that automatically downloads zip files  
>> containing large datasets from another server and then unzips the  
>> files to further process the data.
>
> This smells more like it belongs on comp.lang.python that wb-sig,  
> but here goes...
>
>> The script works fine when two datasets each has several kilobytes  
>> size, but the script stops half way when datasets were about  
>> 11,000KBytes.
>> I think that the execution time is too long and ArcGIS Server just  
>> simply killed the process.
>
> How does ArcGIS execute this script?
>
>> What actions can I try to reduce the execution time?
>> ArcGIS Server only works on the basis of 32 bits and I was told  
>> that the maximum memory it can utilise is 4 MBytes.
>
> For speed analysis, run the script through cProfile:
>
> http://docs.python.org/library/profile.html
>
> For analysis of how much memory your script is using, use heapy:
>
> http://guppy-pe.sourceforge.net/heapy_tutorial.html
>
> However, you mention downloading large files. Are you using httplib,  
> urllib or urllib2 to do this? If so, you could be suffering from  
> this bug:
>
> http://bugs.python.org/issue6838
>
> cheers,
>
> Chris
>
> -- 
> Simplistix - Content Management, Batch Processing & Python Consulting
>           - http://www.simplistix.co.uk

--
Sean