[Chicago] GIS Python and threads

Daniel Griffin dgriff1 at gmail.com
Thu Jan 28 16:21:14 CET 2010

Your best bet would be using a processpool from the
http://docs.python.org/library/multiprocessing.html module. Threads in
python kinda suck but using a process pool should help you here.

On Thu, Jan 28, 2010 at 9:15 AM, Robert Spelich <rlspelich at gmail.com> wrote:

> I work with GIS data and write Python scripts to extract data from very
> large datasets. I have a particular case where it is necessary to query on
> the data and create a separate file object base on the queried data. For
> reasons too difficult to explain, this has to be done over 800 times, once
> for each row in a different dataset. The API that is provided by the GIS
> application vendor is very good but the processing of these scripts is very
> slow. I estimate 30s to a minute each, so the run takes hours. Each
> execution of the script is independent of the other so I was wondering if I
> should use threads. I read up on threads and was like, OK my head hurts now,
> but if this will speed things up I will take a crack at it.
> Is this a good case for threads? Also are they manageable, I don’t want 800
> threads firing at once and crashing the machine.
> Thank you.
> --
> -Bob
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/chicago/attachments/20100128/ad248bad/attachment.htm>

More information about the Chicago mailing list