Execution speed question

Eric Wertman ewertman at gmail.com
Sat Jul 26 21:58:51 CEST 2008


> The number of nodes is very large: millions for sure, maybe tens
> of millions.  If considering (2), take note of my BOLD text above, which
> means I can't remove nodes as I iterate through them in the main loop.

Since your use of 'node' is pretty vague and I don't have a good sense
of what tests you are running and how long they would take,  I'm only
speculating, but a single loop might be the wrong way to go about
that.

If you are going to be frequently running tests and switching nodes
on/off, have you considered a separate set of processes to do both?
For example:


A set of some number of "tester" threads, that loop through and test,
recording thier results (somewhere).


You could then have a separate loop that runs every so often, checks
all the current test values, and runs through the nodes once,
switching them on or off.


I know it's not exactly what you asked, but depending on what your
nodes are exactly, you can avoid a lot of  other problems down the
road.  What if your single loop dies or gets hung on a test?  With a
separate approach, you'll have a lot more resilience too.. if there's
some problem with a single tester or node, it won't keep the rest of
the program from continuing to function.



More information about the Python-list mailing list