Execution speed question
ewertman at gmail.com
Sat Jul 26 21:58:51 CEST 2008
> The number of nodes is very large: millions for sure, maybe tens
> of millions. If considering (2), take note of my BOLD text above, which
> means I can't remove nodes as I iterate through them in the main loop.
Since your use of 'node' is pretty vague and I don't have a good sense
of what tests you are running and how long they would take, I'm only
speculating, but a single loop might be the wrong way to go about
If you are going to be frequently running tests and switching nodes
on/off, have you considered a separate set of processes to do both?
A set of some number of "tester" threads, that loop through and test,
recording thier results (somewhere).
You could then have a separate loop that runs every so often, checks
all the current test values, and runs through the nodes once,
switching them on or off.
I know it's not exactly what you asked, but depending on what your
nodes are exactly, you can avoid a lot of other problems down the
road. What if your single loop dies or gets hung on a test? With a
separate approach, you'll have a lot more resilience too.. if there's
some problem with a single tester or node, it won't keep the rest of
the program from continuing to function.
More information about the Python-list