On Fri, Sep 28, 2012 at 4:36 PM, Chris Leary <cdleary@acm.org> wrote:
Found a red-hot, branchy-looking Python kernel in the wild and naturally I thought of you trace compiler folks! ;-) Hope that it might be useful: I think it could make a nice addition to the speed center, seeing as how it's a CPU bound workload on all the machines I have access to (though I haven't profiled it at all so it could potentially be leaning heavily on paths in some unoptimized builtins).
MalGen is a set of scripts which generate large, distributed data sets suitable for testing and benchmarking software designed to perform parallel processing on large data sets. The data sets can be thought of as site-entity log files. After an initial seeding, the scripts allow for the data generation to be initiated from a single central node to run the generation concurrently on multiple remote nodes of the cluster.
-- http://code.google.com/p/malgen/
Specifically, http://code.google.com/p/malgen/source/browse/trunk/bin/cloud/malgen/malgen.... which gets run thusly:
::
pypy malgen.py -O /tmp/ -o INITIAL.txt 0 50000000 10000000 21
(Where 5e7 is the "initial block size" and 1e7 is the other-than-inital block size.) This generates the initial seeding they were talking about, followed by a run for each of N blocks on each node (in this hypothetical setup, for 5 blocks on each of four nodes the following is run):
::
pypy malgen.py -O /tmp [start_value]
The metadata is read out of the INITIAL.txt file and used to determine the size of the block, and the parameter [start_value] is used to bump to the appropriate start id count for the current block.
Inner loop: http://code.google.com/p/malgen/source/browse/trunk/bin/cloud/malgen/malgen....
Thoughts?
- Leary _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Looks like it could be a good addition, have you run benchmarks on it yourself? (Also, should we be directing any new benchmarks to the python-speed mailing list?) Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero