[IPython-dev] ipython on the sun grid engine

Sun Mar 22 01:38:29 EDT 2009

Hi,

so I made it run.

1. install SPD using this howto:

http://code.google.com/p/qsnake/wiki/SPD

2. create ex9.sge (adjust the path to spd):

$ cat ex9.sge
#/bin/bash
#$ -cwd
#$ -pe Common.pe 40
NUM_WORKER=$(($NSLOTS - 1))
MASTER=$(head -n 1 $TMP/machines)
WORKERS=$(tail -n $NUM_WORKER $TMP/machines)
echo "My Master nodes:"
echo $MASTER
echo "My Worker nodes:"
echo $WORKERS
{
ssh $MASTER '~/spd/spd-0.3/spd -python ~/spd/spd-0.3/local/bin/ipcontroller' &
sleep 3
for x in $WORKERS; do
    ssh $x '~/spd/spd-0.3/spd -python ~/spd/spd-0.3/local/bin/ipengine' &
done
} > runlog.txt
wait

3. submit the job:

$ qsub ex9.sge

4. connect to the ipcontroller:

$ ./spd
----------------------------------------------------------------------
| Source Python Distribution, Version 0.3, Release Date: 2009-03-21  |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
In [1]: from IPython.kernel import client
In [2]: mec = client.MultiEngineClient()
In [3]: mec.get_ids()
Out[3]:
[0,
 1,
 2,
 3,
 4,
 5,
 6,
 7,
 8,
 9,
 10,
 11,
 12,
 13,
 14,
 15,
 16,
 17,
 18,
 19,
 20,
 21,
 22,
 23,
 24,
 25,
 26,
 27,
 28,
 29,
 30,
 31,
 32,
 33,
 34,
 35,
 36,
 37]
In [8]: print mec.get_ids()
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36,
37]
In [9]: mec.activate()
In [10]: mec.block=True
In [11]: %px import numpy
Parallel execution on engines: all
Out[11]:
<Results List>
[0] In [1]: import numpy
[1] In [1]: import numpy
[2] In [1]: import numpy
[3] In [1]: import numpy
[4] In [1]: import numpy
[5] In [1]: import numpy
[6] In [1]: import numpy
[7] In [1]: import numpy
[8] In [1]: import numpy
[9] In [1]: import numpy
[10] In [1]: import numpy
[11] In [1]: import numpy
[12] In [1]: import numpy
[13] In [1]: import numpy
[14] In [1]: import numpy
[15] In [1]: import numpy
[16] In [1]: import numpy
[17] In [1]: import numpy
[18] In [1]: import numpy
[19] In [1]: import numpy
[20] In [1]: import numpy
[21] In [1]: import numpy
[22] In [1]: import numpy
[23] In [1]: import numpy
[24] In [1]: import numpy
[25] In [1]: import numpy
[26] In [1]: import numpy
[27] In [1]: import numpy
[28] In [1]: import numpy
[29] In [1]: import numpy
[30] In [1]: import numpy
[31] In [1]: import numpy
[32] In [1]: import numpy
[33] In [1]: import numpy
[34] In [1]: import numpy
[35] In [1]: import numpy
[36] In [1]: import numpy
[37] In [1]: import numpy

In [12]: %px a = numpy.random.rand(2,2)
Parallel execution on engines: all
Out[12]:
<Results List>
[0] In [2]: a = numpy.random.rand(2,2)
[1] In [2]: a = numpy.random.rand(2,2)
[2] In [2]: a = numpy.random.rand(2,2)
[3] In [2]: a = numpy.random.rand(2,2)
[4] In [2]: a = numpy.random.rand(2,2)
[5] In [2]: a = numpy.random.rand(2,2)
[6] In [2]: a = numpy.random.rand(2,2)
[7] In [2]: a = numpy.random.rand(2,2)
[8] In [2]: a = numpy.random.rand(2,2)
[9] In [2]: a = numpy.random.rand(2,2)
[10] In [2]: a = numpy.random.rand(2,2)
[11] In [2]: a = numpy.random.rand(2,2)
[12] In [2]: a = numpy.random.rand(2,2)
[13] In [2]: a = numpy.random.rand(2,2)
[14] In [2]: a = numpy.random.rand(2,2)
[15] In [2]: a = numpy.random.rand(2,2)
[16] In [2]: a = numpy.random.rand(2,2)
[17] In [2]: a = numpy.random.rand(2,2)
[18] In [2]: a = numpy.random.rand(2,2)
[19] In [2]: a = numpy.random.rand(2,2)
[20] In [2]: a = numpy.random.rand(2,2)
[21] In [2]: a = numpy.random.rand(2,2)
[22] In [2]: a = numpy.random.rand(2,2)
[23] In [2]: a = numpy.random.rand(2,2)
[24] In [2]: a = numpy.random.rand(2,2)
[25] In [2]: a = numpy.random.rand(2,2)
[26] In [2]: a = numpy.random.rand(2,2)
[27] In [2]: a = numpy.random.rand(2,2)
[28] In [2]: a = numpy.random.rand(2,2)
[29] In [2]: a = numpy.random.rand(2,2)
[30] In [2]: a = numpy.random.rand(2,2)
[31] In [2]: a = numpy.random.rand(2,2)
[32] In [2]: a = numpy.random.rand(2,2)
[33] In [2]: a = numpy.random.rand(2,2)
[34] In [2]: a = numpy.random.rand(2,2)
[35] In [2]: a = numpy.random.rand(2,2)
[36] In [2]: a = numpy.random.rand(2,2)
[37] In [2]: a = numpy.random.rand(2,2)

In [13]: %px print numpy.linalg.eigvals(a)
Parallel execution on engines: all
Out[13]:
<Results List>
[0] In [3]: print numpy.linalg.eigvals(a)
[0] Out[3]: [ 0.04392391  0.3245973 ]

[1] In [3]: print numpy.linalg.eigvals(a)
[1] Out[3]: [ 1.55163721  0.20437739]

[2] In [3]: print numpy.linalg.eigvals(a)
[2] Out[3]: [ 1.20045213  0.04814805]

[3] In [3]: print numpy.linalg.eigvals(a)
[3] Out[3]: [ 0.32003643  0.57111291]

[4] In [3]: print numpy.linalg.eigvals(a)
[4] Out[3]: [-0.19009279  0.99128112]

[5] In [3]: print numpy.linalg.eigvals(a)
[5] Out[3]: [ 1.03829332  0.36075953]

[6] In [3]: print numpy.linalg.eigvals(a)
[6] Out[3]: [-0.1517235   1.67516513]

[7] In [3]: print numpy.linalg.eigvals(a)
[7] Out[3]: [ 1.39960728 -0.28465163]

[8] In [3]: print numpy.linalg.eigvals(a)
[8] Out[3]: [ 1.17044538  0.36920544]

[9] In [3]: print numpy.linalg.eigvals(a)
[9] Out[3]: [ 1.29444925  0.03147179]

[10] In [3]: print numpy.linalg.eigvals(a)
[10] Out[3]: [ 1.06269256  0.00629928]

[11] In [3]: print numpy.linalg.eigvals(a)
[11] Out[3]: [ 0.1934025   0.04709656]

[12] In [3]: print numpy.linalg.eigvals(a)
[12] Out[3]: [-0.23157366  0.79206212]

[13] In [3]: print numpy.linalg.eigvals(a)
[13] Out[3]: [ 0.87736399  0.36763171]

[14] In [3]: print numpy.linalg.eigvals(a)
[14] Out[3]: [ 0.75825399  0.42571366]

[15] In [3]: print numpy.linalg.eigvals(a)
[15] Out[3]: [ 0.83819298 -0.33422046]

[16] In [3]: print numpy.linalg.eigvals(a)
[16] Out[3]: [ 0.05119763  1.25161191]

[17] In [3]: print numpy.linalg.eigvals(a)
[17] Out[3]: [ 0.6356765  -0.18668048]

[18] In [3]: print numpy.linalg.eigvals(a)
[18] Out[3]: [ 0.05417072  0.64346778]

[19] In [3]: print numpy.linalg.eigvals(a)
[19] Out[3]: [ 0.94357665  0.08495554]

[20] In [3]: print numpy.linalg.eigvals(a)
[20] Out[3]: [-0.20071214  1.44548662]

[21] In [3]: print numpy.linalg.eigvals(a)
[21] Out[3]: [ 1.20371805  0.17458575]

[22] In [3]: print numpy.linalg.eigvals(a)
[22] Out[3]: [ 1.47189454 -0.25060313]

[23] In [3]: print numpy.linalg.eigvals(a)
[23] Out[3]: [ 0.62675299  0.00833881]

[24] In [3]: print numpy.linalg.eigvals(a)
[24] Out[3]: [-0.61704795  0.86785185]

[25] In [3]: print numpy.linalg.eigvals(a)
[25] Out[3]: [ 0.51454737  1.09251881]

[26] In [3]: print numpy.linalg.eigvals(a)
[26] Out[3]: [ 1.0025157   0.37926302]

[27] In [3]: print numpy.linalg.eigvals(a)
[27] Out[3]: [ 0.07562955  0.69529292]

[28] In [3]: print numpy.linalg.eigvals(a)
[28] Out[3]: [ 0.70206288  0.09943419]

[29] In [3]: print numpy.linalg.eigvals(a)
[29] Out[3]: [-0.41140114  0.76592533]

[30] In [3]: print numpy.linalg.eigvals(a)
[30] Out[3]: [-0.41483813  0.66156836]

[31] In [3]: print numpy.linalg.eigvals(a)
[31] Out[3]: [ 0.06939694  0.82218779]

[32] In [3]: print numpy.linalg.eigvals(a)
[32] Out[3]: [-0.27583508  0.96545541]

[33] In [3]: print numpy.linalg.eigvals(a)
[33] Out[3]: [-0.07448614  0.95500699]

[34] In [3]: print numpy.linalg.eigvals(a)
[34] Out[3]: [ 0.57380518  1.01918719]

[35] In [3]: print numpy.linalg.eigvals(a)
[35] Out[3]: [ 0.90397042  0.06531227]

[36] In [3]: print numpy.linalg.eigvals(a)
[36] Out[3]: [ 0.89331523  0.58087954]

[37] In [3]: print numpy.linalg.eigvals(a)
[37] Out[3]: [ 0.69227186  1.25013097]

Any idea, why the output [3] looks so ugly? Is it some pretty printing
relict from Sage?

the ssh command actually failed to connect to 1 node, as seen from the
ex9.sge.e105489:

Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,gssapi-with-mic,password).

so only 38 engines were registered (e.g. 1 node is for ipcontroller,
38 for engines and 1 failed = 40 nodes).

This rocks!

Ondrej