[IPython-dev] SciPy Sprint summary

[Justin Riley]

Hi Satra/Brian,

I was just going to suggest array jobs and decided to give it a try 
before posting. I've hacked it in starting from Satra's 0.10.1-sge 
branch. I'll commit it to my fork after some testing.

~Justin

On 07/18/2010 12:05 AM, Brian Granger wrote:
> Is the array jobs feature what you want?
>
> http://wikis.sun.com/display/gridengine62u6/Submitting+Jobs
>
> Brian
>
> On Sat, Jul 17, 2010 at 9:00 PM, Brian Granger<ellisonbg at gmail.com>  wrote:
>> On Sat, Jul 17, 2010 at 6:23 AM, Satrajit Ghosh<satra at mit.edu>  wrote:
>>> hi ,
>>>
>>> i've pushed my changes to:
>>>
>>> http://github.com/satra/ipython/tree/0.10.1-sge
>>>
>>> notes:
>>>
>>> 1. it starts cleanly. i can connect and execute things. when i kill using
>>> ctrl-c, the messages appear to indicate that everything shut down well.
>>> however, the sge ipengine jobs are still running.
>>
>> What version of Python and Twisted are you running?
>>
>>> 2. the pbs option appears to require mpi to be present. i don't think one
>>> can launch multiple engines using pbs without mpi or without the workaround
>>> i've applied to the sge engine. basically it submits an sge job for each
>>> engine that i want to run. i would love to know if a single job can launch
>>> multiple engines on a sge/pbs cluster without mpi.
>>
>> I think you are right that pbs needs to use mpirun/mpiexec to start
>> multiple engines using a single PBS job.  I am not that familiar with
>> SGE, can you start mulitple processes without mpi and with just a
>> single SGE job?  If so, let's try to get that working.
>>
>> Cheers,
>>
>> Brian
>>
>>> cheers,
>>>
>>> satra
>>>
>>> On Thu, Jul 15, 2010 at 8:55 PM, Satrajit Ghosh<satra at mit.edu>  wrote:
>>>>
>>>> hi justin,
>>>>
>>>> i hope to test it out tonight. from what fernando and i discussed, this
>>>> should be relatively straightforward. once i'm done i'll push it to my fork
>>>> of ipython and announce it here for others to test.
>>>>
>>>> cheers,
>>>>
>>>> satra
>>>>
>>>>
>>>> On Thu, Jul 15, 2010 at 4:33 PM, Justin Riley<justin.t.riley at gmail.com>
>>>> wrote:
>>>>>
>>>>> This is great news. Right now StarCluster just takes advantage of
>>>>> password-less ssh already being installed and runs:
>>>>>
>>>>> $ ipcluster ssh --clusterfile /path/to/cluster_file.py
>>>>>
>>>>> This works fine for now, however, having SGE support would allow
>>>>> ipcluster's load to be accounted for by the queue.
>>>>>
>>>>> Is Satra on the list? I have experience with SGE and could help with the
>>>>> code if needed. I can also help test this functionality.
>>>>>
>>>>> ~Justin
>>>>>
>>>>> On 07/15/2010 03:34 PM, Fernando Perez wrote:
>>>>>> On Thu, Jul 15, 2010 at 10:34 AM, Brian Granger<ellisonbg at gmail.com>
>>>>>> wrote:
>>>>>>> Thanks for the post.  You should also know that it looks like someone
>>>>>>> is going to add native SGE support to ipcluster for 0.10.1.
>>>>>>
>>>>>> Yes, Satra and I went over this last night in detail (thanks to Brian
>>>>>> for the pointers), and he said he might actually already have some
>>>>>> code for it.  I suspect we'll get this in soon.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> f
>>>>>
>>>>> _______________________________________________
>>>>> IPython-dev mailing list
>>>>> IPython-dev at scipy.org
>>>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>>
>>>
>>>
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>
>>>
>>
>>
>>
>> --
>> Brian E. Granger, Ph.D.
>> Assistant Professor of Physics
>> Cal Poly State University, San Luis Obispo
>> bgranger at calpoly.edu
>> ellisonbg at gmail.com
>>
>
>
>