[IPython-dev] new doc for parallel sessions on clusters

Tue Jun 21 17:41:55 EDT 2011

On Tue, Jun 21, 2011 at 13:45, Johann Cohen-Tanugi
<johann.cohentanugi at gmail.com> wrote:
>
>
> On 06/21/2011 07:58 PM, MinRK wrote:
>>
>> Glad you were able to fix it.
>
> I am still a bit puzzled by what I see in the
> $HOME/.config/ipython/profile_lsf/ipcluster_config.py
> I do not see c.Global, but for instance :
> -bash-3.2$ grep engine_launcher ~/.config/ipython/profile_lsf/*
> /u/ec/cohen/.config/ipython/profile_lsf/ipcluster_config.py:#
> c.IPClusterStart.engine_launcher_class = 'LSFEngineSetLauncher'
> /u/ec/cohen/.config/ipython/profile_lsf/ipcluster_config.py:#
> c.IPClusterEngines.engine_launcher_class = 'LSFEngineSetLauncher'

Sorry, c.Global is not used for anything anymore.  The generated
default config files are accurate.

>
> (where LSF replaces Local as I just edited it)
>
> Johann
>>
>> I just pushed an update to the online docs, so they should be up to date
>> now.
>
> thanks a lot. Where is it?
> http://ipython.org/ipython-doc/dev/parallel/parallel_process.html seems to
> still be the old version....

Oops, I managed to push the update to my own branch.  The real docs
are up to date now.

>
> best,
> JCT
>>
>> On Tue, Jun 21, 2011 at 10:00, Johann Cohen-Tanugi
>> <johann.cohentanugi at gmail.com>  wrote:
>>>
>>> ok found the problem : I still had the ipython-newapp source code in the
>>> same dir as the new ipython, I did everything from this new ipython dir,
>>> but at execution it was looking into ipython-newapp. Obviously there is
>>> something I was not expecting.... Anyway now it works.
>>>
>>> sorry again,
>>> Johann
>>>
>>> On 06/21/2011 06:41 PM, MinRK wrote:
>>>>
>>>> On Tue, Jun 21, 2011 at 09:31, Johann Cohen-Tanugi
>>>> <johann.cohentanugi at gmail.com>    wrote:
>>>>>
>>>>> hmmm ok, after the merge is this the right way to get the code :
>>>>>  git clone git://github.com/ipython/ipython.git ipython
>>>>> I removed the whole source directory, and the local install stuff
>>>>> related
>>>>> to
>>>>> python, and I started from scratch using the git command above.
>>>>> then :
>>>>> python setup.py install
>>>>> --prefix=/afs/slac/g/glast/users/cohen/IPYDEV/local/which python
>>>>>
>>>>> perhaps the problem is that there is an ipython installed with the
>>>>> native
>>>>> python that I use above, and I am wrong to assume that having
>>>>> /afs/slac/g/glast/users/cohen/IPYDEV/local/.../site-packages at the top
>>>>> of
>>>>> PYTHONPATH is enough to ensure that it overrides the one installed with
>>>>> python?
>>>>
>>>> If it was installed with setuptools/easy_install, yes - they inject
>>>> packages at the front sys.path at runtime, ending up ahead of
>>>> PYTHONPATH. easy_installed packages can even come before '.'.
>>>>
>>>> You can check which one you are importing by looking at
>>>> `IPython.__file__`.
>>>>
>>>> It's possible that I've messed something up, but the source looks
>>>> right
>>>>
>>>> (https://github.com/ipython/ipython/blob/master/IPython/core/profileapp.py#L127)
>>>>
>>>>> Anyway, I will continue investigating....
>>>>> sorry for the noise,
>>>>> JOhann
>>>>>
>>>>> On 06/21/2011 05:27 PM, MinRK wrote:
>>>>>>
>>>>>> On Tue, Jun 21, 2011 at 07:27, Johann Cohen-Tanugi
>>>>>> <johann.cohen-tanugi at lupm.univ-montp2.fr>      wrote:
>>>>>>>
>>>>>>> hi there, I tried to build the sphinx doc on my machine after a pull
>>>>>>> of
>>>>>>> the
>>>>>>> head (after Min announced the merge of newapp into master), and I see
>>>>>>> the
>>>>>>> following in docs/source/parallel/parallel_process.txt :
>>>>>>> 1/ A trivial typo at :
>>>>>>> :command:`ipcluster` has a notion of Launchers that can start
>>>>>>> controllers
>>>>>>> and engines with various remote execution schemes.  Currently
>>>>>>> supported
>>>>>>> models include :command:`ssh`, :command`mpiexec`, PBS-style (Torque,
>>>>>>> SGE),
>>>>>>> and Windows HPC Server.
>>>>>>>
>>>>>>>  |_____>
>>>>>>> should be :command:`mpiexec`
>>>>>>> 2/ I can read the following lines :
>>>>>>>    $ ipython profile create --parallel profile=mpi
>>>>>>> or
>>>>>>>    $ ipython profile create --parallel profile=pbs
>>>>>>>
>>>>>>> while ipython profile create does *not* accept a --parallel option
>>>>>>> AFAICT,
>>>>>>> but rather --cluster or --no-cluster.
>>>>>>
>>>>>> Then you do not have current master, because the flag is indeed
>>>>>> '--parallel'.
>>>>>>
>>>>>>> best,
>>>>>>> Johann
>>>>>>>
>>>>>>> On 06/21/2011 07:37 AM, Min RK wrote:
>>>>>>>>
>>>>>>>> Yes, it is going to be standard for the cluster profile and
>>>>>>>> interactive
>>>>>>>> IPython to be different - the profile for the Client is principally
>>>>>>>> for
>>>>>>>> connection info, and the shell profile is for configuring your
>>>>>>>> interactive
>>>>>>>> environment.  There's no reason to change your interactive config
>>>>>>>> just
>>>>>>>> to
>>>>>>>> connect to a different cluster.
>>>>>>>>
>>>>>>>> That said, the default profile of the Client should probably be that
>>>>>>>> of
>>>>>>>> the current application, not just 'default'.
>>>>>>>>
>>>>>>>> -MinRK
>>>>>>>>
>>>>>>>> On Jun 20, 2011, at 22:11, Johann
>>>>>>>> Cohen-Tanugi<johann.cohen-tanugi at lupm.univ-montp2.fr>        wrote:
>>>>>>>>
>>>>>>>>> hi- I don't think we should print the profile name in the default
>>>>>>>>> case,
>>>>>>>>>>
>>>>>>>>>> it's just noise.  I realize we now have a more consistent
>>>>>>>>>> structure
>>>>>>>>>> for profiles and even the default case is now a profile, but we
>>>>>>>>>> should
>>>>>>>>>> keep the amount of printed stuff to a minimum in the default
>>>>>>>>>> cases.
>>>>>>>>>>
>>>>>>>>> Actually I have a question here : I was trying newapp, following
>>>>>>>>> Min's
>>>>>>>>> advice, to try to add the LSF support in parallel.apps. From what I
>>>>>>>>> could
>>>>>>>>> gather
>>>>>>>>> I did
>>>>>>>>> ipcluster start -p lsf -n 2
>>>>>>>>> which created profile_lsf in my $HOME/.ipython directory, but then
>>>>>>>>> when
>>>>>>>>> I
>>>>>>>>> started another terminal window for the ipython session, I typed
>>>>>>>>> ipython profile=lsf
>>>>>>>>> and this loaded the default profile, so that I had to type :
>>>>>>>>>
>>>>>>>>> from IPython.parallel import Client
>>>>>>>>> c = Client(profile='lsf')
>>>>>>>>>
>>>>>>>>> so that unless this is a bug or an operator mistake, there seems to
>>>>>>>>> be
>>>>>>>>> 2
>>>>>>>>> 'profiles' in such a use case : the ipython global one, and the
>>>>>>>>> parallel lsf
>>>>>>>>> one. I find that a bit confusing, and maybe there is a way to merge
>>>>>>>>> the
>>>>>>>>> 2?
>>>>>>>>>
>>>>>>>>> best,
>>>>>>>>> Johann
>