[IPython-dev] new doc for parallel sessions on clusters

[MinRK]

Glad you were able to fix it.

I just pushed an update to the online docs, so they should be up to date now.

On Tue, Jun 21, 2011 at 10:00, Johann Cohen-Tanugi
<johann.cohentanugi at gmail.com> wrote:
> ok found the problem : I still had the ipython-newapp source code in the
> same dir as the new ipython, I did everything from this new ipython dir,
> but at execution it was looking into ipython-newapp. Obviously there is
> something I was not expecting.... Anyway now it works.
>
> sorry again,
> Johann
>
> On 06/21/2011 06:41 PM, MinRK wrote:
>>
>> On Tue, Jun 21, 2011 at 09:31, Johann Cohen-Tanugi
>> <johann.cohentanugi at gmail.com>  wrote:
>>>
>>> hmmm ok, after the merge is this the right way to get the code :
>>>  git clone git://github.com/ipython/ipython.git ipython
>>> I removed the whole source directory, and the local install stuff related
>>> to
>>> python, and I started from scratch using the git command above.
>>> then :
>>> python setup.py install
>>> --prefix=/afs/slac/g/glast/users/cohen/IPYDEV/local/which python
>>>
>>> perhaps the problem is that there is an ipython installed with the native
>>> python that I use above, and I am wrong to assume that having
>>> /afs/slac/g/glast/users/cohen/IPYDEV/local/.../site-packages at the top
>>> of
>>> PYTHONPATH is enough to ensure that it overrides the one installed with
>>> python?
>>
>> If it was installed with setuptools/easy_install, yes - they inject
>> packages at the front sys.path at runtime, ending up ahead of
>> PYTHONPATH. easy_installed packages can even come before '.'.
>>
>> You can check which one you are importing by looking at
>> `IPython.__file__`.
>>
>> It's possible that I've messed something up, but the source looks
>> right
>> (https://github.com/ipython/ipython/blob/master/IPython/core/profileapp.py#L127)
>>
>>> Anyway, I will continue investigating....
>>> sorry for the noise,
>>> JOhann
>>>
>>> On 06/21/2011 05:27 PM, MinRK wrote:
>>>>
>>>> On Tue, Jun 21, 2011 at 07:27, Johann Cohen-Tanugi
>>>> <johann.cohen-tanugi at lupm.univ-montp2.fr>    wrote:
>>>>>
>>>>> hi there, I tried to build the sphinx doc on my machine after a pull of
>>>>> the
>>>>> head (after Min announced the merge of newapp into master), and I see
>>>>> the
>>>>> following in docs/source/parallel/parallel_process.txt :
>>>>> 1/ A trivial typo at :
>>>>> :command:`ipcluster` has a notion of Launchers that can start
>>>>> controllers
>>>>> and engines with various remote execution schemes.  Currently supported
>>>>> models include :command:`ssh`, :command`mpiexec`, PBS-style (Torque,
>>>>> SGE),
>>>>> and Windows HPC Server.
>>>>>
>>>>>  |_____>
>>>>> should be :command:`mpiexec`
>>>>> 2/ I can read the following lines :
>>>>>    $ ipython profile create --parallel profile=mpi
>>>>> or
>>>>>    $ ipython profile create --parallel profile=pbs
>>>>>
>>>>> while ipython profile create does *not* accept a --parallel option
>>>>> AFAICT,
>>>>> but rather --cluster or --no-cluster.
>>>>
>>>> Then you do not have current master, because the flag is indeed
>>>> '--parallel'.
>>>>
>>>>> best,
>>>>> Johann
>>>>>
>>>>> On 06/21/2011 07:37 AM, Min RK wrote:
>>>>>>
>>>>>> Yes, it is going to be standard for the cluster profile and
>>>>>> interactive
>>>>>> IPython to be different - the profile for the Client is principally
>>>>>> for
>>>>>> connection info, and the shell profile is for configuring your
>>>>>> interactive
>>>>>> environment.  There's no reason to change your interactive config just
>>>>>> to
>>>>>> connect to a different cluster.
>>>>>>
>>>>>> That said, the default profile of the Client should probably be that
>>>>>> of
>>>>>> the current application, not just 'default'.
>>>>>>
>>>>>> -MinRK
>>>>>>
>>>>>> On Jun 20, 2011, at 22:11, Johann
>>>>>> Cohen-Tanugi<johann.cohen-tanugi at lupm.univ-montp2.fr>      wrote:
>>>>>>
>>>>>>> hi- I don't think we should print the profile name in the default
>>>>>>> case,
>>>>>>>>
>>>>>>>> it's just noise.  I realize we now have a more consistent structure
>>>>>>>> for profiles and even the default case is now a profile, but we
>>>>>>>> should
>>>>>>>> keep the amount of printed stuff to a minimum in the default cases.
>>>>>>>>
>>>>>>> Actually I have a question here : I was trying newapp, following
>>>>>>> Min's
>>>>>>> advice, to try to add the LSF support in parallel.apps. From what I
>>>>>>> could
>>>>>>> gather
>>>>>>> I did
>>>>>>> ipcluster start -p lsf -n 2
>>>>>>> which created profile_lsf in my $HOME/.ipython directory, but then
>>>>>>> when
>>>>>>> I
>>>>>>> started another terminal window for the ipython session, I typed
>>>>>>> ipython profile=lsf
>>>>>>> and this loaded the default profile, so that I had to type :
>>>>>>>
>>>>>>> from IPython.parallel import Client
>>>>>>> c = Client(profile='lsf')
>>>>>>>
>>>>>>> so that unless this is a bug or an operator mistake, there seems to
>>>>>>> be
>>>>>>> 2
>>>>>>> 'profiles' in such a use case : the ipython global one, and the
>>>>>>> parallel lsf
>>>>>>> one. I find that a bit confusing, and maybe there is a way to merge
>>>>>>> the
>>>>>>> 2?
>>>>>>>
>>>>>>> best,
>>>>>>> Johann
>