[Neuroimaging] [nilearn] Running nilearn on kubernetes cluster.

[Roberto Guidotti]

Hi,

I'm writing since I performed some other attempts to figure out the issue,
using scikit-learn and it turned out that may be an issue with cross_*
functions since I had also a similar problem with a scikit-learn script
without any nilearn's line of code as reported here
https://github.com/dask/distributed/issues/2532.

Bests,
R

On Fri, 15 Feb 2019 at 13:11, Roberto Guidotti <robbenson18 at gmail.com>
wrote:

> Hi all,
>
> I tried to run nilearn on a kubernetes cluster with dask distributed.
> Here's the snippet.
>
> from dask_kubernetes import KubeCluster
> from dask.distributed import Client
> cluster = KubeCluster.from_yaml('pods.yaml')
> cluster.scale(3)
> client = Client(cluster.scheduler_address)
>
> Then I ran the plot_haxby_searchlight.py example, using following line for
> fitting.
>
> from sklearn.externals import joblib
> with joblib.parallel_backend('dask'):
>     searchlight.fit(fmri_img, y)
>
> [Parallel(n_jobs=-1)]: Using backend DaskDistributedBackend with 6
> concurrent workers.
> tornado.application - ERROR - Exception in callback
> functools.partial(<function wrap.<locals>.null_wrapper at 0x7f886a7cd730>,
> <Future finished
> exception=CancelledError(['_group_iter_search_light-batch-18a44db2f247402da4955d757c513610'],)>)
> ...
>
> CancelledError: _fit_and_score-batch-cb239abe1e714386b5a5607c78d927e8
>
> I can run sklearn examples with dask without any problem, but with nilearn
> I had this.
>
> On the kubernetes cluster, I use a docker image that is based on this repo:
> https://github.com/robbisg/docker-dask-k8s
>
> Basically I need to run some nilearn analyses using HPC, is this the less
> painful way to do this or you suggest something else.
>
> Thank you,
> Roberto
>
>
>
> --
> Ing. Roberto Guidotti, PhD.
> PostDoc Fellow
> Institute for Advanced Biomedical Technologies - ITAB
> Department of Neuroscience and Imaging
> University of Chieti "G. D'Annunzio"
> Via dei Vestini, 33
> 66013 Chieti, Italy
> tel: +39 0871 3556919
> e-mail: r.guidotti at unich.it; rguidotti at acm.org
> linkedin: http://it.linkedin.com/in/robertogui/
> twitter: @robbisg
> github: https://github.com/robbisg
>


-- 
Ing. Roberto Guidotti, PhD.
PostDoc Fellow
Institute for Advanced Biomedical Technologies - ITAB
Department of Neuroscience and Imaging
University of Chieti "G. D'Annunzio"
Via dei Vestini, 33
66013 Chieti, Italy
tel: +39 0871 3556919
e-mail: r.guidotti at unich.it; rguidotti at acm.org
linkedin: http://it.linkedin.com/in/robertogui/
twitter: @robbisg
github: https://github.com/robbisg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/neuroimaging/attachments/20190216/a9998af3/attachment.html>