Send scikit-learn mailing list submissions to
        scikit-learn@python.org

To subscribe or unsubscribe via the World Wide Web, visit
        https://mail.python.org/mailman/listinfo/scikit-learn

or, via email, send a message with subject or body 'help' to
        scikit-learn-request@python.org

You can reach the person managing the list at
        scikit-learn-owner@python.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of scikit-learn digest..."


Today's Topics:

   1. GSoC 2017 (Gael Varoquaux)
   2. Control over the inner loop in GridSearchCV (Ludovico Coletta)
   3. Re: Control over the inner loop in GridSearchCV
      (Sebastian Raschka)


----------------------------------------------------------------------

Message: 1
Date: Mon, 27 Feb 2017 11:58:35 +0100
From: Gael Varoquaux <gael.varoquaux@normalesup.org>
To: Scikit-learn user and developer mailing list
        <scikit-learn@python.org>
Subject: [scikit-learn] GSoC 2017
Message-ID: <20170227105835.GC2041043@phare.normalesup.org>
Content-Type: text/plain; charset=iso-8859-1

Hi,

Students have been inquiring about the GSoC (Google Summer of Code) with
scikit-learn, and the core team has been quite silent about team.

I am happy to announce that we will be taking part in the scikit-learn
again. The reason that we decided to do this is to give a chance to the
young, talented, and motivated students.

Importantly, our most limiting resource is the time of our experienced
developers. This is clearly visible from the number of pending pull
requests. Hence, we need students to be very able and independent. This
of course means that they will be getting supervision from mentors. Such
supervision is crucial for moving forward with a good project, that
delivers mergeable code. However, we will need the students to be very
good at interacting efficiently with the mentors. Also, I should stress
that we will be able to take only a very few numbers of students.

With that said, let me introduce the 2017 GSoC for scikit-learn. We have
set up a wiki page which summarizes the experiences from last year and
the ideas for this year:
https://github.com/scikit-learn/scikit-learn/wiki/Google-summer-of-code-(GSOC)-2017


Interested students should declare their interest on the mailing list,
and discuss with possible mentors here. Factors of success will be

* careful work on a good proposal, that takes on of the ideas on the wiki
  but breaks it down in a realistic plan with multiple steps and shows a
  good understanding of the problem.

* demonstration of the required skillset via successful pull requests in
  scikit-learn.

Cheers,

Ga?l


--
    Gael Varoquaux
    Researcher, INRIA Parietal
    NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France
    Phone:  ++ 33-1-69-08-79-68
    http://gael-varoquaux.info            http://twitter.com/GaelVaroquaux




------------------------------

Message: 2
Date: Mon, 27 Feb 2017 14:27:59 +0000
From: Ludovico Coletta <ludo25_90@hotmail.com>
To: "scikit-learn@python.org" <scikit-learn@python.org>
Subject: [scikit-learn] Control over the inner loop in GridSearchCV
Message-ID:
        <BLUPR0301MB2017606E3E103266BBAB5E698C570@BLUPR0301MB2017.namprd03.prod.outlook.com>
       
Content-Type: text/plain; charset="iso-8859-1"

Dear Scikit experts,


we am stucked with GridSearchCV. Nobody else was able/wanted to help us, we hope you will.


We are analysing neuroimaging data coming from 3 different MRI scanners, where for each scanner we have a healthy group and a disease group. We would like to merge the data from the 3 different scanners in order to classify the healthy subjects from the one who have the disease.


The problem is that we can almost perfectly classify the subjects according to the scanner (e.g. the healthy subjects from scanner 1 and scanner 2). We are using a custom cross validation schema to account for the different scanners: when no hyper-parameter (SVM) optimization is performed, everything is straightforward. Problems arise when we would like to perform hyperparameter optimization: in this case we need to balance for the different scanner in the optimization phase as well. We also found a custom cv schema for this, but we are not able to pass it to GridSearchCV object. We would like to get something like the following:


pipeline = Pipeline([('scl', StandardScaler()),
                    ('sel', RFE(estimator,step=0.2)),
                                    ('clf', SVC(probability=True, random_state=42))])


param_grid = [{'sel__n_features_to_select':[22,15,10,2],
                           'clf__C': np.logspace(-3, 5, 100),
                   'clf__kernel':['linear']}]

clf = GridSearchCV(pipeline,
                          param_grid=param_grid,
                  verbose=1,
                                  scoring='roc_auc',
                  n_jobs= -1)

# cv_final is the custom cv for the outer loop (9 folds)

ii = 0

while ii < len(cv_final):
# fit and predict

clf.fit(data[?]], y[[?]])
predictions.append(clf.predict(data[cv_final[ii][1]])) # outer test data
ii = ii + 1


We tried almost everything. When we define clf in the loop, we pass the -ith cv_nested as cv argument, and we fit it on the training data of the -ith custom_cv fold, we get an "Too many values to unpack" error. On the other end, when we try to pass the nested -ith cv fold as cv argument for clf, and we call fit on the same cv_nested fold, we get an "Index out of bound" error.

Two questions:

1) Is there any workaround to avoid the split when clf is called without a cv argument?

2) We suppose that for hyperparameter optimization the test data is removed from the dataset and a  new dataset is created. Is this true? In this case we only have to adjust the indices accordingly


Thank your for your time and sorry for the long text

Ludovico
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scikit-learn/attachments/20170227/e80777cb/attachment-0001.html>

------------------------------

Message: 3
Date: Mon, 27 Feb 2017 11:27:24 -0500
From: Sebastian Raschka <se.raschka@gmail.com>
To: Scikit-learn user and developer mailing list
        <scikit-learn@python.org>
Subject: Re: [scikit-learn] Control over the inner loop in
        GridSearchCV
Message-ID: <FC403FD1-9A00-424A-8453-9D60FE176C92@gmail.com>
Content-Type: text/plain; charset=utf-8

Hi, Ludovico,
what format (shape) is data in? Are these the arrays from a Kfold iterator? In this case, the ?question marks? in your code snippet should simply be the train and validation subset indices generated by the KFold generator. E.g., 

skfold = StratifiedKFold(y=y_train, n_folds=5, shuffle=True, random_state=1)
for outer_train_idx, outer_valid_idx in skfold:
    ?
    gridsearch_object.fit(X_train[outer_train_idx], y_train[outer_train_idx])

>
> On the other end, when we try to pass the nested -ith cv fold as cv argument for clf, and we call fit on the same cv_nested fold, we get an "Index out of bound" error. 
> Two questions:

Are you using an version older than scikit-learn 0.18? Techically, the GridSearchCV, RandomizedSearchCV, cross_val_score ? should all support iterables that of train_ and test_indices e.g.:

outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)

for name, gs_est in sorted(gridcvs.items()):
    nested_score = cross_val_score(gs_est,                
    X=X_train,                     
    y=y_train,                                
   cv=outer_cv,                            
   n_jobs=1)


Best,
Sebastian

> On Feb 27, 2017, at 9:27 AM, Ludovico Coletta <ludo25_90@hotmail.com> wrote:
>
> Dear Scikit experts,
>
> we am stucked with GridSearchCV. Nobody else was able/wanted to help us, we hope you will.
>
> We are analysing neuroimaging data coming from 3 different MRI scanners, where for each scanner we have a healthy group and a disease group. We would like to merge the data from the 3 different scanners in order to classify the healthy subjects from the one who have the disease.
>
> The problem is that we can almost perfectly classify the subjects according to the scanner (e.g. the healthy subjects from scanner 1 and scanner 2). We are using a custom cross validation schema to account for the different scanners: when no hyper-parameter (SVM) optimization is performed, everything is straightforward. Problems arise when we would like to perform hyperparameter optimization: in this case we need to balance for the different scanner in the optimization phase as well. We also found a custom cv schema for this, but we are not able to pass it to GridSearchCV object. We would like to get something like the following:
>
> pipeline = Pipeline([('scl', StandardScaler()),
>                     ('sel', RFE(estimator,step=0.2)),      
>                                     ('clf', SVC(probability=True, random_state=42))])
>                     
>                     
> param_grid = [{'sel__n_features_to_select':[22,15,10,2],
>                            'clf__C': np.logspace(-3, 5, 100),
>                    'clf__kernel':['linear']}]
>
> clf = GridSearchCV(pipeline,
>                           param_grid=param_grid,
>                   verbose=1,
>                                   scoring='roc_auc',
>                   n_jobs= -1)
>
> # cv_final is the custom cv for the outer loop (9 folds)
>
> ii = 0
>
> while ii < len(cv_final): 
> # fit and predict
>
> clf.fit(data[?]], y[[?]])
> predictions.append(clf.predict(data[cv_final[ii][1]])) # outer test data
> ii = ii + 1
>
> We tried almost everything. When we define clf in the loop, we pass the -ith cv_nested as cv argument, and we fit it on the training data of the -ith custom_cv fold, we get an "Too many values to unpack" error. On the other end, when we try to pass the nested -ith cv fold as cv argument for clf, and we call fit on the same cv_nested fold, we get an "Index out of bound" error. 
> Two questions:
> 1) Is there any workaround to avoid the split when clf is called without a cv argument?
> 2) We suppose that for hyperparameter optimization the test data is removed from the dataset and a  new dataset is created. Is this true? In this case we only have to adjust the indices accordingly
>
> Thank your for your time and sorry for the long text
> Ludovico
> _______________________________________________
> scikit-learn mailing list
> scikit-learn@python.org
> https://mail.python.org/mailman/listinfo/scikit-learn




------------------------------

Subject: Digest Footer

_______________________________________________
scikit-learn mailing list
scikit-learn@python.org
https://mail.python.org/mailman/listinfo/scikit-learn



------------------------------

End of scikit-learn Digest, Vol 11, Issue 29
********************************************