Specifying exceptions to ParameterGrid
Hi, Sometimes when using GridSearchCV, I realize that in the grid there are certain combinations of hyperparameters that are either incompatible or redundant. For example, when using an MLP, if I specify the following grid: grid = {'solver': ['sgd', 'adam'], 'learning_rate': ['constant', 'invscaling', 'adaptive']} then it yields the following ParameterGrid: [{'learning_rate': 'constant', 'solver': 'sgd'}, {'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'sgd'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'sgd'}, {'learning_rate': 'adaptive', 'solver': 'adam'}] Now, three of these are redundant, since learning_rate is used only for the sgd solver. Ideally I'd like to specify these cases upfront, and for that I have a simple hack ( https://github.com/jaidevd/jarvis/blob/master/jarvis/cross_validation.py#L38). Using that yields a ParameterGrid as follows: [{'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'adam'}] which is then simply removed from the original ParameterGrid. I wonder if there's a simpler way of doing this. Would it help if we had an additional parameter (something like "grid_exceptions") in GridSearchCV, which would remove these dicts from the list of parameters? Thanks
Hi Jaidev, well, `param_grid` in GridSearchCV can also be a list of dictionaries, so you could directly specify the cases you are interested in (instead of the full grid - exceptions), which might be simpler? On 23/11/16 11:15, Jaidev Deshpande wrote:
Hi,
Sometimes when using GridSearchCV, I realize that in the grid there are certain combinations of hyperparameters that are either incompatible or redundant. For example, when using an MLP, if I specify the following grid:
grid = {'solver': ['sgd', 'adam'], 'learning_rate': ['constant', 'invscaling', 'adaptive']}
then it yields the following ParameterGrid:
[{'learning_rate': 'constant', 'solver': 'sgd'}, {'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'sgd'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'sgd'}, {'learning_rate': 'adaptive', 'solver': 'adam'}]
Now, three of these are redundant, since learning_rate is used only for the sgd solver. Ideally I'd like to specify these cases upfront, and for that I have a simple hack (https://github.com/jaidevd/jarvis/blob/master/jarvis/cross_validation.py#L38). Using that yields a ParameterGrid as follows:
[{'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'adam'}]
which is then simply removed from the original ParameterGrid.
I wonder if there's a simpler way of doing this. Would it help if we had an additional parameter (something like "grid_exceptions") in GridSearchCV, which would remove these dicts from the list of parameters?
Thanks
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
On Wed, 23 Nov 2016 at 19:05 Roman Yurchak <rth.yurchak@gmail.com> wrote:
Hi Jaidev,
well, `param_grid` in GridSearchCV can also be a list of dictionaries, so you could directly specify the cases you are interested in (instead of the full grid - exceptions), which might be simpler?
Actually now that I think of it, I don't know if it will be necessarily simpler. What if I have a massive grid and only few exceptions? Enumerating the complement of that small subset would be much more expensive than specifying the exceptions. What do you think?
On 23/11/16 11:15, Jaidev Deshpande wrote:
Hi,
Sometimes when using GridSearchCV, I realize that in the grid there are certain combinations of hyperparameters that are either incompatible or redundant. For example, when using an MLP, if I specify the following grid:
grid = {'solver': ['sgd', 'adam'], 'learning_rate': ['constant', 'invscaling', 'adaptive']}
then it yields the following ParameterGrid:
[{'learning_rate': 'constant', 'solver': 'sgd'}, {'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'sgd'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'sgd'}, {'learning_rate': 'adaptive', 'solver': 'adam'}]
Now, three of these are redundant, since learning_rate is used only for the sgd solver. Ideally I'd like to specify these cases upfront, and for that I have a simple hack ( https://github.com/jaidevd/jarvis/blob/master/jarvis/cross_validation.py#L38 ). Using that yields a ParameterGrid as follows:
[{'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'adam'}]
which is then simply removed from the original ParameterGrid.
I wonder if there's a simpler way of doing this. Would it help if we had an additional parameter (something like "grid_exceptions") in GridSearchCV, which would remove these dicts from the list of parameters?
Thanks
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
On 24/11/16 09:00, Jaidev Deshpande wrote:
well, `param_grid` in GridSearchCV can also be a list of dictionaries, so you could directly specify the cases you are interested in (instead of the full grid - exceptions), which might be simpler?
Actually now that I think of it, I don't know if it will be necessarily simpler. What if I have a massive grid and only few exceptions? Enumerating the complement of that small subset would be much more expensive than specifying the exceptions.
The solution indicated by Raghav is most concise if that works for you. Otherwise, in general, if you want to define the parameters as the full grid with a few exceptions, without changing the GirdSearchCV API, you could always try something like, ``` from sklearn.model_selection import GridSearchCV, ParameterGrid from sklearn.neural_network import MLPClassifier grid_full = {'solver': ['sgd', 'adam'], 'learning_rate': ['constant', 'invscaling', 'adaptive']} def exception_handler(args): # custom function shaping the domain of valid parameters if args['solver'] == 'adam' and args['learning_rate'] != 'constant': return False else: return True def wrap_strings(args): # all values of dicts provided to GridSearchCV must be lists return {key: [val] for key, val in args.items()} grid_tmp = filter(exception_handler, ParameterGrid(grid_full)) grid = [wrap_strings(el) for el in grid_tmp] gs = GridSearchCV(MLPClassifier(random_state=42), param_grid=grid) ``` That's quite similar to what you were suggesting in the original post.
On Fri, 25 Nov 2016 at 20:24 Roman Yurchak <rth.yurchak@gmail.com> wrote:
On 24/11/16 09:00, Jaidev Deshpande wrote:
well, `param_grid` in GridSearchCV can also be a list of
dictionaries,
so you could directly specify the cases you are interested in
(instead
of the full grid - exceptions), which might be simpler?
Actually now that I think of it, I don't know if it will be necessarily simpler. What if I have a massive grid and only few exceptions? Enumerating the complement of that small subset would be much more expensive than specifying the exceptions.
The solution indicated by Raghav is most concise if that works for you.
Otherwise, in general, if you want to define the parameters as the full grid with a few exceptions, without changing the GirdSearchCV API, you could always try something like,
``` from sklearn.model_selection import GridSearchCV, ParameterGrid from sklearn.neural_network import MLPClassifier
grid_full = {'solver': ['sgd', 'adam'], 'learning_rate': ['constant', 'invscaling', 'adaptive']}
def exception_handler(args): # custom function shaping the domain of valid parameters if args['solver'] == 'adam' and args['learning_rate'] != 'constant': return False else: return True
def wrap_strings(args): # all values of dicts provided to GridSearchCV must be lists return {key: [val] for key, val in args.items()}
grid_tmp = filter(exception_handler, ParameterGrid(grid_full)) grid = [wrap_strings(el) for el in grid_tmp]
gs = GridSearchCV(MLPClassifier(random_state=42), param_grid=grid) ``` That's quite similar to what you were suggesting in the original post.
Yes, also a lot more concise I guess. This way I just have to keep writing an exception handler instead of subclassing. Thanks!
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
Hi! What you could do is specify lists of dicts to group the parameters which apply together in one dict... [{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': 'sgd'}, {'solver': 'adam'}] ```py from sklearn.neural_network import MLPClassifier from sklearn.model_selection import GridSearchCV from sklearn.datasets import make_classification from pandas import DataFrame X, y = make_classification(random_state=42) gs = GridSearchCV(MLPClassifier(random_state=42), param_grid=[{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': ['sgd',]}, {'solver': ['adam',]}]) DataFrame(gs.fit(X, y).cv_results_) ``` Would give [image: Inline image 1] HTH :) On Wed, Nov 23, 2016 at 11:15 AM, Jaidev Deshpande < deshpande.jaidev@gmail.com> wrote:
Hi,
Sometimes when using GridSearchCV, I realize that in the grid there are certain combinations of hyperparameters that are either incompatible or redundant. For example, when using an MLP, if I specify the following grid:
grid = {'solver': ['sgd', 'adam'], 'learning_rate': ['constant', 'invscaling', 'adaptive']}
then it yields the following ParameterGrid:
[{'learning_rate': 'constant', 'solver': 'sgd'}, {'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'sgd'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'sgd'}, {'learning_rate': 'adaptive', 'solver': 'adam'}]
Now, three of these are redundant, since learning_rate is used only for the sgd solver. Ideally I'd like to specify these cases upfront, and for that I have a simple hack (https://github.com/jaidevd/ja rvis/blob/master/jarvis/cross_validation.py#L38). Using that yields a ParameterGrid as follows:
[{'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'adam'}]
which is then simply removed from the original ParameterGrid.
I wonder if there's a simpler way of doing this. Would it help if we had an additional parameter (something like "grid_exceptions") in GridSearchCV, which would remove these dicts from the list of parameters?
Thanks
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
-- Raghav RV https://github.com/raghavrv
On Wed, 23 Nov 2016 at 16:29 Raghav R V <ragvrv@gmail.com> wrote:
Hi!
What you could do is specify lists of dicts to group the parameters which apply together in one dict...
[{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': 'sgd'}, {'solver': 'adam'}]
```py from sklearn.neural_network import MLPClassifier from sklearn.model_selection import GridSearchCV from sklearn.datasets import make_classification
from pandas import DataFrame
X, y = make_classification(random_state=42)
gs = GridSearchCV(MLPClassifier(random_state=42), param_grid=[{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': ['sgd',]}, {'solver': ['adam',]}])
DataFrame(gs.fit(X, y).cv_results_) ```
Would give
[image: image.png]
HTH :)
Haha, this is perfect. I didn't know you could pass a list of dicts to param_grid. Thanks!
On Wed, Nov 23, 2016 at 11:15 AM, Jaidev Deshpande < deshpande.jaidev@gmail.com> wrote:
Hi,
Sometimes when using GridSearchCV, I realize that in the grid there are certain combinations of hyperparameters that are either incompatible or redundant. For example, when using an MLP, if I specify the following grid:
grid = {'solver': ['sgd', 'adam'], 'learning_rate': ['constant', 'invscaling', 'adaptive']}
then it yields the following ParameterGrid:
[{'learning_rate': 'constant', 'solver': 'sgd'}, {'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'sgd'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'sgd'}, {'learning_rate': 'adaptive', 'solver': 'adam'}]
Now, three of these are redundant, since learning_rate is used only for the sgd solver. Ideally I'd like to specify these cases upfront, and for that I have a simple hack ( https://github.com/jaidevd/jarvis/blob/master/jarvis/cross_validation.py#L38). Using that yields a ParameterGrid as follows:
[{'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'adam'}]
which is then simply removed from the original ParameterGrid.
I wonder if there's a simpler way of doing this. Would it help if we had an additional parameter (something like "grid_exceptions") in GridSearchCV, which would remove these dicts from the list of parameters?
Thanks
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
-- Raghav RV https://github.com/raghavrv
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
Raghav's example of [{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': 'sgd'}, {'solver': 'adam'}] was not correct. Should be [{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': ['sgd']}, {'solver': ['adam']}] (Note all values of dicts are lists) On 23 November 2016 at 22:52, Jaidev Deshpande <deshpande.jaidev@gmail.com> wrote:
On Wed, 23 Nov 2016 at 16:29 Raghav R V <ragvrv@gmail.com> wrote:
Hi!
What you could do is specify lists of dicts to group the parameters which apply together in one dict...
[{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': 'sgd'}, {'solver': 'adam'}]
```py from sklearn.neural_network import MLPClassifier from sklearn.model_selection import GridSearchCV from sklearn.datasets import make_classification
from pandas import DataFrame
X, y = make_classification(random_state=42)
gs = GridSearchCV(MLPClassifier(random_state=42), param_grid=[{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': ['sgd',]}, {'solver': ['adam',]}])
DataFrame(gs.fit(X, y).cv_results_) ```
Would give
[image: image.png]
HTH :)
Haha, this is perfect. I didn't know you could pass a list of dicts to param_grid.
Thanks!
On Wed, Nov 23, 2016 at 11:15 AM, Jaidev Deshpande < deshpande.jaidev@gmail.com> wrote:
Hi,
Sometimes when using GridSearchCV, I realize that in the grid there are certain combinations of hyperparameters that are either incompatible or redundant. For example, when using an MLP, if I specify the following grid:
grid = {'solver': ['sgd', 'adam'], 'learning_rate': ['constant', 'invscaling', 'adaptive']}
then it yields the following ParameterGrid:
[{'learning_rate': 'constant', 'solver': 'sgd'}, {'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'sgd'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'sgd'}, {'learning_rate': 'adaptive', 'solver': 'adam'}]
Now, three of these are redundant, since learning_rate is used only for the sgd solver. Ideally I'd like to specify these cases upfront, and for that I have a simple hack (https://github.com/jaidevd/ jarvis/blob/master/jarvis/cross_validation.py#L38). Using that yields a ParameterGrid as follows:
[{'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'adam'}]
which is then simply removed from the original ParameterGrid.
I wonder if there's a simpler way of doing this. Would it help if we had an additional parameter (something like "grid_exceptions") in GridSearchCV, which would remove these dicts from the list of parameters?
Thanks
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
-- Raghav RV https://github.com/raghavrv
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
On Wed, 23 Nov 2016 at 17:31 Joel Nothman <joel.nothman@gmail.com> wrote:
Raghav's example of
[{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': 'sgd'}, {'solver': 'adam'}]
was not correct.
Should be
[{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': ['sgd']}, {'solver': ['adam']}]
(Note all values of dicts are lists)
Ah, thanks! (Just ran into an error as it started iterating over the "sgd".)
On 23 November 2016 at 22:52, Jaidev Deshpande <deshpande.jaidev@gmail.com
wrote:
On Wed, 23 Nov 2016 at 16:29 Raghav R V <ragvrv@gmail.com> wrote:
Hi!
What you could do is specify lists of dicts to group the parameters which apply together in one dict...
[{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': 'sgd'}, {'solver': 'adam'}]
```py from sklearn.neural_network import MLPClassifier from sklearn.model_selection import GridSearchCV from sklearn.datasets import make_classification
from pandas import DataFrame
X, y = make_classification(random_state=42)
gs = GridSearchCV(MLPClassifier(random_state=42), param_grid=[{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': ['sgd',]}, {'solver': ['adam',]}])
DataFrame(gs.fit(X, y).cv_results_) ```
Would give
[image: image.png]
HTH :)
Haha, this is perfect. I didn't know you could pass a list of dicts to param_grid.
Thanks!
On Wed, Nov 23, 2016 at 11:15 AM, Jaidev Deshpande < deshpande.jaidev@gmail.com> wrote:
Hi,
Sometimes when using GridSearchCV, I realize that in the grid there are certain combinations of hyperparameters that are either incompatible or redundant. For example, when using an MLP, if I specify the following grid:
grid = {'solver': ['sgd', 'adam'], 'learning_rate': ['constant', 'invscaling', 'adaptive']}
then it yields the following ParameterGrid:
[{'learning_rate': 'constant', 'solver': 'sgd'}, {'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'sgd'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'sgd'}, {'learning_rate': 'adaptive', 'solver': 'adam'}]
Now, three of these are redundant, since learning_rate is used only for the sgd solver. Ideally I'd like to specify these cases upfront, and for that I have a simple hack ( https://github.com/jaidevd/jarvis/blob/master/jarvis/cross_validation.py#L38). Using that yields a ParameterGrid as follows:
[{'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'adam'}]
which is then simply removed from the original ParameterGrid.
I wonder if there's a simpler way of doing this. Would it help if we had an additional parameter (something like "grid_exceptions") in GridSearchCV, which would remove these dicts from the list of parameters?
Thanks
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
-- Raghav RV https://github.com/raghavrv
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
On Wed, Nov 23, 2016 at 12:59 PM, Joel Nothman <joel.nothman@gmail.com> wrote:
Raghav's example of
[{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': 'sgd'}, {'solver': 'adam'}]
was not correct.
Oops sorry. Ah I ran into that, corrected it in the snipped but forgot to update the line before the snippet... :)
Should be
[{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': ['sgd']}, {'solver': ['adam']}]
(Note all values of dicts are lists)
On 23 November 2016 at 22:52, Jaidev Deshpande <deshpande.jaidev@gmail.com
wrote:
On Wed, 23 Nov 2016 at 16:29 Raghav R V <ragvrv@gmail.com> wrote:
Hi!
What you could do is specify lists of dicts to group the parameters which apply together in one dict...
[{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': 'sgd'}, {'solver': 'adam'}]
```py from sklearn.neural_network import MLPClassifier from sklearn.model_selection import GridSearchCV from sklearn.datasets import make_classification
from pandas import DataFrame
X, y = make_classification(random_state=42)
gs = GridSearchCV(MLPClassifier(random_state=42), param_grid=[{'learning_rate': ['constant', 'invscaling', 'adaptive'], 'solver': ['sgd',]}, {'solver': ['adam',]}])
DataFrame(gs.fit(X, y).cv_results_) ```
Would give
[image: image.png]
HTH :)
Haha, this is perfect. I didn't know you could pass a list of dicts to param_grid.
Thanks!
On Wed, Nov 23, 2016 at 11:15 AM, Jaidev Deshpande < deshpande.jaidev@gmail.com> wrote:
Hi,
Sometimes when using GridSearchCV, I realize that in the grid there are certain combinations of hyperparameters that are either incompatible or redundant. For example, when using an MLP, if I specify the following grid:
grid = {'solver': ['sgd', 'adam'], 'learning_rate': ['constant', 'invscaling', 'adaptive']}
then it yields the following ParameterGrid:
[{'learning_rate': 'constant', 'solver': 'sgd'}, {'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'sgd'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'sgd'}, {'learning_rate': 'adaptive', 'solver': 'adam'}]
Now, three of these are redundant, since learning_rate is used only for the sgd solver. Ideally I'd like to specify these cases upfront, and for that I have a simple hack (https://github.com/jaidevd/ja rvis/blob/master/jarvis/cross_validation.py#L38). Using that yields a ParameterGrid as follows:
[{'learning_rate': 'constant', 'solver': 'adam'}, {'learning_rate': 'invscaling', 'solver': 'adam'}, {'learning_rate': 'adaptive', 'solver': 'adam'}]
which is then simply removed from the original ParameterGrid.
I wonder if there's a simpler way of doing this. Would it help if we had an additional parameter (something like "grid_exceptions") in GridSearchCV, which would remove these dicts from the list of parameters?
Thanks
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
-- Raghav RV https://github.com/raghavrv
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
-- Raghav RV https://github.com/raghavrv
participants (4)
-
Jaidev Deshpande -
Joel Nothman -
Raghav R V -
Roman Yurchak