[scikit-learn] custom loss function
Sebastian Raschka
se.raschka at gmail.com
Wed Sep 13 13:49:26 EDT 2017
> Is it possible to change the loss function in KerasRegressor? I don't have time right now to experiment with hyperparameters of new ANN architectures. I am in urgent need to reproduce in Keras the results obtained with MLPRegressor and the set of hyperparameters that I have optimized for my problem and later change the loss function
Honestly, I don't have much experience with Keras. It may be easy to do that, I don't know.
Alternatively, defining an MLP regressor in TensorFlow is not that hard and only few lines of code. E.g., you could copy the mlp classifier from (cell 4) here:
https://github.com/rasbt/deep-learning-book/blob/master/code/model_zoo/multilayer-perceptron-lowlevel.ipynb
just delete the two last ops in the output layer
out_act = tf.nn.softmax(out_z, name='predicted_probabilities')
out_labels = tf.argmax(out_z, axis=1, name='predicted_labels'))
and replace the loss/cost by
tf.losses.mean_squared_error
and you should have a MLP regressor running in a few lines of code. Then, you could experiment with your loss function by doing your own function. E.g., the usage is quite similar to what you do in NumPy, the mean_squared_error above can be manually defined as e.g.,
cost = tf.reduce_sum(tf.pow(pred-y 2))/(2*n_samples)
Best,
Sebastian
> On Sep 13, 2017, at 1:18 PM, Thomas Evangelidis <tevang3 at gmail.com> wrote:
>
>
> Thanks again for the clarifications Sebastian!
>
> Keras has a Scikit-learn API with the KeraRegressor which implements the Scikit-Learn MLPRegressor interface:
>
> https://keras.io/scikit-learn-api/
>
> Is it possible to change the loss function in KerasRegressor? I don't have time right now to experiment with hyperparameters of new ANN architectures. I am in urgent need to reproduce in Keras the results obtained with MLPRegressor and the set of hyperparameters that I have optimized for my problem and later change the loss function.
>
>
>
> On 13 September 2017 at 18:14, Sebastian Raschka <se.raschka at gmail.com> wrote:
> > What about the SVR? Is it possible to change the loss function there?
>
> Here you would have the same problem; SVR is a constrained optimization problem and you would have to change the calculation of the loss gradient then. Since SVR is a "1-layer" neural net, if you change the cost function to something else, it's not really a SVR anymore.
>
>
> > Could you please clarify what the "x" and "x'" parameters in the default Kernel functions mean? Is "x" a NxM array, where N is the number of observations and M the number of features?
>
> Both x and x' should be denoting training examples. The kernel matrix is symmetric (N x N).
>
>
>
> Best,
> Sebastian
>
> > On Sep 13, 2017, at 5:25 AM, Thomas Evangelidis <tevang3 at gmail.com> wrote:
> >
> > Thanks Sebastian. Exploring Tensorflow capabilities was in my TODO list, but now it's in my immediate plans.
> > What about the SVR? Is it possible to change the loss function there? Could you please clarify what the "x" and "x'" parameters in the default Kernel functions mean? Is "x" a NxM array, where N is the number of observations and M the number of features?
> >
> > http://scikit-learn.org/stable/modules/svm.html#kernel-functions
> >
> >
> >
> > On 12 September 2017 at 00:37, Sebastian Raschka <se.raschka at gmail.com> wrote:
> > Hi Thomas,
> >
> > > For the MLPRegressor case so far my conclusion was that it is not possible unless you modify the source code.
> >
> > Also, I suspect that this would be non-trivial. I haven't looked to closely at how the MLPClassifier/MLPRegressor are implemented but since you perform the weight updates based on the gradient of the cost function wrt the weights, the modification would be non-trivial if the partial derivatives are not computed based on some autodiff implementation -- you would have to edit all the partial d's along the backpropagation up to the first hidden layer. While I think that scikit-learn is by far the best library out there for machine learning, I think if you want an easy solution, you probably won't get around TensorFlow or PyTorch or equivalent, here, for your specific MLP problem unless you want to make your life extra hard :P (seriously, you can pick up any of the two in about an hour and have your MLPRegressor up and running so that you can then experiment with your cost function).
> >
> > Best,
> > Sebastian
> >
> > > On Sep 11, 2017, at 6:13 PM, Thomas Evangelidis <tevang3 at gmail.com> wrote:
> > >
> > > Greetings,
> > >
> > > I know this is a recurrent question, but I would like to use my own loss function either in a MLPRegressor or in an SVR. For the MLPRegressor case so far my conclusion was that it is not possible unless you modify the source code. On the other hand, for the SVR I was looking at setting custom kernel functions. But I am not sure if this is the same thing. Could someone please clarify this to me? Finally, I read about the "scoring" parameter is cross-validation, but this is just to select a Regressor that has been trained already with the default loss function, so it would be harder to find one that minimizes my own loss function.
> > >
> > > For the record, my loss function is the centered root mean square error.
> > >
> > > Thanks in advance for any advice.
> > >
> > >
> > >
> > > --
> > > ======================================================================
> > > Dr Thomas Evangelidis
> > > Post-doctoral Researcher
> > > CEITEC - Central European Institute of Technology
> > > Masaryk University
> > > Kamenice 5/A35/2S049,
> > > 62500 Brno, Czech Republic
> > >
> > > email: tevang at pharm.uoa.gr
> > > tevang3 at gmail.com
> > >
> > > website: https://sites.google.com/site/thomasevangelidishomepage/
> > >
> > >
> > > _______________________________________________
> > > scikit-learn mailing list
> > > scikit-learn at python.org
> > > https://mail.python.org/mailman/listinfo/scikit-learn
> >
> > _______________________________________________
> > scikit-learn mailing list
> > scikit-learn at python.org
> > https://mail.python.org/mailman/listinfo/scikit-learn
> >
> >
> >
> > --
> > ======================================================================
> > Dr Thomas Evangelidis
> > Post-doctoral Researcher
> > CEITEC - Central European Institute of Technology
> > Masaryk University
> > Kamenice 5/A35/2S049,
> > 62500 Brno, Czech Republic
> >
> > email: tevang at pharm.uoa.gr
> > tevang3 at gmail.com
> >
> > website: https://sites.google.com/site/thomasevangelidishomepage/
> >
> >
> > _______________________________________________
> > scikit-learn mailing list
> > scikit-learn at python.org
> > https://mail.python.org/mailman/listinfo/scikit-learn
>
> _______________________________________________
> scikit-learn mailing list
> scikit-learn at python.org
> https://mail.python.org/mailman/listinfo/scikit-learn
>
>
>
> --
> ======================================================================
> Dr Thomas Evangelidis
> Post-doctoral Researcher
> CEITEC - Central European Institute of Technology
> Masaryk University
> Kamenice 5/A35/2S049,
> 62500 Brno, Czech Republic
>
> email: tevang at pharm.uoa.gr
> tevang3 at gmail.com
>
> website: https://sites.google.com/site/thomasevangelidishomepage/
>
>
> _______________________________________________
> scikit-learn mailing list
> scikit-learn at python.org
> https://mail.python.org/mailman/listinfo/scikit-learn
More information about the scikit-learn
mailing list