[scikit-learn] [GSoC 2017] First Draft, request for suggestions - Improve Online Learning of Linear Models.

[Andreas Mueller]

On 03/15/2017 04:48 AM, Karan Desai wrote:
> 4. About a tool to anneal learning rate: I suggest a new approach to 
> look at this - as a callback. I searched through the documentation and 
> I could not find this way of handling tidbits during training of 
> models. We should be able to provide a callback to the constructor of 
> a linear model which can do any dedicated job after every epoch, be it 
> learning rate annealing, saving model checkpoint, getting custom 
> verbose output, or as creative as uploading data to server for real 
> time plots on any website.
There has been some effort on doing adagrad but it was ultimately 
discontinued, I think.
There was quite a bit of complexity to handle.
The problem with callbacks is that for callbacks on each iteration to be 
feasible, they need to be cython functions.
Otherwise they will be too slow. You could do python callbacks, but they 
could not be called at every iteration, and so
they wouldn't be suitable to implement something like adagrad or adam.

Best,
Andy