<div dir="ltr"><div>The current roadmap is amazing. One feature that would be exciting is better support for multilayer stacking with caching and the ability to add models to already trained layers.</div><div><br></div><div>I saw this history: <a href="https://github.com/scikit-learn/scikit-learn/pull/8960">https://github.com/scikit-learn/scikit-learn/pull/8960</a></div><div><br></div><div>This library is very close:</div><div>* API is somewhat awkward, but otherwise good. Does not cache intermediate steps. <a href="https://wolpert.readthedocs.io/en/latest/index.html">https://wolpert.readthedocs.io/en/latest/index.html</a></div><div><br></div><div>These solutions seem to allow only two layers: </div><div>* <a href="https://github.com/scikit-learn/scikit-learn/issues/4816#issuecomment-217817717">https://github.com/scikit-learn/scikit-learn/issues/4816#issuecomment-217817717</a><br></div><div>* <a href="https://rasbt.github.io/mlxtend/user_guide/classifier/StackingClassifier/">https://rasbt.github.io/mlxtend/user_guide/classifier/StackingClassifier/</a><br></div><div>* <a href="https://github.com/scikit-learn/scikit-learn/pull/6674">https://github.com/scikit-learn/scikit-learn/pull/6674</a></div><div><br></div><div>The people who put these other libraries together have made an incredibly welcome effort to solve a real need and it would be amazing to see a payoff for their effort in the form of an addition of stacking to scikit-learn's core library. </div><div><br></div><div>As another data point, I attached a simple implementation I put together to illustrate what I think are core needs of this feature. Feel free to browse the code. Here is the short list:</div><div>* Infinite layers (or at least 3 ;) )</div><div>* Choice of CV or OOB for each model</div><div>* Ability to add a new model to a layer after the stacked ensemble has been trained and refit the pipeline such that only models that must be retrained are retrained (i.e. train the added model and retrain all models in higher layers)</div><div>* All standard scikit-learn pipeline goodness (introspection, grid search, serializability, etc)</div><div><br></div><div>Thanks all! This library is making a real difference for good in the lives of many people.</div><div><br></div><div>Jason</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Sep 28, 2018 at 11:35 AM <<a href="mailto:scikit-learn-request@python.org">scikit-learn-request@python.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send scikit-learn mailing list submissions to<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:scikit-learn-request@python.org" target="_blank">scikit-learn-request@python.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:scikit-learn-owner@python.org" target="_blank">scikit-learn-owner@python.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of scikit-learn digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: [ANN] Scikit-learn 0.20.0 (Sebastian Raschka)<br>
2. Re: [ANN] Scikit-learn 0.20.0 (Andreas Mueller)<br>
3. Re: [ANN] Scikit-learn 0.20.0 (Andreas Mueller)<br>
4. Re: [ANN] Scikit-learn 0.20.0 (Manuel CASTEJ?N LIMAS)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Fri, 28 Sep 2018 11:10:50 -0500<br>
From: Sebastian Raschka <<a href="mailto:mail@sebastianraschka.com" target="_blank">mail@sebastianraschka.com</a>><br>
To: Scikit-learn mailing list <<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a>><br>
Subject: Re: [scikit-learn] [ANN] Scikit-learn 0.20.0<br>
Message-ID:<br>
<<a href="mailto:EFC21EB5-CCC4-48CB-AEB5-2C938689EC5E@sebastianraschka.com" target="_blank">EFC21EB5-CCC4-48CB-AEB5-2C938689EC5E@sebastianraschka.com</a>><br>
Content-Type: text/plain; charset=us-ascii<br>
<br>
> <br>
> > I think model serialization should be a priority.<br>
> <br>
> There is also the ONNX specification that is gaining industrial adoption and that already includes open source exporters for several families of scikit-learn models:<br>
> <br>
> <a href="https://github.com/onnx/onnxmltools" rel="noreferrer" target="_blank">https://github.com/onnx/onnxmltools</a><br>
<br>
<br>
Didn't know about that. This is really nice! What do you think about referring to it under <a href="http://scikit-learn.org/stable/modules/model_persistence.html" rel="noreferrer" target="_blank">http://scikit-learn.org/stable/modules/model_persistence.html</a> to make people aware that this option exists?<br>
Would be happy to add a PR.<br>
<br>
Best,<br>
Sebastian<br>
<br>
<br>
<br>
> On Sep 28, 2018, at 9:30 AM, Olivier Grisel <<a href="mailto:olivier.grisel@ensta.org" target="_blank">olivier.grisel@ensta.org</a>> wrote:<br>
> <br>
> <br>
> > I think model serialization should be a priority.<br>
> <br>
> There is also the ONNX specification that is gaining industrial adoption and that already includes open source exporters for several families of scikit-learn models:<br>
> <br>
> <a href="https://github.com/onnx/onnxmltools" rel="noreferrer" target="_blank">https://github.com/onnx/onnxmltools</a><br>
> <br>
> -- <br>
> Olivier<br>
> _______________________________________________<br>
> scikit-learn mailing list<br>
> <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
> <a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Fri, 28 Sep 2018 13:38:39 -0400<br>
From: Andreas Mueller <<a href="mailto:t3kcit@gmail.com" target="_blank">t3kcit@gmail.com</a>><br>
To: <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
Subject: Re: [scikit-learn] [ANN] Scikit-learn 0.20.0<br>
Message-ID: <<a href="mailto:96edd381-2352-f183-486a-b86e395a78f6@gmail.com" target="_blank">96edd381-2352-f183-486a-b86e395a78f6@gmail.com</a>><br>
Content-Type: text/plain; charset=utf-8; format=flowed<br>
<br>
<br>
<br>
On 09/28/2018 12:10 PM, Sebastian Raschka wrote:<br>
>>> I think model serialization should be a priority.<br>
>> There is also the ONNX specification that is gaining industrial adoption and that already includes open source exporters for several families of scikit-learn models:<br>
>><br>
>> <a href="https://github.com/onnx/onnxmltools" rel="noreferrer" target="_blank">https://github.com/onnx/onnxmltools</a><br>
><br>
> Didn't know about that. This is really nice! What do you think about referring to it under <a href="http://scikit-learn.org/stable/modules/model_persistence.html" rel="noreferrer" target="_blank">http://scikit-learn.org/stable/modules/model_persistence.html</a> to make people aware that this option exists?<br>
> Would be happy to add a PR.<br>
><br>
><br>
I don't think an open source runtime has been announced yet (or they <br>
didn't email me like they promised lol).<br>
I'm quite excited about this as well.<br>
<br>
Javier:<br>
The problem is not so much storing the "model" but storing how to make <br>
predictions. Different versions could act differently<br>
on the same data structure - and the data structure could change. Both <br>
happen in scikit-learn.<br>
So if you want to make sure the right thing happens across versions, you <br>
either need to provide serialization and deserialization for<br>
every version and conversion between those or you need to provide a way <br>
to store the prediction function,<br>
which basically means you need a turing-complete language (that's what <br>
ONNX does).<br>
<br>
We basically said doing the first is not feasible within scikit-learn <br>
given our current amount of resources, and no-one<br>
has even tried doing it outside of scikit-learn (which would be possible).<br>
Implementing a complete prediction serialization language (the second <br>
option) is definitely outside the scope of sklearn.<br>
<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Fri, 28 Sep 2018 13:41:13 -0400<br>
From: Andreas Mueller <<a href="mailto:t3kcit@gmail.com" target="_blank">t3kcit@gmail.com</a>><br>
To: <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
Subject: Re: [scikit-learn] [ANN] Scikit-learn 0.20.0<br>
Message-ID: <<a href="mailto:4cfbb327-7489-70ff-8fa3-a21079ec0068@gmail.com" target="_blank">4cfbb327-7489-70ff-8fa3-a21079ec0068@gmail.com</a>><br>
Content-Type: text/plain; charset=utf-8; format=flowed<br>
<br>
<br>
<br>
On 09/28/2018 01:38 PM, Andreas Mueller wrote:<br>
><br>
><br>
> On 09/28/2018 12:10 PM, Sebastian Raschka wrote:<br>
>>>> I think model serialization should be a priority.<br>
>>> There is also the ONNX specification that is gaining industrial <br>
>>> adoption and that already includes open source exporters for several <br>
>>> families of scikit-learn models:<br>
>>><br>
>>> <a href="https://github.com/onnx/onnxmltools" rel="noreferrer" target="_blank">https://github.com/onnx/onnxmltools</a><br>
>><br>
>> Didn't know about that. This is really nice! What do you think about <br>
>> referring to it under <br>
>> <a href="http://scikit-learn.org/stable/modules/model_persistence.html" rel="noreferrer" target="_blank">http://scikit-learn.org/stable/modules/model_persistence.html</a> to make <br>
>> people aware that this option exists?<br>
>> Would be happy to add a PR.<br>
>><br>
>><br>
> I don't think an open source runtime has been announced yet (or they <br>
> didn't email me like they promised lol).<br>
> I'm quite excited about this as well.<br>
><br>
> Javier:<br>
> The problem is not so much storing the "model" but storing how to make <br>
> predictions. Different versions could act differently<br>
> on the same data structure - and the data structure could change. Both <br>
> happen in scikit-learn.<br>
> So if you want to make sure the right thing happens across versions, <br>
> you either need to provide serialization and deserialization for<br>
> every version and conversion between those or you need to provide a <br>
> way to store the prediction function,<br>
> which basically means you need a turing-complete language (that's what <br>
> ONNX does).<br>
><br>
> We basically said doing the first is not feasible within scikit-learn <br>
> given our current amount of resources, and no-one<br>
> has even tried doing it outside of scikit-learn (which would be <br>
> possible).<br>
> Implementing a complete prediction serialization language (the second <br>
> option) is definitely outside the scope of sklearn.<br>
><br>
><br>
Maybe we should add to the FAQ why serialization is hard?<br>
<br>
<br>
------------------------------<br>
<br>
Message: 4<br>
Date: Fri, 28 Sep 2018 20:34:43 +0200<br>
From: Manuel CASTEJ?N LIMAS <<a href="mailto:mcasl@unileon.es" target="_blank">mcasl@unileon.es</a>><br>
To: Scikit-learn user and developer mailing list<br>
<<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a>><br>
Subject: Re: [scikit-learn] [ANN] Scikit-learn 0.20.0<br>
Message-ID:<br>
<CAAQ3=<a href="mailto:UFntYo02YkR9YwrCjicb8A3cutpN47L4MYZWxeNNYP%2B1A@mail.gmail.com" target="_blank">UFntYo02YkR9YwrCjicb8A3cutpN47L4MYZWxeNNYP+1A@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
How about a docker based approach? Just thinking out loud<br>
Best<br>
Manuel<br>
<br>
El vie., 28 sept. 2018 19:43, Andreas Mueller <<a href="mailto:t3kcit@gmail.com" target="_blank">t3kcit@gmail.com</a>> escribi?:<br>
<br>
><br>
><br>
> On 09/28/2018 01:38 PM, Andreas Mueller wrote:<br>
> ><br>
> ><br>
> > On 09/28/2018 12:10 PM, Sebastian Raschka wrote:<br>
> >>>> I think model serialization should be a priority.<br>
> >>> There is also the ONNX specification that is gaining industrial<br>
> >>> adoption and that already includes open source exporters for several<br>
> >>> families of scikit-learn models:<br>
> >>><br>
> >>> <a href="https://github.com/onnx/onnxmltools" rel="noreferrer" target="_blank">https://github.com/onnx/onnxmltools</a><br>
> >><br>
> >> Didn't know about that. This is really nice! What do you think about<br>
> >> referring to it under<br>
> >> <a href="http://scikit-learn.org/stable/modules/model_persistence.html" rel="noreferrer" target="_blank">http://scikit-learn.org/stable/modules/model_persistence.html</a> to make<br>
> >> people aware that this option exists?<br>
> >> Would be happy to add a PR.<br>
> >><br>
> >><br>
> > I don't think an open source runtime has been announced yet (or they<br>
> > didn't email me like they promised lol).<br>
> > I'm quite excited about this as well.<br>
> ><br>
> > Javier:<br>
> > The problem is not so much storing the "model" but storing how to make<br>
> > predictions. Different versions could act differently<br>
> > on the same data structure - and the data structure could change. Both<br>
> > happen in scikit-learn.<br>
> > So if you want to make sure the right thing happens across versions,<br>
> > you either need to provide serialization and deserialization for<br>
> > every version and conversion between those or you need to provide a<br>
> > way to store the prediction function,<br>
> > which basically means you need a turing-complete language (that's what<br>
> > ONNX does).<br>
> ><br>
> > We basically said doing the first is not feasible within scikit-learn<br>
> > given our current amount of resources, and no-one<br>
> > has even tried doing it outside of scikit-learn (which would be<br>
> > possible).<br>
> > Implementing a complete prediction serialization language (the second<br>
> > option) is definitely outside the scope of sklearn.<br>
> ><br>
> ><br>
> Maybe we should add to the FAQ why serialization is hard?<br>
> _______________________________________________<br>
> scikit-learn mailing list<br>
> <a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
> <a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://mail.python.org/pipermail/scikit-learn/attachments/20180928/f52258e8/attachment.html" rel="noreferrer" target="_blank">http://mail.python.org/pipermail/scikit-learn/attachments/20180928/f52258e8/attachment.html</a>><br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
scikit-learn mailing list<br>
<a href="mailto:scikit-learn@python.org" target="_blank">scikit-learn@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/scikit-learn" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/scikit-learn</a><br>
<br>
<br>
------------------------------<br>
<br>
End of scikit-learn Digest, Vol 30, Issue 25<br>
********************************************<br>
</blockquote></div>