<div dir="ltr">Just a quick ping to share that I've kept playing with this PipeGraph toy. The following example reflects its current state.<div><br></div><div>* As you can see scikit-learn models can be used as steps in the nodes of the graph just by saying so, for example:</div><div><div><span style="font-family:monospace,monospace"><br></span></div><div><span style="font-family:monospace,monospace">'Gaussian_Mixture':</span><br></div><div><font face="monospace, monospace"> {'step': GaussianMixture,</font></div><div><font face="monospace, monospace"> 'kargs': {'n_components': 3},</font></div><div><font face="monospace, monospace"> 'connections': {'X': ('Concatenate_Xy', 'Xy')},</font></div><div><font face="monospace, monospace"> 'use_for': ['fit'],</font></div><div><font face="monospace, monospace"> },</font></div><div><br></div></div><div>* Custom steps need succint declarations with very little code</div><div><div><br></div></div><div>* Graph description is nice to read, in my humble opinion.<br><div><br></div><div>* Optional 'fit' and/or 'run' roles</div><div><br></div><div>* TO-DO: Using memory option to cache and making it compatible with gridSearchCv. I was too busy playing with template methods in order to simplify its use.</div><div><br></div><div>I have convinced some nice colleagues at my university to team up with me and write some nice documentation</div><div><br></div><div>Best wishes</div><div>Manolo</div><div><br></div><div><br></div><div><div><font face="monospace, monospace">import pandas as pd</font></div><div><font face="monospace, monospace">import numpy as np</font></div><div><font face="monospace, monospace">from sklearn.cluster import DBSCAN </font></div><div><font face="monospace, monospace">from sklearn.mixture import GaussianMixture</font></div><div><font face="monospace, monospace">from sklearn.linear_model import LinearRegression</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"># work in progress library: <a href="https://github.com/mcasl/PAELLA/">https://github.com/mcasl/PAELLA/</a></font></div><div><font face="monospace, monospace">from pipeGraph import (PipeGraph,</font></div><div><font face="monospace, monospace"> FirstStep,</font></div><div><font face="monospace, monospace"> LastStep,</font></div><div><font face="monospace, monospace"> CustomStep)</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">from paella import Paella</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">URL = "<a href="https://raw.githubusercontent.com/mcasl/PAELLA/master/data/sin_60_percent_noise.csv">https://raw.githubusercontent.com/mcasl/PAELLA/master/data/sin_60_percent_noise.csv</a>"</font></div><div><font face="monospace, monospace">data = pd.read_csv(URL, usecols=['V1', 'V2'])</font></div><div><font face="monospace, monospace">X, y = data[['V1']], data[['V2']]</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">class CustomConcatenationStep(CustomStep):</font></div><div><font face="monospace, monospace"> def _post_fit(self):</font></div><div><font face="monospace, monospace"> self.output['Xy'] = pd.concat(self.input, axis=1)</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">class CustomCombinationStep(CustomStep):</font></div><div><font face="monospace, monospace"> def _post_fit(self):</font></div><div><font face="monospace, monospace"> self.output['classification'] = np.where(self.input['dominant'] < 0, self.input['dominant'],</font></div><div><font face="monospace, monospace"> self.input['other'])</font></div><div><font face="monospace, monospace">class CustomPaellaStep(CustomStep):</font></div><div><font face="monospace, monospace"> def _pre_fit(self):</font></div><div><font face="monospace, monospace"> self.sklearn_object = Paella(**self.kargs)</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> def _fit(self):</font></div><div><font face="monospace, monospace"> self.sklearn_object.fit(**self.input)</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> def _post_fit(self):</font></div><div><font face="monospace, monospace"> self.output['prediction'] = self.sklearn_object.transform(self.input['X'], self.input['y'])</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">graph_description = {</font></div><div><font face="monospace, monospace"> 'First':</font></div><div><font face="monospace, monospace"> {'step': FirstStep,</font></div><div><font face="monospace, monospace"> 'connections': {'X': X,</font></div><div><font face="monospace, monospace"> 'y': y},</font></div><div><font face="monospace, monospace"> 'use_for': ['fit', 'run'],</font></div><div><font face="monospace, monospace"> },</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> 'Concatenate_Xy':</font></div><div><font face="monospace, monospace"> {'step': CustomConcatenationStep,</font></div><div><font face="monospace, monospace"> 'connections': {'df1': ('First', 'X'),</font></div><div><font face="monospace, monospace"> 'df2': ('First', 'y')},</font></div><div><font face="monospace, monospace"> 'use_for': ['fit'],</font></div><div><font face="monospace, monospace"> },</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> 'Gaussian_Mixture':</font></div><div><font face="monospace, monospace"> {'step': GaussianMixture,</font></div><div><font face="monospace, monospace"> 'kargs': {'n_components': 3},</font></div><div><font face="monospace, monospace"> 'connections': {'X': ('Concatenate_Xy', 'Xy')},</font></div><div><font face="monospace, monospace"> 'use_for': ['fit'],</font></div><div><font face="monospace, monospace"> },</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> 'Dbscan':</font></div><div><font face="monospace, monospace"> {'step': DBSCAN,</font></div><div><font face="monospace, monospace"> 'kargs': {'eps': 0.05},</font></div><div><font face="monospace, monospace"> 'connections': {'X': ('Concatenate_Xy', 'Xy')},</font></div><div><font face="monospace, monospace"> 'use_for': ['fit'],</font></div><div><font face="monospace, monospace"> },</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> 'Combine_Clustering':</font></div><div><font face="monospace, monospace"> {'step': CustomCombinationStep,</font></div><div><font face="monospace, monospace"> 'connections': {'dominant': ('Dbscan', 'prediction'),</font></div><div><font face="monospace, monospace"> 'other': ('Gaussian_Mixture', 'prediction')},</font></div><div><font face="monospace, monospace"> 'use_for': ['fit'],</font></div><div><font face="monospace, monospace"> },</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> 'Paella':</font></div><div><font face="monospace, monospace"> {'step': CustomPaellaStep,</font></div><div><font face="monospace, monospace"> 'kargs': {'noise_label': -1,</font></div><div><font face="monospace, monospace"> 'max_it': 20,</font></div><div><font face="monospace, monospace"> 'regular_size': 400,</font></div><div><font face="monospace, monospace"> 'minimum_size': 100,</font></div><div><font face="monospace, monospace"> 'width_r': 0.99,</font></div><div><font face="monospace, monospace"> 'n_neighbors': 5,</font></div><div><font face="monospace, monospace"> 'power': 30,</font></div><div><font face="monospace, monospace"> 'random_state': None},</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> 'connections': {'X': ('First', 'X'),</font></div><div><font face="monospace, monospace"> 'y': ('First', 'y'),</font></div><div><font face="monospace, monospace"> 'classification': ('Combine_Clustering', 'classification')},</font></div><div><font face="monospace, monospace"> 'use_for': ['fit'],</font></div><div><font face="monospace, monospace"> },</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> 'Regressor':</font></div><div><font face="monospace, monospace"> {'step': LinearRegression,</font></div><div><font face="monospace, monospace"> 'kargs': {},</font></div><div><font face="monospace, monospace"> 'connections': {'X': ('First', 'X'),</font></div><div><font face="monospace, monospace"> 'y': ('First', 'y'),</font></div><div><font face="monospace, monospace"> 'sample_weight': ('Paella', 'prediction')},</font></div><div><font face="monospace, monospace"> 'use_for': ['fit', 'run'],</font></div><div><font face="monospace, monospace"> },</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"> 'Last':</font></div><div><font face="monospace, monospace"> {'step': LastStep,</font></div><div><font face="monospace, monospace"> 'connections': {'prediction': ('Regressor', 'prediction'),</font></div><div><font face="monospace, monospace"> },</font></div><div><font face="monospace, monospace"> 'use_for': ['fit', 'run'],</font></div><div><font face="monospace, monospace"> },</font></div><div><font face="monospace, monospace">}</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">pipegraph = PipeGraph(graph_description)</font></div><div><font face="monospace, monospace">pipegraph.fit()</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">#Fitting: First</font></div><div><font face="monospace, monospace">#Fitting: Concatenate_Xy</font></div><div><font face="monospace, monospace">#Fitting: Dbscan</font></div><div><font face="monospace, monospace">#Fitting: Gaussian_Mixture</font></div><div><font face="monospace, monospace">#Fitting: Combine_Clustering</font></div><div><font face="monospace, monospace">#Fitting: Paella</font></div><div><font face="monospace, monospace">#0 ,</font></div><div><font face="monospace, monospace">#1 ,</font></div><div><font face="monospace, monospace">#2 ,</font></div><div><font face="monospace, monospace">#3 ,</font></div><div><font face="monospace, monospace">#4 ,</font></div><div><font face="monospace, monospace">#5 ,</font></div><div><font face="monospace, monospace">#6 ,</font></div><div><font face="monospace, monospace">#7 ,</font></div><div><font face="monospace, monospace">#8 ,</font></div><div><font face="monospace, monospace">#9 ,</font></div><div><font face="monospace, monospace">#10 ,</font></div><div><font face="monospace, monospace">#11 ,</font></div><div><font face="monospace, monospace">#12 ,</font></div><div><font face="monospace, monospace">#13 ,</font></div><div><font face="monospace, monospace">#14 ,</font></div><div><font face="monospace, monospace">#15 ,</font></div><div><font face="monospace, monospace">#16 ,</font></div><div><font face="monospace, monospace">#17 ,</font></div><div><font face="monospace, monospace">#18 ,</font></div><div><font face="monospace, monospace">#19 ,</font></div><div><font face="monospace, monospace">#Fitting: Regressor</font></div><div><font face="monospace, monospace">#Fitting: Last</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">pipegraph.run()</font></div><div><font face="monospace, monospace">#Running: First</font></div><div><font face="monospace, monospace">#Running: Regressor</font></div><div><font face="monospace, monospace">#Running: Last</font></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">2017-12-19 13:44 GMT+01:00 Manuel Castejón Limas <span dir="ltr"><<a href="mailto:manuel.castejon@gmail.com" target="_blank">manuel.castejon@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Dear all,<div><br></div><div>Kudos to scikit-learn! Having said that, Pipeline is killing me not being able to transform anything other than X.</div><div><br></div><div>My current study case would need:</div><div>- Transformers being able to handle both X and y, e.g. clustering X and y concatenated</div><div>- Pipeline being able to change other params, e.g. sample_weight</div><div><br></div><div>Currently, I'm augmenting X through every step with the extra information which seems to work ok for my_pipe.fit_transform(X_train,<wbr>y_train) but breaks on my_pipe.transform(X_test) for the lack of the y parameter. Ok, I can inherit and modify a descendant from Pipeline class to allow the y parameter which is not ideal but I guess it is an option. The gritty part comes when having to adapt every regressor at the end of the ladder in order to split the extra information from the raw data in X and not being able to generate more than one subproduct from each preprocessing step</div><div><br></div><div>My current research involves clustering the data and using that classification along with X in order to predict outliers which generates sample_weight info and I would love to use that on the final regressor. Currently there seems not to be another option than pasting that info on X.</div><div><br></div><div>All in all, I'm stuck with this API limitation and I would love to learn some tricks from you if you could enlighten me.</div><div><br></div><div>Thanks in advance!</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Manuel Castejón-Limas</div><div><br></div></font></span></div>
</blockquote></div><br></div>