<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hello,</p>
<p>First, thanks for the fantastic scikit-learn library.<br>
</p>
<p>I have the following use case: For a classification problem, I
have a list of sentences and use word2vec and a method (eg. mean,
or weigthed mean, or attention and mean) to transform sentences to
vectors. Because my dataset is very noisy, I may come with
sentences full of words that are not part of word2vec, hence I
can't vectorize them.</p>
<p>I would like to remove those sentences from my dataset X, but
this would mean removing also the corresponding target classes in
y. Afaik, scikit-learn does not implement this possibility. I've
seen a couple of issues about that, but they all seems stalled :
<a class="moz-txt-link-freetext" href="https://github.com/scikit-learn/scikit-learn/issues/9630">https://github.com/scikit-learn/scikit-learn/issues/9630</a>,
<a class="moz-txt-link-freetext" href="https://github.com/scikit-learn/scikit-learn/issues/3855">https://github.com/scikit-learn/scikit-learn/issues/3855</a>,
<a class="moz-txt-link-freetext" href="https://github.com/scikit-learn/scikit-learn/pull/4552">https://github.com/scikit-learn/scikit-learn/pull/4552</a>,
<a class="moz-txt-link-freetext" href="https://github.com/scikit-learn/scikit-learn/issues/4143">https://github.com/scikit-learn/scikit-learn/issues/4143</a></p>
<p>I would like to be able to search for hyper-parameters in a
simple way, so I really would like to be able to use a single
pipeline taking text as input.</p>
<p>My actual conclusion is this one :</p>
<ul>
<li>vectorizer should return None for bad samples (or a specific
vector, like numpy.zeros, or add an extra column marking
valid/invalid samples)<br>
</li>
<li>make all my transformers down the pipeline accept for those
entries and leave them untouched (can be done with a generic
wrapper class)<br>
</li>
<li>have a wrapper around my classifier, to avoid fitting on
those, like jnothman suggested here
<a class="moz-txt-link-freetext" href="https://github.com/scikit-learn/scikit-learn/issues/9630#issuecomment-325202441">https://github.com/scikit-learn/scikit-learn/issues/9630#issuecomment-325202441</a></li>
</ul>
<p>Its a bit tedious, but I can see it working.</p>
<p>Is there any better suggestion ?</p>
</body>
</html>