Supervised prediction of multiple scores for a document
Héllo, I started a natural language processing project a few weeks ago called wikimark <https://github.com/amirouche/wikimark/> (the code is all in wikimark.py <https://github.com/amirouche/wikimark/blob/master/wikimark.py#L1>) Given a text it wants to return a dictionary scoring the input against vital articles categories <https://en.wikipedia.org/api/rest_v1/page/html/Wikipedia%3AVital_articles%2F...>, e.g.: out = wikimark("""Peter Hintjens wrote about the relation between technology and culture. Without using a scientifical tone of state-of-the-art review of the anthroposcene antropology, he gives a fair amount of food for thought. According to Hintjens, technology is doomed to become cheap. As matter of fact, intelligence tools will become more and more accessible which will trigger a revolution to rebalance forces in society.""") for category, score in out: print('{} ~ {}'.format(category, score)) The above program would output something like that: Art ~ 0.1 Science ~ 0.5 Society ~ 0.4 Except not everything went as planned. Mind the fact that in the above example the total is equal to 1, but I could not achieve that at all. I am using gensim to compute vectors of paragraphs (doc2vev) and then submit those vectors to svm.SVR in a one-vs-all strategy ie. a document is scored 1 if it's in that subcategory and zero otherwise. At prediction time, it goes though the same doc2vec pipeline. The computer will score *each paragraph* against the SVR models of wikipedia vital article subcategories and get a value between 0 and 1 for *each paragraph*. I compute the sum and group by subcategory and then I have a score per category for the input document It somewhat works. I made a web ui online you can find it at https://sensimark.com where you can test it. You can directly access the full api e.g. https://sensimark.com/api/v0?url=http://www.multivax.com/last_question.html&all=1 The output JSON document is a list of category dictionary where the prediction key is associated with the average of the "prediction" of the subcategories. If you replace &all=1 by &top=5 you might get something else as top categories e.g. https://sensimark.com/api/v0?url=http://www.multivax.com/last_question.html&top=10 <https://sensimark.com/api/v0?url=http://www.multivax.com/last_question.html&all=1> or https://sensimark.com/api/v0?url=http://www.multivax.com/last_question.html&top=5 I wrote "prediction" with double quotes because the value you see, is the result of some formula. Since, the predictions I get are rather small between 0 and 0.015 I apply the following formula: value = math.exp(prediction) magic = ((value * 100) - 110) * 100 In order to have values to spread between -200 and 200. Maybe this is the symptom that my model doesn't work at all. Still, the top 10 results are almost always near each other (try with BBC <http://www.bbc.com/> articles on https://sensimark.com . It is only when a regression model is disqualified with a score of 0 that the results are simple to understand. Sadly, I don't have an example at hand to support that claim. You have to believe me. I just figured looking at the machine learning map <http://scikit-learn.org/stable/tutorial/machine_learning_map/> that my problem might be classification problem, except I don't really want to know what is *the* class of new documents, I want to how what are the different subjects that are dealt in the document based on a hiearchical corpus; I don't want to guess a hiearchy! I want to now how the document content spread over the different categories or subcategories. I quickly read about multinomal regression, is it something do you recommend I use? Maybe you think about something else? Also, it seems I should benchmark / evaluate my model against LDA. I am rather noob in terms of datascience and my math skills are not so fresh. I more likely looking for ideas on what algorithm, fine tuning and some practice of datascience I must follow that doesn't involve writing my own algorithm. Thanks in advance!
Hi,
I quickly read about multinomal regression, is it something do you recommend I use? Maybe you think about something else?
Multinomial regression (or Softmax Regression) should give you results somewhat similar to a linear SVC (or logistic regression with OvO or OvR). The theoretical difference is that Softmax regression assumes that the classes are mutually exclusive, which is probably not the case in your setting since e.g., an article could be both "Art" and "Science" to some extend or so. Here a quick summary of softmax regression if useful: https://sebastianraschka.com/faq/docs/softmax_regression.html. In scikit-learn, you can use it via LogisticRegression(..., multi_class='ovr'). Howeever, spontaneously, I would say that Latent Dirichlet Allocation could be a better choice in your case. I.e., fit the model on the corpus for a specified number of topics (e.g., 10, but depends on your dataset, I would experiment a bit here), look at the top words in each topic and then assign a topic label to each topic. Then, for a given article, you can assign e.g., the top X labeled topics. Best, Sebastian
On Jun 3, 2018, at 5:03 PM, Amirouche Boubekki <amirouche.boubekki@gmail.com> wrote:
Héllo,
I started a natural language processing project a few weeks ago called wikimark (the code is all in wikimark.py)
Given a text it wants to return a dictionary scoring the input against vital articles categories, e.g.:
out = wikimark("""Peter Hintjens wrote about the relation between technology and culture. Without using a scientifical tone of state-of-the-art review of the anthroposcene antropology, he gives a fair amount of food for thought. According to Hintjens, technology is doomed to become cheap. As matter of fact, intelligence tools will become more and more accessible which will trigger a revolution to rebalance forces in society.""")
for category, score in out: print('{} ~ {}'.format(category, score))
The above program would output something like that:
Art ~ 0.1 Science ~ 0.5 Society ~ 0.4
Except not everything went as planned. Mind the fact that in the above example the total is equal to 1, but I could not achieve that at all.
I am using gensim to compute vectors of paragraphs (doc2vev) and then submit those vectors to svm.SVR in a one-vs-all strategy ie. a document is scored 1 if it's in that subcategory and zero otherwise. At prediction time, it goes though the same doc2vec pipeline. The computer will score each paragraph against the SVR models of wikipedia vital article subcategories and get a value between 0 and 1 for each paragraph. I compute the sum and group by subcategory and then I have a score per category for the input document
It somewhat works. I made a web ui online you can find it at https://sensimark.com where you can test it. You can directly access the full api e.g. https://sensimark.com/api/v0?url=http://www.multivax.com/last_question.html&all=1
The output JSON document is a list of category dictionary where the prediction key is associated with the average of the "prediction" of the subcategories. If you replace &all=1 by &top=5 you might get something else as top categories e.g. https://sensimark.com/api/v0?url=http://www.multivax.com/last_question.html&top=10
or
https://sensimark.com/api/v0?url=http://www.multivax.com/last_question.html&top=5
I wrote "prediction" with double quotes because the value you see, is the result of some formula. Since, the predictions I get are rather small between 0 and 0.015 I apply the following formula: value = math.exp(prediction) magic = ((value * 100) - 110) * 100
In order to have values to spread between -200 and 200. Maybe this is the symptom that my model doesn't work at all.
Still, the top 10 results are almost always near each other (try with BBC articles on https://sensimark.com . It is only when a regression model is disqualified with a score of 0 that the results are simple to understand. Sadly, I don't have an example at hand to support that claim. You have to believe me.
I just figured looking at the machine learning map that my problem might be classification problem, except I don't really want to know what is the class of new documents, I want to how what are the different subjects that are dealt in the document based on a hiearchical corpus; I don't want to guess a hiearchy! I want to now how the document content spread over the different categories or subcategories.
I quickly read about multinomal regression, is it something do you recommend I use? Maybe you think about something else?
Also, it seems I should benchmark / evaluate my model against LDA.
I am rather noob in terms of datascience and my math skills are not so fresh. I more likely looking for ideas on what algorithm, fine tuning and some practice of datascience I must follow that doesn't involve writing my own algorithm.
Thanks in advance! _______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
sorry, I had a copy & paste error, I meant "LogisticRegression(..., multi_class='multinomial')" and not "LogisticRegression(..., multi_class='ovr')"
On Jun 3, 2018, at 5:19 PM, Sebastian Raschka <mail@sebastianraschka.com> wrote:
Hi,
I quickly read about multinomal regression, is it something do you recommend I use? Maybe you think about something else?
Multinomial regression (or Softmax Regression) should give you results somewhat similar to a linear SVC (or logistic regression with OvO or OvR). The theoretical difference is that Softmax regression assumes that the classes are mutually exclusive, which is probably not the case in your setting since e.g., an article could be both "Art" and "Science" to some extend or so. Here a quick summary of softmax regression if useful: https://sebastianraschka.com/faq/docs/softmax_regression.html. In scikit-learn, you can use it via LogisticRegression(..., multi_class='ovr').
Howeever, spontaneously, I would say that Latent Dirichlet Allocation could be a better choice in your case. I.e., fit the model on the corpus for a specified number of topics (e.g., 10, but depends on your dataset, I would experiment a bit here), look at the top words in each topic and then assign a topic label to each topic. Then, for a given article, you can assign e.g., the top X labeled topics.
Best, Sebastian
On Jun 3, 2018, at 5:03 PM, Amirouche Boubekki <amirouche.boubekki@gmail.com> wrote:
Héllo,
I started a natural language processing project a few weeks ago called wikimark (the code is all in wikimark.py)
Given a text it wants to return a dictionary scoring the input against vital articles categories, e.g.:
out = wikimark("""Peter Hintjens wrote about the relation between technology and culture. Without using a scientifical tone of state-of-the-art review of the anthroposcene antropology, he gives a fair amount of food for thought. According to Hintjens, technology is doomed to become cheap. As matter of fact, intelligence tools will become more and more accessible which will trigger a revolution to rebalance forces in society.""")
for category, score in out: print('{} ~ {}'.format(category, score))
The above program would output something like that:
Art ~ 0.1 Science ~ 0.5 Society ~ 0.4
Except not everything went as planned. Mind the fact that in the above example the total is equal to 1, but I could not achieve that at all.
I am using gensim to compute vectors of paragraphs (doc2vev) and then submit those vectors to svm.SVR in a one-vs-all strategy ie. a document is scored 1 if it's in that subcategory and zero otherwise. At prediction time, it goes though the same doc2vec pipeline. The computer will score each paragraph against the SVR models of wikipedia vital article subcategories and get a value between 0 and 1 for each paragraph. I compute the sum and group by subcategory and then I have a score per category for the input document
It somewhat works. I made a web ui online you can find it at https://sensimark.com where you can test it. You can directly access the full api e.g. https://sensimark.com/api/v0?url=http://www.multivax.com/last_question.html&all=1
The output JSON document is a list of category dictionary where the prediction key is associated with the average of the "prediction" of the subcategories. If you replace &all=1 by &top=5 you might get something else as top categories e.g. https://sensimark.com/api/v0?url=http://www.multivax.com/last_question.html&top=10
or
https://sensimark.com/api/v0?url=http://www.multivax.com/last_question.html&top=5
I wrote "prediction" with double quotes because the value you see, is the result of some formula. Since, the predictions I get are rather small between 0 and 0.015 I apply the following formula: value = math.exp(prediction) magic = ((value * 100) - 110) * 100
In order to have values to spread between -200 and 200. Maybe this is the symptom that my model doesn't work at all.
Still, the top 10 results are almost always near each other (try with BBC articles on https://sensimark.com . It is only when a regression model is disqualified with a score of 0 that the results are simple to understand. Sadly, I don't have an example at hand to support that claim. You have to believe me.
I just figured looking at the machine learning map that my problem might be classification problem, except I don't really want to know what is the class of new documents, I want to how what are the different subjects that are dealt in the document based on a hiearchical corpus; I don't want to guess a hiearchy! I want to now how the document content spread over the different categories or subcategories.
I quickly read about multinomal regression, is it something do you recommend I use? Maybe you think about something else?
Also, it seems I should benchmark / evaluate my model against LDA.
I am rather noob in terms of datascience and my math skills are not so fresh. I more likely looking for ideas on what algorithm, fine tuning and some practice of datascience I must follow that doesn't involve writing my own algorithm.
Thanks in advance! _______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
I made a rendering of the result online https://sensimark.com/ Le dim. 3 juin 2018 à 23:22, Sebastian Raschka <mail@sebastianraschka.com> a écrit :
sorry, I had a copy & paste error, I meant "LogisticRegression(..., multi_class='multinomial')" and not "LogisticRegression(..., multi_class='ovr')"
On Jun 3, 2018, at 5:19 PM, Sebastian Raschka <mail@sebastianraschka.com> wrote:
Hi,
I quickly read about multinomal regression, is it something do you recommend I use? Maybe you think about something else?
Multinomial regression (or Softmax Regression) should give you results somewhat similar to a linear SVC (or logistic regression with OvO or OvR). The theoretical difference is that Softmax regression assumes that the classes are mutually exclusive, which is probably not the case in your setting since e.g., an article could be both "Art" and "Science" to some extend or so. Here a quick summary of softmax regression if useful: https://sebastianraschka.com/faq/docs/softmax_regression.html. In scikit-learn, you can use it via LogisticRegression(..., multi_class='ovr').
Howeever, spontaneously, I would say that Latent Dirichlet Allocation could be a better choice in your case. I.e., fit the model on the corpus for a specified number of topics (e.g., 10, but depends on your dataset, I would experiment a bit here), look at the top words in each topic and then assign a topic label to each topic. Then, for a given article, you can assign e.g., the top X labeled topics.
Best, Sebastian
On Jun 3, 2018, at 5:03 PM, Amirouche Boubekki < amirouche.boubekki@gmail.com> wrote:
Héllo,
I started a natural language processing project a few weeks ago called wikimark (the code is all in wikimark.py)
Given a text it wants to return a dictionary scoring the input against vital articles categories, e.g.:
out = wikimark("""Peter Hintjens wrote about the relation between technology and culture. Without using a scientifical tone of state-of-the-art review of the anthroposcene antropology, he gives a fair amount of food for thought. According to Hintjens, technology is doomed to become cheap. As matter of fact, intelligence tools will become more and more accessible which will trigger a revolution to rebalance forces in society.""")
for category, score in out: print('{} ~ {}'.format(category, score))
The above program would output something like that:
Art ~ 0.1 Science ~ 0.5 Society ~ 0.4
Except not everything went as planned. Mind the fact that in the above example the total is equal to 1, but I could not achieve that at all.
I am using gensim to compute vectors of paragraphs (doc2vev) and then submit those vectors to svm.SVR in a one-vs-all strategy ie. a document is scored 1 if it's in that subcategory and zero otherwise. At prediction time, it goes though the same doc2vec pipeline. The computer will score each paragraph against the SVR models of wikipedia vital article subcategories and get a value between 0 and 1 for each paragraph. I compute the sum and group by subcategory and then I have a score per category for the input document
It somewhat works. I made a web ui online you can find it at https://sensimark.com where you can test it. You can directly access the full api e.g. https://sensimark.com/api/v0?url=http://www.multivax.com/last_question.html&all=1
The output JSON document is a list of category dictionary where the prediction key is associated with the average of the "prediction" of the subcategories. If you replace &all=1 by &top=5 you might get something else as top categories e.g. https://sensimark.com/api/v0?url=http://www.multivax.com/last_question.html&top=10
or
https://sensimark.com/api/v0?url=http://www.multivax.com/last_question.html&top=5
I wrote "prediction" with double quotes because the value you see, is
value = math.exp(prediction) magic = ((value * 100) - 110) * 100
In order to have values to spread between -200 and 200. Maybe this is
the result of some formula. Since, the predictions I get are rather small between 0 and 0.015 I apply the following formula: the symptom that my model doesn't work at all.
Still, the top 10 results are almost always near each other (try with
BBC articles on https://sensimark.com . It is only when a regression model is disqualified with a score of 0 that the results are simple to understand. Sadly, I don't have an example at hand to support that claim. You have to believe me.
I just figured looking at the machine learning map that my problem
might be classification problem, except I don't really want to know what is the class of new documents, I want to how what are the different subjects that are dealt in the document based on a hiearchical corpus;
I don't want to guess a hiearchy! I want to now how the document content spread over the different categories or subcategories.
I quickly read about multinomal regression, is it something do you recommend I use? Maybe you think about something else?
Also, it seems I should benchmark / evaluate my model against LDA.
I am rather noob in terms of datascience and my math skills are not so fresh. I more likely looking for ideas on what algorithm, fine tuning and some practice of datascience I must follow that doesn't involve writing my own algorithm.
Thanks in advance! _______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
_______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
participants (2)
-
Amirouche Boubekki -
Sebastian Raschka