[Ncr-Python.in] [New Project] Get Twitter Links
Visit -> https://github.com/nsisodiya/Get-Twitter-Links This is a URL extraction tool which extract URL/links from Twitter Example -- == Get_Twitter_Links_of_User.py == ./Get_Twitter_Links_of_User.py nsisodiya 10 User : nsisodiyahttp://blog.narendrasisodiya.com/2011/08/how-to-learn-php.htmlhttp://narendr... <http://www.softwarefreedom.in/index.php?option=com_content&view=article&id=93&Itemid=80>http://www.amankapaigham.com/2011/08/blog-post.htmlhttp://twitpic.com/5ztnhghttp://groups.google.com/group/ubuntu-uck-remasterhttp://blog.narendrasisodiya.com/2011/08/idea-linux-daily-links.html So the python script Get_Twitter_Links_of_User.py will download latest 10 Tweets from user http://twitter.com/nsisodiya and print URL from those 10 Tweets. It takes two argument, UserName and Number of Tweets to be scanned == Get Twitter Links of Multiple Users == STEP 1 - Add Users narendra@localhost$ gedit listuser STEP 2 - Initialize User narendra@localhost$ ./InitiateUser.py This will list total number tweets of a user. This is needed to calculate Total number of latest Tweets by that user STEP 3 - Wait - you must wait for some time so that user Tweets some latest link/URL STEP 4 - narendra@localhost$ ./Get_Twitter_Links.py This Will print latest URL of users.. -- ┌─────────────────────────┐ │ Narendra Sisodiya │ http://narendrasisodiya.com └─────────────────────────┘
The re for matching URLS is "http://" used with re.search in "def Get_URL_FromText(text):" The re can be improved to consider other cases as well. Even the most basic https isn't supported. Even this tweet won't work if am correctly analyzing : MY DUMMY TWEEThttp://github.com WITHOUT SPACE BETWEEN TEXT AND URL ... It may print TWEEThttp://github.com as the target url if my glance is correct ... best wishes -- Gaurav Kalra +91-9717-620-649 On Mon, Aug 8, 2011 at 17:52, Narendra Sisodiya <narendra@narendrasisodiya.com> wrote:
Visit -> https://github.com/nsisodiya/Get-Twitter-Links
This is a URL extraction tool which extract URL/links from Twitter
Example -- == Get_Twitter_Links_of_User.py == ./Get_Twitter_Links_of_User.py nsisodiya 10 User : nsisodiya http://blog.narendrasisodiya.com/2011/08/how-to-learn-php.html http://narendrasisodiya.com/Portfolio http://www.softwarefreedom.in/index.php?option=com_content&view=article&... http://www.amankapaigham.com/2011/08/blog-post.html http://twitpic.com/5ztnhg http://groups.google.com/group/ubuntu-uck-remaster http://blog.narendrasisodiya.com/2011/08/idea-linux-daily-links.html
So the python script Get_Twitter_Links_of_User.py will download latest 10 Tweets from user http://twitter.com/nsisodiya and print URL from those 10 Tweets.
It takes two argument, UserName and Number of Tweets to be scanned
== Get Twitter Links of Multiple Users ==
STEP 1 - Add Users narendra@localhost$ gedit listuser
STEP 2 - Initialize User
narendra@localhost$ ./InitiateUser.py This will list total number tweets of a user. This is needed to calculate Total number of latest Tweets by that user
STEP 3 - Wait - you must wait for some time so that user Tweets some latest link/URL
STEP 4 - narendra@localhost$ ./Get_Twitter_Links.py
This Will print latest URL of users..
-- ┌─────────────────────────┐ │ Narendra Sisodiya │ http://narendrasisodiya.com └─────────────────────────┘
_______________________________________________ http://mail.python.org/mailman/listinfo/ncr-python.in Mailing list guidelines : http://lug-iitd.org/Mailing_List_Guidelines
On Tue, Aug 9, 2011 at 11:10 PM, Gaurav Kalra <gvkalra@gmail.com> wrote:
The re for matching URLS is "http://" used with re.search in "def Get_URL_FromText(text):"
The re can be improved to consider other cases as well. Even the most basic https isn't supported.
Please improve it and send me the patch... I will add your patch.. I am new to python..
Hi Narendra. Here are few pointers to follow: 1. What you did by parsing the XML from twitter was redundant ... Twitter has very nice JSON interface to it's REST API. 2. Look out for twitter entities : https://dev.twitter.com/docs/tweet-entities .... The urls entity has a field called "url" which contains the URL being used inside the tweet .... 3. Keep the URL as http://api.twitter.com/1/statuses/user_timeline.json?include_entities=true&include_rts=true&count="+ str(count) +"&screen_name=" + user NOTICE include_entities + the JSON format ... 4. Use simplejson for parsing JSON format .... e.g. feed = urllib2.urlopen(URL) feedlist = simplejson.load(feed) #list has as many elements as the count variable in URL so, for each list element (ex. element 2) excess : feedlist[1]["entities"]["urls"] each such output is a list of dictionary and you need to access the "url" key in each such list element ... in case there is only one url inside the tweet text, then there will only be one element inside the list .... so, we'll have : feedlist[1]["entities"]["urls"][0]["url"] as the target url to be printed on the screen .... so that practically reduces your work to ZERO ... all the parsing methods .... regular expressions etc that you have written have no importance ........ play with python / twitter api .... this was also my first shot at it .... cheers ... -- Gaurav Kalra +91-9717-620-649 On Wed, Aug 10, 2011 at 00:26, Narendra Sisodiya <narendra@narendrasisodiya.com> wrote:
On Tue, Aug 9, 2011 at 11:10 PM, Gaurav Kalra <gvkalra@gmail.com> wrote:
The re for matching URLS is "http://" used with re.search in "def Get_URL_FromText(text):"
The re can be improved to consider other cases as well. Even the most basic https isn't supported.
Please improve it and send me the patch... I will add your patch.. I am new to python..
_______________________________________________ http://mail.python.org/mailman/listinfo/ncr-python.in Mailing list guidelines : http://lug-iitd.org/Mailing_List_Guidelines
On Wed, Aug 10, 2011 at 2:41 AM, Gaurav Kalra <gvkalra@gmail.com> wrote:
Hi Narendra.
Here are few pointers to follow:
1. What you did by parsing the XML from twitter was redundant ... Twitter has very nice JSON interface to it's REST API.
Good... I will add JSON Parsing... actually I am comfortable with minidom
2. Look out for twitter entities : https://dev.twitter.com/docs/tweet-entities .... The urls entity has a field called "url" which contains the URL being used inside the tweet ....
3. Keep the URL as
http://api.twitter.com/1/statuses/user_timeline.json?include_entities=true&include_rts=true&count= "+ str(count) +"&screen_name=" + user NOTICE include_entities + the JSON format ...
4. Use simplejson for parsing JSON format .... e.g. feed = urllib2.urlopen(URL) feedlist = simplejson.load(feed) #list has as many elements as the count variable in URL
so, for each list element (ex. element 2) excess : feedlist[1]["entities"]["urls"]
each such output is a list of dictionary and you need to access the "url" key in each such list element ... in case there is only one url inside the tweet text, then there will only be one element inside the list .... so, we'll have : feedlist[1]["entities"]["urls"][0]["url"] as the target url to be printed on the screen ....
Thanks.. I will add these changes.. also, if I use json then 'data to be download' will be more.. RSS has less number of fields.. also url is json will be like http:\/\/ur1.ca\/4v0r4 I have to change it proper url..
Thanks.. I will add these changes.. also, if I use json then 'data to be download' will be more.. RSS has less number of fields..
I would still stick on with json ... Read a very good discussion on "parsing xml" here : http://mail.python.org/pipermail/bangpypers/2011-August/subject.html which may be nearly mapped to this situation as well
_______________________________________________ http://mail.python.org/mailman/listinfo/ncr-python.in Mailing list guidelines : http://lug-iitd.org/Mailing_List_Guidelines
participants (2)
-
Gaurav Kalra -
Narendra Sisodiya