[Tutor] General question rgrd. usage of libraries

Rafael Knuth rafael.knuth at gmail.com
Fri May 5 09:45:00 EDT 2017


Hi there,

I just recently learned how to build a basic web scraper with Python
3.5 (I am learning Python for data analytics purposes). Being new to
coding, I have a question:

How do I know which libraries I need to perform a certain task?
For example, in case of this web scraper (which I built with help of a
tutorial on YouTube) I need to have urrlib and Beautiful Soup

import urllib
import urllib.request
from bs4 import BeautifulSoup

theurl = "https://twitter.com/rafaelknuth"
thepage = urllib.request.urlopen(theurl)
soup = BeautifulSoup(thepage, "html.parser")

print(soup.title.text)

i = 1
for tweets in soup.findAll("div",{"class":"content"}):
    print(i)
    print(tweets.find("p").text)
    i = i + 1

Is there a way I can figure out which libraries I need when drafting my code?
Can you share your experiences? Right now, if I wanted for example to
populate a Google Sheet with my scraped web content - how would I know
which libraries I would need to actually make this happen? I am trying
wondering if there is a process to figure out what I exactly need
library-wise.

Thank you,

Rafael


More information about the Tutor mailing list