[Tutor] urllib ... lost novice's question

Rafael Knuth rafael.knuth at gmail.com
Mon May 8 11:23:15 EDT 2017

Which package should I use to fetch and open an URL?
I am using Python 3.5 and there are presently 4 versions:


Common sense is telling me to use the latest version.
Not sure if my common sense is fooling me here though ;-)

Then, there is another package, along with a dozen other
urllib-related packages (such as aiourllib). I thought this one is
doing what I need:


The latter I found on http://docs.python-requests.org along with these
encouraging words:

"Warning: Recreational use of the Python standard library for HTTP may
result in dangerous side-effects, including: security vulnerabilities,
verbose code, reinventing the wheel, constantly reading documentation,
depression, headaches, or even death."

How do I know where to find the right package - on python.org or elsewhere?
I found some code samples that show how to use urllib.request, now I
am trying to understand why I should use urllib.request.
Would it be also doable to do requests using urllib5 or any other
version? Like 2 or 3? Just trying to understand.

I am lost here. Feeback appreciated. Thank you!

BTW, here's some (working) exemplary code I have been using for
educational purposes:

import urllib.request
from bs4 import BeautifulSoup

theurl = "https://twitter.com/rafaelknuth"
thepage = urllib.request.urlopen(theurl)
soup = BeautifulSoup(thepage, "html.parser")


i = 1
for tweets in soup.findAll("div",{"class":"content"}):
    i = i + 1

I am assuming there are different solutions for fetching and open URLs?
Or is the above the only viable solution?

More information about the Tutor mailing list