[Tutor] Wedscraping Yahoo API

[Danny Yoo]

Hi Joe,


On Mon, Feb 20, 2017 at 6:32 AM, Joe via Tutor <tutor at python.org> wrote:
> Hi,
> I keep getting the following error as I am new to programming and I am following a tutorial and I am using Python 3.6. Can you please point me in the right direction as to what I am doing wrong.


You have made an assumption that you have done something wrong.

Although this might be true, when we program anything substantial that
interacts with other remote services, failures can happen anywhere in
that chain, be it problems in third-party libraries, to problems with
the servers, to even interference from the low-level network!


Read the error message carefully, especially the last part:

######################################################################
RemoteDataError: Unable to read URL:
http://ichart.finance.yahoo.com/table.csv?s=OTEX.TO&a=0&b=1&c=2000&d=11&e=31&f=2016&g=d&ignore=.csv
######################################################################

It says that it can't read the content at that URL.


So there are a few possibilities.  Perhaps there is a problem in your
code.  However, you mention that this script is web-scraping Yahoo.
Web-scraping is known to be an extremely fragile endeavor, because it
introduces factors entirely out of your control:

  * Perhaps that URL is wrong.

  * Perhaps the page is no longer available.

  * Perhaps the page structure has changed.

  * Perhaps the page is available sometimes, but the server recognizes
that it is being scraped for content, and so returns an error response
occasionally as a way to discourage scraping.

  * Perhaps your network is injecting content that breaks parsing.

(The last is not as crazy as it sounds.  See:
https://arstechnica.com/tech-policy/2014/09/why-comcasts-javascript-ad-injections-threaten-security-net-neutrality/.
A HTTPS-only web can't come soon enough.)

Web scraping introduces all this uncertainty, making it a mess even
for professional programmers.

For these reasons, I contend that web scraping is a very bad example
for introductory beginner programming. It's also usually a very bad
idea for professional programmers too.  Without a strong fundamental
background, it's way too easy for a beginner to learn superstitious
beliefs in that kind of environment, because too many things can
happen that are outside of your control.


You mention that you're new to programming.  You might want to try a
tutorial that doesn't depend so much on the external world. I would
strongly recommend not continuing with this example, and go into
something like:

    https://wiki.python.org/moin/BeginnersGuide/Programmers

Have you taken a look at these tutorials?  One of them might be better
suited for you.


Hope that makes sense; good luck!