[Tutor] Extract main text from HTML document

Simon Connah scopensource at gmail.com
Sat May 5 13:59:24 EDT 2018


I'm writing a very simple web scraper. It'll download a page from a
website and then store the result in a database of some sort. The
problem is that this will obviously include a whole heap of HTML,
JavaScript and maybe even some CSS. None of which is useful to me.

I was wondering if there was a way in which I could download a web
page and then just extract the main body of text without all of the

The title is obviously easy but the main body of text could contain
all sorts of HTML and I'm interested to know how I might go about
removing the bits that are not needed but still keep the meaning of
the document intact.

Does anyone have any suggestions on this front at all?

Thanks for any help.


More information about the Tutor mailing list