[Tutor] can I walk or glob a website?

Albert-Jan Roskam fomcl at yahoo.com
Wed May 18 13:23:02 CEST 2011





From: Dave Angel <davea at ieee.org>
To: Alan Gauld <alan.gauld at btinternet.com>
Cc: tutor at python.org
Sent: Wed, May 18, 2011 11:51:35 AM
Subject: Re: [Tutor] can I walk or glob a website?

On 01/-10/-28163 02:59 PM, Alan Gauld wrote:
> 
> "Albert-Jan Roskam" <fomcl at yahoo.com> wrote
>> How can I walk (as in os.walk) or glob a website?
> 
> I don't think there is a way to do that via the web.
> Of course if you have access to the web servers filesystem you can use
> os.walk to do it as for any other filesystem, but I don't think its
> generally possible over http. (And indeed it shouldn''t be for very good
> security reasons!)
> 
> OTOH I've been wrong before! :-)
> 

It has to be (more or less) possible.  That's what google does for their search 
engine.

Three broad issues.

1) Are you violating the terms of service of such a web site?  Are you going to 
be doing this seldom enough that the bandwidth used won't be a DOS attack?  Are 
there copyrights to the material you plan to download?  Is the website protected 
by a login, by cookies, or a VPN?  Does the website present a different view to 
different browsers, different OS's, or different target domains?

===> This crossed my mind too. The advantage of using Python is that it's fun 
and that it saves me from gettng a mouse arm. It is a Dutch government site with 
pdf quality control reports (from the municipal health service) of 
kindergartens. I thought it would be fun and useful to make graphic 
representations of (spider charts) of each kindergarten, so they can be easily 
compared. This is just a hobby project. It just bothers me that they're not very 
easy to compare.

2) Websites vary enormously in their adherence to standards.  There are many 
such standards, and browsers tend to be very tolerant of bugs in the site which 
will be painful for you to accomodate.  And some of the extensions/features are 
very hard to parse, such as flash.  Others, such as javascript, can make it hard 
to do it statically.

===> I checked some of the deep links already. They are of the form  
[/\\\.a-z]+docid[0-9]+resultid[0-9]+ (roughly speaking), e.g.
http://www.landelijkregisterkinderopvang.nl/pp/inzien/Oko/InspectieRapport.jsf?documentId=5547&selectedResultId=5548

I could use a brute force approach and try all the doc/result id combinations. 
But wouldn't that result in a high server load? If so, I could put the program 
to sleep for n seconds.

3) How important is it to do it reliably?  Your code may work perfectly with a 
particular website, and next week they'll make a change which breaks your code 
entirely.  Are you willing to rework the code each time that happens?

===> It should be reliable. Portability to other sites is, of course, cool, but 
not strictly necessary.

Many sites have API's that you can use to access them.  Sometimes this is a 
better answer.

With all of that said, I'll point you to Beautiful Soup, as a library that'll 
parse a page of moderately correct html and give you the elements of it.  If 
it's a static page, you can then walk the elements of the tree that Beautiful 
Soup gives you, and find all the content that interests you.  You can also find 
all the web pages that the first one refers to, and recurse on that.

Notice that you need to limit your scope, since many websites have direct and 
indirect links to most of the web. For example, you might only recurse into 
links that refer to the same domain.  For many websites, that means you won't 
get it all.  So you may want to supply a list of domains and/or subdomains that 
you're willing to recurse into.

See  http://pypi.python.org/pypi/BeautifulSoup/3.2.0

===> Thanks, I'll check BS.

DaveA

Best wishes, 
Albert-Jan

_______________________________________________
Tutor maillist  -  Tutor at python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tutor/attachments/20110518/29c4a5da/attachment.html>


More information about the Tutor mailing list