JavaScript web scraping test cases?

[John J. Lee]

I've put together a Python package for scraping / testing pages that
depend on embedded JavaScript code (without depending on IE, Mozilla
or Konqueror, and with the DOM etc. all implemented in pure Python --
mostly a hacked 4DOM, with some bits from pxdom; the JavaScript
interpreter I'm using ATM is spidermonkey).  It's still missing a lot
and is pre-alpha, but it works, just barely.

Anyway, the point of this post is that I'm looking for pages to test
it on, so if you have a page that you'd like scraped (one that uses
JavaScript in some non-trivial way, of course! -- for dynamically
modifying forms, setting cookies, or whatever), mail me the details:
better that than some randomly-selected site from the Internet.
Obviously, it should be something that doesn't violate any terms &
conditions of use or otherwise cause people trouble, and preferably
that doesn't require any signup.


[In fact, TBH, my completely ad-hoc methodology with this is to write
some web scraping code, discover that the JavaScript breaks things,
often by depending on some nonstandard DOM feature, hack the DOM a
bit, etc.  Hopefully I'll reach a point in understanding where I can
rewrite the DOM from scratch ('scratch' here being 4DOM), properly, to
match some approximation of 'HTML DOM as deployed'...]


John