Python - scraping - javascript

Mike Paul paul.mike25 at gmail.com
Sun Aug 16 02:09:15 CEST 2009


I'm trying to scrap a dynamic page with lot of javascript in it. Inorder to
get all the data from the page i need to access the javascript. But i've no
idea how to do it.

Say I'm scraping some site htttp://www.xyz.com/xyz

request=urllib2.Request("htttp://www.xyz.com/xyz")
response=urllib2.urlopen(request)
data=response.read()


So i get all the data on the initial page. Now i need to access the
javascript on this page to get additional details. I've heard someone
telling me to use spidermonkey. But no idea on how to send javscript as
request and get the response. How hsuld i be sending the javascript request
as ? how can it be sent?

This is the script I need to access.

<a href=# onclick="groups.render_box('data', {&quot;page&quot;: 2});return
false;">Click this to view more items</a>

Can anyone tell me how can i do it very clearly. I've been breaking my head
into this for the past few days with no progress.


Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20090815/aa2e7662/attachment.html>


More information about the Python-list mailing list