Issue with Scrapping Data from a webpage- Noob Question

abhijeet mahagaonkar abhi.forall at gmail.com
Wed Feb 8 07:36:17 CET 2012


Hi Fellow Pythoners,

I'm trying to collect table data from an authenticated webpage (Tool) to
which I have access.

I will have the required data after 'click'ing a submit button on the tool
homepage.
When I inspect the submit button i see
<form action="/Tool/index.do" method="POST">

Thus the tool's homepage is of the form www.example.com/Tool and on
clicking the submit button the data I need will be at
www.example.com/Tool/index.do

The problem that I'm running into is in my below code is giving me the
source of homepage(www.example.com/Tool) and not the of the submitted page(
www.example.com/Tool/index.do)

url="www.example.com/Tool/index.do"
request = urllib2.Request(url, data, {'Authorization': "Basic " +
base64.b64encode("%s:%s" % (username, password))})
Response_Page=urllib2.urlopen(request).read()

Is there a way I can access the source of the submitted page?

PS: Sorry for laying out very tiny details on what I'm trying to do, I just
wanted to explain myself clearly :)

Thanks in advance for your time on this one.

Warm Regards,
Abhi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20120208/aee4a312/attachment.html>


More information about the Python-list mailing list