download web pages that are updated by ajax

Chris Rebert clp2 at
Tue Apr 12 20:30:39 CEST 2011

On Tue, Apr 12, 2011 at 7:47 AM, Jabba Laci <jabba.laci at> wrote:
> Hi,
> I want to download a web page that is updated by AJAX. The page
> requires no human interaction, it is updated automatically:
> If I download it with wget, I get a file of size 97 KB. The source is
> full of AJAX calls, i.e. the content of the page is not expanded.
> If I open it in a browser and save it manually, the result is a file
> of almost 5 MB whose content is expanded.
> (1) How to download such a page with Python? I need the post-AJAX
> version of the page.

I've heard you can drive a web browser using Selenium
( ), have it visit the webpage and
run the JavaScript on it, and then grab the final result.


More information about the Python-list mailing list