using urllib on a more complex site

Adam W. AWasilenko at gmail.com
Mon Feb 25 01:02:02 CET 2013


I'm trying to write a simple script to scrape http://www.vudu.com/movies/#tag/99centOfTheDay/99c%20Rental%20of%20the%20day

in order to send myself an email every day of the 99c movie of the day.

However, using a simple command like (in Python 3.0): 
urllib.request.urlopen('http://www.vudu.com/movies/#tag/99centOfTheDay/99c%20Rental%20of%20the%20day').read()

I don't get the all the source I need, its just the navigation buttons.  Now I assume they are using some CSS/javascript witchcraft to load all the useful data later, so my question is how do I make urllib "wait" and grab that data as well?



More information about the Python-list mailing list