Visit All URLs with selenium python
Deborah Swanson
python at deborahswanson.net
Thu Apr 13 01:29:00 EDT 2017
Nicole wrote, on Wednesday, April 12, 2017 9:49 PM
>
> browser.get('https://www.google.co.uk/search?q=Rashmi&oq=Rashm
> i&aqs=chrome..69i57j69i60l3.6857j0j1&sourceid=chrome&ie=UTF-8#
q=Rashmi+Custom+Tailors')
> time.sleep(5)
>
> try:
> p_links =
> browser.find_elements_by_css_selector('div > h3 > a')
> url_list = []
> for urls in p_links:
> if "Rashmi Custom Tailors" in urls.text:
>
> url = urls.get_attribute("href")
> url_list.append(url)
> for url in url_list:
> browser.get(url)
> time.sleep(4)
>
>
> it just visit first url not all .. Can anybody help how to fix that..
You don't say what module you're using and it would help to see the
"import ..." statement.
But there are a couple things I can think of that could be causing the
problem:
There's only one result with the exact phrase "Rashmi Custom
Tailors" on
the page.
or
The css_selector('div > h3 > a') only occurs for the first
result and
selectors for subsequent results are different. I've seen that
before. If
the div extends all the way down the list until after the last
result,
the results after the first one might have css_selector('h3 >
a'), but
I'm just guessing about how they might be different.
Deborah
More information about the Python-list
mailing list