Eugene Van den Bulke, 02.07.2010 11:53:
Stefan Behnel, 02.07.2010 11:42:
Eugene Van den Bulke, 02.07.2010 10:49:
I am experimenting with web scraping using lxml.
I have played a little with BeautifulSoup in the past and scrapy recently.
I am recoding something I did with scrapy with lxml but encounter a problem I am not sure how to iron out.
With scrapy, hxs is an xpath selector which has a select and re method
types = hxs.select('.//a[@href]/@href').re(r'type=([A-Z]*)')
Which will return a list of the matches in href.
How would I do the same thing with lxml?
types = doc.xpath('.//a[@href]/@href') ...
Note that this is redundant, './/a/@href' is enough.
I did read the doc before I took the liberty to post ... I am afraid I just don't get it.
Personally, I wouldn't even use XPath regular expressions here. I'd rather do something like this: from lxml import html import re parse_type_value = re.compile(r'type=([A-Z]*)').findall root = html.parse(the_file).getroot() for el, attr, link, pos in root.iterlinks(): if 'type=' in link: print el.tag, parse_type_value(link) Note that this will give you all links, not only those in <a> href's. If you really only want those, the XPath expression above will do just fine. Stefan
On Fri, Jul 2, 2010 at 8:13 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
types = doc.xpath('.//a[@href]/@href') ...
Note that this is redundant, './/a/@href' is enough.
I am discovering XPath as well as you can tell :P
Personally, I wouldn't even use XPath regular expressions here. I'd rather do something like this:
from lxml import html import re
parse_type_value = re.compile(r'type=([A-Z]*)').findall
root = html.parse(the_file).getroot()
for el, attr, link, pos in root.iterlinks(): if 'type=' in link: print el.tag, parse_type_value(link)
Note that this will give you all links, not only those in <a> href's. If you really only want those, the XPath expression above will do just fine.
Stefan
Thanks ! -- EuGeNe -- I lend my books on COlivri http://www.colivri.org/user/eugene, do you?
participants (2)
-
Eugene Van den Bulke -
Stefan Behnel