lxml - The Python XML Toolkit

Download

lxml@python.org

July 2024

2 participants
2 discussions

Assistance Needed: Python Code Crashes After Multiple Iterations When Extracting HTML Elements Caixa de entrada
by Carlos Eduardo de Schuller Banjar 29 Jul '24

29 Jul '24

Hello, everyone! I am encountering an issue with my Python code that opens multiple HTML pages and extracts elements from a specific class, if they exist. The code runs fine for a few iterations but crashes after enough iterations, always on the same HTML page. Interestingly, if I process this page individually, the code works without any problems. This issue has been bothering me for some time, and I have tried many approaches. When debugging, I noticed that the code always crashes within the *lxml* package, which is used by *requests_html*. This could be a bug in the library, but what could explain the crash only after several iterations? <https://mail.google.com/mail/u/0?ui=2&ik=50de717578&attid=0.1&permmsgid=msg…> <https://mail.google.com/mail/u/0?ui=2&ik=50de717578&attid=0.1&permmsgid=msg…> A few more information: Python : sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0) lxml.etree : (5, 2, 2, 0) libxml used : (2, 12, 6) libxml compiled : (2, 12, 6) libxslt used : (1, 1, 39) libxslt compiled : (1, 1, 39) I'm attaching the code I mentioned. You can see that the code is quite simple. import pandas as pd import requests_html def main(): df_links = pd.read_csv('./links.csv') session = requests_html.HTMLSession() for i in range(0, len(df_links.index)): url = df_links.iloc[i]['hyperlink'] print(f"[{i}/{len(df_links.index)}]: {url}", flush=True) try: response = session.get(url) if response.status_code == 200: response_html = response.html dateList = response_html.find('relative-time') except Exception as e: print(f"Something went wrong: {e}", flush=True) if __name__ == "__main__": main() The crash always happens at the following line: value = etree.fromstring(html, parser, **kw) in the function document_fromstring in *lxml/html/__init__.py*. Thank you for your time and help. Best regards, Kadu

1 0

Unable to obtain multiple validation errors against schema from a single xml file
by Claudio Pelazza 05 Jul '24

05 Jul '24

Dear all, Hope this email finds you well!! I am trying to obtain multiple error messages from validating a test xml file, which I know in advance it has two or more different errors. However, when attempting to read lxml's log as suggested in the docs, I just get only one (the one which fails first). I'm a bit 'rusty' in python (no pun intended) but the few comments I find around the Web (e.g. Stack Overflow, etc.), even though not super explicit, do suggest it is possible.... am I doing anything wrong, or did I miss anything? (Chances are I did!!) Here is a code snippet (trying to achieve that in Databricks/Spark, lxml version is 5.2.2) from lxml import etree def validate_xml(xml_file, xsd_file): # Parse the XML and XSD files xml_doc = etree.parse(xml_file) with open(xsd_file, 'r') as f: xsd_doc = etree.XML(bytes(f.read(), 'utf-8')) # Create an XMLSchema object schema = etree.XMLSchema(xsd_doc) err_log = "" # Validate the XML document against the schema #is_valid = schema.validate(xml_doc) try: schema.assertValid(xml_doc) except etree.DocumentInvalid as err: err_log = str([error for error in schema.error_log]) #err_log = str(err) return err_log or "Valid" I know error_log is not iterable, but still, I was hoping I would find all the error messages by printing the entire log....what am I missing? isn't this possible?? (is this a SAX type parser only??) Also, looks like I cannot attach files in here?? Anyways, you can find example xml & xsd used at the end of this message. For the kind souls out there: Any hint/ suggestion/ example would be much appreciated!!! Thank you so much in advance!!! Claudio P. ------------------------------------------------- --- xml & xsd follows: ---Schema--- <?xml version="1.0" encoding="utf-8" ?> <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <xsd:element name="BusinessCard"> <xsd:complexType> <xsd:sequence> <xsd:element name="Name" type="xsd:string"/> <xsd:element name="phone" maxOccurs="unbounded"> <xsd:complexType mixed="true"> <xsd:attribute name="type" use="required"> <xsd:simpleType> <xsd:restriction base="xsd:string"> <xsd:enumeration value="mobile"/> <xsd:enumeration value="fax"/> <xsd:enumeration value="work"/> <xsd:enumeration value="home"/> </xsd:restriction> </xsd:simpleType> </xsd:attribute> </xsd:complexType> </xsd:element> <xsd:element name="email" type="xsd:string" minOccurs="0" /> </xsd:sequence> </xsd:complexType> </xsd:element> </xsd:schema> ---------------------------------------------------------- ---xml--- <?xml version="1.0"?> <BusinessCard> <phone type="mobil">(415) 555-4567</phone> <phone type="work">(800) 555-9876</phone> <phone type="fax">(510) 555-1234</phone> <mail>joe(a)joe.com</mail> </BusinessCard>

1 0