Mailman 3 June 2018 - lxml - The Python XML Toolkit

XSLT: transform an XML file into another XML file
by Sylvain.Fejoz＠ville-ge.ch 15 Jun '18

15 Jun '18

Hello :) Python version: 3.6 Python IDE: PyCharm 2018.1 OS: Windows 7 Pro I try to adapt the statements on page http://lxml.de/xpathxslt.html#xslt When running attached python file (on also attached xml file) I get this error message in PyCharm: Traceback (most recent call last): File "C:/Users/Dupont/PycharmProjects/bge_mss_compter_niveaux_de_description/bge_mss_compter_niveaux_de_description_new.py", line 225, in <module> </xsl:stylesheet>''') File "src\lxml\etree.pyx", line 3192, in lxml.etree.XML File "src\lxml\parser.pxi", line 1876, in lxml.etree._parseMemoryDocument File "src\lxml\parser.pxi", line 1764, in lxml.etree._parseDoc File "src\lxml\parser.pxi", line 1126, in lxml.etree._BaseParser._parseDoc File "src\lxml\parser.pxi", line 600, in lxml.etree._ParserContext._handleParseResultDoc File "src\lxml\parser.pxi", line 710, in lxml.etree._handleParseResult File "src\lxml\parser.pxi", line 639, in lxml.etree._raiseParseError File "<string>", line 1 lxml.etree.XMLSyntaxError: Start tag expected, '<' not found, line 1, column 1 Process finished with exit code 1 First question: could you help fixing this bug? Second question: I would like to replace this statement: xslt_fichier = etree.XML('''\ <?xml version="1.0" encoding="UTF-8"?> [continuation of the xslt statements] with a statement calling an XSL file (also attached). How? Thanks for your attention and help. Sylvain

4 5

Re: [lxml] Resolvers - What's going on here?
by Duthie, Roger J.A. 14 Jun '18

14 Jun '18

Update to Question 1 I have created a simpler example which I will share with you, hopefully tomorrow. This uses local resources and simple XSDs which I've crafted. The behaviour, when using my `SimpleFileResolver` class, of failing to expand the full URL is reproduced. I'll share the code and resources when I have time to put it all together in a neat form, as I say, hopefully tomorrow. Also, I've noticed some additional strangeness with the URLs when resolving `<include>` elements for resources referenced by just their filename (perhaps for any relative path). The parser builds the absolute path from the relative one, but in the form `file:{unix_path}/resource.xsd`, instead of `file://{unix_path}/resource.xsd`<file://{unix_path}/resource.xsd>. This means that (e.g.) `urllib3.util.url.parse_url` doesn't recognise this path as a being of `file` type scheme which causes issues. Better explanation of this one tomorrow. I will also try to find where these issues are stemming from in the source, so hopefully I can say something about it with relevance to the lxml code. Any helpful directions on where to look will be appreciated! Cheers, Rg. ________________________________ This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system. ________________________________

1 0

Default parser with/without no_network
by Duthie, Roger J.A. 14 Jun '18

14 Jun '18

Dear lxml mailing list, A little background (next two paragraphs are the same as previous email): I'm creating a bit of code which can validate XML metadata. Initially, I wasn't using any Resolvers when parsing XML, XSD, or schematron, but the default resolver had issues with the some files (which will be the subject of a future question, I suspect). As I work through the issues I have noted and create simplified examples, I am discovering quirks here and there. For the project I'm working on, I am using Python 3.4, lxml 3.8.0, libxml 2.9.3 & libxslt 1.1.29. I have checked the behaviour of the following example using Python 3.6, lxml 4.2.1, libxml 2.9.7 (latest version) & libxslt 1.1.31, but the result is the same. Question 2: When validating an XML file against a schematron, using the lxml.isoschematron module, there are strange differences when using an XMLParser initialised with `no_network=True` and `False`. The `no_network` option doesn't seem to stop remote resources from being read; this might be what has been reported in the bug tracker here: https://bugs.launchpad.net/lxml/+bug/1234114). Moreover, the choice of `True` or `False` has different effects, which I'm not finding easy to unpick. The issues occur at the schematron validation time, when an XSLT transformation is being performed. Up to that point, no remote resources are resolved, it seems. I've noticed that this is different behaviour to XMLSchema objects, which resolve resources at the point when the validator is created. The example I've created to illustrate my point (using SeaDataNet metadata and their custom schematron), I've constructed three tests, with a fourth to highlight that no_network doesn't change the result when using the alternative resolver. I would have liked to created a pared down example, but without debugging messages from the XSLT transform, I am finding it difficult. The test code is here: https://gist.github.com/rogie-bas/e157501d12cff8ac0aafc0cc120ce8dc. The tests (logging output is also annotated with these test numbers): 1. A resolver is added which only outputs which resource is being resolved and then defers to the default parser (SimpleBaseResolver). In this test the `no_network` option is turned on. The process seems to do quite well at resolving a lot of resources, but then also seems to choke at a resource which it has already resolved. This resource will produce a 200 when requested by curl (i.e. no 3XX redirects). However, the actual URL may have had a fragment appended, which is not reported by lxml. With no debugging info from libxslt or lxml, I'm not sure where in the .sch file this problem occurred. Is there a way to get more debugging - e.g. line numbers? 2. Same parser as the first test, but with `no_network=False`. The test chokes at a different resource. Again, this is not the first resource that the parser seems to be handling. In this case, the problematic resource has a redirect (in fact, two); however, it is not the first resource to be parsed with a redirect. 3. I use the Requests package to resolve the HTTP URLs. The processing completes and the validation is successful. 4. The same resolver as 3., but with `no_network=True`. This has no effect, which might be expected; although the documentation doesn't explain that no_network only applies to the default resolver, I can imagine that this might be the case. That said, the no_network option doesn't seem to do anything in other cases where the default parser is used. This latter point might be another question, although I am aware that a bug issue has been raised (#1234114), so I can add my evidence to that instead. To get back to the point, the question is: * Can the behaviour of the default parser be explained in the above tests? Specifically, and in this case it's a bit multi-pronged: 1. What is causing the failures for these particular resources? 2. Why the strange sequence of apparent successful parsing, then subsequent failure? 3. How can I get more useful information from the failing XSLT transformation? So three questions, but I think a little insight into any of these would be helpful to me in understanding the others. I will keep trying to create a simpler example, but that might take time and there may be some insight to be gained in the meantime. I would be happy using the Requests-based resolver for all the resolution, as it seems to work well, but the expansion of <include> URLs is not certain (cf., Question 1 previously sent to the mailing list). Any ideas? All the best, Roger ________________________________ This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system. ________________________________

1 0

lxml, Sphinx, and Intersphinx
by Thomas Schraitle 14 Jun '18

14 Jun '18

Hi, in one of my projects I heavily use the lxml library. :) As I document my API with Sphinx[1], I also mark my datatypes in my docstrings with the respective macro like: :type tree: :class:`lxml.ElementTree` If you enable the Intersphinx[2] extension, Sphinx can automatically link to other projects' documentation. This is very convenient as it gives readers more flexibility as they can follow the link to get more information about a specific type. I would like use this functionality in my documentation to link to lxml's documentation. However, to make this work, the Intersphinx extension needs an "objects.inv" file from lxml. It's a so called "inventory file" which contains all the classes, functions, variables etc. and in which HTML file it is definied, see [3] about the syntax. This file is automatically generated when you write your documentation in Sphinx. As far as I can see, lxml seems to use a different system, therefor no "objects.inv" file is automatically created. At least, I haven't seen one. Did someone used Sphinx/Intersphinx to link to lxml? Or maybe are there any plans to generate the documentation through Sphinx in the future? It would be nice if the lxml project could expose its API as objects.inv so it can be parsed by Intersphinx. ;) Thanks! :) ----- References [1] http://www.sphinx-doc.org [2] http://www.sphinx-doc.org/en/stable/ext/intersphinx.html#module-sphinx.ext.… [3] https://sphobjinv.readthedocs.io/en/latest/syntax.html -- Gruß/Regards, Thomas Schraitle

1 0

pretty_printing XML
by Martin Mueller 14 Jun '18

14 Jun '18

Is there somewhere in this world a tutorial or explanation that will tell someone like me how to format the output of lxml? “pretty_print=True” seems to be a blackbox, at least to me, and I’ve looked around in vain for a starting place. In the project that I’m engaged in it might make sense to invert the indentation rules in the normal “pretty_print.” I deal with texts where every word is wrapped in a <w> element with a fair number of attributes. A single <w> element will occasionally add up to 80 characters or more. So it would save space and provide a more legible and manageable output if leaf nodes like <w> started at the left margin and higher level elements (p, l, div, etc) were indented, whether in a flat or hierarchical manner. I don’t know whether one can do this with xmllint, but the documentation of that program is like the old German Old English dictionaries, where you can only look up a word if you know its root, but if you know its root already you probably don’t need to look it up in the first place. A lot of computer documentation is like that, alas. So I’d be most grateful for any advice on how to construct a pretty_print routine that starts with the leaf nodes of <w> and <pc> elements at the left margin and indents everything else by four spaces or, more elegantly, indents hierarchically by one or two spaces. With thanks in advance Martin Mueller Professor emeritus of English and Classics Northwestern University

1 0

Resolvers - What's going on here?
by Duthie, Roger J.A. 13 Jun '18

13 Jun '18

Dear lxml mailing list, As previously announced, after having done some extra work investigating this aspect of lxml, I have some questions on Resolvers. I think I'll have a few more questions on this topic (hence this is "Question 1"); bear with me, the questions will come as I gather the evidence and form useful examples. A little background: I'm creating a bit of code which can validate XML metadata. Initially, I wasn't using any Resolvers when parsing XML, XSD, or schematron, but the default resolver had issues with the some files (which will be the subject of a future question, I suspect). As I work through the issues I have noted and create simplified examples, I am discovering quirks here and there. For the project I'm working on, I am using Python 3.4, lxml 3.8.0, libxml 2.9.3 & libxslt 1.1.29. I have checked the behaviour of the following example using Python 3.6, lxml 4.2.1, libxml 2.9.7 (latest version) & libxslt 1.1.31, but the result is the same. Question 1: I have created a simple class which inherits from etree.Resolver. I've pasted the code into a GitHub Gist, as I was having issues with being mistaken for spam: https://gist.github.com/rogie-bas/2cb314eee4ce244a1cf33e67e8151d6e). The resolve method of my class uses the Requests package to get the content of a remote resource, then passes the content to the resolve_file method, along with the usl as the base_url. Each time a URI is passed to the custom resolver, the code checks whether the URI has a http, https or file scheme (e.g. file:// prefix), if not then the resolver passes back 'None'. The example uses the ISO19139 standard gco.xsd, which is something I'm using on my main project. The output when logging is set to INFO is given by the GitHub Gist link (the link again: https://gist.github.com/rogie-bas/2cb314eee4ce244a1cf33e67e8151d6e) The crux of the issue is shown on line 10: the dynamicFeature.xsd resource is not fully referenced, with path prefix. This resource, which is featured as an <include> in the resource shown on line 7, has not had the URL added to its path. However, basicTypes.xsd (line 4), which is featured in an <include> in the target XSD (line 1) has indeed had an absolute path resolved, despite being referenced as a relative resource. It might be significant to note that the resource on line 7 was featured in an <import> element of the resource in line 4 - i.e., not an <include> - so the base_url path changes from line 4 resource to line 7 resource. What's going wrong? Is this a bug or a mistake on my part? Thanks in advance, Roger ________________________________ This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system. ________________________________

1 0