[lxml-dev] build & performance issues with 2.0beta2
Hi there, sorry to interrupt the testimonial celebrations ;-) but I'm having some problems with 2.0beta2: First of all, it does not build any more using gcc 2.95.2, yes I know, might old compiler...then again, Cython produces C code, not funky C++ stuff (2.0alpha-r47832 still built without problems). This is the error I get: $ /apps/pydev/hjoukl/bin/python2.4 setup.py build Building with Cython 0.9.6.11b. Building lxml version 2.0.beta2-51091. running build [...] cythoning src/lxml/lxml.etree.pyx to src/lxml/lxml.etree.c building 'lxml.etree' extension creating build/temp.solaris-2.8-sun4u-2.4 creating build/temp.solaris-2.8-sun4u-2.4/src creating build/temp.solaris-2.8-sun4u-2.4/src/lxml gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -I/apps/prod//include -I/apps/prod//include/libxml2 -I/apps/prod/include/libxml2 -I/apps/prod/include -I/apps/pydev/hjoukl/include/python2.4 -c src/lxml/lxml.etree.c -o build/temp.solaris-2.8-sun4u-2.4/src/lxml/lxml.etree.o -w src/lxml/lxml.etree.c: In function `__pyx_PyInt_AsLongLong': src/lxml/lxml.etree.c:110165: parse error before `long' src/lxml/lxml.etree.c:110167: `val' undeclared (first use in this function) src/lxml/lxml.etree.c:110167: (Each undeclared identifier is reported only once src/lxml/lxml.etree.c:110167: for each function it appears in.) src/lxml/lxml.etree.c: In function `__pyx_PyInt_AsUnsignedLongLong': src/lxml/lxml.etree.c:110185: parse error before `long' src/lxml/lxml.etree.c:110187: `val' undeclared (first use in this function) error: command 'gcc' failed with exit status 1 1 pytaf@adevp02 .../lxml-2.0beta2 $ Now, when I switch to use gcc 3.4.4 I can build successfully, but: 0 lb54320@adevp02 .../lxml-2.0beta2 $ LD_LIBRARY_PATH=/apps/prod/gcc/3.4.4/lib python2.4 test.py -p -v '' '!test_schematron_invalid*' /data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/html/__init__.py:22: UserWarning: This version of libxml2 has a known XPath bug. Use it at your own risk. _rel_links_xpath = etree.XPath("descendant-or-self::a[@rel]") TESTED VERSION: 2.0.beta2-51091 Python: (2, 4, 4, 'final', 0) lxml.etree: (2, 0, -98, 51091) libxml used: (2, 6, 27) libxml compiled: (2, 6, 27) libxslt used: (1, 1, 20) libxslt compiled: (1, 1, 20) 855/855 (100.0%): Doctest: xpathxslt.txt ====================================================================== FAIL: Doctest: validation.txt ---------------------------------------------------------------------- Traceback (most recent call last): File "/apps/prod//lib/python2.4/unittest.py", line 260, in run testMethod() File "/apps/prod//lib/python2.4/doctest.py", line 2157, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for validation.txt File "/data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/tests/../../../doc/validation.txt", line 0 ---------------------------------------------------------------------- File "/data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/tests/../../../doc/validation.txt", line 113, in validation.txt Failed example: dtd = etree.DTD(external_id = docbook) # requires catalog support Exception raised: Traceback (most recent call last): File "/apps/prod//lib/python2.4/doctest.py", line 1248, in __run compileflags, 1) in test.globs File "<doctest validation.txt[16]>", line 1, in ? dtd = etree.DTD(external_id = docbook) # requires catalog support File "dtd.pxi", line 50, in lxml.etree.DTD.__init__ DTDParseError: failed to load external entity "-//OASIS//DTD DocBook XML V4.2//EN" ---------------------------------------------------------------------- File "/data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/tests/../../../doc/validation.txt", line 116, in validation.txt Failed example: dtd.assertValid(root) # doctest: +ELLIPSIS Expected: Traceback (most recent call last): DocumentInvalid: Element article content does not follow the DTD, ... Got: Traceback (most recent call last): File "/apps/prod//lib/python2.4/doctest.py", line 1248, in __run compileflags, 1) in test.globs File "<doctest validation.txt[18]>", line 1, in ? dtd.assertValid(root) # doctest: +ELLIPSIS File "lxml.etree.pyx", line 2375, in lxml.etree._Validator.assertValid DocumentInvalid: No declaration for element article, line 1 ====================================================================== FAIL: Doctest: validation.txt ---------------------------------------------------------------------- Traceback (most recent call last): File "/apps/prod//lib/python2.4/unittest.py", line 260, in run testMethod() File "/apps/prod//lib/python2.4/doctest.py", line 2157, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for validation.txt File "/data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/tests/../../../doc/validation.txt", line 0 ---------------------------------------------------------------------- File "/data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/tests/../../../doc/validation.txt", line 113, in validation.txt Failed example: dtd = etree.DTD(external_id = docbook) # requires catalog support Exception raised: Traceback (most recent call last): File "/apps/prod//lib/python2.4/doctest.py", line 1248, in __run compileflags, 1) in test.globs File "<doctest validation.txt[16]>", line 1, in ? dtd = etree.DTD(external_id = docbook) # requires catalog support File "dtd.pxi", line 50, in lxml.etree.DTD.__init__ DTDParseError: failed to load external entity "-//OASIS//DTD DocBook XML V4.2//EN" ---------------------------------------------------------------------- File "/data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/tests/../../../doc/validation.txt", line 116, in validation.txt Failed example: dtd.assertValid(root) # doctest: +ELLIPSIS Expected: Traceback (most recent call last): DocumentInvalid: Element article content does not follow the DTD, ... Got: Traceback (most recent call last): File "/apps/prod//lib/python2.4/doctest.py", line 1248, in __run compileflags, 1) in test.globs File "<doctest validation.txt[18]>", line 1, in ? dtd.assertValid(root) # doctest: +ELLIPSIS File "lxml.etree.pyx", line 2375, in lxml.etree._Validator.assertValid DocumentInvalid: No declaration for element article, line 1 ====================================================================== FAIL: Doctest: validation.txt ---------------------------------------------------------------------- Traceback (most recent call last): File "/apps/prod//lib/python2.4/unittest.py", line 260, in run testMethod() File "/apps/prod//lib/python2.4/doctest.py", line 2157, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for validation.txt File "/data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/tests/../../../doc/validation.txt", line 0 ---------------------------------------------------------------------- File "/data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/tests/../../../doc/validation.txt", line 113, in validation.txt Failed example: dtd = etree.DTD(external_id = docbook) # requires catalog support Exception raised: Traceback (most recent call last): File "/apps/prod//lib/python2.4/doctest.py", line 1248, in __run compileflags, 1) in test.globs File "<doctest validation.txt[16]>", line 1, in ? dtd = etree.DTD(external_id = docbook) # requires catalog support File "dtd.pxi", line 50, in lxml.etree.DTD.__init__ DTDParseError: failed to load external entity "-//OASIS//DTD DocBook XML V4.2//EN" ---------------------------------------------------------------------- File "/data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/tests/../../../doc/validation.txt", line 116, in validation.txt Failed example: dtd.assertValid(root) # doctest: +ELLIPSIS Expected: Traceback (most recent call last): DocumentInvalid: Element article content does not follow the DTD, ... Got: Traceback (most recent call last): File "/apps/prod//lib/python2.4/doctest.py", line 1248, in __run compileflags, 1) in test.globs File "<doctest validation.txt[18]>", line 1, in ? dtd.assertValid(root) # doctest: +ELLIPSIS File "lxml.etree.pyx", line 2375, in lxml.etree._Validator.assertValid DocumentInvalid: No declaration for element article, line 1 ====================================================================== FAIL: Doctest: validation.txt ---------------------------------------------------------------------- Traceback (most recent call last): File "/apps/prod//lib/python2.4/unittest.py", line 260, in run testMethod() File "/apps/prod//lib/python2.4/doctest.py", line 2157, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for validation.txt File "/data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/tests/../../../doc/validation.txt", line 0 ---------------------------------------------------------------------- File "/data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/tests/../../../doc/validation.txt", line 113, in validation.txt Failed example: dtd = etree.DTD(external_id = docbook) # requires catalog support Exception raised: Traceback (most recent call last): File "/apps/prod//lib/python2.4/doctest.py", line 1248, in __run compileflags, 1) in test.globs File "<doctest validation.txt[16]>", line 1, in ? dtd = etree.DTD(external_id = docbook) # requires catalog support File "dtd.pxi", line 50, in lxml.etree.DTD.__init__ DTDParseError: failed to load external entity "-//OASIS//DTD DocBook XML V4.2//EN" ---------------------------------------------------------------------- File "/data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/tests/../../../doc/validation.txt", line 116, in validation.txt Failed example: dtd.assertValid(root) # doctest: +ELLIPSIS Expected: Traceback (most recent call last): DocumentInvalid: Element article content does not follow the DTD, ... Got: Traceback (most recent call last): File "/apps/prod//lib/python2.4/doctest.py", line 1248, in __run compileflags, 1) in test.globs File "<doctest validation.txt[18]>", line 1, in ? dtd.assertValid(root) # doctest: +ELLIPSIS File "lxml.etree.pyx", line 2375, in lxml.etree._Validator.assertValid DocumentInvalid: No declaration for element article, line 1 ---------------------------------------------------------------------- Ran 855 tests in 37.860s FAILED (failures=4) 1 lb54320@adevp02 .../lxml-2.0beta2 $ Compared to 2.0alpha (I rebuilt that also with gcc 3.4.4): 2 pytaf@adevp02 .../current $ LD_LIBRARY_PATH=/apps/prod/gcc/3.4.4/lib python2.4 test.py -p -v '' '!test_schematron_invalid*' /data/pydev/DOWNLOADS/LXML/lxml/versions/lxml-2.0alpha-r47832/src/lxml/html/__init__.py:22: UserWarning: This version of libxml2 has a known XPath bug. Use it at your own risk. _rel_links_xpath = etree.XPath("descendant-or-self::a[@rel]") TESTED VERSION: 2.0.alpha4 Python: (2, 4, 4, 'final', 0) lxml.etree: (2, 0, -196, 0) libxml used: (2, 6, 27) libxml compiled: (2, 6, 27) libxslt used: (1, 1, 20) libxslt compiled: (1, 1, 20) 824/824 (100.0%): Doctest: xpathxslt.txt ---------------------------------------------------------------------- Ran 824 tests in 2.698s OK So basically performance drops by factor >10 for me, on a Sparc Solaris 8 box, python2.4, gcc 3.4.4. I haven't yet looked into the failing tests. Remarks: I currently disable the schematron tests because some of them dump core with my setup. Holger -- GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS. Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail
Hi Holger, thanks for the report. jholg@gmx.de wrote:
but I'm having some problems with 2.0beta2:
First of all, it does not build any more using gcc 2.95.2, yes I know, might old compiler...then again, Cython produces C code, not funky C++ stuff (2.0alpha-r47832 still built without problems). This is the error I get:
gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -I/apps/prod//include -I/apps/prod//include/libxml2 -I/apps/prod/include/libxml2 -I/apps/prod/include -I/apps/pydev/hjoukl/include/python2.4 -c src/lxml/lxml.etree.c -o build/temp.solaris-2.8-sun4u-2.4/src/lxml/lxml.etree.o -w src/lxml/lxml.etree.c: In function `__pyx_PyInt_AsLongLong': src/lxml/lxml.etree.c:110165: parse error before `long' src/lxml/lxml.etree.c:110167: `val' undeclared (first use in this function) src/lxml/lxml.etree.c:110167: (Each undeclared identifier is reported only once src/lxml/lxml.etree.c:110167: for each function it appears in.) src/lxml/lxml.etree.c: In function `__pyx_PyInt_AsUnsignedLongLong': src/lxml/lxml.etree.c:110185: parse error before `long' src/lxml/lxml.etree.c:110187: `val' undeclared (first use in this function) error: command 'gcc' failed with exit status 1
Hmm, guess that's something to fix in Cython. The *LongLong() functions are a recent addition for safe type conversion. The line numbers above differ from mine, though. Could you send me the source code of the lines that failed here?
Now, when I switch to use gcc 3.4.4 I can build successfully, but:
0 lb54320@adevp02 .../lxml-2.0beta2 $ LD_LIBRARY_PATH=/apps/prod/gcc/3.4.4/lib python2.4 test.py -p -v '' '!test_schematron_invalid*' /data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/html/__init__.py:22: UserWarning: This version of libxml2 has a known XPath bug. Use it at your own risk. _rel_links_xpath = etree.XPath("descendant-or-self::a[@rel]")
TESTED VERSION: 2.0.beta2-51091 Python: (2, 4, 4, 'final', 0) lxml.etree: (2, 0, -98, 51091) libxml used: (2, 6, 27) libxml compiled: (2, 6, 27) libxslt used: (1, 1, 20) libxslt compiled: (1, 1, 20)
855/855 (100.0%): Doctest: xpathxslt.txt ====================================================================== FAIL: Doctest: validation.txt ---------------------------------------------------------------------- Traceback (most recent call last): File "/apps/prod//lib/python2.4/unittest.py", line 260, in run testMethod() File "/apps/prod//lib/python2.4/doctest.py", line 2157, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for validation.txt File "/data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/tests/../../../doc/validation.txt", line 0
---------------------------------------------------------------------- File "/data/pydev/DOWNLOADS/LXML/lxml/versions/SVN_CHECKOUTS/TAGS/lxml-2.0beta2/src/lxml/tests/../../../doc/validation.txt", line 113, in validation.txt Failed example: dtd = etree.DTD(external_id = docbook) # requires catalog support Exception raised: Traceback (most recent call last): File "/apps/prod//lib/python2.4/doctest.py", line 1248, in __run compileflags, 1) in test.globs File "<doctest validation.txt[16]>", line 1, in ? dtd = etree.DTD(external_id = docbook) # requires catalog support File "dtd.pxi", line 50, in lxml.etree.DTD.__init__ DTDParseError: failed to load external entity "-//OASIS//DTD DocBook XML V4.2//EN" [...]
It seems to lack catalog support. I thought about adding that test or not. Looks like it's better to leave it out.
---------------------------------------------------------------------- Ran 855 tests in 37.860s
Compared to 2.0alpha (I rebuilt that also with gcc 3.4.4):
---------------------------------------------------------------------- Ran 824 tests in 2.698s
So basically performance drops by factor >10 for me, on a Sparc Solaris 8 box, python2.4, gcc 3.4.4. I haven't yet looked into the failing tests.
I wouldn't dare to compare the numbers here, given a difference of 30 tests (especially not knowing which ones are missing). If you get errors, it naturally takes (a bit) longer. Also, it seems to run much less tests, so I guess you either do not have ElementTree installed for the compat tests (though I actually think that's the case for both runs), or it just takes longer to search the (non-existing) catalogs, or ... If you want real numbers, you should rather run the benchmarks.
Remarks: I currently disable the schematron tests because some of them dump core with my setup.
Hmm, now that Cython supports compile time conditional compilation, maybe we should use that in a couple of places... Stefan
thanks for the report.
I'm actually pretty late... sorry for that. Maybe I should set up s.th. like a nightly or weekly checkout/build/test/bench someday.
Hmm, guess that's something to fix in Cython. The *LongLong() functions are a recent addition for safe type conversion.
The line numbers above differ from mine, though. Could you send me the source code of the lines that failed here?
110149 110150 static INLINE int __Pyx_PyObject_IsTrue(PyObject* x) { 110151 if (x == Py_True) return 1; 110152 else if (x == Py_False) return 0; 110153 else return PyObject_IsTrue(x); 110154 } 110155 110156 static INLINE PY_LONG_LONG __pyx_PyInt_AsLongLong(PyObject* x) { 110157 if (PyInt_CheckExact(x)) { 110158 return PyInt_AS_LONG(x); 110159 } 110160 else if (PyLong_CheckExact(x)) { 110161 return PyLong_AsLongLong(x); 110162 } 110163 else { 110164 PyObject* tmp = PyNumber_Int(x); if (!tmp) return (PY_LONG_LONG)-1; 110165 PY_LONG_LONG val = __pyx_PyInt_AsLongLong(tmp); 110166 Py_DECREF(tmp); 110167 return val; 110168 } 110169 } 110170 110171 static INLINE unsigned PY_LONG_LONG __pyx_PyInt_AsUnsignedLongLong(PyObject* x) { 110172 if (PyInt_CheckExact(x)) { 110173 long val = PyInt_AS_LONG(x); 110174 if (unlikely(val < 0)) { 110175 PyErr_SetString(PyExc_TypeError, "Negative assignment to unsigned type."); 110176 return (unsigned PY_LONG_LONG)-1; 110177 } 110178 return val; 110179 } 110180 else if (PyLong_CheckExact(x)) { 110181 return PyLong_AsUnsignedLongLong(x); 110182 } 110183 else { 110184 PyObject* tmp = PyNumber_Int(x); if (!tmp) return (PY_LONG_LONG)-1; 110185 PY_LONG_LONG val = __pyx_PyInt_AsUnsignedLongLong(tmp); 110186 Py_DECREF(tmp); 110187 return val; 110188 } 110189 } 110190 110191
It seems to lack catalog support. I thought about adding that test or not. Looks like it's better to leave it out.
Is that my libmxl2 that's lacking catalog support? Or do these tests try to access s.th. from the web?
I wouldn't dare to compare the numbers here, given a difference of 30 tests (especially not knowing which ones are missing). If you get errors, it naturally takes (a bit) longer. Also, it seems to run much less tests, so I guess you either do not have ElementTree installed for the compat tests (though I actually think that's the case for both runs), or it just takes longer to search the (non-existing) catalogs, or ...
If you want real numbers, you should rather run the benchmarks.
Your're right, of course, I'm gonna do that. Got a little carried away as I could actually see the test count going up ever so slowly compared to my last build. Btw I noticed that the file bench.py is missing, which the Makefile tries to invoke. Is there any existing tools to compare logs of different benchmark runs? Guess I'm gonna hack up s.th. rather than check differences manually... Thanks, Holger -- GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS. Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail
Hi, jholg@gmx.de wrote:
thanks for the report.
I'm actually pretty late... sorry for that. Maybe I should set up s.th. like a nightly or weekly checkout/build/test/bench someday.
good idea. :)
110165 PY_LONG_LONG val = __pyx_PyInt_AsLongLong(tmp);
Ok, so this line failed with src/lxml/lxml.etree.c:110165: parse error before `long' which lets me assume that PY_LONG_LONG is the usual "long long" on your machine, which apparently fails in gcc 2.95. Let me guess: your Python install was not compiled with gcc 2.95, was it?
It seems to lack catalog support. I thought about adding that test or not. Looks like it's better to leave it out.
Is that my libmxl2 that's lacking catalog support?
Yes, or maybe just the catalog itself. I have the DocBook DTD locally in my catalog under /usr/share/xml/docbook/schema/dtd/
Or do these tests try to access s.th. from the web?
They shouldn't. Though, maybe, ... Guess it's just best to switch them off. :)
Btw I noticed that the file bench.py is missing, which the Makefile tries to invoke.
Ah, had forgotten about that target. Fixed.
Is there any existing tools to compare logs of different benchmark runs? Guess I'm gonna hack up s.th. rather than check differences manually...
That would be very helpful. The benchmarks just output a whole bunch of numbers, but I never got around to making anything more legible from them. Maybe we should have some kind of an "ETstone" or something, that would output a single number for ET performance. Or maybe one for parsing, one for serialising and one for the API or something in that line. Stefan
Hi Stefan,
which lets me assume that PY_LONG_LONG is the usual "long long" on your machine, which apparently fails in gcc 2.95.
Let me guess: your Python install was not compiled with gcc 2.95, was it?
It was compiled with 2.95: 0 pytaf@adevp02 .../lxml-2.0beta2 $ gcc -dumpversion 2.95.2 0 pytaf@adevp02 .../lxml-2.0beta2 $ /apps/pydev/hjoukl/bin/python2.4 setup.py build Building with Cython 0.9.6.11b. Building lxml version 2.0.beta2-51091. running build running build_py writing byte-compilation script '/tmp/tmp61b-gM.py' /apps/pydev/hjoukl/bin/python2.4 -O /tmp/tmp61b-gM.py removing /tmp/tmp61b-gM.py running build_ext building 'lxml.etree' extension gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -I/apps/prod//include -I/apps/prod//include/libxml2 -I/apps/prod/include/libxml2 -I/apps/prod/include -I/apps/pydev/hjoukl/include/python2.4 -c src/lxml/lxml.etree.c -o build/temp.solaris-2.8-sun4u-2.4/src/lxml/lxml.etree.o -w src/lxml/lxml.etree.c: In function `__pyx_PyInt_AsLongLong': src/lxml/lxml.etree.c:110165: parse error before `long' src/lxml/lxml.etree.c:110167: `val' undeclared (first use in this function) src/lxml/lxml.etree.c:110167: (Each undeclared identifier is reported only once src/lxml/lxml.etree.c:110167: for each function it appears in.) src/lxml/lxml.etree.c: In function `__pyx_PyInt_AsUnsignedLongLong': src/lxml/lxml.etree.c:110185: parse error before `long' src/lxml/lxml.etree.c:110187: `val' undeclared (first use in this function) error: command 'gcc' failed with exit status 1 1 pytaf@adevp02 .../lxml-2.0beta2 $ 0 pytaf@adevp02 .../lxml-2.0beta2 $ 0 pytaf@adevp02 .../lxml-2.0beta2 $ 0 pytaf@adevp02 .../lxml-2.0beta2 $ 0 pytaf@adevp02 .../lxml-2.0beta2 $ 0 pytaf@adevp02 .../lxml-2.0beta2 $ 0 pytaf@adevp02 .../lxml-2.0beta2 $/apps/pydev/hjoukl/bin/python2.4 Python 2.4.4 (#1, Mar 6 2007, 11:22:31) [GCC 2.95.2 19991024 (release)] on sunos5 Type "help", "copyright", "credits" or "license" for more information.
0 pytaf@adevp02 .../lxml-2.0beta2 $
So I don't *think* it's s.th. to do with my setup, which is the same I used with the successful 2.0alpha builds (older cython version then, of course). Holger -- Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! Ideal für Modem und ISDN: http://www.gmx.net/de/go/smartsurfer
Hi Holger, jholg@gmx.de wrote:
src/lxml/lxml.etree.c: In function `__pyx_PyInt_AsLongLong': src/lxml/lxml.etree.c:110165: parse error before `long' src/lxml/lxml.etree.c:110167: `val' undeclared (first use in this function) src/lxml/lxml.etree.c:110167: (Each undeclared identifier is reported only once src/lxml/lxml.etree.c:110167: for each function it appears in.) src/lxml/lxml.etree.c: In function `__pyx_PyInt_AsUnsignedLongLong': src/lxml/lxml.etree.c:110185: parse error before `long' src/lxml/lxml.etree.c:110187: `val' undeclared (first use in this function) error: command 'gcc' failed with exit status 1
I forwarded this to the Cython list, let's see what that gives. Stefan
Hi Holger, Stefan Behnel wrote:
jholg@gmx.de wrote:
src/lxml/lxml.etree.c: In function `__pyx_PyInt_AsLongLong': src/lxml/lxml.etree.c:110165: parse error before `long' src/lxml/lxml.etree.c:110167: `val' undeclared (first use in this function) src/lxml/lxml.etree.c:110167: (Each undeclared identifier is reported only once src/lxml/lxml.etree.c:110167: for each function it appears in.) src/lxml/lxml.etree.c: In function `__pyx_PyInt_AsUnsignedLongLong': src/lxml/lxml.etree.c:110185: parse error before `long' src/lxml/lxml.etree.c:110187: `val' undeclared (first use in this function) error: command 'gcc' failed with exit status 1
I forwarded this to the Cython list, let's see what that gives.
And it helped! :) http://comments.gmane.org/gmane.comp.python.cython.devel/588 Here's a fix for Cython. Stefan
I forwarded this to the Cython list, let's see what that gives.
And it helped! :)
http://comments.gmane.org/gmane.comp.python.cython.devel/588
Here's a fix for Cython.
Stefan
Thanks very much, I'll try that out. You guys are lightspeed, as ever. That's one of another big point for lxml, btw: Mailing-list responsiveness by its maintainer, and experienced users. I tried to do a little bit of performance comp. This is the stuff that seems to be >20% slower for me since 2.0alpha: 0 lb54320@adevp02 .../lxml-2.0beta2 $ /data/pydev/hjoukl/python/pysource/tools/lxml_benchcmp.py /data/tmp/pytaf/benchmarks/etree_2.0alpha.log /data/tmp/pytaf/benchmarks/etree_2.0beta2.log --tolerance 20 --loglevel MUCHSLOWER lxe: index_slice_neg (--TR T1 ): 0.02000000 <<< 0.14900000 msec/pass (+6.450000) !!! lxe: index_slice_neg (--TR T4 ): 0.00790000 <<< 0.10700000 msec/pass (+12.544304) !!! lxe: replace_children (--TC T2 ): 0.27700000 <<< 0.39790000 msec/pass (+0.436462) !!! lxe: replace_children (--TC T1 ): 0.03290000 <<< 0.04200000 msec/pass (+0.276596) !!! lxe: index_slice (--TR T3 ): 0.01100000 <<< 0.01410000 msec/pass (+0.281818) !!! lxe: replace_children (--TC T4 ): 0.03290000 <<< 0.04080000 msec/pass (+0.240122) !!! 0 lb54320@adevp02 .../lxml-2.0beta2 $ I hacked up a little script to produce this (attached), not tested at all yet. I won't be able to check the patch until monday, unfortunately (unless I install a 2.95.2 on my linux box at home, that is ;-) Holger -- GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS. Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail
Hi Holger, jholg@gmx.de wrote:
I tried to do a little bit of performance comp. This is the stuff that seems to be >20% slower for me since 2.0alpha:
0 lb54320@adevp02 .../lxml-2.0beta2 $ /data/pydev/hjoukl/python/pysource/tools/lxml_benchcmp.py /data/tmp/pytaf/benchmarks/etree_2.0alpha.log /data/tmp/pytaf/benchmarks/etree_2.0beta2.log --tolerance 20 --loglevel MUCHSLOWER lxe: index_slice_neg (--TR T1 ): 0.02000000 <<< 0.14900000 msec/pass (+6.450000) !!! lxe: index_slice_neg (--TR T4 ): 0.00790000 <<< 0.10700000 msec/pass (+12.544304) !!! lxe: replace_children (--TC T2 ): 0.27700000 <<< 0.39790000 msec/pass (+0.436462) !!! lxe: replace_children (--TC T1 ): 0.03290000 <<< 0.04200000 msec/pass (+0.276596) !!! lxe: index_slice (--TR T3 ): 0.01100000 <<< 0.01410000 msec/pass (+0.281818) !!! lxe: replace_children (--TC T4 ): 0.03290000 <<< 0.04080000 msec/pass (+0.240122) !!! 0 lb54320@adevp02 .../lxml-2.0beta2 $
Hmm, interesting. I'll look over that when I find the time. This is not release critical. Thanks, Stefan
Hi Stefan,
src/lxml/lxml.etree.c: In function `__pyx_PyInt_AsLongLong': src/lxml/lxml.etree.c:110165: parse error before `long' src/lxml/lxml.etree.c:110167: `val' undeclared (first use in this function) src/lxml/lxml.etree.c:110167: (Each undeclared identifier is reported only once src/lxml/lxml.etree.c:110167: for each function it appears in.) src/lxml/lxml.etree.c: In function `__pyx_PyInt_AsUnsignedLongLong': src/lxml/lxml.etree.c:110185: parse error before `long' src/lxml/lxml.etree.c:110187: `val' undeclared (first use in this function) error: command 'gcc' failed with exit status 1
I forwarded this to the Cython list, let's see what that gives.
And it helped! :)
http://comments.gmane.org/gmane.comp.python.cython.devel/588
Here's a fix for Cython.
Stefan
As expected, fix works smoothly! Thank you, Holger -- GMX FreeMail: 1 GB Postfach, 5 E-Mail-Adressen, 10 Free SMS. Alle Infos und kostenlose Anmeldung: http://www.gmx.net/de/go/freemail
Hi Holger, jholg@gmx.de wrote:
Ran 855 tests in 37.860s
Compared to 2.0alpha (I rebuilt that also with gcc 3.4.4):
Ran 824 tests in 2.698s
So basically performance drops by factor >10 for me
Sorry, I had completely forgotten that I switched on garbage collection between test runs somewhere during the alpha cycle. The test run you are comparing to does not explicitly run GC after each test to catch refcounting bugs. That's what makes the difference here, not lxml itself. Stefan
participants (2)
-
jholg＠gmx.de -
Stefan Behnel