[lxml-dev] Faster iteration and deallocation
Hi all, I know, we already had our last beta and 1.0 is pretty close, but I decided to do some more major clean ups anyway and found a lot of code duplication between functions that iterate over the tree. I replaced that by a set of BEGIN/END C macros, which has three effects: * Tree iteration now uses the same code throughout lxml * It's much shorter and thus (hopefully) easier to understand * It's about 20-30% faster than the previous recursive implementations I took another somewhat experimental step and removed the _NodeBase superclass from _Attrib to make it a plain Python object that sits on an _Element. This allowed for another major clean up and a huge simplification of the proxy code, since there is only one proxy type left. I know that these changes impact pretty much every part of lxml and I hope they did not introduce any of those beautiful little bugs which I already squeezed so many of recently. Since they simplify and merge a lot of code, I hope they also make the code easier to maintain. According to the test cases, everything works nicely as before and I will give valgrind a run to see if there's anything suspicious left. It would be a great help if I could get some feedback on the stability of the modified trunk version in other applications to see if the test cases speak the truth. To do this without installing it, checkout the trunk and compile it (make clean inplace). You can then add the 'src' directory to the PYTHONPATH and try running your programs with it. Any feedback is appreciated. Stefan
Hello Stefan, Wednesday, May 31, 2006, 5:17:58 AM, you wrote:
Hi all,
I know, we already had our last beta and 1.0 is pretty close, but I decided to do some more major clean ups anyway and found a lot of code duplication between functions that iterate over the tree.
I replaced that by a set of BEGIN/END C macros, which has three effects: * Tree iteration now uses the same code throughout lxml * It's much shorter and thus (hopefully) easier to understand * It's about 20-30% faster than the previous recursive implementations
I took another somewhat experimental step and removed the _NodeBase superclass from _Attrib to make it a plain Python object that sits on an _Element. This allowed for another major clean up and a huge simplification of the proxy code, since there is only one proxy type left.
I know that these changes impact pretty much every part of lxml and I hope they did not introduce any of those beautiful little bugs which I already squeezed so many of recently. Since they simplify and merge a lot of code, I hope they also make the code easier to maintain.
According to the test cases, everything works nicely as before and I will give valgrind a run to see if there's anything suspicious left.
It would be a great help if I could get some feedback on the stability of the modified trunk version in other applications to see if the test cases speak the truth. To do this without installing it, checkout the trunk and compile it (make clean inplace). You can then add the 'src' directory to the PYTHONPATH and try running your programs with it.
Any feedback is appreciated. What about a beta 2 ? Will help test those new changes made recently, whose were not few. Do you have a planned date for the 1.0 release ?
-- Best regards, Steve mailto:howe@carcass.dhs.org
Steve Howe a écrit :
It would be a great help if I could get some feedback on the stability of the modified trunk version in other applications to see if the test cases speak the truth. To do this without installing it, checkout the trunk and compile it (make clean inplace). You can then add the 'src' directory to the PYTHONPATH and try running your programs with it.
Any feedback is appreciated.
I just did a `cd lxml/trunk && svn up && make test && make bench` and all went well and I'm now successfully using it for some simple parsing / xpath related tests (on a 32bit linux box).
What about a beta 2 ? Will help test those new changes made recently, whose were not few. Do you have a planned date for the 1.0 release ?
+1. Georges told me a couple of days ago that a "make test" was broken on Mac OSX for 1.0.beta but he did not had time to report it on the list so it might be a good idea to have a success report on MacOSX before releasing 1.0 final. -- Olivier
Le 31 mai 2006, à 15:23, Olivier Grisel a écrit :
Steve Howe a écrit :
It would be a great help if I could get some feedback on the stability of the modified trunk version in other applications to see if the test cases speak the truth. To do this without installing it, checkout the trunk and compile it (make clean inplace). You can then add the 'src' directory to the PYTHONPATH and try running your programs with it.
Any feedback is appreciated.
I just did a `cd lxml/trunk && svn up && make test && make bench` and all went well and I'm now successfully using it for some simple parsing / xpath related tests (on a 32bit linux box).
What about a beta 2 ? Will help test those new changes made recently, whose were not few. Do you have a planned date for the 1.0 release ?
+1. Georges told me a couple of days ago that a "make test" was broken on Mac OSX for 1.0.beta but he did not had time to report it on the list so it might be a good idea to have a success report on MacOSX before releasing 1.0 final.
That's right. In fact, I hadn't enough time to check if it my setup was faulty or not. I'll try again more toroughly in a few days and keep the list posted (if noone did it before of course)
-- Olivier
_______________________________________________ lxml-dev mailing list lxml-dev@codespeak.net http://codespeak.net/mailman/listinfo/lxml-dev
Hi George, Georges Racinet wrote:
Le 31 mai 2006, à 15:23, Olivier Grisel a écrit :
Steve Howe a écrit :
What about a beta 2 ? Will help test those new changes made recently, whose were not few.
+1. Georges told me a couple of days ago that a "make test" was broken on Mac OSX for 1.0.beta but he did not had time to report it on the list so it might be a good idea to have a success report on MacOSX before releasing 1.0 final.
That's right. In fact, I hadn't enough time to check if it my setup was faulty or not. I'll try again more toroughly in a few days and keep the list posted (if noone did it before of course)
Ok, I guess that's a reason to postpone. If we have a 1.0, it should compile as nicely as possible on the broadest possible number of machines. It's a pretty crappy thing to bother users with a bug-fix version that only fixes a compilation bug on one platform... Admittedly, I did a couple of changes in setup.py, e.g. for linking against libexslt and to provide a fill-in space for a static linker configuration, so maybe that broke something in your local setup. Anyway, I'd be happy if you could give us some feedback soon, so that we can get 1.0 out of the door. I'll send you a clean tar.gz so that you can try from a real pre-release version. Stefan
Hi Steve, Steve Howe wrote:
Wednesday, May 31, 2006, 5:17:58 AM, you wrote:
I know, we already had our last beta and 1.0 is pretty close, but I decided to do some more major clean ups anyway [ tree iteration and proxy cleanup ] I know that these changes impact pretty much every part of lxml and I hope they did not introduce any of those beautiful little bugs which I already squeezed so many of recently. Since they simplify and merge a lot of code, I hope they also make the code easier to maintain.
It would be a great help if I could get some feedback on the stability of the modified trunk version in other applications to see if the test cases speak the truth. To do this without installing it, checkout the trunk and compile it (make clean inplace). You can then add the 'src' directory to the PYTHONPATH and try running your programs with it.
What about a beta 2 ? Will help test those new changes made recently, whose were not few.
Most of them were bug fixes that are now checked by test cases and confirmed by those who found them. Apart from the two things I mentioned above, there are very few new features. It's the first time since 0.8 that we have a longer fixed-bugs list than new-features list in the ChangeLog. But that's maybe just because we had a beta in-between... :) So, I don't know, I kinda feel tempted to just throw out a 1.0 rather than a new beta. Since both of the bigger changes above are mainly simplifications and code cleanup, I don't really think it's worth bothering people with a second beta. I also checked a few of my own lxml applications to see if they work as expected and I was pretty shocked how fast they were now. I just clicked on 'load file' in "Slow" and the complete GUI was just there, instantaneously. That's pretty baffling if you don't expect it. I mean, Slow uses all sorts of weird XML/XPath/XSLT stuff to work on its internal XML model. It's not supposed to be fast! :) MathDOM's also working nicely and its little test suite is also just breezing by. I really don't see that many things that could go wrong with 1.0.
Do you have a planned date for the 1.0 release ?
Yep, tomorrow. :) Stefan
Hi Steve, Steve Howe wrote:
Ok, please send a message when you freeze the trunk so I can compile the Windows and FreeBSD eggs.
You should better compile them from the source tgz. If you compile from a SVN checkout, setup.py will pick up the SVN revision. I'll upload the signed tgz to cheeseshop and announce it on the list, as usual. I can also send a message to you, George and Olivier first. Note that I'm not sure we'll really make it for tomorrow. Stefan
Hello Stefan, Wednesday, May 31, 2006, 2:13:09 PM, you wrote:
Steve Howe wrote:
Ok, please send a message when you freeze the trunk so I can compile the Windows and FreeBSD eggs.
You should better compile them from the source tgz. If you compile from a SVN checkout, setup.py will pick up the SVN revision. I'll upload the signed tgz to cheeseshop and announce it on the list, as usual. I can also send a message to you, George and Olivier first. Note that I'm not sure we'll really make it for tomorrow. Ok, whenever it happens :)
-- Best regards, Steve mailto:howe@carcass.dhs.org
participants (4)
-
Georges Racinet -
Olivier Grisel -
Stefan Behnel -
Steve Howe