[lxml-dev] reorganisation confusion

Hi there, Tonight I attempted a package reorganisation myself, integrating vlibxml2 into lxml from the other direction, leaving most of lxml's build scripts and testing framework intact. My attempts are here: http://codespeak.net/svn/lxml/branch/lxml-reorg/ The build story works ('make' does the trick). The tests are picked up by the test runner. I noticed that on reorg-20041120 'make' tries to install in site-packages directly, which is something that doesn't happen on my branch. I think it's cleaner not to have to install in site packages just for testing, and the test.py runner takes care of this in lxml. Another thing I tried to do conscientiously on my branch is use 'svn move', so that version control history of various files is not lost during reorganisation. Did you use that, Victor? Everything is currently in the lxml directory (except for vlibxml2_mod.so, but I tried placing it there as well and it's an easy change to setup.py). The problem is that I can't get vlibxml2 tests to get anywhere near passing. When running the tests, I get many, many obscure errors along these lines: Exception exceptions.NameError: 'vlibxml2' in <lxml.vlibxml2_subclasses.xmlNodeSub object at 0x404337cc> ignored Alternate source organisations result in import errors instead.. I tried rearranging the source every which way, but I think the problem remains that vlibxml2_mod uses (through wrapfuncs.pxi) vlibxml2_subclasses.py, where vlibxml2_subclasses.py imports from vlibxml2_mod to actually do the subclassing. This circular setup is driving me rather crazy. I suspect that the original vlibxml2 setup had some magic source code organisation that somehow avoided this problem. I do not know where the magic is though; Victor, could you perhaps explain? Preferably I'd like to break this circular import, as it's obviously hard to maintain. I apparently can't do it. :) Is there any way to accomplish this? I understand that we need to subclass for weak-referencability, but is there any way to do this in .pyx code directly instead of in Python? Or could we make the weakreffable classes the root base class, so there's no need for them to inherit from vlibxml2_mod classes first, and then be used in vlibxml2_mod again? Please help! :) Regards, Martijn

Hi, I haven't been able to do much work on the lxml port as I've been swamped with work and my wrists were acting up (too much typing).
I'm pretty sure I have - the subversion logs seem to indicate that I did. Are you seeing something odd in the repository?
Weird. I haven't had a chance to look at what's going on internally. I'm not really sure how much time I can spend on lxml right now as my spare time can no longer include working on the lxml code.
Not much magic going on. At least none that I was aware of. There's a couple spots in the test code where tests are now no longer valid. You really do need to be checking libxml2 if the memory allocation is fixed. You can't check the node cache as that's not a reliable way of counting bytes allocated.
You currently can't make Pyrex objects have weak references without doing the subclass thing. I don't think there is a circular import going on. I'll see if I can take a look at it later this week, but I'm pretty sure I won't. My fingers hurt. vic

Victor Ng wrote:
I haven't been able to do much work on the lxml port as I've been swamped with work and my wrists were acting up (too much typing).
No problem; I'm intending to finish it if you can only give me a little bit of help. I'd like to preserve some of the lxml infrastructure that your branch throws out (the test runner and the makefile in particular). [breaking the circular import]
Hm, no circular import, I wonder what I am doing in that case. One way to simplify this would be to have two very dummy base classes, and subclass them from Pyrex, if that's possible. That way the dummy base classes don't need to subclass from pyrex objects instead, which causes this odd cycle. I'll try to give the integration effort another try later this week. Perhaps I'll read the code in more detail and try to understand your memory management strategy better and extract just that out as a base. Good luck with your fingers! Thanks, Martijn

Hi, I haven't been able to do much work on the lxml port as I've been swamped with work and my wrists were acting up (too much typing).
I'm pretty sure I have - the subversion logs seem to indicate that I did. Are you seeing something odd in the repository?
Weird. I haven't had a chance to look at what's going on internally. I'm not really sure how much time I can spend on lxml right now as my spare time can no longer include working on the lxml code.
Not much magic going on. At least none that I was aware of. There's a couple spots in the test code where tests are now no longer valid. You really do need to be checking libxml2 if the memory allocation is fixed. You can't check the node cache as that's not a reliable way of counting bytes allocated.
You currently can't make Pyrex objects have weak references without doing the subclass thing. I don't think there is a circular import going on. I'll see if I can take a look at it later this week, but I'm pretty sure I won't. My fingers hurt. vic

Victor Ng wrote:
I haven't been able to do much work on the lxml port as I've been swamped with work and my wrists were acting up (too much typing).
No problem; I'm intending to finish it if you can only give me a little bit of help. I'd like to preserve some of the lxml infrastructure that your branch throws out (the test runner and the makefile in particular). [breaking the circular import]
Hm, no circular import, I wonder what I am doing in that case. One way to simplify this would be to have two very dummy base classes, and subclass them from Pyrex, if that's possible. That way the dummy base classes don't need to subclass from pyrex objects instead, which causes this odd cycle. I'll try to give the integration effort another try later this week. Perhaps I'll read the code in more detail and try to understand your memory management strategy better and extract just that out as a base. Good luck with your fingers! Thanks, Martijn
participants (2)
-
Martijn Faassen
-
Victor Ng