[lxml-dev] Threading redux
Hi all, there is now a branch where I implemented thread concurrency for the parsers. http://codespeak.net/svn/lxml/branch/threading/ This branch is not very well tested. I attached the only test program I have so far. There are likely some bizarre race conditions that I overlooked. Note that parsers are not currently locked, so lxml will crash if multiple threads use /the same/ parser concurrently. I'll see how to do that efficiently. If you have an interest in getting this integrated, please build the branch on your side and do some testing. There are other places where we can try to add concurrency, but that is still to come. Feel free to post a list of personal preferences including little test programs (like the one attached) that show concurrent usage. Stefan
Hi again,
there is now a branch where I implemented thread concurrency for the parsers. http://codespeak.net/svn/lxml/branch/threading/
Here is a new (and slightly more complex) test program that also tests serialisation. The current branch status for parsing and serialisation is: * all in-memory operations (tostring, parse(StringIO), etc.) free the GIL * file operations (on file names) free the GIL * reading from file-like objects frees the GIL and reacquires it for reading * serialisation to file-like objects is single-threaded (high lock overhead) Note that you *must* create independent parsers for each thread. Sharing parsers between threads will serialise the calls or (currently) crash. I'd be very happy if anyone who has a multi-processor machine could give it a try. That would allow us to see if there are 'real' concurrency issues and if lxml achieves to saturate more than one processor. Any feedback is appreciated. Stefan
participants (1)
-
Stefan Behnel