Different behavior of twistd with and without -n
I'm not sure if this belongs in Twisted Python or Twisted Web. I have a twisted.web2 server that, among other things, makes calls to PyLucene. Without going into detail, I'd like to ask a general question to see if there's a high-level explanation for what I'm seeing. When I run twistd without -n, I can take actions in the web UI that fairly reliably cause a call into PyLucene that stops the twistd server. I don't know what's going on, but the server definitely dies (it no longer appears in ps, and you can't telnet to its port). There's nothing in the log file, no core file (I have ulimit -c unlimited), nothing. This is on Mac OS X 10.4.9, PyLucene 2.1.0-2, Python 2.5, Twisted trunk revision 20059. When I stick in debug prints to stderr, I can see that (usually) a call to PyLucene.IndexWriter.optimize and (sometimes) a call to PyLucene.IndexReader.open never returns. The weird thing is that when I run twistd -n using the same .tac file, I can't make the thing fall over. I can click away to my heart's content in the application UI, and it all just works. I also tried running twistd under pydb and the thing wouldn't crash. While I realize there's probably something up with PyLucene, I'm wondering if anyone can suggest why running with twistd -n is preventing the crashing. What else does -n do, other than (presumably) forking and running the server in the child? Regards, Terry
On Thu, 24 May 2007 16:12:28 +0200, Terry Jones <terry@jon.es> wrote:
I'm not sure if this belongs in Twisted Python or Twisted Web.
[snip - errors with twistd and PyLucene, unreproducable with twistd -n]
While I realize there's probably something up with PyLucene, I'm wondering if anyone can suggest why running with twistd -n is preventing the crashing. What else does -n do, other than (presumably) forking and running the server in the child?
A completely wild guess is that forking is confusing PyLucene in a fatal way. Are you importing PyLucene in the .tac file itself? If so, it may help to avoid doing this, so that no code from PyLucene even gets a chance to run until after the process has already daemonized. Jean-Paul
A completely wild guess is that forking is confusing PyLucene in a fatal way.
FWIW, I have an application running CherryPy over Twisted, using PyLucene. Since PyLucene isn't thread-safe, every thread must subclass PyLucene.PythonThread, so the underlying "java runtime garbage collector is aware about the python thread upon their creation" (http://lists.osafoundation.org/pipermail/pylucene-dev/2004-June/000037.html). Without this, the apps simply crashes (segfault, see also http://chandlerproject.org/PyLucene/ThreadingInPyLucene). Maybe there's something about that while twistd is daemonizing the process ? Hope that helps. Cheers, Seb -- Sébastien LELONG http://www.sirloon.net sebastien.lelong[at]sirloon.net
Terry Jones napisał(a):
While I realize there's probably something up with PyLucene, I'm wondering if anyone can suggest why running with twistd -n is preventing the crashing. What else does -n do, other than (presumably) forking and running the server in the child?
I have other issues regarding forking in twistd and PyLucene. My server works reliably (on linux, didn't try on OSX) only if I run it without daemonizong twistd. If I daemonize twistd, the memory consumption grows out of control and finally system's oomkiller starts killing processes. There's one Queue object that holds indexing requests and it grows indefinitely without freeing any memory, so the whole twistd process eats more and more memory while requests come to the system. As a temporary solution we start the server under screen session. -- Jarek Zgoda "We read Knuth so you don't have to."
participants (4)
-
Jarek Zgoda -
Jean-Paul Calderone -
Sébastien LELONG -
Terry Jones