[Twisted-Python] Thread Consumption Problem in Daemon?
![](https://secure.gravatar.com/avatar/951c999448b4939261b2aea4a753a511.jpg?s=120&d=mm&r=g)
Hi, I am using twisted to run my Flask app via WSGI like so. twistd --pidfile $PORT/pidfile -l $PORT/logfile -n web --port tcp:$PORT --wsgi my.app Naturally, I have functions representing routes that enter and exit just fine. However, I notice the twisted daemon process is :"gathering threads". Eventually system runs out of them. Here's a full status for one twisted server. 504 threads??? Name: twistd Umask: 0077 State: S (sleeping) Tgid: 54855 Ngid: 35415 Pid: 54855 PPid: 1 TracerPid: 0 Uid: 4052 4052 4052 4052 Gid: 4052 4052 4052 4052 FDSize: 256 Groups: 4052 VmPeak: 34240104 kB VmSize: 34239336 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 1942708 kB VmRSS: 1871884 kB RssAnon: 1834800 kB RssFile: 37080 kB RssShmem: 4 kB VmData: 33310576 kB VmStk: 284 kB VmExe: 4 kB VmLib: 234176 kB VmPTE: 8876 kB VmSwap: 0 kB Threads: 504 SigQ: 1/1546652 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000001001007 SigCgt: 00000001800146e8 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: 0000001fffffffff CapAmb: 0000000000000000 Seccomp: 0 Speculation_Store_Bypass: thread vulnerable Cpus_allowed: ffffff,ffffffff Cpus_allowed_list: 0-55 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 358534596 nonvoluntary_ctxt_switches: 31738
![](https://secure.gravatar.com/avatar/fbd473e7e3b6675a84cd3c3b4a2c1972.jpg?s=120&d=mm&r=g)
On maandag 19 november 2018 12:40:20 CET Darren Govoni wrote:
I have a server running inside twistd which uses exactly 1 thread after running for a few weeks, so the problem may not be in twistd itself. I'm using a reverse-proxy HTTP setup though, not WSGI. Maybe the problem is specific to WSGI, Flask or your application? Bye, Maarten
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On Mon, Nov 19, 2018 at 8:26 AM Maarten ter Huurne <maarten@treewalker.org> wrote:
Twisted's WSGI support definitely uses threads (as this is essentially a requirement of WSGI). It uses the reactor thread pool (if you launch it from the CLI with twistd) which used to be limited to 10 threads. I don't know if the same limit is in place these days. Jean-Paul
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Nov 19, 2018, at 6:16 AM, Darren Govoni <darrenuspto@gmail.com> wrote:
I tried to find out if there is a way to limit the thread pool size from command line for twisted web and found nothing. Does it exist?
The thread pool is limited to 10. While this is configurable via the API, no command line option is exposed to tune it. (This would be a great contribution if you were so inclined!) It seems likely to me that Flask is spawning background threads for some reason; given the way Twisted's threadpool works, leaks like this are not common. However, anything is possible: you probably want to gather some information about what all those threads are doing. Can you add a route to your Flask application to call https://docs.python.org/3/library/faulthandler.html#dumping-the-traceback <https://docs.python.org/3/library/faulthandler.html#dumping-the-traceback> and then save the output somewhere? That will at least give you a hint as to whether Twisted's threadpool is at fault, and if it is, what it is blocked doing. -g
![](https://secure.gravatar.com/avatar/599519579a707ab348b35cf68477df08.jpg?s=120&d=mm&r=g)
On 22/11/2018 02:30, Glyph wrote:
Some ideas on this front: - pstree/ps and strace will tell you at a low level - http://pyrasite.com/ and then use Python's thread introspection stuff. cheers, Chris
![](https://secure.gravatar.com/avatar/951c999448b4939261b2aea4a753a511.jpg?s=120&d=mm&r=g)
Thanks. I added Tipper to my program and will see what it shows when I ping the process. https://pypi.org/project/tipper/ On Thu, Nov 22, 2018 at 6:43 AM Chris Withers <chris@withers.org> wrote:
![](https://secure.gravatar.com/avatar/fbd473e7e3b6675a84cd3c3b4a2c1972.jpg?s=120&d=mm&r=g)
On maandag 19 november 2018 12:40:20 CET Darren Govoni wrote:
I have a server running inside twistd which uses exactly 1 thread after running for a few weeks, so the problem may not be in twistd itself. I'm using a reverse-proxy HTTP setup though, not WSGI. Maybe the problem is specific to WSGI, Flask or your application? Bye, Maarten
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On Mon, Nov 19, 2018 at 8:26 AM Maarten ter Huurne <maarten@treewalker.org> wrote:
Twisted's WSGI support definitely uses threads (as this is essentially a requirement of WSGI). It uses the reactor thread pool (if you launch it from the CLI with twistd) which used to be limited to 10 threads. I don't know if the same limit is in place these days. Jean-Paul
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Nov 19, 2018, at 6:16 AM, Darren Govoni <darrenuspto@gmail.com> wrote:
I tried to find out if there is a way to limit the thread pool size from command line for twisted web and found nothing. Does it exist?
The thread pool is limited to 10. While this is configurable via the API, no command line option is exposed to tune it. (This would be a great contribution if you were so inclined!) It seems likely to me that Flask is spawning background threads for some reason; given the way Twisted's threadpool works, leaks like this are not common. However, anything is possible: you probably want to gather some information about what all those threads are doing. Can you add a route to your Flask application to call https://docs.python.org/3/library/faulthandler.html#dumping-the-traceback <https://docs.python.org/3/library/faulthandler.html#dumping-the-traceback> and then save the output somewhere? That will at least give you a hint as to whether Twisted's threadpool is at fault, and if it is, what it is blocked doing. -g
![](https://secure.gravatar.com/avatar/599519579a707ab348b35cf68477df08.jpg?s=120&d=mm&r=g)
On 22/11/2018 02:30, Glyph wrote:
Some ideas on this front: - pstree/ps and strace will tell you at a low level - http://pyrasite.com/ and then use Python's thread introspection stuff. cheers, Chris
![](https://secure.gravatar.com/avatar/951c999448b4939261b2aea4a753a511.jpg?s=120&d=mm&r=g)
Thanks. I added Tipper to my program and will see what it shows when I ping the process. https://pypi.org/project/tipper/ On Thu, Nov 22, 2018 at 6:43 AM Chris Withers <chris@withers.org> wrote:
participants (5)
-
Chris Withers
-
Darren Govoni
-
Glyph
-
Jean-Paul Calderone
-
Maarten ter Huurne