On Apr 2, 2015, at 4:03 PM, Burak Arslan <burak.arslan@arskom.com.tr> wrote:

Hi,

I happen to have the new logging hotness(tm) running in production as the backend to python's own logging module in one of the daemons that I'm responsible for.

Thanks for trying out this code and helping us validate it!

I also have dowser[1] integrated with this one so tryng to optimize memory usage is almost fun :) So I noticed one day LimitedHistoryObserver was responsible for holding on to ~30k dict instances. I fixed this with [2]. Resident set size with 48hrs uptime is now down ~300M thanks to that function (and is flat).

That's... a lot of dict instances :). I guess not a very "limited" history observer ;-).

That's a bit too much for my tastes so I thought maybe you can reconsider that 64 * 1024 entry limit or choose to get rid of the default observer automatically once the global observer has a proper user-supplied sink.

Can you file this bug on the Twisted tracker so we don't lose it? I am pretty sure that 300M of memory for buffered logs is not something anybody really wants.

Best regards, Burak

PS: Neurons will be a web framework that ties a lot of stuff like sqlalchemy, spyne and twisted together. Currently it does only two things quite well: Forking and rendering html forms :)

[1] https://github.com/plq/neurons/tree/master/neurons/daemon/dowser [2] https://github.com/plq/neurons/blob/417836a729706f2d1449109ccbb7d4a510227050... <https://github.com/plq/neurons/blob/417836a729706f2d1449109ccbb7d4a510227050...>

Awesome, I can't wait to see the results.

On Apr 2, 2015, at 4:03 PM, Burak Arslan <burak.arslan@arskom.com.tr> wrote:

Hi,

I happen to have the new logging hotness(tm) running in production as the backend to python's own logging module in one of the daemons that I'm responsible for.

Thanks for trying out this code and helping us validate it!

I also have dowser[1] integrated with this one so tryng to optimize memory usage is almost fun :) So I noticed one day LimitedHistoryObserver was responsible for holding on to ~30k dict instances. I fixed this with [2]. Resident set size with 48hrs uptime is now down ~300M thanks to that function (and is flat).

That's... a lot of dict instances :). I guess not a very "limited" history observer ;-).

That's a bit too much for my tastes so I thought maybe you can reconsider that 64 * 1024 entry limit or choose to get rid of the default observer automatically once the global observer has a proper user-supplied sink.

Can you file this bug on the Twisted tracker so we don't lose it? I am pretty sure that 300M of memory for buffered logs is not something anybody really wants.

Best regards, Burak

PS: Neurons will be a web framework that ties a lot of stuff like sqlalchemy, spyne and twisted together. Currently it does only two things quite well: Forking and rendering html forms :)

[1] https://github.com/plq/neurons/tree/master/neurons/daemon/dowser [2] https://github.com/plq/neurons/blob/417836a729706f2d1449109ccbb7d4a510227050... <https://github.com/plq/neurons/blob/417836a729706f2d1449109ccbb7d4a510227050...>

Awesome, I can't wait to see the results.