[Twisted-Python] Metrics and all that jazz
![](https://secure.gravatar.com/avatar/433365de0f787faa3ed3e6dd1da5884f.jpg?s=120&d=mm&r=g)
I've been looking at metrics - currently the model is multiple machines, connecting to a central metrics server that stores metrics in postgres. I'm going to need something like that, so that's good, but it should also scale downwards. Oh, and Sean, I couldn't actually get values added by test-client.py to show up in the web interface - maybe try recreating your test environment from scratch and see what step are missing from Twisted? I keep getting these errors: 26/12/2001 16:16 [Broker,0,127.0.0.1] ERROR: unknown variable counter3 So, what metrics needs: 1. Ability to store metrics in memory or in a round-robin database, and on the local server too (for smaller setups and for easier testing.) See the Round Robin Archives section of http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/manual/index.html for details on what it is - it's a more reasonable way of storing data than a RDBMS for this sort of thing in a lot of cases. 2. OS statistics for Unix - does resource.rusage() give the necessary info? 3. Docs, explanations, examples of protocol specific statistics (e.g. Web). 4. An additional way of keeping track of open connections and their metrics, which should combine with the current metrics system. 5. Exporting stats to Windows NT Performance Monitor so it can do pretty graphs and stuff.
![](https://secure.gravatar.com/avatar/588326b78898c23564d107c4fa70c2f0.jpg?s=120&d=mm&r=g)
yes, i must write some docs. did you run the populate.sql script? each metrics variable must exist in the metrics_variables table. rddtool looks interesting. i will take a look in more detail. all good feedback! thanks. Unfortunately I currently don't have a production system that this is running on - it is for a system that isn't built yet. It turned out that implementing it took about the same time as writing the technical specification document for it in the required format, so I did them both simultaneously. Could you elaborate on "An additional way of keeping track of open connections and their metrics"? -----Original Message----- From: twisted-python-admin@twistedmatrix.com [mailto:twisted-python-admin@twistedmatrix.com]On Behalf Of Itamar Shtull-Trauring Sent: Wednesday, December 26, 2001 10:14 AM To: twisted-python@twistedmatrix.com Subject: [Twisted-Python] Metrics and all that jazz I've been looking at metrics - currently the model is multiple machines, connecting to a central metrics server that stores metrics in postgres. I'm going to need something like that, so that's good, but it should also scale downwards. Oh, and Sean, I couldn't actually get values added by test-client.py to show up in the web interface - maybe try recreating your test environment from scratch and see what step are missing from Twisted? I keep getting these errors: 26/12/2001 16:16 [Broker,0,127.0.0.1] ERROR: unknown variable counter3 So, what metrics needs: 1. Ability to store metrics in memory or in a round-robin database, and on the local server too (for smaller setups and for easier testing.) See the Round Robin Archives section of http://people.ee.ethz.ch/~oetiker/webtools/rrdtrool/manual/index.html for details on what it is - it's a more reasonable way of storing data than a RDBMS for this sort of thing in a lot of cases. 2. OS statistics for Unix - does resource.rusage() give the necessary info? 3. Docs, explanations, examples of protocol specific statistics (e.g. Web). 4. An additional way of keeping track of open connections and their metrics, which should combine with the current metrics system. 5. Exporting stats to Windows NT Performance Monitor so it can do pretty graphs and stuff. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
![](https://secure.gravatar.com/avatar/433365de0f787faa3ed3e6dd1da5884f.jpg?s=120&d=mm&r=g)
Sean Riley wrote:
did you run the populate.sql script? each metrics variable must exist in the metrics_variables table.
There is no populate.sql for metrics last I checked.
Could you elaborate on "An additional way of keeping track of open connections and their metrics"?
Consider a SMTP server. It has clients connect and send emails through it. I would like to be able (through a web interface) to see which clients are currently connected to it, and how many messages they each have sent through (e.g. user from foo.example.com port 67349 has just sent 10000 messages, perhaps he's a spammer.) Or to be able to see who is logged in from where to a PB server and how many method calls each perspective has sent, etc..
![](https://secure.gravatar.com/avatar/588326b78898c23564d107c4fa70c2f0.jpg?s=120&d=mm&r=g)
yes, i must write some docs. did you run the populate.sql script? each metrics variable must exist in the metrics_variables table. rddtool looks interesting. i will take a look in more detail. all good feedback! thanks. Unfortunately I currently don't have a production system that this is running on - it is for a system that isn't built yet. It turned out that implementing it took about the same time as writing the technical specification document for it in the required format, so I did them both simultaneously. Could you elaborate on "An additional way of keeping track of open connections and their metrics"? -----Original Message----- From: twisted-python-admin@twistedmatrix.com [mailto:twisted-python-admin@twistedmatrix.com]On Behalf Of Itamar Shtull-Trauring Sent: Wednesday, December 26, 2001 10:14 AM To: twisted-python@twistedmatrix.com Subject: [Twisted-Python] Metrics and all that jazz I've been looking at metrics - currently the model is multiple machines, connecting to a central metrics server that stores metrics in postgres. I'm going to need something like that, so that's good, but it should also scale downwards. Oh, and Sean, I couldn't actually get values added by test-client.py to show up in the web interface - maybe try recreating your test environment from scratch and see what step are missing from Twisted? I keep getting these errors: 26/12/2001 16:16 [Broker,0,127.0.0.1] ERROR: unknown variable counter3 So, what metrics needs: 1. Ability to store metrics in memory or in a round-robin database, and on the local server too (for smaller setups and for easier testing.) See the Round Robin Archives section of http://people.ee.ethz.ch/~oetiker/webtools/rrdtrool/manual/index.html for details on what it is - it's a more reasonable way of storing data than a RDBMS for this sort of thing in a lot of cases. 2. OS statistics for Unix - does resource.rusage() give the necessary info? 3. Docs, explanations, examples of protocol specific statistics (e.g. Web). 4. An additional way of keeping track of open connections and their metrics, which should combine with the current metrics system. 5. Exporting stats to Windows NT Performance Monitor so it can do pretty graphs and stuff. _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
![](https://secure.gravatar.com/avatar/433365de0f787faa3ed3e6dd1da5884f.jpg?s=120&d=mm&r=g)
Sean Riley wrote:
did you run the populate.sql script? each metrics variable must exist in the metrics_variables table.
There is no populate.sql for metrics last I checked.
Could you elaborate on "An additional way of keeping track of open connections and their metrics"?
Consider a SMTP server. It has clients connect and send emails through it. I would like to be able (through a web interface) to see which clients are currently connected to it, and how many messages they each have sent through (e.g. user from foo.example.com port 67349 has just sent 10000 messages, perhaps he's a spammer.) Or to be able to see who is logged in from where to a PB server and how many method calls each perspective has sent, etc..
participants (2)
-
Itamar Shtull-Trauring
-
Sean Riley