[Twisted-Python] twisted pysnmp status
Hi, I just wanted to know what the latest status is with twisted and pysnmp. Is pysnmp 4.x supported? Is full SNMPv3 supported? What is the roadmap and/or plans for twisted and pysnmp ??? I want to write and SNMPv3 GUI manager in python :) I'm not sure if the complexity and dependency of twisted is worth the effort? Then again on the other hand it may provide lots of things I would probably need to write anyway ? Given I would also like SSH and maybe even WBEM and/or WS-Management protocols too, then maybe twisted provides a better framework ??? Cheers, Brendan.
Brendan Simon wrote:
Hi,
I just wanted to know what the latest status is with twisted and pysnmp. Is pysnmp 4.x supported? Is full SNMPv3 supported? What is the roadmap and/or plans for twisted and pysnmp ???
I want to write and SNMPv3 GUI manager in python :) I'm not sure if the complexity and dependency of twisted is worth the effort? Then again on the other hand it may provide lots of things I
It is. Although I don't use the pysnmp library, I do use Twisted for a large SNMP polling application (~1200 nodes). It goes much faster than anything else I've tried, with the possible exception of a custom event loop I wrote ages ago that's virtually unmaintainable and has all sorts of magic values. One thing you might want to investigate is forking off N child processes which are simple 2-protocol bridges - they listen on Perspective Broker for high-level RPC calls such as "GetSnmpTable(base, cols)" and execute SNMP walks. I've found this can overcome some scalability issues with very large numbers of clients on a single UDP socket. The problem is emptying the UDP socket buffer fast enough - you have to yield back into the reactor ASAP, and any code you execute should be behind a reactor.callLater(0, handler). It also gives you multiprocessor speedups, *and* you can trivially move those processes elsewhere later on e.g. if you're monitoring boxes down the end of a WAN link, put one process down there. Then the large numbers of SNMP packets don't have to flow over the WAN, just the PB link which is flow-controlled TCP sending tiny request/result PDUs.
Brendan Simon wrote:
Hi,
I just wanted to know what the latest status is with twisted and pysnmp. Is pysnmp 4.x supported? Is full SNMPv3 supported? What is the roadmap and/or plans for twisted and pysnmp ??? PySNMP 4.x was, the last time I looked at it (probably more than a year ago, now), *far* too slow for our client's purposes (monitoring 10,000+ cable modems). In fact, even PySNMP 3.x was too slow (even with our speed enhanced version). We *needed* to be able to scale up to well beyond 40,000+ cable modems per (reasonable) scanning box to have a chance in the market where DSLAMs/CMTSs have that number of modems on a single chasis (and 100s of thousands per head end). As a result, we had to defer on the product's biggest potential market.
SNMPv3 is not supported with TwistedSNMP. We hacked in PySNMP 4.x, discovered it just couldn't be used for our purposes and dropped the porting effort before it was completed (which would have included the SNMPv3 interfaces). As for the roadmap, I'm not currently working on the project as we aren't working on the product that was footing the bill any more. I believe Zenoss is using the package to monitor (smaller) networks, so they may have some requirements they need and decide to contribute or pay for. There are others using it for small installations as well, they may be interested in working as well. If there are people who want to work on the project I'm certainly willing to add them to the developer list on SourceForge. HTH, Mike -- ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com
On Thu, 2007-03-29 at 15:41 -0400, Mike C. Fletcher wrote:
Brendan Simon wrote:
Hi,
I just wanted to know what the latest status is with twisted and pysnmp. Is pysnmp 4.x supported? Is full SNMPv3 supported? What is the roadmap and/or plans for twisted and pysnmp ??? PySNMP 4.x was, the last time I looked at it (probably more than a year ago, now), *far* too slow for our client's purposes (monitoring 10,000+ cable modems). In fact, even PySNMP 3.x was too slow (even with our speed enhanced version). We *needed* to be able to scale up to well beyond 40,000+ cable modems per (reasonable) scanning box to have a chance in the market where DSLAMs/CMTSs have that number of modems on a single chasis (and 100s of thousands per head end). As a result, we had to defer on the product's biggest potential market.
SNMPv3 is not supported with TwistedSNMP. We hacked in PySNMP 4.x, discovered it just couldn't be used for our purposes and dropped the porting effort before it was completed (which would have included the SNMPv3 interfaces).
The performance of PySNMP (many years ago) was the main reason we wrote
libsnmp; PySNMP seemed to have been written from a very CompSci
perspective with full abstract ASN.1 model that gets subclassed to SNMP
via many steps, all of which slow it down. We went for a more
engineering approach, hardcoding lookup tables for values that never
change for the specific ASN.1 instanciation that is SNMP, for example.
libsnmp isn't integrated into twisted, though we do use it with twisted
with great success in seafelt; there has been some interest in
integrating libsnmp with twisted in recent times. libsnmp also lacks
SNMPv3 support, only because we haven't had a need to add it yet, but
this means it probably doesn't meet your needs.
However, I am interested in both of these goals: twisted integration,
and SNMPv3 support, so perhaps if there enough people interested in such
an outcome we could work together to achieve it?
--
Justin Warren
Justin Warren wrote: ...
The performance of PySNMP (many years ago) was the main reason we wrote libsnmp; PySNMP seemed to have been written from a very CompSci perspective with full abstract ASN.1 model that gets subclassed to SNMP via many steps, all of which slow it down. We went for a more engineering approach, hardcoding lookup tables for values that never change for the specific ASN.1 instanciation that is SNMP, for example.
libsnmp isn't integrated into twisted, though we do use it with twisted with great success in seafelt; there has been some interest in integrating libsnmp with twisted in recent times. libsnmp also lacks SNMPv3 support, only because we haven't had a need to add it yet, but this means it probably doesn't meet your needs.
Actually, would meet our client's needs well enough. SNMPv3 isn't a requirement, just a small nicety for them.
However, I am interested in both of these goals: twisted integration, and SNMPv3 support, so perhaps if there enough people interested in such an outcome we could work together to achieve it?
Looking over your code it seems like it would be fairly easy to make it an option in TwistedSNMP. Currently TwistedSNMP has an abstraction point that allows regular or speed-enhanced PySNMP to be used. By the look of it a very simple wrapper module would let libsnmp provide the (small set of) features we use (basically just encode/decode and translation of result values to/from Pythonic values with a few "type" operations as well (OID, EndOfMibView, that kind of thing)). BTW, I notice that none of your code uses GetBulk (though you define it)? Bulk operations were a *huge* speed win for our application. Hopefully you're doing it in your real-world code. Anyway, I'll see if there's any interest from the customer in trying out libsnmp with the code-base. That would let me take company time to work on integrating it into TwistedSNMP. They almost certainly don't care enough to have it do v3 operations, but raw speed on v2 operations is something they may care about enough to pay for a few hours of work. Have fun, Mike -- ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com
On Fri, 2007-03-30 at 12:57 -0400, Mike C. Fletcher wrote:
Justin Warren wrote: ...
The performance of PySNMP (many years ago) was the main reason we wrote libsnmp; PySNMP seemed to have been written from a very CompSci perspective with full abstract ASN.1 model that gets subclassed to SNMP via many steps, all of which slow it down. We went for a more engineering approach, hardcoding lookup tables for values that never change for the specific ASN.1 instanciation that is SNMP, for example.
libsnmp isn't integrated into twisted, though we do use it with twisted with great success in seafelt; there has been some interest in integrating libsnmp with twisted in recent times. libsnmp also lacks SNMPv3 support, only because we haven't had a need to add it yet, but this means it probably doesn't meet your needs.
Actually, would meet our client's needs well enough. SNMPv3 isn't a requirement, just a small nicety for them.
Neato. That was our approach also, given the relatively small number of devices, etc. that implement SNMPv3 and the development effort required. It will get added when there is sufficient need.
However, I am interested in both of these goals: twisted integration, and SNMPv3 support, so perhaps if there enough people interested in such an outcome we could work together to achieve it?
Looking over your code it seems like it would be fairly easy to make it an option in TwistedSNMP. Currently TwistedSNMP has an abstraction point that allows regular or speed-enhanced PySNMP to be used. By the look of it a very simple wrapper module would let libsnmp provide the (small set of) features we use (basically just encode/decode and translation of result values to/from Pythonic values with a few "type" operations as well (OID, EndOfMibView, that kind of thing)).
That should certainly be fairly simple with libsnmp, yes.
BTW, I notice that none of your code uses GetBulk (though you define it)? Bulk operations were a *huge* speed win for our application. Hopefully you're doing it in your real-world code.
Not that I recall, probably because we haven't hit a speed hurdle that would be solved by using it thus far. We've had more issues with SNMP bugs on remote devices, such as certain brands of wireless router crashing when polled with multi-variable SNMP packets. :)
Anyway, I'll see if there's any interest from the customer in trying out libsnmp with the code-base. That would let me take company time to work on integrating it into TwistedSNMP. They almost certainly don't care enough to have it do v3 operations, but raw speed on v2 operations is something they may care about enough to pay for a few hours of work.
Cool. Let me know how you get on.
--
Justin Warren
Justin Warren wrote:
On Fri, 2007-03-30 at 12:57 -0400, Mike C. Fletcher wrote:
Justin Warren wrote:
...
Anyway, I'll see if there's any interest from the customer in trying out libsnmp with the code-base. That would let me take company time to work on integrating it into TwistedSNMP. They almost certainly don't care enough to have it do v3 operations, but raw speed on v2 operations is something they may care about enough to pay for a few hours of work.
Cool. Let me know how you get on.
No one jumped all over the idea. Probably not going to happen as a result. Sorry about that, Mike -- ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com
On Wed, 2007-04-18 at 13:13 -0400, Mike C. Fletcher wrote:
Justin Warren wrote:
On Fri, 2007-03-30 at 12:57 -0400, Mike C. Fletcher wrote:
Justin Warren wrote:
...
Anyway, I'll see if there's any interest from the customer in trying out libsnmp with the code-base. That would let me take company time to work on integrating it into TwistedSNMP. They almost certainly don't care enough to have it do v3 operations, but raw speed on v2 operations is something they may care about enough to pay for a few hours of work.
Cool. Let me know how you get on.
No one jumped all over the idea. Probably not going to happen as a result.
Sorry about that, Mike
Hey, no problem. It'll probably happen when I need to add some new
features into our code, or someone else will have a pressing need.
--
Justin Warren
participants (4)
-
Brendan Simon -
Justin Warren -
Mike C. Fletcher -
Phil Mayers