Mailman 3 December 2001 - Twisted

[Twisted-Python] Twisted Python 0.7.0 Prerelease Testing
by Glyph Lefkowitz 05 Aug '21

05 Aug '21

hi there, folks: I'd really like to release 0.7.0 but I would like it to be at least a little bit tested before I do so. Could those of you with CVS trees check everything out and see if it performs as advertised? Deeper bugs than that will have to wait for the next release, but I'd at least like to know if it works for someone other than me. Thanks. ______ __ __ _____ _ _ | ____ | \_/ |_____] |_____| |_____| |_____ | | | | @ t w i s t e d m a t r i x . c o m http://www.twistedmatrix.com/~glyph/

4 5

[Twisted-Python] PB for Java status report
by Itamar Shtull-Trauring 06 Jan '02

06 Jan '02

It's now possible to create Referenceables in Java - meanwhile simple remote_xxx style ones only, no support for Perspectives or Viewables yet. In other words, you can now create PB services in Java as well - I think it's pluggable enough that you could integrate this with any Java server architecture that uses streams (which is probably all of them :).

3 4

[Twisted-Python] newbie question: error handling
by Ivo van der Wijk 01 Jan '02

01 Jan '02

Hi All, I'm currently migrating my irc client to twisted, and things look promising. However, I can't figure out the following things: My protocol is derived from protocols.basic.LineReceiver (which in turn derives from protocol.Protocol.protocol). How can I determine what specific error condition caused connectionLost() or connectionFailed()? I.e. connection refused, unknown hostname, no route to host, etc (the standard errno stuff). Also, how can I determine if Linereceived.sendLine() (or Protocol.Transport.write()) failed? It doesn't seem to return an error value or raise Exceptions, and depending on the invocation of connectionLost makes things perhaps too asynchronous.. Cheers, Ivo -- Drs. I.R. van der Wijk -=- Brouwersgracht 132 Amaze Internet Services V.O.F. 1013 HA Amsterdam, NL -=- Tel: +31-20-4688336 Linux/Web/Zope/SQL/MMBase Fax: +31-20-4688337 Network Solutions Web: http://www.amaze.nl/ Consultancy Email: ivo(a)amaze.nl -=-

2 3

[Twisted-Python] Passport, my ideas about an LDAP backend
by Tommi Virtanen 31 Dec '01

31 Dec '01

Here's a sketch of how I imagine an LDAP backend to passport will work: LDAP is a logically simple (implementation horrible:) TCP protocol with messages passing in both directions. User authentication is one possible message type, and the server replies with either success or an error message. For just searching and browsing, you can bind anonymously. Most servers allow read-only access by default. But for any kind of modification, we need to bind. So far, this seems pretty much like with databases. However, here's the big catch: the normal way to use databases is to have one userid and password per application, and perhaps store actual userids and passwords within a table in the database. The normal way to use LDAP is to bind _as the user who is doing the operation_. Here is a list of requirements I can think, that need to be fulfilled in order for me to be able to use passport. Some of these may be very trivial, or taken for granted by now -- I'm just making sure they are seen. Requirement #1: Passport needs to be async. In order to be able to use LDAP as the backend for passport-like user verification, the backend needs to be able to talk to a possibly slow server. Requirement #2: Passport must be able to store data in the Perspective during authentication. However, if the application wants to use LDAP for more than just user authentication, things get a bit worse. We could just store the password we saw the user enter, and later bind to the LDAP server as this user. However, I dislike storing plain-text passwords in virtual memory. Requirement #3: Passport must be able to ask users to reauthenticate. A better way is to remember that we already did bind (authenticate) to the LDAP server as that user. Just reserve that connection for that user alone. As long as the connection is alive, we can keep on talking to the LDAP server. Now, programs can be restarted, TCP connections can die, and servers can utilize idle timeouts, maximum time limits, or limit maximum number of operations per connection. The TCP connection can die any moment. As we didn't store the user password, we need to ask it again when we see the connection has closed and the user tries an operation that requires talking to the LDAP server. This requires "invalidating" the users previous authentication. -- tv(a){{hq.yok.utu,havoc,gaeshido}.fi,{debian,wanderer}.org,stonesoft.com} double a,b=4,c;main(){for(;++a<2e6;c-=(b=-b)/a++);printf("%f\n",c);}

2 2

[Twisted-Python] How to test metrics
by Itamar Shtull-Trauring 27 Dec '01

27 Dec '01

0. make sure postgres and pypgsql are installed (apt-get install python-pgsql). 1. run "createdb twisted" 2. using psql, create the tables listed in the schemas in twisted/enterprise/dbpassport.py and twisted/metrics/metricsdb.py 3. $ cat twisted/enterprise/populate.sql | psql twisted $ cat twisted/metrics/populate.sql | psql twisted 4. $ twisted -n -y doc/examples/metrics.tac 5. $ PYTHONPATH=. python twisted/metrics/test-client.py 6. Point your browser at http://locahost:8485/ Perhaps there should be a doc/examples/metrics/ and the populate.sql and test-client.py, test-os.py and metrics.tac should be there?

1 0

[Twisted-Python] Metrics and all that jazz
by Itamar Shtull-Trauring 26 Dec '01

26 Dec '01

I've been looking at metrics - currently the model is multiple machines, connecting to a central metrics server that stores metrics in postgres. I'm going to need something like that, so that's good, but it should also scale downwards. Oh, and Sean, I couldn't actually get values added by test-client.py to show up in the web interface - maybe try recreating your test environment from scratch and see what step are missing from Twisted? I keep getting these errors: 26/12/2001 16:16 [Broker,0,127.0.0.1] ERROR: unknown variable counter3 So, what metrics needs: 1. Ability to store metrics in memory or in a round-robin database, and on the local server too (for smaller setups and for easier testing.) See the Round Robin Archives section of http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/manual/index.html for details on what it is - it's a more reasonable way of storing data than a RDBMS for this sort of thing in a lot of cases. 2. OS statistics for Unix - does resource.rusage() give the necessary info? 3. Docs, explanations, examples of protocol specific statistics (e.g. Web). 4. An additional way of keeping track of open connections and their metrics, which should combine with the current metrics system. 5. Exporting stats to Windows NT Performance Monitor so it can do pretty graphs and stuff.

2 2

[Twisted-Python] Metrics and all that jazz
by Itamar Shtull-Trauring 26 Dec '01

26 Dec '01

I've been looking at metrics - currently the model is multiple machines, connecting to a central metrics server that stores metrics in postgres. I'm going to need something like that, so that's good, but it should also scale downwards. Oh, and Sean, I couldn't actually get values added by test-client.py to show up in the web interface - maybe try recreating your test environment from scratch and see what step are missing from Twisted? I keep getting these errors: 26/12/2001 16:16 [Broker,0,127.0.0.1] ERROR: unknown variable counter3 So, what metrics needs: 1. Ability to store metrics in memory or in a round-robin database, and on the local server too (for smaller setups and for easier testing.) See the Round Robin Archives section of http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/manual/index.html for details on what it is - it's a more reasonable way of storing data than a RDBMS for this sort of thing in a lot of cases. 2. OS statistics for Unix - does resource.rusage() give the necessary info? 3. Docs, explanations, examples of protocol specific statistics (e.g. Web). 4. An additional way of keeping track of open connections and their metrics, which should combine with the current metrics system. 5. Exporting stats to Windows NT Performance Monitor so it can do pretty graphs and stuff.

1 0

[Twisted-Python] Object prevalence (advogato article)
by Tommi Virtanen 26 Dec '01

26 Dec '01

http://advogato.org/article/398.html Short summary: -keep all state in RAM, no traditional database. -write snapshots of state to file every n hours. -write a journal of all commands coming in to journal. -in case of crash, restore state+rerun journal. This is pretty close to what Python and Twisted .taps can do. Let's have a goal of being able to do it all, easily. I think I'll add .tap rewriting every n seconds to MC Foo now :) -- tv(a){{hq.yok.utu,havoc,gaeshido}.fi,{debian,wanderer}.org,stonesoft.com} double a,b=4,c;main(){for(;++a<2e6;c-=(b=-b)/a++);printf("%f\n",c);}

1 0

[Twisted-Python] First go at connectTCP and connectSSL
by Itamar Shtull-Trauring 24 Dec '01

24 Dec '01

if I don't get complaints about this, it gets checked in and I move on to shutting down connectors, and so on. Index: twisted/internet/app.py =================================================================== RCS file: /cvs/Twisted/twisted/internet/app.py,v retrieving revision 1.4 diff -c -r1.4 app.py *** twisted/internet/app.py 13 Dec 2001 07:01:59 -0000 1.4 --- twisted/internet/app.py 24 Dec 2001 14:52:29 -0000 *************** *** 125,130 **** --- 125,132 ---- self.name = name # a list of (tcp, ssl, udp) Ports self.ports = [] + # a list of (tcp, ssl, udp) Connectors + self.connectors = [] # a list of twisted.python.delay.Delayeds self.delayeds = [] # a list of twisted.internet.passport.Services *************** *** 152,159 **** self.lock() ! persistenceVersion = 3 ! def upgradeToVersion3(self): """Version 3 Persistence Upgrade """ --- 154,166 ---- self.lock() ! persistenceVersion = 4 ! ! def upgradeToVersion4(self): ! """Version 4 Persistence Upgrade ! """ ! self.connectors = [] ! def upgradeToVersion3(self): """Version 3 Persistence Upgrade """ *************** *** 235,240 **** --- 242,266 ---- if self.running: port.startListening() + def connectTCP(self, host, port, factory): + """Connect a given client protocol factory to a specific TCP server.""" + from twisted.internet import tcp + self.addConnector(tcp.Connector(host, port, factory)) + + def connectSSL(self, host, port, factory, ctxFactory=None): + """Connect a given client protocol factory to a specific SSL server.""" + from twisted.internet import ssl + c = ssl.Connector(host, port, factory) + if ctxFactory: + c.contextFactory = ctxFactory + self.addConnector(c) + + def addConnector(self, connector): + """Add a connector to this Application.""" + self.connectors.append(connector) + if self.running: + connector.startConnecting() + def addDelayed(self, delayed): """ Adds a twisted.python.delay.Delayed object for execution in my event loop. *************** *** 326,331 **** --- 352,359 ---- except socket.error: print 'port %s already bound' % port.port return + for connector in self.connectors: + connector.startConnecting() for port in self.ports: port.factory.startFactory() resolver = self.resolver Index: twisted/internet/tcp.py =================================================================== RCS file: /cvs/Twisted/twisted/internet/tcp.py,v retrieving revision 1.35 diff -c -r1.35 tcp.py *** twisted/internet/tcp.py 28 Nov 2001 07:17:29 -0000 1.35 --- twisted/internet/tcp.py 24 Dec 2001 14:52:30 -0000 *************** *** 239,251 **** s = '<%s to %s at %x>' % (self.__class__, self.addr, id(self)) return s class Connector: ! def __init__(self, host, factory, portno, timeout=30): self.host = host self.portno = portno ! self.factory = factory ! self.portno = portno ! def connectionFailed(self): self.startConnecting() --- 239,256 ---- s = '<%s to %s at %x>' % (self.__class__, self.addr, id(self)) return s + class Connector: ! """Connect a protocol to a server using TCP and if it fails make a new one.""" ! ! transportFactory = Client ! ! def __init__(self, host, portno, protocolFactory, timeout=30): self.host = host self.portno = portno ! self.factory = protocolFactory ! self.timeout = timeout ! def connectionFailed(self): self.startConnecting() *************** *** 254,260 **** def startConnecting(self): proto = self.factory.buildProtocol((self.host, self.portno)) ! Client(self.host, self.portno, proto, 30, self) --- 259,265 ---- def startConnecting(self): proto = self.factory.buildProtocol((self.host, self.portno)) ! self.transportFactory(self.host, self.portno, proto, self.timeout, self) Index: twisted/internet/ssl.py =================================================================== RCS file: /cvs/Twisted/twisted/internet/ssl.py,v retrieving revision 1.6 diff -c -r1.6 ssl.py *** twisted/internet/ssl.py 29 Nov 2001 12:20:32 -0000 1.6 --- twisted/internet/ssl.py 24 Dec 2001 14:52:30 -0000 *************** *** 88,96 **** """I am an SSL client. """ ! def __init__(self, host, port, protocol, ctxFactory, timeout=None): self.ctxFactory = ctxFactory ! apply(tcp.Client.__init__, (self, host, port, protocol), {'timeout': timeout}) def createInternetSocket(self): """(internal) create an SSL socket --- 88,96 ---- """I am an SSL client. """ ! def __init__(self, host, port, protocol, ctxFactory, timeout=None, connector=None): self.ctxFactory = ctxFactory ! apply(tcp.Client.__init__, (self, host, port, protocol), {'timeout': timeout, 'connector': connector}) def createInternetSocket(self): """(internal) create an SSL socket *************** *** 146,148 **** --- 146,159 ---- protocol.makeConnection(transport, self) except: traceback.print_exc(file=log.logfile) + + + class Connector(tcp.Connector): + """Connect a protocol to a server using SSL and if it fails make a new one.""" + + transportFactory = Client + contextFactory = ClientContextFactory + + def startConnecting(self): + proto = self.factory.buildProtocol((self.host, self.portno)) + self.transportFactory(self.host, self.portno, proto, self.contextFactory, self.timeout, self)

1 0

[Twisted-Python] Why Zoteca Uses Twisted
by Itamar Shtull-Trauring 24 Dec '01

24 Dec '01

1 Twisted 1.1 Open Source Advantage Zoteca believes that Open Source software provides a considerable business advantage. Many parts of any software platform are generic, and not restricted to the task at hand. Using Open Source software for these components allows us to concentrate R&D resources on our core competencies and proprietary advantages, while still using an underlying technology that is unparalleled in terms of cost, flexibility, scalability and robustness. 1.1.1 Open Source Saves Money The costs for development for Open Source components is spread out over a number of organizations and multiple developers. As a result, for every person-year of investment on our part, we receive tens of person-years of development in return, at no additional cost. 1.1.2 Open Source is Flexible Open Source development allows us to cooperate with organizations and individuals from other industries, bringing ideas and knowledge from many different fields of competency to bear upon our common infrastructure needs. This development process takes advantage of orthogonal synergy between different industries' software requirements. Hence the resultant software is far more generic and flexible than had it been developed for one specific use. Of course, total access to the underlying source code also allows customized changes for particular applications. 1.1.3 Open Source is Scalable and Robust Developers on open source projects are not engaged in an intellectual exercise nor are they removed from the application space of the software under development. They are developing the software for their own use for real needs. They are not only developers, but users and testers as well. Hence the resultant software is far more field-ready and scalable to real-world applications than proprietary alternatives. The multiple environments the components are used in, also ensures that they are robust and relatively bug free. Being cross-platform is the icing on the cake. 1.2 Twisted Network Server A prime example of this Open Source approach is the Twisted network server. Twisted is written in Python, a cross-platform, high-level object-oriented programming language. Twisted includes a web server, a telnet server, a multiplayer RPG engine, a generic client and server for remote object access, and APIs for creating new protocols and services. Twisted is being developed and used by individuals and organizations from across the world, for uses ranging from massive multiplayer games to streaming audio servers. Some of Twisted's advanced features include: * Abstraction layers for protocols and transports, allowing design of servers that transparently support multiple transports (e.g. TCP and SSL). * Pluggable event loops, supporting integration with GUI toolkits and platform-specific event loops (e.g. wxPython, Java or Unix/Win32 specific APIs). * Multiple protocol support inside a process, allowing cross-protocol applications, integrated web control panels, etc.. * A metrics system supporting centralized collection of protocol and process statistics. * Perspective Broker, a powerful, asynchronous and efficient remote object protocol, which can also be used from Java. * Integrated event-based and threaded programming, using the Async/Half-Async design pattern. The use of Twisted and Python has had many benefits for Zoteca when developing our software: * Low cost of development, since basic infrastructure needed by any network server is built in to the platform. * Cross-platform and platform specific support (Twisted, and thus Zoteca's servers, can run and take advantage of the Unix, Windows and Java platforms). * Flexibility and speed of development, allowing easy customization of the software. * Wide scale testing of low-level infrastructure in a variety of situations. * The ability to customize and enhance Twisted to support additional functionality. 1.3 About Zoteca Zoteca has developed a new, patented, proprietary technology which provides reliable, available, private and secure ("RAPS") data sharing in networked environments. Zoteca's mission is to create a family of product toolkits, which integrate into customized versions of data sharing applications. These Zoteca-based products provide RAPS services to end-users --- both businesses and consumers. In current networks it is nearly impossible to achieve reliability and accessibility simultaneously --- hence the constant shuffling of data between centralized servers and the organizational edge. Similarly the more private and secure networks are, the less efficient data sharing becomes. Zoteca creates spaces where individuals and groups share data in a total RAPS environment, simultaneously efficient (R & A) and safe (P & S). The core of our innovation involves decoupling computing entities (data, files, etc.) from specific physical computers or locations, thereby eliminating single point of failure. By using distributed protocols, redundant and immutable data stores, sophisticated authentication schemes and encryption techniques, we address the problem at the infrastructure level, allowing us to offer guaranteed levels of service in terms of the four RAPS dimensions. For more information about Zoteca and our products, please visit http://www.zoteca.com.

1 0

Twisted December 2001