hi there, folks:
I'd really like to release 0.7.0 but I would like it to be at least a
little bit tested before I do so. Could those of you with CVS trees check
everything out and see if it performs as advertised? Deeper bugs than
that will have to wait for the next release, but I'd at least like to know
if it works for someone other than me.
Thanks.
______ __ __ _____ _ _
| ____ | \_/ |_____] |_____|
|_____| |_____ | | | |
@ t w i s t e d m a t r i x . c o m
http://www.twistedmatrix.com/~glyph/
It's now possible to create Referenceables in Java - meanwhile simple
remote_xxx style ones only, no support for Perspectives or Viewables yet.
In other words, you can now create PB services in Java as well - I think
it's pluggable enough that you could integrate this with any Java server
architecture that uses streams (which is probably all of them :).
Hi All,
I'm currently migrating my irc client to twisted, and things look
promising.
However, I can't figure out the following things:
My protocol is derived from protocols.basic.LineReceiver
(which in turn derives from protocol.Protocol.protocol).
How can I determine what specific error condition caused connectionLost()
or connectionFailed()? I.e. connection refused, unknown hostname,
no route to host, etc (the standard errno stuff).
Also, how can I determine if Linereceived.sendLine() (or
Protocol.Transport.write()) failed? It doesn't seem to return an
error value or raise Exceptions, and depending on the invocation
of connectionLost makes things perhaps too asynchronous..
Cheers,
Ivo
--
Drs. I.R. van der Wijk -=-
Brouwersgracht 132 Amaze Internet Services V.O.F.
1013 HA Amsterdam, NL -=-
Tel: +31-20-4688336 Linux/Web/Zope/SQL/MMBase
Fax: +31-20-4688337 Network Solutions
Web: http://www.amaze.nl/ Consultancy
Email: ivo(a)amaze.nl -=-
Here's a sketch of how I imagine an LDAP backend to passport will
work:
LDAP is a logically simple (implementation horrible:) TCP protocol
with messages passing in both directions. User authentication is one
possible message type, and the server replies with either success or
an error message.
For just searching and browsing, you can bind anonymously. Most
servers allow read-only access by default.
But for any kind of modification, we need to bind.
So far, this seems pretty much like with databases. However, here's
the big catch: the normal way to use databases is to have one userid
and password per application, and perhaps store actual userids and
passwords within a table in the database. The normal way to use LDAP
is to bind _as the user who is doing the operation_.
Here is a list of requirements I can think, that need to be fulfilled
in order for me to be able to use passport. Some of these may be very
trivial, or taken for granted by now -- I'm just making sure they are
seen.
Requirement #1: Passport needs to be async.
In order to be able to use LDAP as the backend for passport-like user
verification, the backend needs to be able to talk to a possibly slow
server.
Requirement #2: Passport must be able to store data in the
Perspective during authentication.
However, if the application wants to use LDAP for more than just user
authentication, things get a bit worse. We could just store the
password we saw the user enter, and later bind to the LDAP server as
this user. However, I dislike storing plain-text passwords in virtual
memory.
Requirement #3: Passport must be able to ask users to reauthenticate.
A better way is to remember that we already did bind (authenticate) to
the LDAP server as that user. Just reserve that connection for that
user alone. As long as the connection is alive, we can keep on talking
to the LDAP server.
Now, programs can be restarted, TCP connections can die, and servers
can utilize idle timeouts, maximum time limits, or limit maximum
number of operations per connection. The TCP connection can die any
moment. As we didn't store the user password, we need to ask it again
when we see the connection has closed and the user tries an operation
that requires talking to the LDAP server. This requires "invalidating"
the users previous authentication.
--
tv(a){{hq.yok.utu,havoc,gaeshido}.fi,{debian,wanderer}.org,stonesoft.com}
double a,b=4,c;main(){for(;++a<2e6;c-=(b=-b)/a++);printf("%f\n",c);}
0. make sure postgres and pypgsql are installed (apt-get install python-pgsql).
1. run "createdb twisted"
2. using psql, create the tables listed in the schemas in
twisted/enterprise/dbpassport.py and twisted/metrics/metricsdb.py
3. $ cat twisted/enterprise/populate.sql | psql twisted
$ cat twisted/metrics/populate.sql | psql twisted
4. $ twisted -n -y doc/examples/metrics.tac
5. $ PYTHONPATH=. python twisted/metrics/test-client.py
6. Point your browser at http://locahost:8485/
Perhaps there should be a doc/examples/metrics/ and the populate.sql and
test-client.py, test-os.py and metrics.tac should be there?
I've been looking at metrics - currently the model is multiple machines,
connecting to a central metrics server that stores metrics in postgres. I'm
going to need something like that, so that's good, but it should also scale
downwards.
Oh, and Sean, I couldn't actually get values added by test-client.py to show
up in the web interface - maybe try recreating your test environment from
scratch and see what step are missing from Twisted? I keep getting these errors:
26/12/2001 16:16 [Broker,0,127.0.0.1] ERROR: unknown variable counter3
So, what metrics needs:
1. Ability to store metrics in memory or in a round-robin database, and on
the local server too (for smaller setups and for easier testing.) See the
Round Robin Archives section of
http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/manual/index.html for
details on what it is - it's a more reasonable way of storing data than a
RDBMS for this sort of thing in a lot of cases.
2. OS statistics for Unix - does resource.rusage() give the necessary info?
3. Docs, explanations, examples of protocol specific statistics (e.g. Web).
4. An additional way of keeping track of open connections and their metrics,
which should combine with the current metrics system.
5. Exporting stats to Windows NT Performance Monitor so it can do pretty
graphs and stuff.
I've been looking at metrics - currently the model is multiple machines,
connecting to a central metrics server that stores metrics in postgres. I'm
going to need something like that, so that's good, but it should also scale
downwards.
Oh, and Sean, I couldn't actually get values added by test-client.py to show
up in the web interface - maybe try recreating your test environment from
scratch and see what step are missing from Twisted? I keep getting these errors:
26/12/2001 16:16 [Broker,0,127.0.0.1] ERROR: unknown variable counter3
So, what metrics needs:
1. Ability to store metrics in memory or in a round-robin database, and on
the local server too (for smaller setups and for easier testing.) See the
Round Robin Archives section of
http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/manual/index.html for
details on what it is - it's a more reasonable way of storing data than a
RDBMS for this sort of thing in a lot of cases.
2. OS statistics for Unix - does resource.rusage() give the necessary info?
3. Docs, explanations, examples of protocol specific statistics (e.g. Web).
4. An additional way of keeping track of open connections and their metrics,
which should combine with the current metrics system.
5. Exporting stats to Windows NT Performance Monitor so it can do pretty
graphs and stuff.
http://advogato.org/article/398.html
Short summary:
-keep all state in RAM, no traditional database.
-write snapshots of state to file every n hours.
-write a journal of all commands coming in to journal.
-in case of crash, restore state+rerun journal.
This is pretty close to what Python and Twisted .taps can do.
Let's have a goal of being able to do it all, easily.
I think I'll add .tap rewriting every n seconds to MC Foo now :)
--
tv(a){{hq.yok.utu,havoc,gaeshido}.fi,{debian,wanderer}.org,stonesoft.com}
double a,b=4,c;main(){for(;++a<2e6;c-=(b=-b)/a++);printf("%f\n",c);}
1 Twisted
1.1 Open Source Advantage
Zoteca believes that Open Source software provides a considerable
business advantage. Many parts of any software platform
are generic, and not restricted to the task at hand. Using
Open Source software for these components allows us to concentrate
R&D resources on our core competencies and proprietary advantages,
while still using an underlying technology that is unparalleled
in terms of cost, flexibility, scalability and robustness.
1.1.1 Open Source Saves Money
The costs for development for Open Source components is spread
out over a number of organizations and multiple developers.
As a result, for every person-year of investment on our
part, we receive tens of person-years of development in
return, at no additional cost.
1.1.2 Open Source is Flexible
Open Source development allows us to cooperate with organizations
and individuals from other industries, bringing ideas and
knowledge from many different fields of competency to bear
upon our common infrastructure needs. This development process
takes advantage of orthogonal synergy between different
industries' software requirements. Hence the resultant software
is far more generic and flexible than had it been developed
for one specific use. Of course, total access to the underlying
source code also allows customized changes for particular
applications.
1.1.3 Open Source is Scalable and Robust
Developers on open source projects are not engaged in an
intellectual exercise nor are they removed from the application
space of the software under development. They are developing
the software for their own use for real needs. They are
not only developers, but users and testers as well. Hence
the resultant software is far more field-ready and scalable
to real-world applications than proprietary alternatives.
The multiple environments the components are used in, also
ensures that they are robust and relatively bug free. Being
cross-platform is the icing on the cake.
1.2 Twisted Network Server
A prime example of this Open Source approach is the Twisted
network server. Twisted is written in Python, a cross-platform,
high-level object-oriented programming language. Twisted
includes a web server, a telnet server, a multiplayer RPG
engine, a generic client and server for remote object access,
and APIs for creating new protocols and services. Twisted
is being developed and used by individuals and organizations
from across the world, for uses ranging from massive multiplayer
games to streaming audio servers.
Some of Twisted's advanced features include:
* Abstraction layers for protocols and transports, allowing
design of servers that transparently support multiple
transports (e.g. TCP and SSL).
* Pluggable event loops, supporting integration with GUI
toolkits and platform-specific event loops (e.g. wxPython,
Java or Unix/Win32 specific APIs).
* Multiple protocol support inside a process, allowing cross-protocol
applications, integrated web control panels, etc..
* A metrics system supporting centralized collection of protocol
and process statistics.
* Perspective Broker, a powerful, asynchronous and efficient
remote object protocol, which can also be used from Java.
* Integrated event-based and threaded programming, using
the Async/Half-Async design pattern.
The use of Twisted and Python has had many benefits for Zoteca
when developing our software:
* Low cost of development, since basic infrastructure needed
by any network server is built in to the platform.
* Cross-platform and platform specific support (Twisted,
and thus Zoteca's servers, can run and take advantage
of the Unix, Windows and Java platforms).
* Flexibility and speed of development, allowing easy customization
of the software.
* Wide scale testing of low-level infrastructure in a variety
of situations.
* The ability to customize and enhance Twisted to support
additional functionality.
1.3 About Zoteca
Zoteca has developed a new, patented, proprietary technology
which provides reliable, available, private and secure ("RAPS")
data sharing in networked environments. Zoteca's mission
is to create a family of product toolkits, which integrate
into customized versions of data sharing applications. These
Zoteca-based products provide RAPS services to end-users
--- both businesses and consumers.
In current networks it is nearly impossible to achieve reliability
and accessibility simultaneously --- hence the constant
shuffling of data between centralized servers and the organizational
edge. Similarly the more private and secure networks are,
the less efficient data sharing becomes. Zoteca creates
spaces where individuals and groups share data in a total
RAPS environment, simultaneously efficient (R & A) and safe
(P & S).
The core of our innovation involves decoupling computing
entities (data, files, etc.) from specific physical computers
or locations, thereby eliminating single point of failure.
By using distributed protocols, redundant and immutable
data stores, sophisticated authentication schemes and encryption
techniques, we address the problem at the infrastructure
level, allowing us to offer guaranteed levels of service
in terms of the four RAPS dimensions.
For more information about Zoteca and our products, please
visit http://www.zoteca.com.
