Mailman 3 [Twisted-Python] Backup Agent - Twisted

Feb. 17, 2004

On 17 Feb 04, at 10:21 AM, twisted-python-request@twistedmatrix.com 
wrote:
...
Message: 1
Date: Mon, 16 Feb 2004 10:03:37 -0500
To: twisted-python@twistedmatrix.com
From: "David A. Leedom" <daleedom@hightowergroup.com>
Subject: [Twisted-Python] Backup Agent
Reply-To: twisted-python@twistedmatrix.com
--=====================_580045343==.ALT
Content-Type: text/plain; charset="us-ascii"; format=flowed
OK Guys I Need your help.
I own a small computer consulting firm and am struggling with the basic
sales and marketing issues. However, I think I have finally stumbled 
onto
a great "no brainer" idea:
A Remote Backup Service for Small businesses.
Yes I know there are a bunch out their, but I am thinking of the 
Hundreds
of Businesses around my general geographic area who don't know how to 
find
them and who don't backup their data. I would charge a monthly fee 
for a
given amount of space used ( no more than 500 Meg) and provide a 
software
that would backup and restore their data.
My dilemma is that my margins would be to small to make it affordable 
for
me to resale someone else's software. So....I would like to roll my
own. Python makes a lot of sense for many reasons that I don't need 
to get
into here.
My struggle is that although I have been programming since the 
mid-70s, I
am new to Python and feel like a man groping around in the dark. Also
being new to Twisted it is like the room is full of chairs as well. I 
did
get some sample code to work over the weekend, but I got a serious 
brain
cramp trying to figure it all out.
The basic architecture that I am thinking of is client/server. There 
would
be three basic applications running to make things work. First, would 
be
an Agent program that does most of the work. The Agent would run as a
service on each computer being backed up. Second would be a GUI 
program
running locally to control the agents. Finally there would be a "Web
Service" that received backup sets from each agent or a designated 
master
agent. The Web Service would handle authentication and other house 
keeping
things.
Finally the Question!!!!
There are many other details that I have in mind as well (using 
patches to
reduce size, compression, encryption...), but right now I am working on
issues related to the communication between the GUI and Agents. 
Assuming
that I have one or more number of agents running on a network I want 
to do
a number of things. 1) Have the GUI request that the agents identify
themselves so I don't have to tell the GUI where they are. 2) 
Transmit a
catalog list of the file system to the GUI so I can build a TreeView 
and
pick which files to backup on each computer. 3) Finally I may want to
identify a "Master" agent that collects all the backups on the network 
and
transmits them as one session to the Web Service. This would allow for
local "On Site" backups for fast restores due to normal computer 
crashes
and failures.
1. What would you guys suggest I do to I get the GUI to ask all 
the
agents to identify themselves?
2. What would be the best approach using twisted to transmit a 
file
system catalog across the local wire. I was playing with some remote
procedure calls over the weekend that seemed to work. I am thinking I
could create a class that would gather the local file system and send 
it
over the wire as one lump.
3. If I am going to transmit a backup file (<500meg) over the 
wire to
another local computer what is the best Twisted approach to take? Due 
to
WinXP security issues and the desire the run in a cross platform
environment I don't want to rely on normal file shares.
There are a hundred other questions running though my mind right now, 
but
this will due for now,
Thanks for any help you can give,
David A. Leedom
The Hightower Group, Inc.
Custom Software Solutions Designed To Fit Your Business Like A Glove.
165 West Airport Road/Lititz, PA 17543
V:717-560-4002, 877-560-4002 x: 114
F:717-560-2825
www.hightowergroup.com
--=====================_580045343==.ALT
Content-Type: text/html; charset="us-ascii"
<html>
<body>
OK Guys I Need your help. 
I own a small computer consulting firm and am struggling with the basic
sales and marketing issues.  However, I think I have finally
stumbled onto a great "no brainer" idea:  
<x-tab>        </x-tab>A Remote
Backup Service for Small businesses.  
Yes I know there are a bunch out their, but I am thinking of the 
Hundreds
of Businesses around my general geographic area who don't know how to
find them and who don't backup their data.  I would charge a 
monthly
fee for a given amount of space used ( no more than 500 Meg) and 
provide
a software that would backup and restore their data. 
My dilemma is that my margins would be to small to make it affordable 
for
me to resale someone else's software.  So....I would like to roll 
my
own.  Python makes a lot of sense for many reasons that I don't 
need
to get into here. 
My struggle is that although I have been programming since the 
mid-70s, I
am new to Python and feel like a man groping around in the dark. 
Also being new to Twisted it is like the room is full of chairs as
well.  I did get some sample code to work over the weekend, but I
got a serious brain cramp trying to figure it all out. 
The basic architecture that I am thinking of is client/server. 
There would be three basic applications running to make things
work.  First, would be an Agent program that does most of the
work.  The Agent would run as a service on each computer being
backed up.  Second would be a GUI program running locally to 
control
the agents.  Finally there would be a "Web Service" that
received backup sets from each agent or a designated master agent. The
Web Service would handle authentication and other house keeping
things. 
Finally the Question!!!! 
There are many other details that I have in mind as well (using
patches to reduce size, compression, encryption...), but right now I am
working on issues related to the communication between the GUI and
Agents.  Assuming that I have one or more number of agents running
on a network I want to do a number of things.  1)  Have the 
GUI
request that the agents identify themselves so I don't have to tell the
GUI where they are.  2) Transmit a catalog list of the file system
to the GUI so I can build a TreeView and pick which files to backup on
each computer.  3)  Finally I may want to identify a
"Master" agent that collects all the backups on the network 
and
transmits them as one session to the Web Service. This would allow for
local "On Site" backups for fast restores due to normal
computer crashes and failures.  
1.<x-tab>      </x-tab>What would you 
guys
suggest I do to I get the GUI to ask all the agents to identify
themselves? 
2.<x-tab>      </x-tab>What would be the
best approach using twisted to transmit a file system catalog across 
the
local wire.  I was playing with some remote procedure calls over 
the
weekend that seemed to work.  I am thinking I could create a class
that would gather the local file system and send it over the wire as 
one
lump. 
3.<x-tab>      </x-tab>If I am going to
transmit a backup file (<500meg) over the wire to another local
computer what is the best Twisted approach to take?  Due to WinXP
security issues and the desire the run in a cross platform environment 
I
don't want to rely on normal file shares. 
There are a hundred other questions running though my mind right now, 
but
this will due for now, 
 
Thanks for any help you can give, 
David A. Leedom 
 
<x-sigsep></x-sigsep>
The Hightower Group, Inc. 
Custom Software Solutions Designed To Fit Your Business Like A
Glove. 
165 West Airport Road/Lititz, PA 17543 
V:717-560-4002, 877-560-4002 x: 114 
F:717-560-2825 
<a href="http://www.hightowergroup.com/" 
eudora="autourl">www.hightowergroup.com</a>
</body>
</html>
--=====================_580045343==.ALT--
--__--__--
Message: 2
Date: Mon, 16 Feb 2004 18:23:03 +0100
From: Michal Pasternak <michal@pasternak.w.lub.pl>
To: twisted-python@twistedmatrix.com
Subject: Re: [Twisted-Python] Backup Agent
Reply-To: twisted-python@twistedmatrix.com
David A. Leedom [Mon, Feb 16, 2004 at 10:03:37AM -0500]:
...
1. What would you guys suggest I do to I get the GUI to ask all 
the
agents to identify themselves?
Well, that's a good question. In some app I did, I used machine's IP 
address
+ some operating system info it run (sys.platform, os.uname) to build 
a kind
of machine identifier. You could as well generate some UID, then save 
it,
then send it together with machine's IP to identify it on server, by 
hand.
Using OS-specific procedures could help you with this.
...
2. What would be the best approach using twisted to transmit a 
file
system catalog across the local wire. I was playing with some remote
procedure calls over the weekend that seemed to work. I am thinking I
could create a class that would gather the local file system and send 
it
over the wire as one lump.
Remote procedures have limit of 640 KB for their parameters (because 
"640 KB
should be enough to anyone"). You should use Pager and Collector.
...
3. If I am going to transmit a backup file (<500meg) over the 
wire to
another local computer what is the best Twisted approach to take?
See above. Also, please remember, that memory issues matter. If you're
paging the data (send small chunks), you should save them on the backup
server, as they are received. If you are transfering a big file, 
keeping all
the data received in some buffer is a bad idea.
--__--__--
Message: 3
Date: Mon, 16 Feb 2004 13:00:47 -0800
To: Twisted <twisted-python@twistedmatrix.com>
From: stephan <mailinglists@shechen.at>
Subject: [Twisted-Python] p2p, gnutella
Reply-To: twisted-python@twistedmatrix.com
I am looking forward to add p2p functionality to my app. Does anybody 
know
what the current status of twisted's gnutella implementation is?
There seems to have been a semi-finished implemention in 1.0.2alpha4 
which
I can't find in 1.1.1 anymore.
I might also be willing to implement missing parts but I would need to 
get
a short briefing on how things are.
thanks,
_stephan
--__--__--
Message: 4
Date: Mon, 16 Feb 2004 18:13:13 -0500
To: peak@eby-sarna.com
From: "Phillip J. Eby" <pje@telecommunity.com>
Cc: zope3-dev@zope.org, twisted-python@twistedmatrix.com
Subject: [Twisted-Python] PEAK 0.5a3 released
Reply-To: twisted-python@twistedmatrix.com
After several months of feature creep, the third alpha release of PEAK 
0.5
is now available for download at:
http://peak.telecommunity.com/dist/
Some highlights of the changes since 0.5a2 include:
* A simplified and consolidated binding API: there are now only four 
kinds
of bindings (Make, Obtain, Require, and Delegate) that provide a wider
variety of functionality than the old API. You can also now create 
your
own IRecipe or IComponentKey implementations for use with Make and 
Obtain.
* The 'n2' (namespace navigator) tool provides a handy, extensible 
shell
(with completion and history) for exploring PEAK naming systems and SQL
connections.
* A greatly enhanced configuration file parser, with lots of new
convenience features such as [Import on Demand], [Named Services], and
[Component Factories] sections, based on its new extensible parsing
framework. You can define new section types of your own, even from 
inside
an .ini file.
* New configuration key features, such as multi-keys (similar to some 
of
Zope X3's "adaptergeddon" features), the ability to iterate over 
defined
keys in a hierarchy (useful for finding/registering various kinds of
"plugins"), new convenience features for defining wildcard property 
rules,
* A new 'peak.events' framework that supports event sources, listeners,
subscriptions, generator-based pseudothreads ("tasks"), scheduling, 
and I/O
events. Using 'peak.events', you can write event-driven code in a more
natural, sequential, "untwisted" style, but without giving up access to
Twisted's many great features. 'peak.events' can use PEAK's built-in 
event
loop, or adapt a Twisted reactor for use as an event loop. You can 
also
use Twisted's "Deferred" objects as event sources, which means you can 
use
them in your Tasks (pseudothreads).
* A new 'peak.ddt' "document-driven testing" framework, similar in 
form and
function to the FIT ( http://fit.c2.com/ ) framework, but in Python and
well-integrated with the rest of PEAK. 'peak.ddt' lets you write test 
data
as tables in HTML documents (created with Word, Open Office, Mozilla, 
or
virtually any other tool), and then view the results in your
browser. Correct results are highlighted in green, incorrect answers 
are
highlighted in red, and errors are highlighted in yellow, with a 
small-font
traceback added to the table cell. It's a great way to do data-driven
acceptance testing, or to manage a project's progress using example 
data as
part of a requirements document.
* Many, many more features, both large and small, like a pre-forking
multiprocess supervisor for FastCGI and other forking servers. Quick
access to documentation via the 'peak help' command, which uses pydoc 
to
display reference documentation. Early versions of the 'peak.security'
(access control), 'peak.web' (Zope X3-based web publishing), and
'peak.query' (declarative relational and conceptual queries w/SQL
generation) packages. The 'whenImported()' function that lets you 
call a
hook when a named module is first used. "Mock" DBAPI and socket
implementations for testing. Hooks for "thunking" database-specific 
SQL
and functions. And many, many more examples of unbelievably blatant
feature creep!
There is also now a nice "Hello World" introduction to PEAK available 
at:
http://peak.telecommunity.com/DevCenter/IntroToPeak
that presents a guided tour of some of PEAK's "vertical" frameworks, 
like
command-line application support, database access, and so on.
For more information about PEAK, and access to community resources
including documentation, the Wiki, mailing list archives, IRC channel 
logs,
CVS access, and more, please visit:
http://peak.telecommunity.com/
--__--__--
Message: 5
To: twisted-python@twistedmatrix.com
From: ralf@brainbot.com
Date: Tue, 17 Feb 2004 02:05:41 +0100
Subject: [Twisted-Python] [PATCH] nasty bug in 
twisted.names.client.createResolver
Reply-To: twisted-python@twistedmatrix.com
The parameter variable 'hosts' gets overwritten by the import statement
in createResolver. hosts.Resolver constructor is called with a module
instead of a filename.
- Ralf
===================================================================
RCS file: /cvs/Twisted/twisted/names/client.py,v
retrieving revision 1.45
diff -u -r1.45 client.py

--- twisted/names/client.py     4 Feb 2004 19:55:34 -0000       1.45
+++ twisted/names/client.py     17 Feb 2004 01:01:53 -0000
@@ -303,20 +303,20 @@
         return p
-def createResolver(servers = None, resolvconf = None, hosts = None):
+def createResolver(servers = None, resolvconf = None, hostsfile = 
None):
     import resolve, cache, hosts, root
     if platform.getType() == 'posix':
         if resolvconf is None:
             resolvconf = '/etc/resolv.conf'
-        if hosts is None:
-            hosts = '/etc/hosts'
+        if hostsfile is None:
+            hostsfile = '/etc/hosts'
         theResolver = Resolver(resolvconf, servers)
-        hostResolver = hosts.Resolver(hosts)
+        hostResolver = hosts.Resolver(hostsfile)
     else:
-        if hosts is None:
-            hosts = r'c:\windows\hosts'
+        if hostsfile is None:
+            hostsfile = r'c:\windows\hosts'
         bootstrap = ThreadedResolver()
-        hostResolver = hosts.Resolver(hosts)
+        hostResolver = hosts.Resolver(hostsfile)
         theResolver = root.bootstrap(bootstrap)
L = [hostResolver, cache.CacheResolver(), theResolver]
-- 
brainbot technologies ag
boppstrasse 64 . 55118 mainz . germany
fon +49 6131 211639-1 . fax +49 6131 211639-2
http://brainbot.com/  mailto:ralf@brainbot.com
--__--__--
Message: 6
Subject: Re: [Twisted-Python] [PATCH] nasty bug in
  twisted.names.client.createResolver
From: Itamar Shtull-Trauring <itamar@itamarst.org>
To: twisted-python@twistedmatrix.com
Organization: http://itamarst.org
Date: Mon, 16 Feb 2004 20:33:55 -0500
Reply-To: twisted-python@twistedmatrix.com
On Mon, 2004-02-16 at 20:05, ralf@brainbot.com wrote:
...
The parameter variable 'hosts' gets overwritten by the import 
statement
in createResolver. hosts.Resolver constructor is called with a module
instead of a filename.
Please add patches and bugs to twistedmatrix.com/bugs/
--__--__--
Message: 7
Subject: Re: [Twisted-Python] Advice on doing thousands of 
simultaneous UDP
  queries with Twisted...
From: Itamar Shtull-Trauring <itamar@itamarst.org>
To: twisted-python@twistedmatrix.com
Organization: http://itamarst.org
Date: Mon, 16 Feb 2004 20:40:06 -0500
Reply-To: twisted-python@twistedmatrix.com
On Mon, 2004-02-09 at 05:51, Mike C. Fletcher wrote:
...
Writing a simple asynchronous loop myself (poll on a simple socket, 
send
message from queue when writable, read one into other queue when
readable) allowed for doing a few thousand queries simultaneously), 
with
only a few dozen dropped messages.  However, with the Twisted 
equivalent
(UDPTransport with my simple protocol object), I was seeing huge drop
rates, so, gathering that Twisted isn't queueing up the UDP requests,
Possibly we need some generic UDP queuing, rather than current system.
Patches are welcome.
-- 
Itamar Shtull-Trauring    http://itamarst.org
Looking for a job: http://itamarst.org/resume.html
--__--__--
Message: 8
To: twisted-python@twistedmatrix.com
From: ralf@brainbot.com
Date: Tue, 17 Feb 2004 02:47:19 +0100
Subject: [Twisted-Python] [PATCH] twisted.names.hosts not working 
correct with getHostByName
Reply-To: twisted-python@twistedmatrix.com
Even with my prior patch I'm not able to use 
twisted.names.client.getHostByName.
Seems like twisted.names.hosts.Resolver calls itself with bad 
parameters
and even doesn't use the ip address it extracted from /etc/hosts. Also 
the return
value could not be handled by the code in 
twisted.names.common.extractRecord.
The following patch seems to solve those problems.
- Ralf
===================================================================
RCS file: /cvs/Twisted/twisted/names/hosts.py,v
retrieving revision 1.14
diff -u -r1.14 hosts.py
--- twisted/names/hosts.py      19 Jul 2003 10:13:50 -0000      1.14
+++ twisted/names/hosts.py      17 Feb 2004 01:45:44 -0000
@@ -49,11 +49,11 @@
def query(self, query, timeout = None):
-        return self._lookup(str(query.name), query.type, query.type, 
timeout)
+        return self._lookup(str(query.name), query.cls, query.type, 
timeout)
def _lookup(self, name, cls, type, timeout):
-        if cls != dns.IN or type != dns.A:
+        if cls != dns.IN or (type != dns.A and type != 
dns.ALL_RECORDS): # getHostByName passes type==dns.ALL_RECORDS. what 
else do we need here?
             return 
defer.fail(failure.Failure(NotImplementedError(type, cls)))
         return self.lookupAddress(
             name, timeout
@@ -64,7 +64,7 @@
         res = searchFileFor(self.file, name)
         if res:
             return defer.succeed([
-                [dns.RRHeader(name, dns.A, dns.IN, self.ttl)], (), ()
+                [dns.RRHeader(name, dns.A, dns.IN, self.ttl, 
dns.Record_A(res, self.ttl))], [], []
             ])
         return defer.fail(failure.Failure(dns.DomainError(name)))
-- 
brainbot technologies ag
boppstrasse 64 . 55118 mainz . germany
fon +49 6131 211639-1 . fax +49 6131 211639-2
http://brainbot.com/  mailto:ralf@brainbot.com
--__--__--
Message: 9
Date: Mon, 16 Feb 2004 22:47:47 -0300
From: Daniel Henrique Debonzi <debonzi@gwyddion.com>
To:  twisted-python@twistedmatrix.com
Subject: [Twisted-Python] Client Prompt
Reply-To: twisted-python@twistedmatrix.com
Hi everybody,
Maybe this is a very stupid question, but I realy couldn't find a way 
to
solve it by myself.
Would be enough for my aplication if this example works.
So, I want to have a client that read a string from the stdin and then
send it to the server. Wich changes should I do in this code to this
client work like a discrebed
#################################################################
from twisted.internet import reactor, protocol
class EchoClient(protocol.Protocol):
def connectionMade(self):
  """I don't want to write it here
     I want to take data from stdin and send o server,
     but were and how to do it? Too much dificult?
  """
         self.transport.write("hello, world!")
def dataReceived(self, data):
         print data
def connectionLost(self, reason):
         print "connection lost"
         from twisted.internet import reactor
         reactor.stop()
class EchoFactory(protocol.ClientFactory):
     protocol = EchoClient
def clientConnectionFailed(self, connector, reason):
         print "Connection failed - goodbye!"
         reactor.stop()
def clientConnectionLost(self, connector, reason):
         print "Connection lost - goodbye!"
         reactor.stop()
def main():
     f = EchoFactory()
     reactor.connectTCP("localhost", 5555, f)
     reactor.run()
if __name__ == '__main__':
     main()
#################################################################
Thanks for all
-- 
Daniel Henrique Debonzi  Linux user number 166903
Gwyddion Embedded Free-Software Development
www.gwyddion.com
São Carlos, São Paulo
Brazil
--__--__--
Message: 10
To: twisted-python@twistedmatrix.com
Subject: Re: [Twisted-Python] [PATCH] nasty bug in
 twisted.names.client.createResolver
From: ralf@brainbot.com
Date: Tue, 17 Feb 2004 03:20:28 +0100
Reply-To: twisted-python@twistedmatrix.com
Itamar Shtull-Trauring <itamar@itamarst.org> writes:
...
On Mon, 2004-02-16 at 20:05, ralf@brainbot.com wrote:
...
The parameter variable 'hosts' gets overwritten by the import 
statement
in createResolver. hosts.Resolver constructor is called with a module
instead of a filename.
Please add patches and bugs to twistedmatrix.com/bugs/
Lots of bug reports in the bug tracker have status 'unread'. That's
not very motivating. Just committing this patch -even if it was wrong-
would not make things worse, as the original code is definitely
broken. But, I'll add it to the bugtracker tomorrow.
- Ralf
-- 
brainbot technologies ag
boppstrasse 64 . 55118 mainz . germany
fon +49 6131 211639-1 . fax +49 6131 211639-2
http://brainbot.com/  mailto:ralf@brainbot.com
--__--__--
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
End of Twisted-Python Digest

    

[Twisted-Python] Backup Agent

Frankie Chu

tags

participants (1)