Mailman 3 March 2004 - Twisted

[Twisted-Python] Deterministic TimerService
by Norm Petterson March 4, 2004

March 4, 2004

Hi, Thanks for such a useful framework. While working on an industrial data logger using Twisted, I found that application.internet.TimerService calls my own callLater callable function before reissuing the next callLater. Consequently, instead of a time period c, the callable function runs every c+x, where x is the duration of my own function, and thus appears to slip over time. Based on my past experiences with DEC RSX "mark time" and VMS "set timer" in similar situations, I recommend … [View More]

3 3

[Twisted-Python] Running twisted tap file as NT service.
by Max Thiercy March 4, 2004

March 4, 2004

I'm trying to implement a small dedicated service on both Windows and Unix. So, I'm interesting in the capability of converting a twisted .tap file to an NT/2K/XP service. I read some useful mails from Cory Dodt (end of january 2004) but these mails mention a tool named 'tap2ntsvc' that I can't find anywhere. Could you tell me where I can find this tool ? Thanks in advance. Max THIERCY.

3 2

[Twisted-Python] deferDirWalk code snipped
by Michal Pasternak March 4, 2004

March 4, 2004

Hi, attached is a simple defer analogue to os.path.walk. It can be paused, it can be restarted. I've wrote this code making some simple gtk backup application. Perhaps this could be placed in examples somewhere (or even included in the library). If someone would like to comment on my code, feel free to do it, I always like constructive criticism ;) Keep hacking, -- Michal Pasternak :: http://pasternak.w.lub.pl :: http://winsrc.sf.net "There's so much comedy on television. Does that cause … [View More]

3 2

[Twisted-Python] Re: [Stackless] Stackless and Twisted
by Christopher Armstrong March 4, 2004

March 4, 2004

Ok, here's a completely rewritten stackless support module for Twisted. I used to have one that did all kinds of stupid unnecessary stuff with the reactor and junk, but it's gone now. For you CVS Twisted users, this is in sandbox/radix/threadless.py. It offers two functions: blockOn and deferredTasklet. deferredTasklet is a function decorator that you should use on your stacklessy code that you want to integrate with Twisted. blockOn is a function that you should pass a Deferred to wait … [View More]

1 0

Re: [Twisted-Python] [Request] IMAP Server Sample code
by exarkun＠divmod.com March 4, 2004

March 4, 2004

On Tue, 02 Mar 2004 11:49:29 +0000, David Burton <david(a)zeos.org> wrote: >Hi guys, I'm new to the list, but I've had a look through the site and > the mailing list archives, looking for some sample code which would help > me in setting up the basics of an IMAP server - specifically to get mail > from a POP account (which is easy enough), store it in a database > (again, easy enough), and set up the interface to the database messages > (and possible other non-email … [View More]

1 0

[Twisted-Python] resource.render_METHOD
by Richard.Townsend＠edl.uk.eds.com March 3, 2004

March 3, 2004

> Hmm. I suppose render() could grope through methods on self to find all > 'render_FOO' methods when an exception occurs.. OTOH, we could just get > rid of the allowed methods reporting. What's the use? Perhaps the Resource.render() method could check for self.allowedMethods instead of server.UnsupportedMethod.allowedMethods, then I could set it as a class attribute in each derived class?

2 1

[Twisted-Python] resource.render_METHOD
by Richard.Townsend＠edl.uk.eds.com March 2, 2004

March 2, 2004

I have a server application that creates a number of web resources (derived from resource.Resource). Some of the resources only handle GET requests, while others only handle POSTs. Currently the resources only override the render() method. I want to change them to override render_GET() or render_POST() as appropriate. I notice that if a resource receives a request it doesn't handle, then an UnsupportedMethod exception is raised. This returns HTML to the client containing a message that uses … [View More]

2 1

[Twisted-Python] [Request] IMAP Server Sample code
by David Burton March 2, 2004

March 2, 2004

Hi guys, I'm new to the list, but I've had a look through the site and the mailing list archives, looking for some sample code which would help me in setting up the basics of an IMAP server - specifically to get mail from a POP account (which is easy enough), store it in a database (again, easy enough), and set up the interface to the database messages (and possible other non-email goodies) using IMAP (currently not a clue). I've tried looking through the API and the source code, but I don'… [View More]

2 1

[Twisted-Python] Using Twisted together with Windows Namespace Extensions
by Thorsten Henninger March 2, 2004

March 2, 2004

Hi everybody, I try to implement an ActiveX component for a Windows Namespace Extension, such that the data of my server, a twisted perspective broker, can be accessed and browsed via the Windows explorer. The challenge on writing something like this is, that once Windows asks my ActiveX component to open a folder, a file or any other object, Windows expects an immediate answer. But asking by twisted server just returns a deferred. To emulate those synchronous calls, I could use the "… [View More]

3 3

[Twisted-Python] Running a reactor in IPython
by Thorsten Henninger March 2, 2004

March 2, 2004

Hi all, At my company, we heavily use IPython for debugging and testing. The following code shows, how to run a reactor together with IPython: from twisted.internet import reactor from twisted.python import threadable threadable.init(1) import threading class MyThread(threading.Thread): def run(self): print "starting reactor.." reactor.run(installSignalHandlers=False) if __name__=="__main__": m=MyThread() m.start() from IPython.Shell import … [View More]

1 0

[Twisted-Python] WebDAV for Twisted
by Thorsten Henninger March 2, 2004

March 2, 2004

Hi all, since some people were interested in using WebDAV together with Twisted recently, I decided to send you my current work on this. I essentially used the "pydav" implementation: http://www.comlounge.net/webdav/ and wrote a twisted extension. Once you are familar with "pydav", the twisted extension is self-explaning. I hope, the short README.txt will help you. By the way, I tested this implementation together with the Windows Explorer and cadaver, a command line WebDAV client. In my … [View More]

1 0

[Twisted-Python] Collaborative file storage via ftp/sftp/smb and html
by Andy Gayton March 2, 2004

March 2, 2004

I'm investigating setting up a collaborative remote file share. There are a number of traditional ways for doing this, each has their strengths and weaknesses: ftp - not secure, fast and robust file transfer, supports resume, difficult to administer as unless chrooted users have access to the entire file system, if chrooted its difficult to set up group, no vendor lockin on the client sftp - same as ftp but secure smb - secureish, slow over wan, doesn't support resume, great to … [View More]

3 4

[Twisted-Python] Help about adbapi and sqlreflector
by Celso Providelo March 1, 2004

March 1, 2004

Maybe it might be a dummy question, but I couln't solve my doubts with the set of examples from Twisted about the use of a PostgreSQL DB Server in the Twisted way. Will be better if I explain exactly my case: I have a set of CLIENTS invoking random methods from a SERVER throught my protocol, one of these should add/update an entry in my DB. As I can see, if I just add: conn = PgSQL.connect(database='test'); ..... """my method !!!!!""" curs = conn.cursor(); curs.execute(… [View More]

2 1

RE: [Twisted-Python] SSH connection, redux!
by Schollnick, Benjamin March 1, 2004

March 1, 2004

Folks, Thanks for the support, so far... I downloaded the precompiled version of the PyCrypto and everything is working fine... Except for the fact, that I am attempting to figure out how to make this work the way, that I *believe* it needs to work... By the way, I was unable to get the "Writing a client with Twisted.Conch" FAQ to work.... I did find the sshSimpleClient.py example, and that is working okay... But I am having a problem, trying to figure out how to adapt the reactor … [View More]

2 1

RE: [Twisted-Python] SSH connection, redux!
by Schneider, Michael March 1, 2004

March 1, 2004

Benjamin, I just posted some code for review for spawning jobs off and capturing jobs. I posted this code for review, since I am new to twisted, and trying to lean the "twisted way". This approach may work for you, Mike ---------------------------------------------------------------- Michael Schneider Senior Software Engineering Consultant UGS PLM Solutions - an EDS Company "The Greatest Performance Improvement Is the transitioning from a non-working state to the working state" > --… [View More]

1 0

[Twisted-Python] RE: tracking spawned jobs in a twisted application (code review request)
by Schneider, Michael March 1, 2004

March 1, 2004

Sorry, I hit the send button too soon, Mike ---------------------------------------------------------------- Michael Schneider Senior Software Engineering Consultant UGS PLM Solutions - an EDS Company "The Greatest Performance Improvement Is the transitioning from a non-working state to the working state" > -----Original Message----- > From: Schneider, Michael > Sent: Monday, March 01, 2004 11:28 AM > To: 'twisted-python(a)twistedmatrix.com' > Subject: tracking spawned … [View More]jobs in a twisted application (code review > request) > > > Hello All, > > I have been pushing some code around. > > My problem is that I need to execute OS command line > jobs with twisted. > > I started by morphing the twisted tutorial app into a > job runner. > > > My first path (It was wrong!!) was to morph the database connection > pool to a compute pool using popen2. > > This group, suggested that I use reactor.spawn instead of popen2 . > > Attached is the resulting code. > > Basic Approach for Command protocol.processProtocol: > > 1) crate a JobPool Object to manage list of running jobs > 2) Create Job Runner Object to wrap individual process info, > and capture process output. > (start time, stop time ...) > 3) Create CmdProcProtcol to manage interaction with the reactor > > > I morphed the twisted application: (Coded in comment section below) > 1) Create JobPoolService > 2) Create JobRunnerProtocol > 3) Create XmlRpc Job Runner Interface that dispatches > to job runner service. > > > > My questions are: > - Is the the twisted way? > - What are some alternative approaches? > - Is there a better way? > > Thank you very much, > Mike > > ---------------------------------------------------------------- > Michael Schneider > Senior Software Engineering Consultant > UGS PLM Solutions - an EDS Company > > "The Greatest Performance Improvement Is the transitioning > from a non-working state to the working state" > # Twisted, the Framework of Your Internet """ Manage execution of multiple processes\ Usage Model: 1)Create Job runner Service in App 2)Create Job Runner Protocol in App 3) Create Xmlrpc to call ----- from service code in twisted app m = RunTaskPool() cmdLine="dir d:\" def = m.runJob(cmdLine) class JobRunnerService(service.Service): __implements__ = service.Service.__implements__, IJobRunnerService def __init__(self): self.taskPool = None def runJob(self, command): print "In Twisted runJob, command = " + command self.verifyRunTaskPool() return self.taskPool.runJob(command) def verifyRunTaskPool(self): if(self.taskPool == None): self.taskPool = RunTaskPool() class JobRunnerProtocol(basic.LineReceiver): def lineReceived(self, command): d = self.factory.runJob(command) d.addErrback(catchError) def writeValue(value): self.transport.write(value) self.transport.write('\n\n') self.transport.loseConnection() d.addCallback(writeValue) class JobStatusXR(xmlrpc.XMLRPC): def __init__(self, service): xmlrpc.XMLRPC.__init__(self) self.service = service def xmlrpc_runJob(self, command): return self.service.runJob(command) def xmlrpc_isAlive(self): return 1 """ from twisted.internet import protocol from twisted.internet import reactor from twisted.spread import pb from twisted.python import reflect, log from twisted.internet import defer from time import sleep from time import time import os class CmdProcProtocol(protocol.ProcessProtocol): """Twisted Protocol Class to run command line in a sub process """ def __init__(self, parentJobRunner): self._exitStatus = 0 self.parentJobRunner = parentJobRunner self.data = "" def connectionMade(self): """Called at the start of the execution""" #print "connectionMade!" pass def outReceived(self, data): """Called when process writes output""" #print "outReceived! with %d bytes!" % len(data) self.data = self.data + data #print self.data def errReceived(self, data): #print "errReceived! with %d bytes!" % len(data) pass def inConnectionLost(self): #print "inConnectionLost! stdin is closed! (we probably did it)" pass def outConnectionLost(self): """Program has closed stdout (program terminated)""" #print "outConnectionLost! The child closed their stdout!" #print "# now is the time to examine what they wrote" #print "dir:", self.data pass def errConnectionLost(self): """Program has closed stderr (program terminated)""" #print "errConnectionLost! The child closed their stderr." pass def processEnded(self, status_object): """Child process exited""" self._exitStatus = status_object.value.exitCode print "processEnded, status %d" % self._exitStatus print "quitting" print self.data #notify parent with data returned , and exit status self.parentJobRunner.signalRunComplete(self._exitStatus, self.data) def kill(self): """Kill this process""" #os.kill(self.transport.pid, signal.SIGKILL) class JobRunner: """ I am a lightweight wrapper for run job. """ def __init__(self, parentJobRunnerPool, jobRunDeferred): #command to execute in Command Proc Protocol self.command = "" #set self.parentJobRunnerPool = parentJobRunnerPool self._startTime = -1 self._endTime = -1 self._timeLimit = -1 self._exitcode = None self._log = "job not yet started" self._jobRunDeferred = jobRunDeferred def getLogCallback(self,*args, **kw): """Called by Process Run Protocol when child process execution is complete """ #Notify JobRunner Pool That this job has completed execution self.parentJobRunnerPool.jobCompleteCallback(self) #return output of job return self._log def getExitCode(self): return self._exitcode def getStartTime(self): """Get Time Job Started Executing """ return self._startTime def getEndTime(self): """Get Time Job Stopped Executing """ return self._endTime def getRunTime(self): """ get time executing""" if(self._startTime < 0): #job has not started yet, return -1 return -1 if(self._endTime < 0): #job is running, but not yet finished, return time spent so far return (time() - self._startTime) #job is complete, return run time return (self._endTime - self._startTime) def runJob(self, *args, **kw): """Execute Job: NOTE: this function executes in its own thread """ #set execution start time self.command = (args[0])[0] print "In Run Job daemon command : " + self.command self._startTime = time() print "Start Running Command : " + str(self.command) + " " + str(self.getStartTime()) self._log = "Job is Running: " #setup reactor to run job self.runCommand(self.command) def runCommand(self, commandStr): """Create Command Line Protocol, and call spawnProcess""" commandStr = 'cmd.exe /c ' + commandStr pp= CmdProcProtocol(self) reactor.spawnProcess(pp, commandStr , env=os.environ) def signalRunComplete(self, exitCode, stdoutString): # Pass Control back to caller to self.runJob(..) # by triggering defered callback, # this callback will return this object self._exitCode = exitCode self._log = stdoutString # set execution time self._endTime = time() print "Done Running Job " + self.command reactor.callFromThread(self._jobRunDeferred.callback, self) class RunTaskPool(pb.Referenceable): running = 0 # true when the pool is operating def __init__(self, *args, **kw ): """See RunTaskPool.__doc__ """ self.args = args self.kw = kw self._jobRunnerJobs = [] self.jobs = {} # running Jobs, hashed on thread id self.startID = reactor.callWhenRunning(self.start) def start(self): """Start of execution """ if not self.running: self.shutdownID = reactor.addSystemEventTrigger('during', 'shutdown', self.finalClose) self.running = 1 def jobCompleteCallback(self, jobRunner): """Callback triggered by JobRunner when execution of job is complete """ print "Deleting Job Object from running queue" if jobRunner in self._jobRunnerJobs: self._jobRunnerJobs.remove(jobRunner) def runJob(self, *args, **kw ): """run job and return the result as a defered object """ print "MLS: runJob - setting up to run in thread " d = defer.Deferred() jobRunner = JobRunner(self,d) #Add to list of JobRunnerJobs managed self._jobRunnerJobs.append(jobRunner) print "MLS: in _runJobRunner " # run Job in thread pool thread jobRunner.runJob(args, kw) # return defered used by jobRunner d.addCallback(jobRunner.getLogCallback) return d def close(self): """Close all pool connections and shutdown the pool.""" from twisted.internet import reactor if self.shutdownID: reactor.removeSystemEventTrigger(self.shutdownID) self.shutdownID = None if self.startID: reactor.removeSystemEventTrigger(self.startID) self.startID = None self.finalClose() def finalClose(self): """This should only be called by the shutdown trigger.""" self.running = 0 for job in self.jobs.values(): if self.noisy: log.msg(' closing: %s%s' % ( self.kw or '')) job.close() self.jobs.clear() def __getstate__(self): return {'noisy': self.noisy, 'min': self.min, 'max': self.max, 'kw': self.kw} def __setstate__(self, state): self.__dict__ = state apply(self.__init__, self.kw) def startup(): for x in range(1,14): d = m.runJob() d.addCallback(printSelf) def printSelf(jobRunner): print "Job Runner Done" [View Less]

1 0

[Twisted-Python] tracking spawned jobs in a twisted application (code review request)
by Schneider, Michael March 1, 2004

March 1, 2004

Hello All, I have been pushing some code around. My problem is that I need to execute OS command line jobs with twisted. I started by morphing the twisted tutorial app into a job runner. My first path (It was wrong!!) was to morph the database connection pool to a compute pool using popen2. This group, suggested that I use reactor.spawn instead of popen2 . Attached is the resulting code. Basic Approach for Command protocol.processProtocol: 1) crate a JobPool Object to manage list of … [View More]running jobs 2) Create Job Runner Object to wrap individual process info, and capture process output. (start time, stop time ...) 3) Create CmdProcProtcol to manage interaction with the reactor I morphed the twisted application: (Coded in comment section below) 1) Create JobPoolService 2) Create JobRunnerProtocol 3) Create XmlRpc Job Runner Interface that dispatches to job runner service. My questions are: - Is the the twisted way? - What are some alternative approaches? - Is there a better way? Thank you very much, Mike ---------------------------------------------------------------- Michael Schneider Senior Software Engineering Consultant UGS PLM Solutions - an EDS Company "The Greatest Performance Improvement Is the transitioning from a non-working state to the working state" > -----Original Message----- > From: twisted-python-admin(a)twistedmatrix.com > [mailto:twisted-python-admin@twistedmatrix.com]On Behalf Of > Schollnick, > Benjamin > Sent: Monday, March 01, 2004 10:26 AM > To: 'twisted-python(a)twistedmatrix.com' > Subject: RE: [Twisted-Python] SSH connection, redux! > > > Folks, > > Thanks for the support, so far... I downloaded the > precompiled version of the PyCrypto and everything is working fine... > > Except for the fact, that I am attempting to figure out > how to make this work the way, that I *believe* it needs to work... > > By the way, I was unable to get the "Writing a client with > Twisted.Conch" FAQ to work.... I did find the sshSimpleClient.py > example, and that is working okay... > > But I am having a problem, trying to figure out how to > adapt the > reactor model with my program... > > The application has, for example, 6 "ssh queries" that > it needs to > send, for example a "/usr/sbin/prtconf -v", and then > "capture" the output, > and return the output > for processing... > > Then it proceeds to the next query....etc... > > Is there a method that I can setup the communication, > and say.... > > reactor.send ("asdfasdf") > data = reactor.recv () > > More along the telnetlib design? This way, I could branch the > telnet code > I use? > > Once, I start the reactor, I do not see where I can get > data in and > out, > without using some form of a global variable... (Or > embedding the reactor > inside > another object) > > - Benjamin > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python(a)twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > [View Less]

1 0