Mailman 3 January 2003 - Twisted

[Twisted-Python] Large project (IMS) architecture
by Rick Richardson Jan. 16, 2003

Jan. 16, 2003

I have had the task of the architecture of a lab information management system. If you're not familiar with a LIMS it basically tracks a sample of something, be it a soil sample, a solution generated in a lab, whatever, through its creation/logging through instrument testing, analysis, verification and disposal. It will be a replacement for our current project which is written currently in C++ Builder 4, is about 1900 source files, 52 applications and.. approx 180 different screens/docked-panels. One of the challenges is the heavy relationships between all of the different entities in the database, as well as some meta concepts of batching and QC. Also the there is the maintenance of the overall system as well as roles of admin/employee/customer. Much of the current app is redundant and could probably be reduced to about 60 screens that do perform quite a bit of computation of one sort or another. This is also a somewhat distributed system, having a single system (either client server or p2p) used on a multitude of desktops, possibly geographically seperated. My vision for the project is to, first off, create an Information Management System, then customize it into a Lab IMS. I want to take a single (or very few) screen approach that is data and context sensitive. So that it will dynamically create most of the screens, leaving custom development for only a handful of computationally intensive tasks that still take advantage of an sweet component structure. For this task I turn to Twisted. Partically because I'm enamoured with the way that it takes advantage of, and complements pythons dynamic nature. And partially I think because it is so large and amorphous that I'm sure there's a solution somewhere in there for me. My questions: In what areas should I begin looking for components that will help me? Is a single, dynamic screen approach reasonable? What is people's preference? A centralized object center and RMI? Or purely central data and all logic on the client? I know this is a rather vague request for information but anything is helpful. Rick Richardson Quidquid latine dictum sit, altum videtur. (Anything said in Latin sounds profound.)

3 2

[Twisted-Python] Some high-level python 2.2 generator toys
by Bob Ippolito Jan. 15, 2003

Jan. 15, 2003

I just spent an hour or two screwing around with generators and deferreds. I threw this together, I think it's pretty slick. I'm kinda tired, so the naming conventions and "documentation" might be stupid, but it does work.

1 0

[Twisted-Python] Producers and Consumers
by Andrew Bennetts Jan. 15, 2003

Jan. 15, 2003

Hi all, It seems to me that the IProducer and IConsumer interfaces in twisted.internet.interfaces are effectively useless. Say you wanted to write a class with a method like: class MyConsumer: def foo(self, producer): """@type producer: L{IProducer}""" # keep a reference to the producer, to consume stuff from it later How do you actually implement that method? IProducer only specifies {resume,pause,stop}Producing. IConsumer states that it expects its write() method to be called by the producer. But there is no way to find out the IConsumer's producer using the methods documented in IProducer/IConsumer (the interface doesn't explicitly say it, but it's fairly obvious that there can only be one consumer per producer, and vice versa). Similarly, you can't write a method like this: def streamData(self): """@returns: a L{Deferred}, which will be called with an object implementing L{IProducer}.""" # ... You'd use this like: d = source.streamData() d.addCallback(lambda p: self.registerProducer(p, streaming=1)) But of course, without knowing something extra about the producer beyond what IProducer tells you, you can't implement this, because there's no way to tell an IProducer what IConsumer to write to. (Incidentally, the 'streaming' argument to registerProducer isn't very clearly explained... after several readings, I *think* all it means is that the producer starts producing immediately, rather than waiting for a call to resumeProducing.) Finally, I also see that there is no way for an producer to signal to its consumer that it has finished. Perhaps for the intended use cases of producers and consumers, this is superfluous, but it would be very useful to me! My motivation for all this (in case you hadn't guessed :), is FTP. I've previously discussed the way the code currently works (by taking a Protocol instance as an argument, so that downloads can be fed directly to it), and I'm looking for clean ways to improve it, so I can add the STOR command (i.e. uploads). Now, I could do FTPClient.store similarly to how I do FTPClient.retrieve, i.e. def store(self, path, protocol): """ ... @param path: The path to store data to. @param protocol: A L{Protocol} instance that will write the data to be stored upon a call to C{connectionMade}, and will call C{self.transport.loseConnection()} when done. """ # ... But everyone assures me that requiring protocol instances as parameters is a bad way to do things, and my gut feeling agrees -- I just don't feel comfortable with asking Twisted users to instantiate a Protocol without involving a Factory. The obvious alternative that occured to me was to use producers and consumers -- after all, a download is just a stream of bytes waiting to be consumed, and an upload is just a connection waiting for data to be produced -- but IConsumer and IProducer don't fit this use-case. So, are IConsumer and IProducer fundamentally broken, or is it intentional that all Consumer/Producer implementations are co-dependent? And what is the right interface for FTPClient? If I'm missing the Zen of Consumption and Production, then please enlighten me, so that I may improve the existing docstrings and maybe write a howto? :) Regards, -Andrew.

1 1

[Twisted-Python] RE: How do I upload files using FTPClient?
by Daniel Burr Jan. 14, 2003

Jan. 14, 2003

WARNING! Long-winded, and perhaps clueless post ahead! As discussed earlier in this thread, here is a rough sketch of how I think a high-level interface to twisted.protocols.ftp.FTPClient should look. Currently, twisted.protocols.ftp contains mostly FTP-related classes; there is only one module-level method, "parsePWDResponse". I am thinking that we could just implement the high-level client interface as methods within the current twisted.protocols.ftp module. Ideally, I'd like to be able to use it like so: --------------------------------------------------------------------------- import os from twisted.protocols import ftp def deploy(self, appRoot, filePaths): """ Deploy code to an ftp server appRoot - string containing the root directory of some app filePaths - list or tuple containing absolute filepaths """ try: ftp.connect(host='foo.bar.org', port=21, user='anonymous' password='anonymous(a)anonymous.net') except ftp.ConnectionFailed: return False else: for filePath in filePaths: fileDirectory, fileName = os.path.split(filePath) uploadDirectory = '%s/%s' % (appRoot, fileDirectory) if ftp.pwd() != (uploadDirectory): try: ftp.cd(uploadDirectory) except ftp.DirDoesNotExist: makeDirectory(uploadDirectory) ftp.cd(uploadDirectory) ftp.binary() ftp.put(fileName) ftp.quit() return True def makeDirectory(self, path): """ Recursively create directories on the FTP server """ parent, base = os.path.split(path) try: ftp.cd(parent) ftp.mkdir(base) except ftp.DirDoesNotExist: makeDirectory(parent) ftp.cd(parent) ftp.mkdir(base) --------------------------------------------------------------------------- This code is pretty naive, to be sure, and not at all Twisted. However, it does hopefully illustrate the way I would like to interact with an FTP client. The "deploy" function could be refactored to use Deferreds pretty easily; the "makeDirectory" function would be similarly simple. As for the interface to FTPClient, I've outlined my thoughts below. Note that my immediate itch only calls for "cd", "pwd", "mkdir", and "put", of which only "mkdir" and "put" need to be implemented. It'd be cool, down the road, to implement "mget" and friends, so that my sample code above could be reduced even further, by removing the need to loop through the list of file paths and transfer files one at a time. My overall, grand ambition is to tie all of this together into a handy code deployment tool that can pull versioned code from multiple source control solutions, and deploy it using a variety of protocols. --------------------------------------------------------------------------- # a module level variable to which we will bind an instance of FTPClient theClient = None def append(localFile, remoteFile=localFile) """ Append the contents of a file on the client to a file on the server localFile - string containing the name of the file on the client remoteFile - string containing the name of the file on the server; defaults to the same name as the file on the client """ def ascii(): """ Set the transfer type to ascii (TYPE A) """ def binary(): """ Set the transfer type to binary (TYPE I) """ def bye(): """ Close the connection and exit the client """ def cd(remoteDirectory): """ Change directory on the server (CWD) remoteDirectory - string containing the directory name """ def close(): """ Close the connection, but do not exit the client """ def connect(host, port, userID='anonymous', password='foo(a)bar.org'): """ Connect to an FTP server This function will create an instance of FTPClient and bind it to "theClient", a module-level variable which we will treat like a singleton, so all these functions have an instance to work with """ def delete(remoteFile): """ Delete a file from the server remoteFile - string containing the name of the file on the server """ def debug(): """ Toggle debugging mode on/off If debugging mode is on, debug info will be logged to some default file """ def disconnect(): """ Close the connection, but do not exit the client """ def get(remoteFile, localFile=remoteFile): """ Transfer a file from the server to the client remoteFile - string containing the name of the file on the server localFile - string containing the name of the file on the client; defaults to the same name as the file on the server """ def literal(command): """ Send arbitrary FTP command command - string containing the line to be sent as a command """ def mdelete(remoteFiles): """ Delete multiple files from the server remoteFiles - list or tuple of filepaths on the server """ def mget(remoteFiles): """ Transfer multiple files from the server to the client remoteFiles - list or tuple of filepaths on the server """ def mkdir(directory): """ Create a directory on the server directory - string containing an absolute or relative path """ def mput(localFiles): """ Transfer multiple file from the client to the server localFiles - list or tuple of filepaths on the client """ def put(localFile, remoteFile=localFile): """ Transfer a file from the client to the server localFile - string containing the name of the file on the client remoteFile - string containing the name of the file on the server; defaults to the same name as the file on the client """ def quit(): """ Close the connection and exit the client """ def rmdir(directory): """ Delete a directory on the server directory - string containing an absolute or relative path """ --------------------------------------------------------------------------- Well, that's my brain dump for now. Be as harsh as you like in criticizing; I have a thick hide, and I'm trying to learn, so don't pull any punches. If anyone thinks it should be entirely different, that's fantastic, and I want to hear about it. L. Daniel Burr

3 2

[Twisted-Python] Woven
by Xavier Antoviaque Jan. 14, 2003

Jan. 14, 2003

Hello all, I have spent most of the day looking at the documentation of Twisted, and I am quite impressed. What a wonderful toy ! It is like Christmas a second time in the year. Congratulations to you happy hackers, you have accomplished a tremendous job. However, I have a question regarding Woven; as the howto ends quite abruptly, I have tried to figure out how I could build a web interface I planned to develop in a more usual way. But I quickly became puzzled by the lack of examples, and a quick look on the sources only helped a bit. Is Woven usable at this state ? This question is based on a limitation I encountered while trying to make the menu of my interface : I could not find a way to build a link (Anchor) with a text different from the href parameter. I found a message from Donovan Preston in this list's archives (http://twistedmatrix.com/pipermail/twisted-python/2002-December/002426 .html) stating that it was not possible with the actual Anchor widget. I would have the option to write a subclass of Anchor implementing what I need, but finding such a limitation on the first widget I look at is disturbing. :-) If Woven is usable, can you point some examples of realizations using it, and/or some good example sources to get started ? Thank you ! -- Xavier.

3 5

[Twisted-Python] First contribution...
by Sylane Jan. 14, 2003

Jan. 14, 2003

Hello twisted people, I actually working on the simplistic implementation of a complex spread example... soon in your best mailing list... But for now a little diff for pb... I hop it will be applied because my example don't work very nice without it... --snip-snip-------------------------------------------------------- --- ./twisted/spread/pb-orig.py Tue Jan 14 17:12:47 2003 +++ ./twisted/spread/pb.py Tue Jan 14 17:23:51 2003 @@ -543,8 +543,18 @@ try: d.errback(failure.Failure(PBConnectionLost())) except: - log.deferr() - for notifier in self.disconnects: + log.deferr() + # Assure all Cacheable.stoppedObserving are called + for lobj in self.remotelyCachedObjects.values(): + cacheable = lobj.object + perspective = lobj.perspective + try: + cacheable.stoppedObserving(perspective, RemoteCacheObserver(self, cacheable, perspective)) + except: + log.deferr() + # Loop on a copy to prevent notifiers to mixup + # the list by calling dontNotifyOnDisconnect + for notifier in self.disconnects[:]: try: notifier() except: --snip-snip-------------------------------------------------------- -- Best regards, Sylane mailto:twisted@sylane.com

1 0

[Twisted-Python] RE: How do I upload files using FTPClient?
by Daniel Burr Jan. 13, 2003

Jan. 13, 2003

On Tue, 14 Jan 2003 at 13:21:54 +1100, Andrew Bennetts wrote: >On Mon, Jan 13, 2003 at 05:14:26PM -0800, Daniel Burr wrote: >> That would be awesome; I'd be interested in contributing some code/tests/docs >> for the revised module. > >Now *that* would be awesome! :) Well, I think I grok the whole Deferred thing now, and I'm poring over the Twisted developer docs for information about contributing. The only hurdle I have to clear is getting up to speed with CVS. It looks like the best place to start is writing some tests for the proposed "high-level" ftp interface. That should help me to define it. >> I've been going over the existing python ftplib.FTP code, as well as >> twisted.protocols.ftp.FTPClient, and I'm left wondering one thing: Why don't >> either of these classes implement the client commands in the manner that your >> typical command-line FTP client does? I think FTPClient should have simple >> methods, like "binary", "get", "mget", "mkdir", et cetera. From the >> developer's viewpoint, you'd just use FTPClient in the same manner as you >> would an interactive ftp client session; the actual queueing of commands, >> creation of Deferreds, and callbacks could be hidden away a little more easily >> then, and make life easier for developers just looking to roll a simple FTP >> client application using Twisted. > >Indeed. FTPClient is already almost this simple >for simple FTP commands, ones that don't require >opening a data socket. This is stuff like CWD, >MKD,etc. The only unfriendly thing about it is >it currently doesn't offer much help with parsing >the response, beyond stating that it is a success >or a failure (by calling your Deferred's callback >or errback) -- but that's probably all you care >about. Yep, for those basic commands I think we are well covered. As for parsing the response, I noticed that the effbot has some ftp-parsing code, I'll take a look at that, and see if it contains any ideas. >There are a fair few of these I haven't wrapped >up yet, for instance "binary" (which is the "TYPE >I" command, and I issue that automatically on log >in anyway -- is there ever any time when you >*don't* want this?). Not that I can think of; the first thing I always do in an interactive FTP session is to issue the "binary" command. >The problem is "get", which is not that simple, >let alone stuff like mget that has no one-to-one >analogy with the raw protocol. I don't think >FTPClient should be restricted to thinking >everything is a file on disk -- you may want to >process the data incrementally in another way [I >certainly did :) ]. Of course, most of the time, >you probably want to just write it to a file. To be honest, I can't think of a single scenario in which I wouldn't be using FTP to put a file to disk or get a file from disk. Granted, I am not blessed with much in the way of imagination... ;) >Thus I think I may need two layers; the current >FTPClient is the low-level plumbing that you >don't interact with unless you need it's >generality, and then there's a higher-level >interface that provides things like mget. It's >entirely possible that I can fit this all into >one class after all, but I'm not sure yet. I think putting it all into twisted.protocols.ftp would suffice. All in one class sounds harder. >So in summary, I want to allow for nifty/insane >things like directly funnelling data from e.g. an >HTTP download into an FTP upload -- but I also >want a class that's straightforward to use in the >common case as well. This sounds fine to me; obviously I'm really only interested in the common case, but nifty/insane stuff is cool too. ;) >What I'd really like from you, if you don't mind, >is a sketch of what *you* think the API should >look like for the high-level interface. >Hopefully that'll help me design this in a way >that satisfies everyone :) > >-Andrew. I think the API for the high-level client should basically be identical to what you would use for an interactive FTP session. I'll do some thinking on this and post a rough draft by the weekend. L. Daniel Burr

1 0

[Twisted-Python] RE: How do I upload files using FTPClient?
by Daniel Burr Jan. 13, 2003

Jan. 13, 2003

On Wed, Jan 08, 2003 at 23:53:17 +1100 Andrew Bennetts wrote: >> Yes, that's my fault. I haven't gotten around >> to that yet -- I might have a stab at it >> tonight, it shouldn't actually be that hard. >> I'll try to make sure I get this done in time >> for the 1.0.2 release. That would be awesome; I'd be interested in contributing some code/tests/docs for the revised module. > > Just letting you know I haven't forgotten about > this, but solving this properly really involves > refactoring the mess in FTPClient.retrieve... > and if I do that properly, it'll make my planned > rewrite of the FTP server alot easier, as it > ought to be able to re-use the code. > Unfortunately this means it might take a few > more days before this happens. > > -Andrew. While you are still thinking about the solution, might I suggest something with regard to FTPClient ? I've been going over the existing python ftplib.FTP code, as well as twisted.protocols.ftp.FTPClient, and I'm left wondering one thing: Why don't either of these classes implement the client commands in the manner that your typical command-line FTP client does? I think FTPClient should have simple methods, like "binary", "get", "mget", "mkdir", et cetera. From the developer's viewpoint, you'd just use FTPClient in the same manner as you would an interactive ftp client session; the actual queueing of commands, creation of Deferreds, and callbacks could be hidden away a little more easily then, and make life easier for developers just looking to roll a simple FTP client application using Twisted. Thanks for listening, L. Daniel Burr

2 1

[Twisted-Python] twisted.python.plugin
by Pavel Pergamenshchik Jan. 13, 2003

Jan. 13, 2003

Hi. I extended t.py.p to use deferreds. The code is somewhat hideous and about three times as slow, but at least it's asynchronous. There's a much more elegant way to do it using generators. Hopefully, at some point in the future Python 2.2 will be the minimum supported version. diff is at http://people.cornell.edu/pages/pp64/tpypdiff

2 1

[Twisted-Python] In case anyone is bored... fun with tests!
by Andrew Bennetts Jan. 13, 2003

Jan. 13, 2003

There's a little bit of work to be done :) Try running: python -Werror bin/trial -p twisted.test (i.e. turn warnings into errors, thus making them register with trial... hmm, perhaps trial should somehow hook into the warnings framework to make it log these errors automagically?) Under Python 2.2, I currently see: FAILED (failures=4, errors=22) Python 2.3a1 adds extra fun, thanks to the "hex/oct constants > sys.maxint will return positive values in Python 2.4 and up" warning. Also, I discovered that if I hit ^S then ^Q during testReadLimit, then all subsequent tests fail, apparently due to the pendingTimedCalls never getting cleared? I'm a bit surprised by this one, but I did manage to reproduce it, so I guess it must be real ;) -Andrew.

2 1