Mailman 3 July 2003 - Twisted

[Twisted-Python] Does anybody have a working project ...
by Thomas Weholt July 4, 2003

July 4, 2003

I could take a peek at? Hopefully, something simple. I'm currently trying to get a grip on how to build a site using Twisted methods and use Issues Alpha 1 as a foundation for that. But there are too many things going on and it's hard to get a hang of why things are done they way they are. What I need is something that : 1. don't use rpys, but Site and Application 2. starts a service which holds some persistent data and does a scheduled call to a long-running, blocking method once a while 3. … [View More]

7 9

[Twisted-Python] minor beginner questions
by Achim Domma (ProCoders) July 4, 2003

July 4, 2003

Hi, playing around with twisted.web, I have some minor questions: - In another message a function 'deferToThread' was mentioned. I tought that somethink like must be available in twisted but could not find it. Is there an example on how to use it? - Is it possible to create resources in my startup script, where I create my application. Instead of having a simple *.rpy with resouce=SomeClassInstance() I would like to use addChild on the root of my app. Is that possible? - If I want to keep a … [View More]

2 1

[Twisted-Python] Extending t.w.spider
by janko＠mind-nest.com July 4, 2003

July 4, 2003

Bok, I am sending this as suggested by "forgot who" on #twisted. I extended t.w.spider , this isn't finished. It seems to work however. You can download, run it and look it crawl ower certain domain ... It prints out quite few information doing work so you know what is going on ( mainly for testing purposes ). I am hoping just for a quick scan thru code and run by someone more experienced in this. So I will know if I did it python/twisted/sane way, before I go further and conect it to … [View More]various stuff. ( also read NOTE at the end of email ) You can get this at http://www.mind-nest.com/downloads/walker.tgz or http://www.mind-nest.com/downloads/walker.zip When finished I plan give it back to twisted... of course if they will accept it. note for the maillist moderator: I sent this mail when I was not yet a member some week ago, I got response that you must first approve it. As it didn't happen in 1 week and I am a member now I sent it again as a member. lp :janko janko(a)mind-nest.com www.mind-nest.com *** This module extends twisted.web.spider.SpiderSender class into WalkerSender (and also extends htmllib.HTMLParser and t.w.c.HTTPClientFactory so WalkerSender can use them): LinkParser -collects links on a page -also collects frame scr-es, to crawl ower frames -can also collect images, links(css,js..) for link/img validating purposes HTTPCollector -Doesnt store content of page to file but to variable -Can be easily set to diferent link-parsers or page-downloaders(*).. -Returns *self* to the callback, so that links, content, or anything else it collects can be retrieved by the callback method. (**) WalkerSender -Can be easily set to diferent http-collectors/downloaders(***).. -Uses dictionary instead of list for queue now.. explaind below -Has 4 more plugins/events, some existent are cahnged to be more powerfull -Plugin to filter links to whatever you wanth (extensions, domains...) -Plugin to fill with some algorithm to prevent from looping -Plugin to notify that download failed -Plugin to tell that all links found while crawling were crawled and there is nowwhere else to go (Likely/hopefully to occur if doing One Site/domain crawler as I was, and when timeouting or some other shit happens) -Plugin notifyDownloadEnd has aditional argument downloader which holds anything you prepare in dowloader class(****) some smaller things were made to get it working -preventing from starting downloader on page/url that is already downloading -queue (is now dictionary) so it can't have multiple same pages in it (the depth of the first ocurrence is stored) -removes fragments from urls (www.a.org/index.html#fragment) so we get multiple -same- pages that are then filtered -it doesnt remove ?queries as they often mean new content.. I made OneSiteWalkerSender as an example.. I intend to make one site search engine (with pyndex probably) and a anchor/link/img... validating script. OneSiteWalkerSender has just now crawled ower 525 pages of www.google.com. I also tested it with other sites. NOTE: Don't shoot me or something if I made something very stupid, I am very new to Python and Twisted and don't understand many important issues on any of them. Where I marked with * in upper description I am a little suspicious with my way of doing it. *** [View Less]

1 0

[Twisted-Python] ldaptor feedback
by Clark C. Evans July 4, 2003

July 4, 2003

Tommi, Howdy. Thank you for your initial help with ldaptor; it looks like I'll be spending alot of time on LDAP in the next few months. 1. I would like to see ldap in twisted proper, it is really a core protocol (like imap, etc.). From what I heared there are two arguments against, size and licensing. As for size, I think that LDAP is so important that it really belongs in twisted. It is at the same level as IMAP. I do not want to have to go to an external module, … [View More]hence LDAP is in-scope so the size argument does not matter to me. As for licensing, I understand your concerns. However, it is very important for enforcement of copyright and for the ability to 'fix' the license that a single entity have the ability to license the and enforce the work as needed. Long term I'd like to see Glyph form some sort of non-profit to hold the copyrights, or at the very least present a 'will' of sorts describing what happens to the copyrights in the event that he is incapacitated or gets hit by a MAC truck. For now, I'm not too worried about this. As for Glyph selling the software and making a profit... well, as a bunch of individual packages this just is not an option at all, so better someone than no one. But seriously, I trust that Glyph will spread the wealth to those who have really contributed to Twisted, perhaps by simply paying for Flights and Hotels for Python/Twisted conference if he ever makes bank with the copyright. In reality though, I think that this is a moot point; the most important point is the ability to fix the copyright and pursue violators... without copyright powers this is too hard. So, Glyph approving, would you consider contributing ldaptor to twisted? 2. Your package hierarchy needs help... lots of help. Let's start with this example... from ldaptor.protocols.ldap.ldapsyntax import LDAPJournalOperation_Replace Arg! First, ldaptor has one protocol... ldap. So "protocols.ldap" is quite redundant. Second, ldapsyntax is describing a ldap proxy object with the ability to journal its changes and save. It has nothing to do with "syntax" let alone "ldapsyntax", how about 'object' or 'proxy'?. Third, I count 3 occurances of LDAP... Uncle! Fourth, the class name is *huge* and using prefixes this way sucks, JournalOperations should be in a file called 'journal'. In short, what I'd want to see... from ldaptor import ldap op = ldap.journal.Replace() The 'low-level' pureber and pureldap are really not protocol items so much as they are parsers/emitters. You could stuff these, as well as the filter, ldif, and schema all into another part of ldaptor, ldaptor.parse.ldif, etc. Anyway, for ldaptor to be understandable it really needs a refactor like this. Just look at your commands as to how ugly the packaging is. Good packaging and naming takes quite a bit of effort and thought. 3. I can help convert your public interfaces to use Deferreds, and I can rewrite your examples to use flow (so that they are more readable). In particular, I was playing with a flow version of your ldap client and it makes things quite clean; to use the client you have a generator which yield ldap commands... If I'm to help, I'll need commit access and you can't be completely opposed to me restructuring/commenting things as I move along. If you need an example of my work, see flow.py Anyway, I'm curious how open you are to these items. Adding this to Twisted (with alot of work) is probably a good bargin; you get more eyes helping your project along, documentation, etc. Best, Clark [View Less]

6 15

[Twisted-Python] Can't get ssh-connections to work
by Anders Bruun Olsen July 3, 2003

July 3, 2003

Hi, I am brand-spanking new at twisted but wanted to make an app that connects through ssh to a remote machine and fetches some info, which I thought twisted to be perfect for. I am trying to get the ssh example to work but it just hangs! I have entered it exactly as it is on the webpage (cut-and-paste and the fixed the indentions). The only thing changed is the host to connect to, there I have chosen a server I have running on my LAN. I have also added a call to log.startLogging(sys.stdout, … [View More]

2 2

[Twisted-Python] Re: IMAP fixes
by Jp Calderone July 3, 2003

July 3, 2003

On Wed, Jul 02, 2003 at 11:14:22PM +0200, Anders Hammarquist wrote: > Hi! > > Here are the changes for handling literals I talked about earlier. > So far I've only looked at the server part, but it seems to work > nicely. There's still work to do on improving the argument parser > functions, as some arguments can probably be parsed into a nicer > format than used now. Great stuff, I really really like your solution to the parsing problem. The patch is now applied, with … [View More]

2 2

[Twisted-Python] cgi on win32 fails, process only available in this reactor on POSIX
by eltronic＠juno.com July 3, 2003

July 3, 2003

am I missing something to get cgi working? running a web.tas created from a dos window, when I navigate to http://localhost/moin.cgi **edited [HTTPChannel,0,127.0.0.1] Traceback File "...\twisted\web\twcgi.py", line 130, in runProcess reactor.spawnProcess(p, self.filename, [self.filename]+qargs, env, os.path.dirname(self.filename)) File "...\twisted\internet\default.py", line 168, in spawnProcess raise NotImplementedError, "process only available in this reactor on POSIX" judging from … [View More]

1 0

[Twisted-Python] [patch] win32serialport win98 support
by Chris Liechti July 3, 2003

July 3, 2003

i'd call it a "crappy OS support patch" ;-) there are a few flaws in win98, e.g. reading zero bytes blocks forever and such. this patch provides workarounds so that the serial port can be used on both, Win98/ME and the NT/2k/XP series. i tested on win98, nt4, 2k and xp. chris Index: twisted/internet/serialport/win32serialport.py =================================================================== RCS file: /cvs/Twisted/twisted/internet/serialport/win32serialport.py,v retrieving revision 1.2 … [View More]diff -u -r1.2 win32serialport.py --- twisted/internet/serialport/win32serialport.py 5 Apr 2003 00:50:37 -0000 1.2 +++ twisted/internet/serialport/win32serialport.py 3 Jul 2003 00:42:59 -0000 @@ -64,7 +64,7 @@ self.protocol = protocol self.protocol.makeConnection(self) self._overlappedRead = win32file.OVERLAPPED() - self._overlappedRead.hEvent = win32event.CreateEvent(None, 0, 0, None) + self._overlappedRead.hEvent = win32event.CreateEvent(None, 1, 0, None) self._overlappedWrite = win32file.OVERLAPPED() self._overlappedWrite.hEvent = win32event.CreateEvent(None, 0, 0, None) @@ -83,24 +83,31 @@ first = str(self.read_buf[:n]) #now we should get everything that is already in the buffer flags, comstat = win32file.ClearCommError(self._serial.hComPort) - rc, buf = win32file.ReadFile(self._serial.hComPort, - win32file.AllocateReadBuffer(comstat.cbInQue), - self._overlappedRead) - n = win32file.GetOverlappedResult(self._serial.hComPort, self._overlappedRead, 1) - #handle all the received data: - self.protocol.dataReceived(first + str(buf[:n])) + if comstat.cbInQue: + win32event.ResetEvent(self._overlappedRead.hEvent) + rc, buf = win32file.ReadFile(self._serial.hComPort, + win32file.AllocateReadBuffer(comstat.cbInQue), + self._overlappedRead) + n = win32file.GetOverlappedResult(self._serial.hComPort, self._overlappedRead, 1) + #handle all the received data: + self.protocol.dataReceived(first + str(buf[:n])) + else: + #handle all the received data: + self.protocol.dataReceived(first) #set up next one + win32event.ResetEvent(self._overlappedRead.hEvent) rc, self.read_buf = win32file.ReadFile(self._serial.hComPort, win32file.AllocateReadBuffer(1), self._overlappedRead) def write(self, data): - if self.writeInProgress: - self.outQueue.append(data) - else: - self.writeInProgress = 1 - win32file.WriteFile(self._serial.hComPort, data, self._overlappedWrite) + if data: + if self.writeInProgress: + self.outQueue.append(data) + else: + self.writeInProgress = 1 + win32file.WriteFile(self._serial.hComPort, data, self._overlappedWrite) def serialWriteEvent(self): try: [View Less]

1 0

[Twisted-Python] twisted.trial.unittest.TestCase now now has an mktemp() method
by Pavel Pergamenshchik July 3, 2003

July 3, 2003

Hi. Please don't use tempfile.mktemp() in tests -- use "self.mktemp" instead. It gives names from nice, organized directory structure under _trial_temp/, which is much better than putting stuff with cryptic names all over /tmp. I modified all tests that use mktemp. Yell at me if something suddenly breaks. Pahan

1 0

[Twisted-Python] logfile.LogFile
by Richard.Townsend＠edl.uk.eds.com July 2, 2003

July 2, 2003

I was trying to set the defaultMode arg in LogFile.__init__() but it didn't seem to have any effect on the generated log file's permissions. In BaseLogFile.__init__() - if defaultMode is not None, it sets self.defaultMode to None. Should it not set it to defaultMode ?

1 0