Mailman 3 March 2005 - Twisted

[Twisted-Python] Twisted based LDAP protocol implemenation
by Andrzej Leszczynski 15 Mar '05

15 Mar '05

I am looking for a Twisted based LDAP protocol implementation (both versions 2 and 3). Is anybody aware of such, even in very early development phase. If there are neither any activity in that area nor actual plans, would Twisted be interested in adopting one into its code base? What would be level of interest of Twisted community? I am interested in both client and server side. Please advice, AndyL

3 2

RE: [Twisted-Python] Maximum sockets open
by Sinang, Danny 15 Mar '05

15 Mar '05

Thanks Ken. How do we determine (from the O.S.) how many sockets are currently open ? Regards, Danny -----Original Message----- From: twisted-python-bounces(a)twistedmatrix.com [mailto:twisted-python-bounces@twistedmatrix.com] On Behalf Of Ken Kinder Sent: Tuesday, March 15, 2005 12:47 AM To: Twisted general discussion Subject: Re: [Twisted-Python] Maximum sockets open A while ago, we were opening up a ton of sockets and got a "Too many files open" error. I don't know what you're getting, but if that's it, you might try (1) opening fewer sockets, (2) upping operating system limit (Twisted doesn't control that). According to our admin, on Linux, that would be /proc/sys/fs/file-max and use "sysctl -w 'fs.file-max=<new limit>'" to up the limit. -Ken Sinang, Danny wrote: > Hello, > > Is there a maximum number of sockets that Twisted supports ? If so, > what is the maximum, how do I see what it's current value, and how to > I set it to another value ? > > Regards, > Danny > > > >----------------------------------------------------------------------- >- > >_______________________________________________ >Twisted-Python mailing list >Twisted-Python(a)twistedmatrix.com >http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > _______________________________________________ Twisted-Python mailing list Twisted-Python(a)twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

2 1

[Twisted-Python] More on PB Copyable Errors
by Justin Johnson 15 Mar '05

15 Mar '05

I would really like to be able to pass custom exceptions over the wire in PB. Previous discussion on this list ended with the conclusion that it currently isn't possible without hacking a local copy of Twisted. I have done that to get the result I want, with minimal testing, and would like to get some discussion going around this topic. Attached are 3 test files, s.py for the server, c.py for the client, and e.py for the error definitions which are imported by both s.py and c.py. If I make the following modification to my local copy of twisted/spread/pb.py in the CopyableFailure's getStateToCopy method, the custom error is passed back as I want. #state['value'] = str(self.value) # Exception instance state['value'] = self.value # Exception instance I was wondering if there is a reason we wouldn't want to change the code to check for registered unjellyables for the exception, and return them (or rather go through the jelly/unjelly process) if they exist, but return the string representation otherwise. Maybe that is more complicated than I realize. Any thoughts or suggestions on how we might make this possible? Thanks. -Justin

5 21

[Twisted-Python] Weekly Bug Summary
by James Y Knight 14 Mar '05

14 Mar '05

Bug summary ______________________ Since 2005-03-05 04:13:21.060000: Bugs opened: 9 Bugs closed: 17 Total open bugs: 326 (-8) Urgent: 9 (+2) Bug: 118 (-13) Feature: 131 (+2) Wish: 68 (+1) New / Reopened Bugs ______________________ ==== Bug ==== twisted.python.rebuild fails if python 2.4's decimal module is imported (1 1/4 hours ago) http://twistedmatrix.com/bugs/issue921 opened by marienz e-mail followup rejected (2 days ago) CLOSED http://twistedmatrix.com/bugs/issue920 opened by zooko roundup exception (2 days ago) CLOSED http://twistedmatrix.com/bugs/issue919 opened by zooko [CORE] If program has > 1024 fds, the select() used in t.i.process can fail (3 days ago) http://twistedmatrix.com/bugs/issue918 opened by itamarst [DOCUMENTATION] Not all the examples in the twisted.application.strports docstring are valid (3 days ago) http://twistedmatrix.com/bugs/issue917 opened by exarkun xmlrpc.QueryFactory failed to catch some exceptions (5 days ago) http://twistedmatrix.com/bugs/issue914 opened by maoy ==== Feature ==== .buildProtocol((host, port)) is unsuitable for e.g. UNIX domain sockets; use t.i.address (4 days ago) http://twistedmatrix.com/bugs/issue915 opened by Tv [CONCH] signal to conch to reconnect (4 days ago) http://twistedmatrix.com/bugs/issue916 opened by z3p ==== Wish ==== Add test cases for twisted.words.protocols.jabber.client (12 minutes ago) http://twistedmatrix.com/bugs/issue922 opened by ralphm Bugs Closed ______________________ ==== Urgent ==== [PATCH] fix twisted.words.protocols.jabber.jid to add stringprep (1 week ago) http://twistedmatrix.com/bugs/issue912 opened by ralphm, closed by ralphm plugins.tml doesn't work with split up packages (2 weeks ago) http://twistedmatrix.com/bugs/issue906 opened by exarkun, closed by exarkun [WIN32] problems with Twisted-2.0.0a1 on Windows (1 month ago) http://twistedmatrix.com/bugs/issue891 opened by manlio, closed by radix setgroups() raises when given too many groups, and nothing handles the error (11 months ago) http://twistedmatrix.com/bugs/issue600 opened by exarkun, closed by jknight [TRIAL] t.trial.util.wait deprecates, but doesn't replace (3 months ago) http://twistedmatrix.com/bugs/issue795 opened by teratorn, closed by exarkun [RELEASE MANAGEMENT] Change setup.cfg so bdist_rpm uses admin/twisted.spec (18 months ago) http://twistedmatrix.com/bugs/issue260 opened by wkv, closed by radix ==== Bug ==== [CHAT] [PATCH] Digest jabber authentication never tried (14 months ago) http://twistedmatrix.com/bugs/issue448 opened by ralphm, closed by ralphm e-mail followup rejected (2 days ago) http://twistedmatrix.com/bugs/issue920 opened by zooko, closed by radix roundup exception (2 days ago) http://twistedmatrix.com/bugs/issue919 opened by zooko, closed by radix twisted.cred.portal.IRealm docstring for parameters (5 months ago) http://twistedmatrix.com/bugs/issue740 opened by thomasvs, closed by radix adbapi prints value of connection password to log file (1 month ago) http://twistedmatrix.com/bugs/issue871 opened by itamarst, closed by radix t.test.test_application.TestConvert fails without t.web (1 month ago) http://twistedmatrix.com/bugs/issue877 opened by marienz, closed by radix [DOCUMENTATION] Make a separate tutorial index page (1 month ago) http://twistedmatrix.com/bugs/issue889 opened by hypatia, closed by radix t.names.client.ThreadedResolver should be in core? (3 months ago) http://twistedmatrix.com/bugs/issue790 opened by jknight, closed by exarkun [CORE] [PATCH] t.p.usage uses passed arguments rather than real defaults when displaying help (5 months ago) http://twistedmatrix.com/bugs/issue743 opened by itamarst, closed by jml [CONCH] _scanDirectory broken (2 weeks ago) http://twistedmatrix.com/bugs/issue904 opened by spiv, closed by z3p [CONCH] No ProtocolWrapper defined in twisted/conch/ssh/session.py (2 weeks ago) http://twistedmatrix.com/bugs/issue905 opened by spiv, closed by z3p

1 0

[Twisted-Python] Re: Twisted and the Posh Module
by Ed Suominen 14 Mar '05

14 Mar '05

On Monday 14 March 2005 10:13 am, twisted-python-request(a)twistedmatrix.com wrote: > Has anyone tried playing with Twisted and the posh module? I need to > do some CPU-intensive stuff inside the reactor, which unfortunately > holds on to the GIL. I was thinking of just making a deferToPosh() > type of method in the spirit of deferToThread(), but forking on each > operation would be pretty expensive. Which brings up the question of > having process pools. > > Has anyone done anything like this? I don't really need any > interaction between the worker processes and Twisted, so in theory I > could spawn off some worker processes early and just communicate > queues of tasks to complete, independent of Twisted and the reactor. I think having some sort of process pool mangagement in Twisted is a great idea, especially with multi-core CPUs emerging on the scene. I have access to a dual-core Pentium Prescott CPU and it would be great to have that available to keep both cores humming on a certain CPU-intensive project I'm considering. However, I'm not sure the best way to go about it would be with the posh module, pypar (see http://datamining.anu.edu.au/~ole/pypar/), or just using Perspective Broker as an underlying message-passing mechanism with UNIX sockets and/or TCP. One thought might be to have a single master process start up and act as a PB server and process pool manager. Subsidiary processes could then make authenticated PB connections to the server to "volunteer" for work in the process pool. Note that pypar lets you easily find out how many CPUs you have under kernel control, with pypar.size(). Thus, the main process could start the process pool by spawning a subsidiary "volunteer" process for each CPU core present. --- Ed Suominen Registered Patent Agent Open-Source Software Author (yes, both...) Web Site: http://www.eepatents.com

4 4

[Twisted-Python] twisted.web.server and memory
by Lenny G Arbage 14 Mar '05

14 Mar '05

A couple of questions about twisted.web server's memory management: 1- Is it normal for the server process to consume large amounts of memory (and never free it) if large requests are sent? 2- Assuming the answer to #1 is 'no,' what might I be doing wrong? I have subclassed Resource and added it to the Site tree with putChild. My Resource subclass processes mutlipart/form-data style file uploads. It looks to me like the Request object that is passed in contains /all/ the data in the request. So if the client sends a 100MB file, server consumes 100MB of memory. What I'm seeing is that if I send small file uploads, server never consumes more than about 30MB (7MB resident). If I send a single ~100MB file, usage mushrooms to over 200MB (~140MB resident), and the memory is never returned to the system. 3- Assuming the answer to #1 is 'yes,' what is the correct way to architect a solution. Do I need to subclass web.resource.Resource or web.site.Request to do my own management of memory (doing partial reads of the incoming data)? Or is there some other pattern that I should be using rather than .web.server? Note that I've run gc with DEBUG_LEAK trying to spot any problems, and it doesn't report anything. If a code snippet that reproduces the above would help, I'd be happy to produce one. Thanks -- Lenny G. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1

Re: [Twisted-Python] twisted.web.server and memory
by Lenny G Arbage 14 Mar '05

14 Mar '05

Run the following web server (memoryhogserver.py) and load the provided form (post.html) into a browser of your choice and choose a large file (~100MB) to submit. Notice the memory usage of the python process (on my machine, around 150MB resident). memoryhogserver.py: from twisted.web.resource import Resource from twisted.web import server, resource from twisted.internet import reactor import os class ROOT(Resource): def __init__(self): Resource.__init__(self) def getChild(self, name, request): if name == "": return self return Resource.getChild(self, name, request) def render_GET(self, request): return "<html>root dir</hrml>" class UPLOAD(ROOT): def render_POST(self, request): f = open("/tmp/testfile","wb") f.write(request.args.get('filename')[0]) f.close() return "<html>upload successful</html>" def render_GET(self, request): return "'upload' must use POST" if __name__ == '__main__': root = ROOT() root.putChild('upload',UPLOAD()) site = server.Site(root) reactor.listenTCP(7000, site) print "webserver listening on port 7000" reactor.run() post.html: <html> <form method="POST" enctype="multipart/form-data" action="http://localhost:7000/upload"> <input type="file" name="filename"> <input type="submit"> </form> </html> (all in the spirit of replying to my own posts :) -- Lenny G. --- Lenny G Arbage <alengarbage(a)yahoo.com> wrote: > > A couple of questions about twisted.web server's > memory management: ... > If a code snippet that reproduces the above would > help, I'd be happy to produce one. __________________________________ Do you Yahoo!? Yahoo! Small Business - Try our new resources site! http://smallbusiness.yahoo.com/resources/

1 0

[Twisted-Python] sending queries and responses with the new dns module
by jarrod roberson 14 Mar '05

14 Mar '05

can someone give me a quick code snippet on how to say construct and send a simple query and construct and send a simple response using the new dns module? I have a nice mdns listener written and I can see all the traffic flying around but I have not had any luck in sending queries or responses . . .

1 1

[Twisted-Python] Twisted and the Posh Module
by Ken Kinder 14 Mar '05

14 Mar '05

Has anyone tried playing with Twisted and the posh module? I need to do some CPU-intensive stuff inside the reactor, which unfortunately holds on to the GIL. I was thinking of just making a deferToPosh() type of method in the spirit of deferToThread(), but forking on each operation would be pretty expensive. Which brings up the question of having process pools. Has anyone done anything like this? I don't really need any interaction between the worker processes and Twisted, so in theory I could spawn off some worker processes early and just communicate queues of tasks to complete, independent of Twisted and the reactor. -Ken

1 0

[Twisted-Python] Python Xlib + Twisted
by Ed Suominen 14 Mar '05

14 Mar '05

Below is what I sent to python-xlib-users this morning. Hopefully there's some interest in getting a Twisted-driven version of Python Xlib done. I think I've learned the innards of Twisted well enough by now to do it myself (writing a new protocol for X events), but help and advice is of course welcome. --- Ed Suominen Registered Patent Agent Open-Source Software Author (yes, both...) Web Site: http://www.eepatents.com ----- On Monday 14 March 2005 08:08 am, Peter Liljenberg wrote: > Simon Forman, calroc99(a)users.sourceforge.net, has kindly offered to > maintain Python Xlib from now on. Give him a big hand, folks. > > I'll resign as project admin, but will still lurk around and may even > do the occasional CVS commit (here's hoping). > > /Peter Liljenberg Python Xlib is a neat project, and I'm glad to see the continued activity. One of the projects I've been planning for the near future is to add some functionality to the connection code of Python Xlib so that it can operate asynchronously using the Twisted networking framework rather than relying on blocking and threads. I'm planning to integrate the patched Python Xlib into WinDictator, a Windows-Linux bridge I wrote and use for running dictation software in VMware and having the keystrokes typed with X events. Briefly, the Windows copy of WinDictator is a TCP server to a Linux client and presents a dictation window on the Windows machine. Using the dictation software's "direct dictation" feature, the user generates text in the dictation window, which is sent immediately to the Linux client. The Linux client then generates an X "KeyPress" event (followed by an X "KeyRelease" event) corresponding to the character sent from the Windows client. It is assumed that the (virtual) Windows machine is insecure and will be firewalled off from the Internet anyhow, so it is the TCP server. Plus, multiple Linux clients can connect to and use a single Windows server at different times. Advantageously, the client-server pair allows "direct dictation" from a text entry window running on Windows (along with the Windows-only dictation software) to whatever word processor the user is employing under Linux. The text entry window can be viewed on a separate screen of a separate Windows box, or can appear somewhere on the Linux computer's display. The latter can be accomplished by emulation with VMware or (I think) QEMU, or by VNC operation of the separate Windows box. Twisted's "Perspective Broker" maintains the client-server communication very nicely at a high level. WinDicator works great with the xte program "typing" the keystrokes, and I use it every day. But I want to improve it and need the finer-grained control of Python Xlib to do so. Ideally, I would like to see Python Xlib's networking integrated smoothly into Twisted's asynchronous event loop for performance, compatibility, and because it is just plain cool. See http://twistedmatrix.com/services/twisted-advantage for more info about Twisted, if you're interested. It runs my web site plus my boss's site (http://valuablepatents.com), among other things, off of a single main python thread. Offers of cooperation, either to add Twisted capability to Python Xlib or with WinDictator (GPL) itself, would of course be welcomed.

1 0