Mailman 3 November 2009 - Twisted

Re: [Twisted-Python] Twisted webserver performance issues (was volunteer...)
by exarkun＠twistedmatrix.com 08 Nov '09

08 Nov '09

On 10:41 pm, leyssw(a)ihug.co.nz wrote: > >Isn't the simplest option to place a decent reverse proxy between the >webserver and our clients? Possibly so! I gave this a half-hearted attempted a few weeks ago but quickly gave up. If someone who is familiar with configuring such a proxy would like to help out, perhaps we can get something useful up and running. > >For example, varnish will cache and proxy simple GETs, so simple views >of the front page and docs (which I imagine is a large chunk of the >traffic) will be offloaded from the webserver: >http://varnish.projects.linpro.no/wiki/VarnishFeatures > >Also, is the current installation (Db, apache + trac) all on one >machine or are the DB and the webserver split already? They're all on one machine. Changing this is an expensive proposition with a hard to predict payoff, so it's rather low on the list of ideas to try. >These would be easy ways to throw more hardware at the problem. > >What is the current web-server configuration? Can you be more specific about which configuration you're interested in? >Can you share traffic stats for the top 20 pages, so we get an idea of >where the load is? Yep. Here are counts for the last few days. Rank, request count, URL. 1) 22058 /trac/ 2) 8742 /trac/attachment/wiki/TSF/SponsorLogos/seomoves.png?format=raw 3) 8646 /trac/attachment/wiki/TSF/SponsorLogos/zenoss.2.png?format=raw 4) 8550 /trac/attachment/wiki/TSF/SponsorLogos/Appropriate- Solutions-Inc.gif?format=raw 5) 8499 /trac/attachment/wiki/TSF/SponsorLogos/securitymetrics.png?format=raw 6) 3075 /trac/wiki/Documentation 7) 2615 /trac/timeline?changeset=on&max=100&daysback=10&format=rss 8) 2448 /trac/wiki/Downloads 9) 1788 /trac/wiki/FrequentlyAskedQuestions 10) 1356 /trac/wiki 11) 1019 /trac/wiki/TwistedProject 12) 966 /trac/wiki/TracGuide 13) 828 /trac/wiki/TwistedWeb 14) 814 /trac/robots.txt 15) 781 /trac/search 16) 710 /trac/wiki/TwistedProjects 17) 662 /trac/wiki/WikiStart?format=txt 18) 605 /trac/wiki/TwistedAdvantage 19) 595 /trac/report/9 20) 590 /trac/timeline?milestone=on&ticket=on&changeset=on&wiki=on&max=50&daysback=90&format=rss Looking at this, I'm reminded that we should try moving the sponsor logos to somewhere else Unfortunately one thing we don't know is the relative cost of serving each of these URLs. The sponsor logos, while accounting for a significant number requests, may actually be a very small contribution to overall load. Anyhow, thanks! Jean-Paul

2 3

[Twisted-Python] Twisted webserver performance issues (was volunteer...)
by Tom Leys 07 Nov '09

07 Nov '09

Isn't the simplest option to place a decent reverse proxy between the webserver and our clients? For example, varnish will cache and proxy simple GETs, so simple views of the front page and docs (which I imagine is a large chunk of the traffic) will be offloaded from the webserver: http://varnish.projects.linpro.no/wiki/VarnishFeatures Also, is the current installation (Db, apache + trac) all on one machine or are the DB and the webserver split already? These would be easy ways to throw more hardware at the problem. What is the current web-server configuration? Can you share traffic stats for the top 20 pages, so we get an idea of where the load is? -Tom > From: "ssteinerX(a)gmail.com" <ssteinerx(a)gmail.com> > Subject: Re: [Twisted-Python] Volunteer for twisted code Reviews and > Commits > To: Twisted general discussion <twisted-python(a)twistedmatrix.com> > Message-ID: <C38535FC-8267-4162-B67C-1F27822CBC8D(a)gmail.com> > Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes > > > On Nov 7, 2009, at 8:51 AM, David Lyon wrote: > > >> Hi Jean-Paul, >> >> On Sat, 07 Nov 2009 13:32:28 -0000, exarkun(a)twistedmatrix.com wrote: >> >> >>> A server farm doesn't buy us anything, as trac can't be scaled up >>> across >>> multiple hosts (or even multiple CPUs on a single host). >>> >> I'm surprised to hear that. >> >> Usually replication of servers for read requests works well >> with dns sharing. >> >> Does Trac not run a database? that can go on another machine and >> lighten the load. >> >> Anyway, it's your call. Keep up the good work. >> > > I think the best option may just be to collect machine, database, and > usage stats and ask the trac guys for help. > > S > > > > > ------------------------------ > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python(a)twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > End of Twisted-Python Digest, Vol 68, Issue 12 > ********************************************** > >

1 0

[Twisted-Python] twisted server for recording streaming audio
by Josef Novak 06 Nov '09

06 Nov '09

Hi, I'm interested in using twisted to build a server for recording streaming audio. I've spent some time googling and looking through the archives but haven't been able to find much in the way of examples for this sort of usage. To be clear, I'm not trying to stream media from the server to the client, but want to stream the audio from (potentially multiple) clients to the server, where it will be recorded and possibly later annotated for reference. The closest possible example of this sort of usage scenario that I was able to find was shtoom, but I wasn't able to get very far with that, as the source does not seem to be available, despite the site still being up (perhaps I just couldn't find it ). Does anyone know of any working examples of such usage that might be easily adapted. I'm really only looking for the simplest possible base example that would allow me to connect one or more clients over localhost and stream the audio to the server. On the other hand, perhaps it is better to just go with the bundled SocketServer module to start with? Cheers

2 1

[Twisted-Python] Gridspy - a new project using Twisted
by Tom Leys 06 Nov '09

06 Nov '09

Hi Everyone I would like to introduce my project for the http://twistedmatrix.com/trac/wiki/ProjectsUsingTwisted page. We use twisted as the central server that talks to many power measurement sensors in the field. " Gridspy provides you with an interactive view of resource usage in your building. It gives you hard data on your consumption patterns and helps you to make informed decisions. ... The Gridspy allows you to access and monitor your consumption patterns in real-time using a standard web browser on your PC, laptop or mobile phone. The data is presented in high resolution and updated each second as you watch. The moment a light is turned on in your house, you can see the change on your Gridspy dashboard from across the room or across the planet. " Our homepage is here http://www.gridspy.co.nz/ a running demo is here http://your.gridspy.co.nz/powertech And you can read more about the role of our twisted backend here: http://blog.gridspy.co.nz/2009/10/realtime-data-from-sensors-to-browsers.ht… " ... The nexus then collects those sensor samples from the nearby Gridspies, bundles it up with its own data and uploads it to our central servers. To collect the data we use a custom application written using the excellent Twisted framework <http://twistedmatrix.com/>. Every single reading gets instant attention: it is evaluated for real-time events, queued to be stored in the database and finally forwarded to watching dashboard users. Processing the data stream live on the server opens up many exciting possible features that I look forward to discussing in future blog posts. By the time that the sample reaches the server, it is about 0.2 seconds old. ... " It has been a great framework to work with. My sincere thanks goes to everyone who has ever contributed to this project. Job very well done! -Tom

3 3

[Twisted-Python] Logging to both console and file (using twisted logging API)?
by naman jain 06 Nov '09

06 Nov '09

Hi, I wanted to log in a file and keep the console also going with it. I am using log.startLogging(open(logname, 'w')) whatever documentation I could find about this API, it suggested me to pass setStdout=False(in order to keep stdout away from putting into log) somewhere in the startlogging function. Something like this I tried: log.startLogging(sys.stdout,setStdout=False) Doesnt help in solving the problem of getting both things: Requirement: log.msg("messageforlog") -> goes to log print "message for console" -> goes to console Could anyone give some ideas on this? Regards Naman

3 7

[Twisted-Python] help needed with flow-control for push protocol
by Pietro Niccoli 05 Nov '09

05 Nov '09

Hi, I need assistance figuring out how to create a push protocol that implements flow control. To keep things simple, I'm trying to build a push-based echo server. It should (1) accept incoming tcp connections (2) on each connection, all data rec'd should be echo'd unchanged. (3) should the output transport request that transmission be "paused," the input transport (in this case, the same socket) should be similarly blocked. This would seem to be a simple problem. I think my "PushEcho" class should implement both IConsumer and IProducer interfaces; As a consumer, its dataRecieved method would cause it to act like a producer and call the transport's dataReceived method. Similarly, as a producer, it's pause, and resume methods would be called, and they would (as a consumer) just call the same functions in the transport. However, I can't get the plumbing right. Can someone help by providing pointers or some sample code? Thanks.

2 2

[Twisted-Python] Twisted and wxPython
by Jon Mills 05 Nov '09

05 Nov '09

Can anybody point me towards a nice up-to-date example of the use of Twisted with wxPython? I assume (hope) there's some way to bind the wx event loop into Twisted's Reactor? Or do they need to run in separate threads/processes? Thanks in advance, Jon Mills

3 3

[Twisted-Python] Need replacement for subprocess.call within a pyAMF method - any suggestions?
by Read Roberts 05 Nov '09

05 Nov '09

I am trying to use the twisted reactor as a basis for a Python server for a Flex app, using pyAMF. My problem is that one of the server methods requires me to run a series of command-line tools to complete parsing data from a file into an XML object, with lots of Python processing on the results of one command-line tool to provide the input to the next one. I have been using subprocess.call to invoke the command-line tools, but these intermittently - and frequently - fail with an error "IOError: [Errno 4] Interrupted system call". I do know that this was supposed to be fixed in twisted for Python 2.6, but it doesn't seem to be. I have python 2.6.3 under Mac OSX 10.5.8, and twisted 8.2.0. Is there a work-around for this problem, or a better way to do this with twisted? I do know that I can make a command-line call with reactor.callLater, and that I can return a deferred from my server method, but I don't see how to easily accomplish the chain of calls that I need, short of breaking up my serverMethod1 into the several sequential parts, and calling each from the result handler of the previous one. I'd rather not go that route, as the logic of which command-line call gets made when is actually quite complicated. What I have now looks like (enormously simplified): class MyPythonServer: def serverMethod1(self, dummyPost, inputFilePath): # Open inputFilePath, do stuff, save results to file2 subprocess.call(["command1", "file2", "file3"]) # Open file3, do stuff to the data, save results to file4 subprocess.call(["command2", "file4", "file5"]) # Open file5, convert data to a large string containing XML. return xmlString - Read Roberts

2 2

Re: [Twisted-Python] Trial: testing multiple test modules in one go?
by Paul Goins 05 Nov '09

05 Nov '09

Tim Allen wrote: > Paul Goins <general(a)vultaire.net> wrote: >>> Does "trial tests" not work? (you probably need a tests/__init__.py) >> Nope: >> >> $ trial tests >> >> >> ------------------------------------------------------------------------------- >> >> PASSED >> >> Nothing executes if I run it that way. And I do have the >> tests/__init__.py. It's totally empty, in case it matters. > > Just to be sure, your test modules are named "tests/test_foo.py", > "tests/test_bar.py" and so forth? That appears to be it. I thought it was just the function names that needed to follow the test_<foo> pattern. I don't seem to see this documented, either... Guess I'll add a comment to #2443. Thanks for the help!

1 0

[Twisted-Python] Trial: testing multiple test modules in one go?
by Paul Goins 05 Nov '09

05 Nov '09

My project has 160 or so test functions spread across 29 test modules. The layout is something like this: project_root/ <project>/ <module>.py <module>.py ... tests/ <test_module>.py <test_module>.py ... unit_tests.py Running tests one-by-one ("trial tests.<test_module>") works perfectly. However, I want to test them all in one go, and can't find a "proper" way to do it. Currently to test it, I have a module, tests.unit_tests, which imports all the tests from all other modules in the tests namespace via: from tests.<test_module_1> import * from tests.<test_module_2> import * ... However, this is less than ideal, since if someone makes a unit test class with the same name as one in another module, they overlap and one ends up untested. (We just got blindsided by this to some degree, hence the e-mail.) I'll probably end up just attaching the test module's name as a prefix to each test class, but I wanted to ask, is there a better, "proper" way of doing this? Thanks. - Paul Goins

2 1