On 10:41 pm, leyssw(a)ihug.co.nz wrote:
>Isn't the simplest option to place a decent reverse proxy between the
>webserver and our clients?
Possibly so! I gave this a half-hearted attempted a few weeks ago but
quickly gave up. If someone who is familiar with configuring such a
proxy would like to help out, perhaps we can get something useful up and
>For example, varnish will cache and proxy simple GETs, so simple views
>of the front page and docs (which I imagine is a large chunk of the
>traffic) will be offloaded from the webserver:
>Also, is the current installation (Db, apache + trac) all on one
>machine or are the DB and the webserver split already?
They're all on one machine. Changing this is an expensive proposition
with a hard to predict payoff, so it's rather low on the list of ideas
>These would be easy ways to throw more hardware at the problem.
>What is the current web-server configuration?
Can you be more specific about which configuration you're interested in?
>Can you share traffic stats for the top 20 pages, so we get an idea of
>where the load is?
Yep. Here are counts for the last few days. Rank, request count, URL.
1) 22058 /trac/
4) 8550 /trac/attachment/wiki/TSF/SponsorLogos/Appropriate-
6) 3075 /trac/wiki/Documentation
7) 2615 /trac/timeline?changeset=on&max=100&daysback=10&format=rss
8) 2448 /trac/wiki/Downloads
9) 1788 /trac/wiki/FrequentlyAskedQuestions
10) 1356 /trac/wiki
11) 1019 /trac/wiki/TwistedProject
12) 966 /trac/wiki/TracGuide
13) 828 /trac/wiki/TwistedWeb
14) 814 /trac/robots.txt
15) 781 /trac/search
16) 710 /trac/wiki/TwistedProjects
17) 662 /trac/wiki/WikiStart?format=txt
18) 605 /trac/wiki/TwistedAdvantage
19) 595 /trac/report/9
Looking at this, I'm reminded that we should try moving the sponsor
logos to somewhere else
Unfortunately one thing we don't know is the relative cost of serving
each of these URLs. The sponsor logos, while accounting for a
significant number requests, may actually be a very small contribution
to overall load.
Isn't the simplest option to place a decent reverse proxy between the
webserver and our clients?
For example, varnish will cache and proxy simple GETs, so simple views
of the front page and docs (which I imagine is a large chunk of the
traffic) will be offloaded from the webserver:
Also, is the current installation (Db, apache + trac) all on one machine
or are the DB and the webserver split already?
These would be easy ways to throw more hardware at the problem.
What is the current web-server configuration?
Can you share traffic stats for the top 20 pages, so we get an idea of
where the load is?
> From: "ssteinerX(a)gmail.com" <ssteinerx(a)gmail.com>
> Subject: Re: [Twisted-Python] Volunteer for twisted code Reviews and
> To: Twisted general discussion <twisted-python(a)twistedmatrix.com>
> Message-ID: <C38535FC-8267-4162-B67C-1F27822CBC8D(a)gmail.com>
> Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes
> On Nov 7, 2009, at 8:51 AM, David Lyon wrote:
>> Hi Jean-Paul,
>> On Sat, 07 Nov 2009 13:32:28 -0000, exarkun(a)twistedmatrix.com wrote:
>>> A server farm doesn't buy us anything, as trac can't be scaled up
>>> multiple hosts (or even multiple CPUs on a single host).
>> I'm surprised to hear that.
>> Usually replication of servers for read requests works well
>> with dns sharing.
>> Does Trac not run a database? that can go on another machine and
>> lighten the load.
>> Anyway, it's your call. Keep up the good work.
> I think the best option may just be to collect machine, database, and
> usage stats and ask the trac guys for help.
> Twisted-Python mailing list
> End of Twisted-Python Digest, Vol 68, Issue 12
I'm interested in using twisted to build a server for recording streaming
audio. I've spent some time googling and looking through the archives but
haven't been able to find much in the way of examples for this sort of
usage. To be clear, I'm not trying to stream media from the server to the
client, but want to stream the audio from (potentially multiple) clients to
the server, where it will be recorded and possibly later annotated for
The closest possible example of this sort of usage scenario that I was
able to find was shtoom, but I wasn't able to get very far with that, as the
source does not seem to be available, despite the site still being up
(perhaps I just couldn't find it ).
Does anyone know of any working examples of such usage that might be
easily adapted. I'm really only looking for the simplest possible base
example that would allow me to connect one or more clients over localhost
and stream the audio to the server.
On the other hand, perhaps it is better to just go with the bundled
SocketServer module to start with?
I would like to introduce my project for the
http://twistedmatrix.com/trac/wiki/ProjectsUsingTwisted page. We use
twisted as the central server that talks to many power measurement
sensors in the field.
Gridspy provides you with an interactive view of resource usage in your
building. It gives you hard data on your consumption patterns and helps
you to make informed decisions.
The Gridspy allows you to access and monitor your consumption patterns
in real-time using a standard web browser on your PC, laptop or mobile
phone. The data is presented in high resolution and updated each second
as you watch. The moment a light is turned on in your house, you can see
the change on your Gridspy dashboard from across the room or across the
Our homepage is here http://www.gridspy.co.nz/
a running demo is here http://your.gridspy.co.nz/powertech
And you can read more about the role of our twisted backend here:
... The nexus then collects those sensor samples from the nearby
Gridspies, bundles it up with its own data and uploads it to our central
servers. To collect the data we use a custom application written using
the excellent Twisted framework <http://twistedmatrix.com/>. Every
single reading gets instant attention: it is evaluated for real-time
events, queued to be stored in the database and finally forwarded to
watching dashboard users. Processing the data stream live on the server
opens up many exciting possible features that I look forward to
discussing in future blog posts. By the time that the sample reaches the
server, it is about 0.2 seconds old. ...
It has been a great framework to work with. My sincere thanks goes to
everyone who has ever contributed to this project. Job very well done!
I wanted to log in a file and keep the console also going with it.
I am using
whatever documentation I could find about this API, it suggested me to pass
setStdout=False(in order to keep stdout away from putting into log)
somewhere in the startlogging function.
Something like this I tried:
Doesnt help in solving the problem of getting both things:
log.msg("messageforlog") -> goes to log
print "message for console" -> goes to console
Could anyone give some ideas on this?
I need assistance figuring out how to create a push protocol that implements
To keep things simple, I'm trying to build a push-based echo server. It
(1) accept incoming tcp connections
(2) on each connection, all data rec'd should be echo'd unchanged.
(3) should the output transport request that transmission be "paused,"
the input transport (in this case, the same socket) should be similarly
This would seem to be a simple problem. I think my "PushEcho" class
should implement both IConsumer and IProducer interfaces; As a consumer,
its dataRecieved method would cause it to act like a producer and call
the transport's dataReceived method. Similarly, as a producer, it's
pause, and resume methods would be called, and they would (as a
consumer) just call the same functions in the transport.
However, I can't get the plumbing right.
Can someone help by providing pointers or some sample code?
Can anybody point me towards a nice up-to-date example of the use of
Twisted with wxPython?
I assume (hope) there's some way to bind the wx event loop into
Twisted's Reactor? Or do they need to run in separate
Thanks in advance,
I am trying to use the twisted reactor as a basis for a Python server for a
Flex app, using pyAMF. My problem is that one of the server methods
requires me to run a series of command-line tools to complete parsing data
from a file into an XML object, with lots of Python processing on the
results of one command-line tool to provide the input to the next one. I
have been using subprocess.call to invoke the command-line tools, but
these intermittently - and frequently - fail with an error "IOError: [Errno
4] Interrupted system call".
I do know that this was supposed to be fixed in twisted for Python 2.6, but
it doesn't seem to be. I have python 2.6.3 under Mac OSX 10.5.8, and twisted
8.2.0. Is there a work-around for this problem, or a better way to do this
with twisted? I do know that I can make a command-line call with
reactor.callLater, and that I can return a deferred from my server method,
but I don't see how to easily accomplish the chain of calls that I need,
short of breaking up my serverMethod1 into the several sequential parts, and
calling each from the result handler of the previous one. I'd rather not go
that route, as the logic of which command-line call gets made when is
actually quite complicated.
What I have now looks like (enormously simplified):
def serverMethod1(self, dummyPost, inputFilePath):
# Open inputFilePath, do stuff, save results to file2
subprocess.call(["command1", "file2", "file3"])
# Open file3, do stuff to the data, save results to file4
subprocess.call(["command2", "file4", "file5"])
# Open file5, convert data to a large string containing XML.
- Read Roberts
Tim Allen wrote:
> Paul Goins <general(a)vultaire.net> wrote:
>>> Does "trial tests" not work? (you probably need a tests/__init__.py)
>> $ trial tests
>> Nothing executes if I run it that way. And I do have the
>> tests/__init__.py. It's totally empty, in case it matters.
> Just to be sure, your test modules are named "tests/test_foo.py",
> "tests/test_bar.py" and so forth?
That appears to be it. I thought it was just the function names that
needed to follow the test_<foo> pattern.
I don't seem to see this documented, either... Guess I'll add a comment
Thanks for the help!
My project has 160 or so test functions spread across 29 test modules.
The layout is something like this:
Running tests one-by-one ("trial tests.<test_module>") works perfectly.
However, I want to test them all in one go, and can't find a "proper"
way to do it.
Currently to test it, I have a module, tests.unit_tests, which imports
all the tests from all other modules in the tests namespace via:
from tests.<test_module_1> import *
from tests.<test_module_2> import *
However, this is less than ideal, since if someone makes a unit test
class with the same name as one in another module, they overlap and one
ends up untested. (We just got blindsided by this to some degree, hence
I'll probably end up just attaching the test module's name as a prefix
to each test class, but I wanted to ask, is there a better, "proper" way
of doing this?
- Paul Goins