[Twisted-Python] Volunteer for twisted code Reviews and Commits
Hi All, Glyph mentioned on python-dev that he was after assistance with code reviews. I would like to offer some assistance. I've been developing python for some time and I have a knowledge of prior C++ async patterns such as ACE. More currently I have a reasonable knowledge of protocols such as jabber and so forth. If there is any way that I might be of assistance, please don't hesitate to ask. David
On 10:10 am, david.lyon@preisshare.net wrote:
Hi All,
Glyph mentioned on python-dev that he was after assistance with code reviews.
I would like to offer some assistance. I've been developing python for some time and I have a knowledge of prior C++ async patterns such as ACE. More currently I have a reasonable knowledge of protocols such as jabber and so forth.
If there is any way that I might be of assistance, please don't hesitate to ask.
Hi David, Thanks for the offer! Twisted would absolutely benefit from more assistance. :) There are quite a few things that you could do which would help out immensely: - Ticket reviews, as Glyph mentioned. Until you get more familiar with various parts of Twisted, you probably want to avoid giving a proposed change (ie a branch or a patch) a thumbs up - but it's rare that the first time a change is submitted for review it's ready for a thumbs up anyway, so reviews you can do of tickets to point out what additional work is necessary would be super useful. This lets the contributor get on with doing useful work (instead of waiting and waiting for a review) and lets people with more Twisted experience focus on things that their experience is actually necessary for. - Development, either on tickets with the "easy" keyword (at least initially), or, even better, on tickets which have already been reviewed one or more times and still require some work to be acceptable. There are unfortunately quite a few tickets which have a proposed change which has been reviewed but for which the original contributor has disappeared. These are often pretty straightforward to get into acceptable shape, so this is a high impact area in which to do work. - Triage. Lots of duplicates are filed in the issue tracker. Closing these as such (with a link to the existing ticket, and another link on the existing ticket to the duplicate) makes it that much easier to make sense of all the other open tickets (of which there are too many even without counting duplicates). - Website. Our front page needs an overhaul. It's too hard for people new to Twisted to get the information they need from the current front page. We should have a goal-oriented page that makes it easy for someone who just knows "I want to write a jabber client" to get to the Twisted Words page, the jabber/xmpp examples, the Twisted Words examples, etc. If any of these sound interesting to you, dive in! You might want to hop on IRC, #twisted on irc.freenode.net, to talk about things in more detail, too. Jean-Paul
On Nov 6, 2009, at 5:49 PM, exarkun@twistedmatrix.com wrote:
- Website. Our front page needs an overhaul. It's too hard for people new to Twisted to get the information they need from the current front page. We should have a goal-oriented page that makes it easy for someone who just knows "I want to write a jabber client" to get to the Twisted Words page, the jabber/xmpp examples, the Twisted Words examples, etc.
I wonder whether I could offer anything in terms of getting the Twisted site onto better hosting. I t's often unreachable, and also often unusably slow but I'm not sure whether that's a horsepower or software issue, or both. I have lots of server space in good quarters so if that's of interest please have whoever would be in charge of that contact me off-list. S
On 12:48 am, ssteinerx@gmail.com wrote:
On Nov 6, 2009, at 5:49 PM, exarkun@twistedmatrix.com wrote:
- Website. Our front page needs an overhaul. It's too hard for people new to Twisted to get the information they need from the current front page. We should have a goal-oriented page that makes it easy for someone who just knows "I want to write a jabber client" to get to the Twisted Words page, the jabber/xmpp examples, the Twisted Words examples, etc.
I wonder whether I could offer anything in terms of getting the Twisted site onto better hosting. I t's often unreachable, and also often unusably slow but I'm not sure whether that's a horsepower or software issue, or both.
I have lots of server space in good quarters so if that's of interest please have whoever would be in charge of that contact me off-list.
As far as I can tell, these problems are due solely to the quality of trac. There is no indication that they will disappear through only an improvement of the hardware the site is hosted on (unless we consider a really amazing improvement, something on the order of 10x faster CPU and disk than what we have now, a 3GHz P4 and contemporary disks). I would be happy to be mistaken in this, though. Jean-Paul
On Nov 6, 2009, at 7:54 PM, exarkun@twistedmatrix.com wrote:
As far as I can tell, these problems are due solely to the quality of trac. There is no indication that they will disappear through only an improvement of the hardware the site is hosted on (unless we consider a really amazing improvement, something on the order of 10x faster CPU and disk than what we have now, a 3GHz P4 and contemporary disks).
Not to be overly blunt, but there are other trac installations running on far less capable hardware that don't suck anywhere near as hard. That's not to say trac is great, but it isn't near this bad in many, many other places. I don't know anything about the history of this particular installation, what else is running on the server etc. but I don't think it's trac as trac. It may be trac as installed in this particular situation, with whatever historical mutilations have been performed on it, with the particular versions of this and that. Dunno, just trying to find a way to help with my very, very limited time. S
On 01:11 am, ssteinerx@gmail.com wrote:
On Nov 6, 2009, at 7:54 PM, exarkun@twistedmatrix.com wrote:
As far as I can tell, these problems are due solely to the quality of trac. There is no indication that they will disappear through only an improvement of the hardware the site is hosted on (unless we consider a really amazing improvement, something on the order of 10x faster CPU and disk than what we have now, a 3GHz P4 and contemporary disks).
Not to be overly blunt, but there are other trac installations running on far less capable hardware that don't suck anywhere near as hard. That's not to say trac is great, but it isn't near this bad in many, many other places.
Do you know roughly how much traffic any of those installations deal with? We do around 50k hits per day, which is basically maxing out our hardware. I have the sense that most trac sites don't handle this much traffic.
I don't know anything about the history of this particular installation, what else is running on the server etc. but I don't think it's trac as trac. It may be trac as installed in this particular situation, with whatever historical mutilations have been performed on it, with the particular versions of this and that.
Dunno, just trying to find a way to help with my very, very limited time.
Several people have spent non-trivial amounts of time looking at this. I'm not sure what improvements can be made with only limited additional input. Do you know of any relevant resources which we may have overlooked? Are there guides to running a working trac site? Tips about special indexes to create in the database? Something? Jean-Paul
On Nov 6, 2009, at 9:59 PM, exarkun@twistedmatrix.com wrote:
On 01:11 am, ssteinerx@gmail.com wrote:
On Nov 6, 2009, at 7:54 PM, exarkun@twistedmatrix.com wrote:
As far as I can tell, these problems are due solely to the quality of trac. There is no indication that they will disappear through only an improvement of the hardware the site is hosted on (unless we consider a really amazing improvement, something on the order of 10x faster CPU and disk than what we have now, a 3GHz P4 and contemporary disks).
Not to be overly blunt, but there are other trac installations running on far less capable hardware that don't suck anywhere near as hard. That's not to say trac is great, but it isn't near this bad in many, many other places.
Do you know roughly how much traffic any of those installations deal with? We do around 50k hits per day, which is basically maxing out our hardware. I have the sense that most trac sites don't handle this much traffic.
I'm not sure. That sounds like a lot for trac but I don't really know .
Dunno, just trying to find a way to help with my very, very limited time.
Several people have spent non-trivial amounts of time looking at this. I'm not sure what improvements can be made with only limited additional input.
I know, and I'm not trying to be flip or anything like it. I hate to ask, but how much memory is in the box? If it's memory bound with that many hits, another couple of gigs could work wonders. Hell, I'll even buy them send them to the hosting company if it comes down to it.
Do you know of any relevant resources which we may have overlooked? Are there guides to running a working trac site? Tips about special indexes to create in the database? Something?
Sorry to say, my early experiences with Trac were so bad, I just quit. Maybe the thing to do is pull together the statistics (is there a module for this), and just ask the trac devs for some help. I'd be happy to spearhead that if I could be of some help in that capacity. S
On Sat, 07 Nov 2009 00:54:38 -0000, exarkun@twistedmatrix.com wrote:
On 12:48 am, ssteinerx@gmail.com wrote: As far as I can tell, these problems are due solely to the quality of trac. There is no indication that they will disappear through only an improvement of the hardware the site is hosted on (unless we consider a really amazing improvement, something on the order of 10x faster CPU and disk than what we have now, a 3GHz P4 and contemporary disks).
I suspect that Steve has access to a server farm that has processing power several orders of magnitude above the single machine you have. It might be worth following it up. David
On 04:43 am, david.lyon@preisshare.net wrote:
On Sat, 07 Nov 2009 00:54:38 -0000, exarkun@twistedmatrix.com wrote:
On 12:48 am, ssteinerx@gmail.com wrote: As far as I can tell, these problems are due solely to the quality of trac. There is no indication that they will disappear through only an improvement of the hardware the site is hosted on (unless we consider a really amazing improvement, something on the order of 10x faster CPU and disk than what we have now, a 3GHz P4 and contemporary disks).
I suspect that Steve has access to a server farm that has processing power several orders of magnitude above the single machine you have.
It might be worth following it up.
A server farm doesn't buy us anything, as trac can't be scaled up across multiple hosts (or even multiple CPUs on a single host). Jean-Paul
Hi Jean-Paul, On Sat, 07 Nov 2009 13:32:28 -0000, exarkun@twistedmatrix.com wrote:
A server farm doesn't buy us anything, as trac can't be scaled up across multiple hosts (or even multiple CPUs on a single host).
I'm surprised to hear that. Usually replication of servers for read requests works well with dns sharing. Does Trac not run a database? that can go on another machine and lighten the load. Anyway, it's your call. Keep up the good work. David
On 01:51 pm, david.lyon@preisshare.net wrote:
Hi Jean-Paul,
On Sat, 07 Nov 2009 13:32:28 -0000, exarkun@twistedmatrix.com wrote:
A server farm doesn't buy us anything, as trac can't be scaled up across multiple hosts (or even multiple CPUs on a single host).
I'm surprised to hear that.
Usually replication of servers for read requests works well with dns sharing.
Does Trac not run a database? that can go on another machine and lighten the load.
Anyway, it's your call. Keep up the good work.
Part of the trouble is that there are no trac requests which are read- only. Even anonymous view requests do things to the session table. That could probably be overcome with some coding. Perhaps someone would like to work on that? It would be useful, although there may be better ways to spend trac maintenance time (but maybe not, who knows!). Jean-Paul
On Nov 7, 2009, at 8:51 AM, David Lyon wrote:
Hi Jean-Paul,
On Sat, 07 Nov 2009 13:32:28 -0000, exarkun@twistedmatrix.com wrote:
A server farm doesn't buy us anything, as trac can't be scaled up across multiple hosts (or even multiple CPUs on a single host).
I'm surprised to hear that.
Usually replication of servers for read requests works well with dns sharing.
Does Trac not run a database? that can go on another machine and lighten the load.
Anyway, it's your call. Keep up the good work.
I think the best option may just be to collect machine, database, and usage stats and ask the trac guys for help. S
On Nov 6, 2009, at 11:43 PM, David Lyon wrote:
On Sat, 07 Nov 2009 00:54:38 -0000, exarkun@twistedmatrix.com wrote:
On 12:48 am, ssteinerx@gmail.com wrote: As far as I can tell, these problems are due solely to the quality of trac. There is no indication that they will disappear through only an improvement of the hardware the site is hosted on (unless we consider a really amazing improvement, something on the order of 10x faster CPU and disk than what we have now, a 3GHz P4 and contemporary disks).
I suspect that Steve has access to a server farm that has processing power several orders of magnitude above the single machine you have.
Uh, I have access to all of Rackspace, Amazon, Elastic Cloud etc., just like anyone else and also some servers of our own, the smallest of which is a dual quad-core beast with 16gb of memory and 500gb of Raid 10 storage constantly mirrored to a SAN. It isn't all dedicated to a single job (obviously) but there's some horsepower to spare ;-). S
<exarkun <at> twistedmatrix.com> writes:
As far as I can tell, these problems are due solely to the quality of trac. There is no indication that they will disappear through only an improvement of the hardware the site is hosted on (unless we consider a really amazing improvement, something on the order of 10x faster CPU and disk than what we have now, a 3GHz P4 and contemporary disks).
You've probably checked this already but just in case: - What kind of database it uses? If it is SQLite then Postgres will improve things for sure. - The about page shows that you are using 0.11dev - this is not good. During 0.11 development they fixed a lot of performance/memory problems. The current version is 0.11.5. You definitely should upgrade. Regards, Mikhail
participants (4)
-
David Lyon -
exarkun＠twistedmatrix.com -
Mikhail -
ssteinerX＠gmail.com