[Twisted-Python] CPUShare-Twisted
Hello everyone, if you've pending patches that have not been applied or you've no time for unittests until your project works, you can consider switching to CPUShare-Twisted. Mercurial allows distributed development so we can merge our branches against each other with very little effort. If you've suggestion for betters names you're welcome. I could have done it over sourceforge but they lack mercurial AFIK and since I've the setup on cpushare.com I was quick to add Twisted to it. We can move it elsewhere later if needed (especially if it consumes too much network bandwidth ;). I have no interest to maintain twisted except for the fact I'm using it in semi-production and I plan to use it in production, and so I need it to be stable, and with all possible bugfixes and features applied without formalities. I assume other business usages of twisted may have the same needs, so perhaps we can join our efforts with this project. My choice was to either dump twisted or to maintain it locally, and so I clearly have an interest to make the CPUShare-Twisted branch public and to include other people fixes and features (I was already maintaining it over the last year except it was just a patch with fixes that I failed to push in SVN, and not a real fork). You can find the quickstart, basic objectives (including the not trying to reinvent the wheel) and mailing list details on the project page: http://www.cpushare.com/twisted I'll keep tracking SVN. I'll also keep submitting patches upstream since one can always hope for the best. Also note, over the last year I've fixed at least one bug in core twisted basic protocols that would have never been found with any unittest out there. Careful auditing and reading code and filtering of the patches, and thinking deeply about the design before writing code (to write the code in a way that won't break easily over time), is much more important than spending time on unittests. Unittests still makes perfect sense after stuff is included and works in basic testing, but they should be separated from the logic of committing valid patches to the tree. NOTE: before clicking on that page you may want to also answer these questions: 1) is python much better than ruby and all other language on earth? 2) is twisted much better than any other framework to write network apps? 3) are twisted and python fast and scalable enough for all applications? 4) is the single threaded model scalable enough for all applications in smp? 5) is async programming using deferreds simpler to code for a webserver that is stateless and that only does sql queries over the network? 6) would you rather prefer to go broke than to use code "not-invented-here" or not written with python and twisted? (of course the python interpreter the c compiler and the underlying operative system are magically excluded from the not-invented-here clause for whatever unknown reason) I'm welcome anybody who wants to join cpushare-twisted, but if the answer of _any_ of the above questions is "yes", I think you may be better off ignoring this email.
On Sun, 22 Jan 2006 02:53:12 +0100, Andrea Arcangeli <andrea@cpushare.com> wrote:
If you've pending patches that have not been applied, would you please consider instead to agitate for those patches on the mailing lists, and add information to them in the tracker? Discussing an abandoned patch can cast some light where it is needed. Putting it into the CPUShare fork rather than the bugtracker virtually guarantees that no core Twisted developer will ever look at it. Not that you can't do both - I just want to make it clear that we're going to keep working off the bugtracker and SVN, not off of random mercurial (or git or bzr or monotone - not to pick on hg) repositories run by users. Also, Andrea, would you mind making sure that your codebase is visually distinct from Twisted in some way that is identifiable when it is run? My main concern here is that if CPUShare-Twisted is to become popular, it seems likely that users might confuse it with the actual Twisted. Since you have loudly declared your animosity towards filing bugs, I assume CPUShare-Twisted will not have its own bugtracker, and since you are opposed to test-driven development, I am sure that it will be full of bugs. I would prefer it if we could avoid asking the Twisted team to deal with those bugs as well as actual bugs in Twisted, when users discover the only Twisted bugtracker is on twistedmatrix.com. The best option would be to avoid using the same module name, but I understand that tracking trunk would be made more difficult with a diff that touched every import. Perhaps instead you could change the version from SVN-Trunk to 'HG-CPUShare', so that the CPUShare-ness of the code is visible at the beginning of any logs, and change the names of the command line tools to e.g. cpushare-twistd and cpushare-mktap? This would save time since developers could reject bugs immediately upon seeing one of those commands or the telltale log message rather than having to identify that the line numbers are all wrong before doing so.
On Sat, Jan 21, 2006 at 11:55:44PM -0500, glyph@divmod.com wrote:
If you've pending patches that have not been applied, would you please consider instead to agitate for those patches on the mailing lists, and add
These are the very old ones (ignore the web2 part that is recent). http://www.cpushare.com/hg/Twisted/?cs=400da64bd5a6 IIRC you said that disconnecting isn't present in all protocols, not sure if you added disconnecting to all protocols. There was a discussion to add more names, but I didn't see any fix going into SVN trunk. I need it in tcp and these fixes are good enough for my usage (call it dirty fix but it works for me). The other is a fundamental feature I need from the proxy, it's backwards compatible so it doesn't add up any risk but I never received feedback on it. Without it, it's impossible to safely stack one server on top of the other without losing the client info. I'm in discussion how to add it to web2 with headers instead of path mangling.
information to them in the tracker? Discussing an abandoned patch can cast some light where it is needed. Putting it into the CPUShare fork rather
I agree, but I always needed the fix since day zero. I had to maintain it for so long. The fact I'm exporting the tree in a public way can't make things worse as far as SVN integration is concerned. If something it can make things better, since with a public tree there may be more interest from your part in reducing the diff to nothing ;).
Sure I understand that.
My developmnt is generally test-driven. I'm only opposed to unit-test mandatory development to fix bugs or add new features. Writing unit-test isn't the only way to test code. It's nice to have a unit test, even a simple one, but it shouldn't be mandatory.
[..] I am sure that it will be full of bugs.
Time will tell. Since you made your prediction I'll make mine. I'm sure axiom is wasted time in its current API (at least as far as twisted is concerned). I don't see how you can advertize axiom saying "We do plan to add some later, and perhaps also support other databases in the future.". Sure you can add it, but if you do it, the whole axiom api will fall apart unless you want to make synchronous queries over the network. The only two deferreds you have are during startup and in the testsuite, just grep for the word Deferred. Making synchronous sql queries in the twisted async model is unacceptable for anything serious. Infact even sqllite queries are obviously unacceptable once the db grows beyond the size of the cache (and for sure you can't scale the queries over different servers to have more ram-cache when using sqllite). Ironically axiom current api would have a chance to work well with threads, with twisted single threaded async model not. There are good python storage packages to use with twisted and threads (heck there is even pgasync if you're religious about not using threads, why do you think people like me, Valentino and the pgasync author care about using non-blocking solutions while doing queries to the db?). I can't imagine why you insist on making your inferior solution with a design that can't work well with twisted. I'm feeling guilty for risking hitting the harddisk for a few msec when people clicks on the mailing list archives, and infact I keep two webservers exactly to avoid hurting the scalability of the ssl one.
Sure.
Ok, I'll make this change right away. I already did that for the web2 side.
This is a very fair requirement (changing the version is trivial). However I don't see much point in changing the commands if the module name is the same. Either I change both, or none. I mean, changing cpushare-twistd won't prevent a python app importing the reactor to get mine instead of the official one, so you would still risk to get bogus bugreports unless I rename both. I think I can rename the module later if some people uses it, right now I've no idea if people will use it, or if I'll remain the only user (clearly as long as I'm the only user I don't need to rename the module ;).
At 2006-01-22 07:52 AM +0100, you wrote:
My developmnt is generally test-driven. I'm only opposed to unit-test mandatory development to fix bugs or add new features.
Just my 2 cents worth: It might help if you think of it not as a unit test, but as a simplified test program that demonstrates the failure. If you understand the problem, it's generally possible to create a simplified program that captures its essence for others to understand.
Automated unit testing *is* the only way to *effectively* test code. It's also the only way to ensure that someone doesn't make a change at some later date that re-introduces a bug; the unit test will point the finger at the guilty party and the (newly-)broken code. In fact, even if you don't write a unit test case or program, you are doing unit testing when you duplicate and report a bug. You're either unit testing using automatic testing methods (i.e., a test program) or you're unit testing manually. Guess which tests will actually be run? You're either unit testing a constrained chunk of code with a limited, reasonably-well-defined range of behavior, or you're unit testing a larger system with a wide range of behavior and trying to determine, from the aggregate behavior, whether a small change somewhere has changed the system's behavior in an undesired fashion. Guess which test will efficiently catch problems? *Without* automated tests, you're testing the system instead of the module--which is far less effective-- AND you're doing it manually--which means the test will probably never be repeated to verify behavior after some *other* change is made. If you want stable, dependable code, WRITE THE TEST CASES. Further IMO: As a *user* of a variety of development tools and libraries who *knows* how difficult it can be to duplicate a bug, I always try to create a test program that demonstrates the bug, and which can either become a test case or be used, in whole or in part, to create a test case. As a professional software developer, I automatically reject any bug report (from the QA folks) that doesn't provide sufficient information to duplicate a bug. QA personnel are paid to find and document bugs; if they don't, it's their problem, not mine. On the other hand, I know that "users" of an open-source module generally don't think they signed up to do QA (though it's implicit in the open-source development model). As someone who has released open-source software in the past, and plans to do so again soon, the presence or absence of a test program (AKA unit test) that demonstrates a bug is often the deciding factor when I devote some of my limited time to fixing bugs. Practical considerations (mainly available time, but interest as well) tend to result in the issue going to the end of the queue, and therefore never being addressed (without a specific need on my part). So I see no reason why a unit test or test program should not be required. Twisted, like most open-source development projects, is a *cooperative* development effort. And, as find myself telling my coworkers *far* too often, if the code isn't tested, there's no assurance that it works as intended or designed. FINALLY (I sincerly hope!): More generally, I think everyone on the list is now well aware of both the range of opinions and the Twisted requirements WRT test cases for reported bugs. The project is Twisted, and has guidelines for participation; it's to everyone's benefit to follow the guidelines. Now... Can we PLEASE agree to disagree, stop sniping, and devote mailing list bandwidth to "real" issues?! - Sam __________________________________________________________ Spinward Stars, LLC Samuel Reynolds Software Consulting and Development 303-805-1446 http://SpinwardStars.com/ sam@SpinwardStars.com
On Sun, Jan 22, 2006 at 06:10:24PM -0700, Samuel Reynolds wrote:
This is just a dream for me, the bug I've fixed this month as a professional software developer (in my real job, nothing that has to do with twisted) were not reproducible, not just by me, but by the customers too. I found it after nailing in the crash dump for several hours and there's no way I could ever hope to write a testcase to reproduce it reliably even after I've fixed it (furthermore not all hardware can reproduce it at all).
So I see no reason why a unit test or test program should not be required. Twisted, like most open-source
We may argue about new features, but for bugfixes it makes no sense to require them, and my mind isn't going to change about that.
Can we PLEASE agree to disagree, stop sniping, and devote mailing list bandwidth to "real" issues?!
Yep.
On Mon, Jan 23, 2006 at 02:50:06AM +0100, Andrea Arcangeli wrote: [...]
Tests for bugfixes help ensure the bug remains fixed, just like how tests for features help ensure the feature keeps working. The difference is minimal, except that bugs are usually much easier to specify and thus write tests for. An anecdote: at one point, the FTP code in Twisted got rewritten, and as part of that the person doing the rewrite decided it would be easiest to just remove some tests rather than figure out how to update them[1]. Some time later, a bug report got filed and I thought "I remember fixing this already" -- sure enough I had, but the test that would have detected it had been thrown away in the rewrite, so no-one noticed when the bug was reintroduced until after it had been released. Tests being thrown away almost never happens -- it was clearly a mistake that time. Based on the fact that our range of automated tests is always increasing (aside from the rare hiccup like that one), I speculate that the number of bugs prevented from recurring is probably quite high. It's hard to measure; generally code that breaks tests isn't committed. Anyway, my point is this: the reason why Twisted is of such good quality (even though there's still room to do much better) is because of the tests. That's why we're so fussy about having them. -Andrew. [1] To be fair the original tests weren't particularly well written, but neither was the rest of the FTP code...
I think nobody here is claiming that you're not a useful to the Twisted effort. Mr. Lefkowitz has already apologized for the delays of applying patches (mostly due to lack of unit tests and manpower) but stating that a great eye on code is better than automatic testing I think it's wron and I don't know where one excludes the other. You can can write test, use your experience to read through it and find if it's correct and submit to the Twisted folks. It's very normal to apply a pending patch on your local Twisted tree while waiting that they apply it on the trunk. Delays? Why you don't join the project?
NOTE: before clicking on that page you may want to also answer these questions:
Let me say that reading those kind of question I'm starting to think that Twisted it's not what you need in first place, so why bother rewriting your app N times or wasting your time here if Twisted is not what you need? I specifically refer to the question #5
1) is python much better than ruby and all other language on earth?
Than Ruby? The answer is no. It's a matter of tastes. Than all the languages? Who knows them all?
2) is twisted much better than any other framework to write network apps?
Which other framework do you know? I'm aware of ACE for C++ and another for Perl. Full stop. There are languages like E, like Oz having concurrency support builtin but they aren't "frameworks"
3) are twisted and python fast and scalable enough for all applications?
Twisted and Python are scalable enough for all applications that uses them. If you need something powerful why using them? I don't understand your kind of questions.
4) is the single threaded model scalable enough for all applications in smp?
The same as above.
5) is async programming using deferreds simpler to code for a webserver that is stateless and that only does sql queries over the network?
That's my main concern. If you have stateless stuff and you do only sql queries, do you really need all the twisted power?
What's the meaning of such a question?
I wish you luck but this clause terrifies me: "If the CPUShare-Twisted fork fails to be successful, CPUShare will stop using Twisted, so the use of this fork is at your own risk." If you have find time to write countless emails on this mailing list, to say that Nevow is ugly and slow, to whatever... why didn't write unit tests for patches and stop? I don't understand the real point to your argues. -- Lawrence http://www.oluyede.org/blog
On Sun, Jan 22, 2006 at 11:33:56AM +0100, Lawrence Oluyede wrote:
think it's wron and I don't know where one excludes the other. You can
I've never said one excludes the other, I say that unit-test should not be mandatory, and if you should write the unit-test _after_ reviewing the code, not before.
I started to think the same indeed, especially the more time I spend answering email like this one ;). However I still feel that it's quicker to get things working with twisted in the short term, and given all my founding are my savings I've to be dirty and quick (I can't pretend to debug memory corruption or memory leaks with so little resources, so an interpreter is a sane choice), but if I had more resources I could afford not using it and it would be a lot faster and it would scale to a larger number of users using the same hardware resources.
That's my main concern. If you have stateless stuff and you do only sql queries, do you really need all the twisted power?
No my app isn't stateless, or I would be using the thread model too. The very cpushare protocol is complex in the way it handles race conditions and async event like disconnects, twisted makes life easy at the expense of scalability. This helps getting things working quick. And I use pb to attach the webserver to the cpushare server, this is why it's confortable for me to use twisted on the web side too (I'm not making queries to the db only). But most people with simpler projects would be better off with threads to write web apps. Infact twisted web already provides a model like this, my mercurial export already use it, moinmoin also uses it. the webserver it's twisted.web for both and they're threaded.
I wrote it to scare people indeed, I'm careful not to generate any hype, you know what you get when you work with me. If you get on the CPUShare-Twisted project it probably means you already had to maintain your own set of patches for over one year, so joining efforts won't make thing worse even if I decide to dump twisted from my proejct (I can only tell that my twisted branch has solid backups and I can guarantee an export will remain availble if things go wrong).
Even before you write the unittest you should fix nevow, and that's not going to happen, it didn't happen in one year after I sent the first performance bottleneck reports, it sure can't happen in the few hours I spent writing these emails. Plus it'd be terrible to waste time on nevow when Cheetah is already an order of magnitude better and faster (IMHO of course).
I don't understand the real point to your argues.
The point is: if you think the same way I think, if you've similar needs to mine, switch to CPUShare-Twisted.
On 1/22/06, Andrea Arcangeli <andrea@cpushare.com> wrote:
I think that tests MUST BE mandatory in every project used and/or developed from more than 1 people. As Mr. Lefkowitz said in the other message Twisted is not test driven, requires only test for submitted code and I think it's the right way to go if you won't break older or future code.
Ok so you need Twisted, no question about that.
Why didn't you ask to join the project?
Plus it'd be terrible to waste time on nevow when Cheetah is already an order of magnitude better and faster (IMHO of course).
Ok we are in the tastes domain :)
The point is: if you think the same way I think, if you've similar needs to mine, switch to CPUShare-Twisted.
Understood. -- Lawrence http://www.oluyede.org/blog
On Sun, 22 Jan 2006 02:53:12 +0100, Andrea Arcangeli <andrea@cpushare.com> wrote:
If you've pending patches that have not been applied, would you please consider instead to agitate for those patches on the mailing lists, and add information to them in the tracker? Discussing an abandoned patch can cast some light where it is needed. Putting it into the CPUShare fork rather than the bugtracker virtually guarantees that no core Twisted developer will ever look at it. Not that you can't do both - I just want to make it clear that we're going to keep working off the bugtracker and SVN, not off of random mercurial (or git or bzr or monotone - not to pick on hg) repositories run by users. Also, Andrea, would you mind making sure that your codebase is visually distinct from Twisted in some way that is identifiable when it is run? My main concern here is that if CPUShare-Twisted is to become popular, it seems likely that users might confuse it with the actual Twisted. Since you have loudly declared your animosity towards filing bugs, I assume CPUShare-Twisted will not have its own bugtracker, and since you are opposed to test-driven development, I am sure that it will be full of bugs. I would prefer it if we could avoid asking the Twisted team to deal with those bugs as well as actual bugs in Twisted, when users discover the only Twisted bugtracker is on twistedmatrix.com. The best option would be to avoid using the same module name, but I understand that tracking trunk would be made more difficult with a diff that touched every import. Perhaps instead you could change the version from SVN-Trunk to 'HG-CPUShare', so that the CPUShare-ness of the code is visible at the beginning of any logs, and change the names of the command line tools to e.g. cpushare-twistd and cpushare-mktap? This would save time since developers could reject bugs immediately upon seeing one of those commands or the telltale log message rather than having to identify that the line numbers are all wrong before doing so.
On Sat, Jan 21, 2006 at 11:55:44PM -0500, glyph@divmod.com wrote:
If you've pending patches that have not been applied, would you please consider instead to agitate for those patches on the mailing lists, and add
These are the very old ones (ignore the web2 part that is recent). http://www.cpushare.com/hg/Twisted/?cs=400da64bd5a6 IIRC you said that disconnecting isn't present in all protocols, not sure if you added disconnecting to all protocols. There was a discussion to add more names, but I didn't see any fix going into SVN trunk. I need it in tcp and these fixes are good enough for my usage (call it dirty fix but it works for me). The other is a fundamental feature I need from the proxy, it's backwards compatible so it doesn't add up any risk but I never received feedback on it. Without it, it's impossible to safely stack one server on top of the other without losing the client info. I'm in discussion how to add it to web2 with headers instead of path mangling.
information to them in the tracker? Discussing an abandoned patch can cast some light where it is needed. Putting it into the CPUShare fork rather
I agree, but I always needed the fix since day zero. I had to maintain it for so long. The fact I'm exporting the tree in a public way can't make things worse as far as SVN integration is concerned. If something it can make things better, since with a public tree there may be more interest from your part in reducing the diff to nothing ;).
Sure I understand that.
My developmnt is generally test-driven. I'm only opposed to unit-test mandatory development to fix bugs or add new features. Writing unit-test isn't the only way to test code. It's nice to have a unit test, even a simple one, but it shouldn't be mandatory.
[..] I am sure that it will be full of bugs.
Time will tell. Since you made your prediction I'll make mine. I'm sure axiom is wasted time in its current API (at least as far as twisted is concerned). I don't see how you can advertize axiom saying "We do plan to add some later, and perhaps also support other databases in the future.". Sure you can add it, but if you do it, the whole axiom api will fall apart unless you want to make synchronous queries over the network. The only two deferreds you have are during startup and in the testsuite, just grep for the word Deferred. Making synchronous sql queries in the twisted async model is unacceptable for anything serious. Infact even sqllite queries are obviously unacceptable once the db grows beyond the size of the cache (and for sure you can't scale the queries over different servers to have more ram-cache when using sqllite). Ironically axiom current api would have a chance to work well with threads, with twisted single threaded async model not. There are good python storage packages to use with twisted and threads (heck there is even pgasync if you're religious about not using threads, why do you think people like me, Valentino and the pgasync author care about using non-blocking solutions while doing queries to the db?). I can't imagine why you insist on making your inferior solution with a design that can't work well with twisted. I'm feeling guilty for risking hitting the harddisk for a few msec when people clicks on the mailing list archives, and infact I keep two webservers exactly to avoid hurting the scalability of the ssl one.
Sure.
Ok, I'll make this change right away. I already did that for the web2 side.
This is a very fair requirement (changing the version is trivial). However I don't see much point in changing the commands if the module name is the same. Either I change both, or none. I mean, changing cpushare-twistd won't prevent a python app importing the reactor to get mine instead of the official one, so you would still risk to get bogus bugreports unless I rename both. I think I can rename the module later if some people uses it, right now I've no idea if people will use it, or if I'll remain the only user (clearly as long as I'm the only user I don't need to rename the module ;).
At 2006-01-22 07:52 AM +0100, you wrote:
My developmnt is generally test-driven. I'm only opposed to unit-test mandatory development to fix bugs or add new features.
Just my 2 cents worth: It might help if you think of it not as a unit test, but as a simplified test program that demonstrates the failure. If you understand the problem, it's generally possible to create a simplified program that captures its essence for others to understand.
Automated unit testing *is* the only way to *effectively* test code. It's also the only way to ensure that someone doesn't make a change at some later date that re-introduces a bug; the unit test will point the finger at the guilty party and the (newly-)broken code. In fact, even if you don't write a unit test case or program, you are doing unit testing when you duplicate and report a bug. You're either unit testing using automatic testing methods (i.e., a test program) or you're unit testing manually. Guess which tests will actually be run? You're either unit testing a constrained chunk of code with a limited, reasonably-well-defined range of behavior, or you're unit testing a larger system with a wide range of behavior and trying to determine, from the aggregate behavior, whether a small change somewhere has changed the system's behavior in an undesired fashion. Guess which test will efficiently catch problems? *Without* automated tests, you're testing the system instead of the module--which is far less effective-- AND you're doing it manually--which means the test will probably never be repeated to verify behavior after some *other* change is made. If you want stable, dependable code, WRITE THE TEST CASES. Further IMO: As a *user* of a variety of development tools and libraries who *knows* how difficult it can be to duplicate a bug, I always try to create a test program that demonstrates the bug, and which can either become a test case or be used, in whole or in part, to create a test case. As a professional software developer, I automatically reject any bug report (from the QA folks) that doesn't provide sufficient information to duplicate a bug. QA personnel are paid to find and document bugs; if they don't, it's their problem, not mine. On the other hand, I know that "users" of an open-source module generally don't think they signed up to do QA (though it's implicit in the open-source development model). As someone who has released open-source software in the past, and plans to do so again soon, the presence or absence of a test program (AKA unit test) that demonstrates a bug is often the deciding factor when I devote some of my limited time to fixing bugs. Practical considerations (mainly available time, but interest as well) tend to result in the issue going to the end of the queue, and therefore never being addressed (without a specific need on my part). So I see no reason why a unit test or test program should not be required. Twisted, like most open-source development projects, is a *cooperative* development effort. And, as find myself telling my coworkers *far* too often, if the code isn't tested, there's no assurance that it works as intended or designed. FINALLY (I sincerly hope!): More generally, I think everyone on the list is now well aware of both the range of opinions and the Twisted requirements WRT test cases for reported bugs. The project is Twisted, and has guidelines for participation; it's to everyone's benefit to follow the guidelines. Now... Can we PLEASE agree to disagree, stop sniping, and devote mailing list bandwidth to "real" issues?! - Sam __________________________________________________________ Spinward Stars, LLC Samuel Reynolds Software Consulting and Development 303-805-1446 http://SpinwardStars.com/ sam@SpinwardStars.com
On Sun, Jan 22, 2006 at 06:10:24PM -0700, Samuel Reynolds wrote:
This is just a dream for me, the bug I've fixed this month as a professional software developer (in my real job, nothing that has to do with twisted) were not reproducible, not just by me, but by the customers too. I found it after nailing in the crash dump for several hours and there's no way I could ever hope to write a testcase to reproduce it reliably even after I've fixed it (furthermore not all hardware can reproduce it at all).
So I see no reason why a unit test or test program should not be required. Twisted, like most open-source
We may argue about new features, but for bugfixes it makes no sense to require them, and my mind isn't going to change about that.
Can we PLEASE agree to disagree, stop sniping, and devote mailing list bandwidth to "real" issues?!
Yep.
On Mon, Jan 23, 2006 at 02:50:06AM +0100, Andrea Arcangeli wrote: [...]
Tests for bugfixes help ensure the bug remains fixed, just like how tests for features help ensure the feature keeps working. The difference is minimal, except that bugs are usually much easier to specify and thus write tests for. An anecdote: at one point, the FTP code in Twisted got rewritten, and as part of that the person doing the rewrite decided it would be easiest to just remove some tests rather than figure out how to update them[1]. Some time later, a bug report got filed and I thought "I remember fixing this already" -- sure enough I had, but the test that would have detected it had been thrown away in the rewrite, so no-one noticed when the bug was reintroduced until after it had been released. Tests being thrown away almost never happens -- it was clearly a mistake that time. Based on the fact that our range of automated tests is always increasing (aside from the rare hiccup like that one), I speculate that the number of bugs prevented from recurring is probably quite high. It's hard to measure; generally code that breaks tests isn't committed. Anyway, my point is this: the reason why Twisted is of such good quality (even though there's still room to do much better) is because of the tests. That's why we're so fussy about having them. -Andrew. [1] To be fair the original tests weren't particularly well written, but neither was the rest of the FTP code...
I think nobody here is claiming that you're not a useful to the Twisted effort. Mr. Lefkowitz has already apologized for the delays of applying patches (mostly due to lack of unit tests and manpower) but stating that a great eye on code is better than automatic testing I think it's wron and I don't know where one excludes the other. You can can write test, use your experience to read through it and find if it's correct and submit to the Twisted folks. It's very normal to apply a pending patch on your local Twisted tree while waiting that they apply it on the trunk. Delays? Why you don't join the project?
NOTE: before clicking on that page you may want to also answer these questions:
Let me say that reading those kind of question I'm starting to think that Twisted it's not what you need in first place, so why bother rewriting your app N times or wasting your time here if Twisted is not what you need? I specifically refer to the question #5
1) is python much better than ruby and all other language on earth?
Than Ruby? The answer is no. It's a matter of tastes. Than all the languages? Who knows them all?
2) is twisted much better than any other framework to write network apps?
Which other framework do you know? I'm aware of ACE for C++ and another for Perl. Full stop. There are languages like E, like Oz having concurrency support builtin but they aren't "frameworks"
3) are twisted and python fast and scalable enough for all applications?
Twisted and Python are scalable enough for all applications that uses them. If you need something powerful why using them? I don't understand your kind of questions.
4) is the single threaded model scalable enough for all applications in smp?
The same as above.
5) is async programming using deferreds simpler to code for a webserver that is stateless and that only does sql queries over the network?
That's my main concern. If you have stateless stuff and you do only sql queries, do you really need all the twisted power?
What's the meaning of such a question?
I wish you luck but this clause terrifies me: "If the CPUShare-Twisted fork fails to be successful, CPUShare will stop using Twisted, so the use of this fork is at your own risk." If you have find time to write countless emails on this mailing list, to say that Nevow is ugly and slow, to whatever... why didn't write unit tests for patches and stop? I don't understand the real point to your argues. -- Lawrence http://www.oluyede.org/blog
On Sun, Jan 22, 2006 at 11:33:56AM +0100, Lawrence Oluyede wrote:
think it's wron and I don't know where one excludes the other. You can
I've never said one excludes the other, I say that unit-test should not be mandatory, and if you should write the unit-test _after_ reviewing the code, not before.
I started to think the same indeed, especially the more time I spend answering email like this one ;). However I still feel that it's quicker to get things working with twisted in the short term, and given all my founding are my savings I've to be dirty and quick (I can't pretend to debug memory corruption or memory leaks with so little resources, so an interpreter is a sane choice), but if I had more resources I could afford not using it and it would be a lot faster and it would scale to a larger number of users using the same hardware resources.
That's my main concern. If you have stateless stuff and you do only sql queries, do you really need all the twisted power?
No my app isn't stateless, or I would be using the thread model too. The very cpushare protocol is complex in the way it handles race conditions and async event like disconnects, twisted makes life easy at the expense of scalability. This helps getting things working quick. And I use pb to attach the webserver to the cpushare server, this is why it's confortable for me to use twisted on the web side too (I'm not making queries to the db only). But most people with simpler projects would be better off with threads to write web apps. Infact twisted web already provides a model like this, my mercurial export already use it, moinmoin also uses it. the webserver it's twisted.web for both and they're threaded.
I wrote it to scare people indeed, I'm careful not to generate any hype, you know what you get when you work with me. If you get on the CPUShare-Twisted project it probably means you already had to maintain your own set of patches for over one year, so joining efforts won't make thing worse even if I decide to dump twisted from my proejct (I can only tell that my twisted branch has solid backups and I can guarantee an export will remain availble if things go wrong).
Even before you write the unittest you should fix nevow, and that's not going to happen, it didn't happen in one year after I sent the first performance bottleneck reports, it sure can't happen in the few hours I spent writing these emails. Plus it'd be terrible to waste time on nevow when Cheetah is already an order of magnitude better and faster (IMHO of course).
I don't understand the real point to your argues.
The point is: if you think the same way I think, if you've similar needs to mine, switch to CPUShare-Twisted.
On 1/22/06, Andrea Arcangeli <andrea@cpushare.com> wrote:
I think that tests MUST BE mandatory in every project used and/or developed from more than 1 people. As Mr. Lefkowitz said in the other message Twisted is not test driven, requires only test for submitted code and I think it's the right way to go if you won't break older or future code.
Ok so you need Twisted, no question about that.
Why didn't you ask to join the project?
Plus it'd be terrible to waste time on nevow when Cheetah is already an order of magnitude better and faster (IMHO of course).
Ok we are in the tastes domain :)
The point is: if you think the same way I think, if you've similar needs to mine, switch to CPUShare-Twisted.
Understood. -- Lawrence http://www.oluyede.org/blog
participants (5)
-
Andrea Arcangeli -
Andrew Bennetts -
glyph＠divmod.com -
Lawrence Oluyede -
Samuel Reynolds