Hello everyone, if you've pending patches that have not been applied or you've no time for unittests until your project works, you can consider switching to CPUShare-Twisted. Mercurial allows distributed development so we can merge our branches against each other with very little effort. If you've suggestion for betters names you're welcome. I could have done it over sourceforge but they lack mercurial AFIK and since I've the setup on cpushare.com I was quick to add Twisted to it. We can move it elsewhere later if needed (especially if it consumes too much network bandwidth ;). I have no interest to maintain twisted except for the fact I'm using it in semi-production and I plan to use it in production, and so I need it to be stable, and with all possible bugfixes and features applied without formalities. I assume other business usages of twisted may have the same needs, so perhaps we can join our efforts with this project. My choice was to either dump twisted or to maintain it locally, and so I clearly have an interest to make the CPUShare-Twisted branch public and to include other people fixes and features (I was already maintaining it over the last year except it was just a patch with fixes that I failed to push in SVN, and not a real fork). You can find the quickstart, basic objectives (including the not trying to reinvent the wheel) and mailing list details on the project page: http://www.cpushare.com/twisted I'll keep tracking SVN. I'll also keep submitting patches upstream since one can always hope for the best. Also note, over the last year I've fixed at least one bug in core twisted basic protocols that would have never been found with any unittest out there. Careful auditing and reading code and filtering of the patches, and thinking deeply about the design before writing code (to write the code in a way that won't break easily over time), is much more important than spending time on unittests. Unittests still makes perfect sense after stuff is included and works in basic testing, but they should be separated from the logic of committing valid patches to the tree. NOTE: before clicking on that page you may want to also answer these questions: 1) is python much better than ruby and all other language on earth? 2) is twisted much better than any other framework to write network apps? 3) are twisted and python fast and scalable enough for all applications? 4) is the single threaded model scalable enough for all applications in smp? 5) is async programming using deferreds simpler to code for a webserver that is stateless and that only does sql queries over the network? 6) would you rather prefer to go broke than to use code "not-invented-here" or not written with python and twisted? (of course the python interpreter the c compiler and the underlying operative system are magically excluded from the not-invented-here clause for whatever unknown reason) I'm welcome anybody who wants to join cpushare-twisted, but if the answer of _any_ of the above questions is "yes", I think you may be better off ignoring this email.
On Sun, 22 Jan 2006 02:53:12 +0100, Andrea Arcangeli <andrea@cpushare.com> wrote:
if you've pending patches that have not been applied or you've no time for unittests until your project works, you can consider switching to CPUShare-Twisted. Mercurial allows distributed development so we can merge our branches against each other with very little effort.
If you've pending patches that have not been applied, would you please consider instead to agitate for those patches on the mailing lists, and add information to them in the tracker? Discussing an abandoned patch can cast some light where it is needed. Putting it into the CPUShare fork rather than the bugtracker virtually guarantees that no core Twisted developer will ever look at it. Not that you can't do both - I just want to make it clear that we're going to keep working off the bugtracker and SVN, not off of random mercurial (or git or bzr or monotone - not to pick on hg) repositories run by users. Also, Andrea, would you mind making sure that your codebase is visually distinct from Twisted in some way that is identifiable when it is run? My main concern here is that if CPUShare-Twisted is to become popular, it seems likely that users might confuse it with the actual Twisted. Since you have loudly declared your animosity towards filing bugs, I assume CPUShare-Twisted will not have its own bugtracker, and since you are opposed to test-driven development, I am sure that it will be full of bugs. I would prefer it if we could avoid asking the Twisted team to deal with those bugs as well as actual bugs in Twisted, when users discover the only Twisted bugtracker is on twistedmatrix.com. The best option would be to avoid using the same module name, but I understand that tracking trunk would be made more difficult with a diff that touched every import. Perhaps instead you could change the version from SVN-Trunk to 'HG-CPUShare', so that the CPUShare-ness of the code is visible at the beginning of any logs, and change the names of the command line tools to e.g. cpushare-twistd and cpushare-mktap? This would save time since developers could reject bugs immediately upon seeing one of those commands or the telltale log message rather than having to identify that the line numbers are all wrong before doing so.
On Sat, Jan 21, 2006 at 11:55:44PM -0500, glyph@divmod.com wrote:
If you've pending patches that have not been applied, would you please consider instead to agitate for those patches on the mailing lists, and add
These are the very old ones (ignore the web2 part that is recent). http://www.cpushare.com/hg/Twisted/?cs=400da64bd5a6 IIRC you said that disconnecting isn't present in all protocols, not sure if you added disconnecting to all protocols. There was a discussion to add more names, but I didn't see any fix going into SVN trunk. I need it in tcp and these fixes are good enough for my usage (call it dirty fix but it works for me). The other is a fundamental feature I need from the proxy, it's backwards compatible so it doesn't add up any risk but I never received feedback on it. Without it, it's impossible to safely stack one server on top of the other without losing the client info. I'm in discussion how to add it to web2 with headers instead of path mangling.
information to them in the tracker? Discussing an abandoned patch can cast some light where it is needed. Putting it into the CPUShare fork rather
I agree, but I always needed the fix since day zero. I had to maintain it for so long. The fact I'm exporting the tree in a public way can't make things worse as far as SVN integration is concerned. If something it can make things better, since with a public tree there may be more interest from your part in reducing the diff to nothing ;).
than the bugtracker virtually guarantees that no core Twisted developer will ever look at it. Not that you can't do both - I just want to make it clear that we're going to keep working off the bugtracker and SVN, not off of random mercurial (or git or bzr or monotone - not to pick on hg) repositories run by users.
Sure I understand that.
Also, Andrea, would you mind making sure that your codebase is visually distinct from Twisted in some way that is identifiable when it is run? My main concern here is that if CPUShare-Twisted is to become popular, it seems likely that users might confuse it with the actual Twisted. Since you have loudly declared your animosity towards filing bugs, I assume CPUShare-Twisted will not have its own bugtracker, and since you are opposed to test-driven development, [..]
My developmnt is generally test-driven. I'm only opposed to unit-test mandatory development to fix bugs or add new features. Writing unit-test isn't the only way to test code. It's nice to have a unit test, even a simple one, but it shouldn't be mandatory.
[..] I am sure that it will be full of bugs.
Time will tell. Since you made your prediction I'll make mine. I'm sure axiom is wasted time in its current API (at least as far as twisted is concerned). I don't see how you can advertize axiom saying "We do plan to add some later, and perhaps also support other databases in the future.". Sure you can add it, but if you do it, the whole axiom api will fall apart unless you want to make synchronous queries over the network. The only two deferreds you have are during startup and in the testsuite, just grep for the word Deferred. Making synchronous sql queries in the twisted async model is unacceptable for anything serious. Infact even sqllite queries are obviously unacceptable once the db grows beyond the size of the cache (and for sure you can't scale the queries over different servers to have more ram-cache when using sqllite). Ironically axiom current api would have a chance to work well with threads, with twisted single threaded async model not. There are good python storage packages to use with twisted and threads (heck there is even pgasync if you're religious about not using threads, why do you think people like me, Valentino and the pgasync author care about using non-blocking solutions while doing queries to the db?). I can't imagine why you insist on making your inferior solution with a design that can't work well with twisted. I'm feeling guilty for risking hitting the harddisk for a few msec when people clicks on the mailing list archives, and infact I keep two webservers exactly to avoid hurting the scalability of the ssl one.
I would prefer it if we could avoid asking the Twisted team to deal with those bugs as well as actual bugs in Twisted, when users discover the only Twisted bugtracker is on twistedmatrix.com.
Sure.
The best option would be to avoid using the same module name, but I understand that tracking trunk would be made more difficult with a diff that touched every import. Perhaps instead you could change the version from SVN-Trunk to 'HG-CPUShare', so that the CPUShare-ness of the code is
Ok, I'll make this change right away. I already did that for the web2 side.
visible at the beginning of any logs, and change the names of the command line tools to e.g. cpushare-twistd and cpushare-mktap? This would save time since developers could reject bugs immediately upon seeing one of those commands or the telltale log message rather than having to identify that the line numbers are all wrong before doing so.
This is a very fair requirement (changing the version is trivial). However I don't see much point in changing the commands if the module name is the same. Either I change both, or none. I mean, changing cpushare-twistd won't prevent a python app importing the reactor to get mine instead of the official one, so you would still risk to get bogus bugreports unless I rename both. I think I can rename the module later if some people uses it, right now I've no idea if people will use it, or if I'll remain the only user (clearly as long as I'm the only user I don't need to rename the module ;).
Also note, over the last year I've fixed at least one bug in core twisted basic protocols that would have never been found with any unittest out there. Careful auditing and reading code and filtering of the patches, and thinking deeply about the design before writing code (to write the code in a way that won't break easily over time), is much more important than spending time on unittests. Unittests still makes perfect sense after stuff is included and works in basic testing, but they should be separated from the logic of committing valid patches to the tree.
I think nobody here is claiming that you're not a useful to the Twisted effort. Mr. Lefkowitz has already apologized for the delays of applying patches (mostly due to lack of unit tests and manpower) but stating that a great eye on code is better than automatic testing I think it's wron and I don't know where one excludes the other. You can can write test, use your experience to read through it and find if it's correct and submit to the Twisted folks. It's very normal to apply a pending patch on your local Twisted tree while waiting that they apply it on the trunk. Delays? Why you don't join the project?
NOTE: before clicking on that page you may want to also answer these questions:
Let me say that reading those kind of question I'm starting to think that Twisted it's not what you need in first place, so why bother rewriting your app N times or wasting your time here if Twisted is not what you need? I specifically refer to the question #5
1) is python much better than ruby and all other language on earth?
Than Ruby? The answer is no. It's a matter of tastes. Than all the languages? Who knows them all?
2) is twisted much better than any other framework to write network apps?
Which other framework do you know? I'm aware of ACE for C++ and another for Perl. Full stop. There are languages like E, like Oz having concurrency support builtin but they aren't "frameworks"
3) are twisted and python fast and scalable enough for all applications?
Twisted and Python are scalable enough for all applications that uses them. If you need something powerful why using them? I don't understand your kind of questions.
4) is the single threaded model scalable enough for all applications in smp?
The same as above.
5) is async programming using deferreds simpler to code for a webserver that is stateless and that only does sql queries over the network?
That's my main concern. If you have stateless stuff and you do only sql queries, do you really need all the twisted power?
6) would you rather prefer to go broke than to use code "not-invented-here" or not written with python and twisted? (of course the python interpreter the c compiler and the underlying operative system are magically excluded from the not-invented-here clause for whatever unknown reason)
What's the meaning of such a question?
I'm welcome anybody who wants to join cpushare-twisted, but if the answer of _any_ of the above questions is "yes", I think you may be better off ignoring this email.
I wish you luck but this clause terrifies me: "If the CPUShare-Twisted fork fails to be successful, CPUShare will stop using Twisted, so the use of this fork is at your own risk." If you have find time to write countless emails on this mailing list, to say that Nevow is ugly and slow, to whatever... why didn't write unit tests for patches and stop? I don't understand the real point to your argues. -- Lawrence http://www.oluyede.org/blog
On Sun, Jan 22, 2006 at 11:33:56AM +0100, Lawrence Oluyede wrote:
think it's wron and I don't know where one excludes the other. You can
I've never said one excludes the other, I say that unit-test should not be mandatory, and if you should write the unit-test _after_ reviewing the code, not before.
Let me say that reading those kind of question I'm starting to think that Twisted it's not what you need in first place, so why bother rewriting your app N times or wasting your time here if Twisted is not what you need? I specifically refer to the question #5
I started to think the same indeed, especially the more time I spend answering email like this one ;). However I still feel that it's quicker to get things working with twisted in the short term, and given all my founding are my savings I've to be dirty and quick (I can't pretend to debug memory corruption or memory leaks with so little resources, so an interpreter is a sane choice), but if I had more resources I could afford not using it and it would be a lot faster and it would scale to a larger number of users using the same hardware resources.
That's my main concern. If you have stateless stuff and you do only sql queries, do you really need all the twisted power?
No my app isn't stateless, or I would be using the thread model too. The very cpushare protocol is complex in the way it handles race conditions and async event like disconnects, twisted makes life easy at the expense of scalability. This helps getting things working quick. And I use pb to attach the webserver to the cpushare server, this is why it's confortable for me to use twisted on the web side too (I'm not making queries to the db only). But most people with simpler projects would be better off with threads to write web apps. Infact twisted web already provides a model like this, my mercurial export already use it, moinmoin also uses it. the webserver it's twisted.web for both and they're threaded.
I wish you luck but this clause terrifies me: "If the CPUShare-Twisted fork fails to be successful, CPUShare will stop using Twisted, so the use of this fork is at your own risk."
I wrote it to scare people indeed, I'm careful not to generate any hype, you know what you get when you work with me. If you get on the CPUShare-Twisted project it probably means you already had to maintain your own set of patches for over one year, so joining efforts won't make thing worse even if I decide to dump twisted from my proejct (I can only tell that my twisted branch has solid backups and I can guarantee an export will remain availble if things go wrong).
If you have find time to write countless emails on this mailing list, to say that Nevow is ugly and slow, to whatever... why didn't write unit tests for patches and stop?
Even before you write the unittest you should fix nevow, and that's not going to happen, it didn't happen in one year after I sent the first performance bottleneck reports, it sure can't happen in the few hours I spent writing these emails. Plus it'd be terrible to waste time on nevow when Cheetah is already an order of magnitude better and faster (IMHO of course).
I don't understand the real point to your argues.
The point is: if you think the same way I think, if you've similar needs to mine, switch to CPUShare-Twisted.
On 1/22/06, Andrea Arcangeli <andrea@cpushare.com> wrote:
On Sun, Jan 22, 2006 at 11:33:56AM +0100, Lawrence Oluyede wrote:
think it's wron and I don't know where one excludes the other. You can
I've never said one excludes the other, I say that unit-test should not be mandatory, and if you should write the unit-test _after_ reviewing the code, not before.
I think that tests MUST BE mandatory in every project used and/or developed from more than 1 people. As Mr. Lefkowitz said in the other message Twisted is not test driven, requires only test for submitted code and I think it's the right way to go if you won't break older or future code.
No my app isn't stateless, or I would be using the thread model too. The very cpushare protocol is complex in the way it handles race conditions and async event like disconnects, twisted makes life easy at the expense of scalability. This helps getting things working quick. And I use pb to attach the webserver to the cpushare server, this is why it's confortable for me to use twisted on the web side too (I'm not making queries to the db only).
Ok so you need Twisted, no question about that.
Even before you write the unittest you should fix nevow, and that's not going to happen, it didn't happen in one year after I sent the first performance bottleneck reports, it sure can't happen in the few hours I spent writing these emails.
Why didn't you ask to join the project?
Plus it'd be terrible to waste time on nevow when Cheetah is already an order of magnitude better and faster (IMHO of course).
Ok we are in the tastes domain :)
The point is: if you think the same way I think, if you've similar needs to mine, switch to CPUShare-Twisted.
Understood. -- Lawrence http://www.oluyede.org/blog
participants (3)
-
Andrea Arcangeli -
glyph＠divmod.com -
Lawrence Oluyede