Having reactor run at the main thread make things harder for late adopters
I will first describe my application in a single sentence. Write a server that can serve TCP/UDP clients on several ports, and let the clients communicate with each other in a predefined protocol, while each can 'talk' to the server in a different protocol/port. In my case where I picked up Twisted relatively late in the project I need Twisted to plugged into my applocation and not the my application plugged-into Twisted (at least at this stage). Is there a way to by-pass this limitation? At this stage the main problem I see is having transport.write will have data transmitted immediately when called from another thread.
On Tue, 2008-11-04 at 22:20 +0200, Tzury Bar Yochay wrote:
I will first describe my application in a single sentence.
Write a server that can serve TCP/UDP clients on several ports, and let the clients communicate with each other in a predefined protocol, while each can 'talk' to the server in a different protocol/port.
Are the clients communicating directly or through the server? Are the ports specific to either client/server communication or (presuming the answer to the first question) client/client communication?
In my case where I picked up Twisted relatively late in the project I need Twisted to plugged into my applocation and not the my application plugged-into Twisted (at least at this stage).
Is this because the client is already written with a specific protocol in mind?
Is there a way to by-pass this limitation?
At this stage the main problem I see is having transport.write will have data transmitted immediately when called from another thread.
You already have threads in your server? Is this the limitation? If so, are the threads there to handle client connections? -- George Pauly Ring Development www.ringdevelopment.com
On Tue, Nov 4, 2008 at 10:56 PM, George Pauly <george@ringdevelopment.com> wrote:
On Tue, 2008-11-04 at 22:20 +0200, Tzury Bar Yochay wrote:
I will first describe my application in a single sentence.
Write a server that can serve TCP/UDP clients on several ports, and let the clients communicate with each other in a predefined protocol, while each can 'talk' to the server in a different protocol/port.
Are the clients communicating directly or through the server?
clients are to communicate via server only (they are behind NATs)
Are the ports specific to either client/server communication or (presuming the answer to the first question) client/client communication?
Clients can communicate with the server using either one of predefined available ports. You never know what port/protocol the firewall will allow them
In my case where I picked up Twisted relatively late in the project I need Twisted to plugged into my applocation and not the my application plugged-into Twisted (at least at this stage).
Is this because the client is already written with a specific protocol in mind?
That is correct!
Is there a way to by-pass this limitation?
At this stage the main problem I see is having transport.write will have data transmitted immediately when called from another thread.
You already have threads in your server? Is this the limitation? If so, are the threads there to handle client connections?
I am using (thread-safe) queue.Queue and have one for incoming-requests and another one for pending-responses. I have one thread which read on request at a time and handle it. once done, puts a response in the response-queue. Bare in mind that the response might be forwarded to another client or to the original sender depend on the case. If I won't find a decent way to hack this I might going to rewrite the more code than I can allow myself at this point of time.
-- George Pauly Ring Development www.ringdevelopment.com
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
-- Tzury Bar Yochay
On Tue, 2008-11-04 at 23:03 +0200, Tzury Bar Yochay wrote: ...
I am using (thread-safe) queue.Queue and have one for incoming-requests and another one for pending-responses. I have one thread which read on request at a time and handle it. once done, puts a response in the response-queue. Bare in mind that the response might be forwarded to another client or to the original sender depend on the case.
If I won't find a decent way to hack this I might going to rewrite the more code than I can allow myself at this point of time.
This kind of thread work is what Twisted can do for you. Either do it with threads or with Twisted. Queue may be more than you need to handle the client-client connections. Referring to the FAQ Glyph recommended to you, http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeinputo... Instead of broadcasting to a list of connections set up a method to obtain the single appropriate connection and write to that. Please respond only to the list. Thanks, George -- George Pauly Ring Development www.ringdevelopment.com
This kind of thread work is what Twisted can do for you. Either do it with threads or with Twisted. Queue may be more than you need to handle the client-client connections.
Referring to the FAQ Glyph recommended to you,
http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeinputo...
Instead of broadcasting to a list of connections set up a method to obtain the single appropriate connection and write to that.
Thinking about this over again I decided to bring my application to Twisted. I will give up the pending queues and will make the code more readable and control-flow much more traceable. Thanks very much for your advice and sorry for the reply. (i simply pressed `r` in my gmail and started typing...) Tzury
Tzury Bar Yochay wrote:
I will first describe my application in a single sentence.
Write a server that can serve TCP/UDP clients on several ports, and let the clients communicate with each other in a predefined protocol, while each can 'talk' to the server in a different protocol/port.
In my case where I picked up Twisted relatively late in the project I need Twisted to plugged into my applocation and not the my application plugged-into Twisted (at least at this stage).
Is there a way to by-pass this limitation?
You can run the reactor in any thread you like. You probably need to pass installSignalHandlers=False to reactor.run in a non-main thread, but otherwise there's nothing stopping you running the reactor in a separate, non-main thread if that's what's easiest for you.
At this stage the main problem I see is having transport.write will have data transmitted immediately when called from another thread.
When calling into Twisted from non-Twisted threads, use reactor.callFromThread. See <http://twistedmatrix.com/projects/core/documentation/howto/threading.html>. -Andrew.
At this stage the main problem I see is having transport.write will have data transmitted immediately when called from another thread.
When calling into Twisted from non-Twisted threads, use reactor.callFromThread. See <http://twistedmatrix.com/projects/core/documentation/howto/threading.html>.
I actually saw this reactor.callFromThread but weren't sure whether it is a singelton i.e. I can use it in another module by simply importing reactor and calling it and it will affect the same I called .run on in another module. I checked it right now and it works. thank you, - Tzury
Tzury Bar Yochay wrote:
At this stage the main problem I see is having transport.write will have data transmitted immediately when called from another thread.
When calling into Twisted from non-Twisted threads, use reactor.callFromThread. See <http://twistedmatrix.com/projects/core/documentation/howto/threading.html>.
I actually saw this reactor.callFromThread but weren't sure whether it is a singelton i.e. I can use it in another module by simply importing reactor and calling it and it will affect the same I called .run on in another module. I checked it right now and it works.
Yes, it is a singleton. As <http://twistedmatrix.com/projects/core/documentation/howto/reactor-basics.ht...> says: You can get to the reactor object using the following code: from twisted.internet import reactor The documentation could perhaps be clearer about this (that document implies, but doesn't directly state, that there's only one reactor instance). -Andrew.
On Wed, 5 Nov 2008 15:11:08 +1100, Andrew Bennetts <andrew-twisted@puzzling.org> wrote:
Tzury Bar Yochay wrote:
At this stage the main problem I see is having transport.write will have data transmitted immediately when called from another thread.
When calling into Twisted from non-Twisted threads, use reactor.callFromThread. See <http://twistedmatrix.com/projects/core/documentation/howto/threading.html>.
I actually saw this reactor.callFromThread but weren't sure whether it is a singelton i.e. I can use it in another module by simply importing reactor and calling it and it will affect the same I called .run on in another module. I checked it right now and it works.
Yes, it is a singleton. As <http://twistedmatrix.com/projects/core/documentation/howto/reactor-basics.ht...> says:
You can get to the reactor object using the following code:
from twisted.internet import reactor
The documentation could perhaps be clearer about this (that document implies, but doesn't directly state, that there's only one reactor instance).
Although I don't see how it matters if it's a singleton or not. :) If it weren't, you could just pass the reactor you were using to the thread that needed to use it. Jean-Paul
Jean-Paul Calderone wrote:
On Wed, 5 Nov 2008 15:11:08 +1100, Andrew Bennetts <andrew-twisted@puzzling.org> wrote: [...]
Yes, it is a singleton. As <http://twistedmatrix.com/projects/core/documentation/howto/reactor-basics.ht...> says:
You can get to the reactor object using the following code:
from twisted.internet import reactor
The documentation could perhaps be clearer about this (that document implies, but doesn't directly state, that there's only one reactor instance).
Although I don't see how it matters if it's a singleton or not. :) If it weren't, you could just pass the reactor you were using to the thread that needed to use it.
It matters just because it makes it clear that yes, “from twisted.internet import reactor” really does Just Work, all the time. -Andrew.
On Wed, 5 Nov 2008 17:41:44 +1100, Andrew Bennetts <andrew-twisted@puzzling.org> wrote:
Jean-Paul Calderone wrote:
On Wed, 5 Nov 2008 15:11:08 +1100, Andrew Bennetts <andrew-twisted@puzzling.org> wrote: [...]
Yes, it is a singleton. As <http://twistedmatrix.com/projects/core/documentation/howto/reactor-basics.ht...> says:
You can get to the reactor object using the following code:
from twisted.internet import reactor
The documentation could perhaps be clearer about this (that document implies, but doesn't directly state, that there's only one reactor instance).
Although I don't see how it matters if it's a singleton or not. :) If it weren't, you could just pass the reactor you were using to the thread that needed to use it.
It matters just because it makes it clear that yes, “from twisted.internet import reactor” really does Just Work, all the time.
Sure. But even if it didn't, you'd be alright. Plus, parameterized reactor makes unit testing easier. Jean-Paul
On 01:05 pm, exarkun@divmod.com wrote:
On Wed, 5 Nov 2008 17:41:44 +1100, Andrew Bennetts <andrew- twisted@puzzling.org> wrote:
It matters just because it makes it clear that yes, 1Cfrom twisted.internet import reactor 1D really does Just Work, all the time.
Sure. But even if it didn't, you'd be alright.
Plus, parameterized reactor makes unit testing easier.
As with most things that make testing easier, parameterizing the reactor makes a lot of other things easier too. My long-term plan (very, very long term) is to deprecate and remove the twisted.internet.reactor import and eliminate the singleton. However, that won't happen until you can get the reactor from everywhere interesting that might need it. See ticket http://twistedmatrix.com/trac/ticket/3205 for more information.
On Wed, Nov 05, 2008 at 03:11:08PM +1100, Andrew Bennetts wrote:
The documentation could perhaps be clearer about this (that document implies, but doesn't directly state, that there's only one reactor instance).
I've always struggled with reactor being a singleton. It makes it very difficult to use Twisted on a small scale in a larger existing project. Sometimes it would be really nice to stop a reactor and later start another one. Unit testing is one of several situations where this can be a problem. I suppose this wouldn't be quite as severe of a problem if it were easy to restart a stopped reactor, but even then, it would be nice to be able to have two threads that each have their own reactor. But I guess I'm just ranting now. :) -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On 03:08 pm, amcnabb@mcnabbs.org wrote:
On Wed, Nov 05, 2008 at 03:11:08PM +1100, Andrew Bennetts wrote:
The documentation could perhaps be clearer about this (that document implies, but doesn't directly state, that there's only one reactor instance).
I've always struggled with reactor being a singleton. It makes it very difficult to use Twisted on a small scale in a larger existing project. Sometimes it would be really nice to stop a reactor and later start another one. Unit testing is one of several situations where this can be a problem.
I suppose this wouldn't be quite as severe of a problem if it were easy to restart a stopped reactor, but even then, it would be nice to be able to have two threads that each have their own reactor. But I guess I'm just ranting now. :)
I completely agree, architecturally speaking, for Twisted, that this is a good idea. I already linked to <http://twistedmatrix.com/trac/ticket/3205> in this thread, which is a broad outline of my long-term plan to eliminate the reactor's singleton- ness. On the other hand, the approach you're describing here, using Twisted "just a little", starting up a reactor and spinning it in some obscure corner of an application, is basically an architectural antipattern, which should be avoided as much as possible. One of the hardest things to do on a modern software project is to keep a coherent high-level system design that makes enough sense to introduce new people to it. Concurrency, in particular, is a property which is basically impossible to introduce into a software system without a comprehensive rethinking of how the whole thing works. I can definitely think of a few "larger projects" - names withheld to protect the guilty - whose designs have been influenced and rehabilitated by their decision to use Twisted. I worry about making it easier to throw Twisted into a situation where it will make the overall design of a system into even more of an incomprehensible mess. Just imagine trying to decipher a GUI project using Twisted for networking, where there are 3 or 4 different global reactors, each of which is used by a different window or dialog box, and you have to figure out which one you need based on what context your Twisted-using code is running in. It's nice to be able to say "you can't do that, it's impossible", when someone asks how to set up something like that.
On Wed, Nov 05, 2008 at 07:24:31PM -0000, glyph@divmod.com wrote:
I completely agree, architecturally speaking, for Twisted, that this is a good idea. I already linked to <http://twistedmatrix.com/trac/ticket/3205> in this thread, which is a broad outline of my long-term plan to eliminate the reactor's singleton- ness.
Thanks. I must have missed the link earlier.
On the other hand, the approach you're describing here, using Twisted "just a little", starting up a reactor and spinning it in some obscure corner of an application, is basically an architectural antipattern, which should be avoided as much as possible.
I agree that it should be avoided as much as possible, but I don't think that it should be impossible to do. Suppose, for example, that you are refactoring a project to use Twisted. It's nice to be able to use it just a little, then iteratively add more to it. I agree that you shouldn't stay in this state long, but it sure would be a nice crutch during refactoring.
I can definitely think of a few "larger projects" - names withheld to protect the guilty - whose designs have been influenced and rehabilitated by their decision to use Twisted. I worry about making it easier to throw Twisted into a situation where it will make the overall design of a system into even more of an incomprehensible mess. Just imagine trying to decipher a GUI project using Twisted for networking, where there are 3 or 4 different global reactors, each of which is used by a different window or dialog box, and you have to figure out which one you need based on what context your Twisted-using code is running in.
Give me any API, and I can create a program that misuses it. :) I'm guessing that the GUI project would be much more decipherable if instead of having 3 or 4 global reactors, there were 3 or 4 reactors that were nicely encapsulated. Anyway, if there's a good reason for having 3 or 4 reactors, I'm sure there's a good way to organize the code so that things don't get out of hand. I think the following use cases for mutliple reactors outweigh the potential risks: - starting and stopping a reactor multiple times, particularly for testing - making it easier to refactor existing code to use the reactor pattern - k threads that each have their own reactor And I'm sure we could come up with others.
It's nice to be able to say "you can't do that, it's impossible", when someone asks how to set up something like that.
If I wanted to be constantly told "you can't do that, it's impossible", I wouldn't be a Python guy. If I want to shoot myself in the foot, that's my problem. :) -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On 5 Nov, 09:19 pm, amcnabb@mcnabbs.org wrote:
On Wed, Nov 05, 2008 at 07:24:31PM -0000, glyph@divmod.com wrote:
On the other hand, the approach you're describing here, using Twisted "just a little", ... is basically an architectural antipattern, which should be avoided as much as possible.
I agree that it should be avoided as much as possible, but I don't think that it should be impossible to do. Suppose, for example, that you are refactoring a project to use Twisted.
Sure.
Give me any API, and I can create a program that misuses it. :)
Oh, come on. This is a totally bogus argument, and I suspect you already know that :). First of all, it's not true: in some languages (E comes to mind) it is possible to create APIs which are literally not abusable (for some suitable definition of "abuse", this one about global state included). It's the thing Perl people say when you corner them and make a strong case that Python is objectively more readable than Perl. "Sure, but you can create an unreadable program in any language". That's not the point; the point is, does tool X make it more *likely* that you'll create maintainable programs? Does X help people learn how to create maintainable programs? Twisted should be doing those things.
I think the following use cases for mutliple reactors outweigh the potential risks:
This is almost certainly true, and as I've already said I am keenly interested in de-globalizing the reactor myself; making it easier to test things is a key reason for doing so. However, we should be aware of the risks and try to mitigate them. It could well be that we could come up with an API which is smooth as glass for the "good" use-cases we have in mind here, and raises exceptions or emits helpful warnings when used for the "bad" ones. Maybe that's not possible, maybe we can only get halfway there: but it would be nice to try. In fact, it would be a much better influence on architecture to have intentional, helpful error messages rather than buggy, non-deterministic and unsupported behavior.
- k threads that each have their own reactor
This is a whole other, almost unrelated can of worms. You can only use Twisted from one thread at a time. There are things which make assumptions about non-reentrancy and mutual exclusion of global module- level state. You can find and fix every instance of this if you want, but don't bundle it in with multiple reactors :).
It's nice to be able to say "you can't do that, it's impossible", when someone asks how to set up something like that.
If I wanted to be constantly told "you can't do that, it's impossible", I wouldn't be a Python guy. If I want to shoot myself in the foot, that's my problem. :)
Another bogus argument. There are lots of things that are "impossible" in Python. For example, you "can't" change the values of a tuple, or the contents of a string. I mean, there's terrible stuff you can do with ctypes, but if it breaks, you get to keep both pieces. Similarly, there are lots of things that already kind of work, by accident, if you instantiate multiple reactors; you can kind of re-set a reactor if you know exactly what to twiddle. When I say it's nice to be able to say that something's impossible, that doesn't mean that I don't want to support all the reasonable use-cases, just that boundaries are a useful tool. This is very much the Python idiom - consider your options for indenting your code in strange ways, for example :).
On Thu, Nov 6, 2008 at 3:55 AM, <glyph@divmod.com> wrote:
On 5 Nov, 09:19 pm, amcnabb@mcnabbs.org wrote:
On Wed, Nov 05, 2008 at 07:24:31PM -0000, glyph@divmod.com wrote:
On the other hand, the approach you're describing here, using Twisted "just a little", ... is basically an architectural antipattern, which should be avoided as much as possible.
I agree that it should be avoided as much as possible, but I don't think that it should be impossible to do. Suppose, for example, that you are refactoring a project to use Twisted.
Sure.
Give me any API, and I can create a program that misuses it. :)
Oh, come on. This is a totally bogus argument, and I suspect you already know that :). First of all, it's not true: in some languages (E comes to mind) it is possible to create APIs which are literally not abusable (for some suitable definition of "abuse", this one about global state included).
On the other hand, there's a related invalid argument that gets used a lot by library and framework authors: "if we provide X, people might misuse it, so we should not provide X". This argument is also bogus (that way lies Java). Provide safe, well-documented alternatives for the abusers and let those who want to shoot themselves in the foot do so: sometimes they might actually know better than you. I don't particularly care about the broader issue under discussion, I just wanted to make that point. jml
On Thu, Nov 06, 2008 at 10:48:23AM +0100, Jonathan Lange wrote:
On the other hand, there's a related invalid argument that gets used a lot by library and framework authors: "if we provide X, people might misuse it, so we should not provide X". This argument is also bogus (that way lies Java). Provide safe, well-documented alternatives for the abusers and let those who want to shoot themselves in the foot do so: sometimes they might actually know better than you.
And this is a great way to sum up the point I was trying to make. -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Thu, Nov 06, 2008 at 02:55:37AM -0000, glyph@divmod.com wrote:
This is almost certainly true, and as I've already said I am keenly interested in de-globalizing the reactor myself; making it easier to test things is a key reason for doing so. However, we should be aware of the risks and try to mitigate them. It could well be that we could come up with an API which is smooth as glass for the "good" use-cases we have in mind here, and raises exceptions or emits helpful warnings when used for the "bad" ones. Maybe that's not possible, maybe we can only get halfway there: but it would be nice to try.
I think we really agree for the most part.
- k threads that each have their own reactor
This is a whole other, almost unrelated can of worms. You can only use Twisted from one thread at a time. There are things which make assumptions about non-reentrancy and mutual exclusion of global module- level state. You can find and fix every instance of this if you want, but don't bundle it in with multiple reactors :).
Now that you mention that, I have noticed that a few times when reading Twisted code. I think that long-term, it's nice to remove global module-level state wherever possible, but in the meantime, just putting the above paragraph in the docs as a warning is probably sufficient.
If I wanted to be constantly told "you can't do that, it's impossible", I wouldn't be a Python guy. If I want to shoot myself in the foot, that's my problem. :)
Another bogus argument. There are lots of things that are "impossible" in Python. For example, you "can't" change the values of a tuple, or the contents of a string. I mean, there's terrible stuff you can do with ctypes, but if it breaks, you get to keep both pieces. Similarly, there are lots of things that already kind of work, by accident, if you instantiate multiple reactors; you can kind of re-set a reactor if you know exactly what to twiddle.
Let me rephrase that, then. I think that in Python, the goal is usually to create a flexible API and to document how it should be properly used rather than creating barriers to stop people from doing something just because it might be a bad idea in some contexts.
When I say it's nice to be able to say that something's impossible, that doesn't mean that I don't want to support all the reasonable use-cases, just that boundaries are a useful tool. This is very much the Python idiom - consider your options for indenting your code in strange ways, for example :).
Again, I think we agree for the most part. -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
I just read through this whole thread and don't understand why no one has raised suggested running your "little bit of twisted" in another process. That gives good separation, no multiple thread or singleton issues and should be quite controllable and it breaks any GIL issues that might come up as a freebie too. Of course as you get past dipping your toes in it'll be rather difficult, but so will anything else. -Andy Fundinger On Thu, Nov 6, 2008 at 1:46 PM, Andrew McNabb <amcnabb@mcnabbs.org> wrote:
On Thu, Nov 06, 2008 at 02:55:37AM -0000, glyph@divmod.com wrote:
This is almost certainly true, and as I've already said I am keenly interested in de-globalizing the reactor myself; making it easier to test things is a key reason for doing so. However, we should be aware of the risks and try to mitigate them. It could well be that we could come up with an API which is smooth as glass for the "good" use-cases we have in mind here, and raises exceptions or emits helpful warnings when used for the "bad" ones. Maybe that's not possible, maybe we can only get halfway there: but it would be nice to try.
I think we really agree for the most part.
- k threads that each have their own reactor
This is a whole other, almost unrelated can of worms. You can only use Twisted from one thread at a time. There are things which make assumptions about non-reentrancy and mutual exclusion of global module- level state. You can find and fix every instance of this if you want, but don't bundle it in with multiple reactors :).
Now that you mention that, I have noticed that a few times when reading Twisted code. I think that long-term, it's nice to remove global module-level state wherever possible, but in the meantime, just putting the above paragraph in the docs as a warning is probably sufficient.
If I wanted to be constantly told "you can't do that, it's impossible", I wouldn't be a Python guy. If I want to shoot myself in the foot, that's my problem. :)
Another bogus argument. There are lots of things that are "impossible" in Python. For example, you "can't" change the values of a tuple, or the contents of a string. I mean, there's terrible stuff you can do with ctypes, but if it breaks, you get to keep both pieces. Similarly, there are lots of things that already kind of work, by accident, if you instantiate multiple reactors; you can kind of re-set a reactor if you know exactly what to twiddle.
Let me rephrase that, then. I think that in Python, the goal is usually to create a flexible API and to document how it should be properly used rather than creating barriers to stop people from doing something just because it might be a bad idea in some contexts.
When I say it's nice to be able to say that something's impossible, that doesn't mean that I don't want to support all the reasonable use-cases, just that boundaries are a useful tool. This is very much the Python idiom - consider your options for indenting your code in strange ways, for example :).
Again, I think we agree for the most part.
-- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
-- Blog: http://channel3b.wordpress.com Second Life Name: Ciemaar Flintoff Watch out for the invisible man.
Andrew McNabb wrote:
On Wed, Nov 05, 2008 at 03:11:08PM +1100, Andrew Bennetts wrote:
The documentation could perhaps be clearer about this (that document implies, but doesn't directly state, that there's only one reactor instance).
I've always struggled with reactor being a singleton. It makes it very difficult to use Twisted on a small scale in a larger existing project. Sometimes it would be really nice to stop a reactor and later start another one. Unit testing is one of several situations where this can be a problem.
I suppose this wouldn't be quite as severe of a problem if it were easy to restart a stopped reactor, but even then, it would be nice to be able to have two threads that each have their own reactor. But I guess I'm just ranting now. :)
Oh, I completely agree. I was just talking about the documentation for what we have now. -Andrew. -
participants (8)
-
Andrew Bennetts -
Andrew McNabb -
Andy Fundinger -
George Pauly -
glyph＠divmod.com -
Jean-Paul Calderone -
Jonathan Lange -
Tzury Bar Yochay