Kickin' it IRC style --
Here are some responses interspersed with the dialogue. Hope this helps :)
On Sat, 28 Jul 2001, Kevin Turner wrote:
<itamar|home> you coifgure them and then can pickle them to disk
... OR you run them, depending on your requirements. The default 'twistd' binary does UNIX daemonization; but if you wanted to do an NT service, you'd obviously have to have some different code.
<Tv> Acapnotic: But all that does is make your "config settings" into "command line settings".
It makes your *initial* "config settings" into "command line settings". Those scripts exist more for examples than for actual configuration; the "settings" they provide are a small subset of the full set available in the code.
<Tv> I don't see it making things very easy. <Tv> And think about version upgrades.
We here at Twisted Matrix Laboratories have a commitment to backwards compatibility :-). *IF* we had more users, the 0.9.0 upgrade would have gone much more smoothly; we actually had a backwards compatibility plan and decided to forego it for efficiency because we (rightly) believed that nobody would care.
My configuration (e.g. the www.twistedmatrix.com site) is fairly complex, and I've been maintaining it with Twisted since the inception of Twisted.Web (which I think was .7 or so). It's never taken me more than 3 or 4 minutes to replicate this with simple python scripts.
<Tv> I see trouble with the approach; mostly losing the unix ubiquitousness (sp?). I can grep my config files; I store them in CVS.
1: learn how to use add +kb :-)
2: use pickle's "text" mode.
3: version control a script which generates and runs your monumentally byzantine configuration, then store THAT in cvs.
About option 3: radix, moshez, and I have been thinking about, and prototyping configuration systems for quite a while. I have yet to think of a system which can't be easily beaten by "just using Python". For example, here's an example XML solution;
<Site name="myvhost"> <NameVirtualHost> <Host name="myhost.com"> <Resource> ... </Resource> </Host> </NameVirtualHost> </Site> ... <WebServer port=80 site="myvhost"/>
And here's the actual Twisted code which will do the same thing (I haven't had an opportunity to test it, but you get the idea):
--- from twisted.web.server import Site from twisted.web.vhost import NameVirtualHost from twisted.internet.main impot Application a = Application("web") v = NameVirtualHost() r = SomeResource(...) v.addHost("myhost.com", r) s = Site() s.resource = v a.listenOn(80, s) ---
It's equally concise *and* more flexible.
<Tv> And don't give a trivial example.. <Tv> I have multi-hundred-line config files here. <Tv> Shared between n hosts, via CVS. <Tv> I'd love to hear more of the twisted philosophy..
If you have multi-hundred-line configuration files some of which allow programmability), I'm guessing that you have considerable applications that are written in buggy half-thought-out programming languages with no module systems and you're using version control as part of a crutch to make up for their many and varied deficiencies.
Since Twisted will allow you to use Python (or save your configuration in Pickles, which is usually more convenient -- keep in mind I use this software myself; pickles are how I use it -- to configure your servers, you can encapsulate parts of your configuration or segment it however you like. and VC it too.
<Tv> Personally, I have _needed_ programmability in config files many times.
And you've got it ;)
<Tv> Acapnotic: Oh, I like version control. I wish I could put more things into it.
Hopefully jasbahr will jump in here with a little comment about potential future ideas for version control within Twisted itself... ;-)
<Tv> Have you ever screwed up a config file? <Tv> Have you ever downgraded something, and found out it won't understand the new config file, and had to recreate the config to match the older version? <Tv> The same reasons you version control source code.
<Acapnotic> tv: you should post to the list at twisted-python@twistedmatrix.com <Tv> Acapnotic: I'm not following any more lists until I have migrated my mail to another domain :) <Acapnotic> tv: bah, you don't have to follow it, just post ;)
I would *really* appreciate it if more people took advantage of resources like the mailing list, and my personal email address :-), to resolve issues like this. Thanks, kevin, for posting this log!
<Acapnotic> Tv: so if you want programmable config, is there a problem with just using python? In which case your configuration process is a script that does "app = pickle.load(...) app['web']. port = 8080 ; app.append(ChickenServer()) ; ... ; pickle.dump(app, ...) ? <Tv> Acapnotic: That may very well be what I'd do. But then the mechanism needs to support _both_ getting config programmatically and state from pickle, at the same time.
Just make your "configuration file" as described, except put "app.run()" at the end of it. This lets you do what "configuration" really is; e.g. to make a customized version of the application for your site.
<Acapnotic> Well, if I follow this train, there is only the state of the application, since it is persistant. That state may be changed or examined programmaticly.
It's *optionally* persistent.
The point of Twisted's configuration philosophy is that "configuration" is not really a disjoint set of options, but rather the custom development of a personal application. Twisted provides you with a big pile of very well-developed pieces (so you hopefully don't have to spend time fixing the little pieces while you're building your big one), but you still have to put them together somehow. The mk*server scripts provide one way to do that; hopefully matsaleh's upcoming configuration protoserver will allow it to be done interactively so that users can discover more aspects of the software.
In eschewing the ubiquity of "configuration files" on UNIX, twisted also eschews the necessity of dedicating a large body of code to parsing, interpreting, and understanding an underpowered mini-language designed only for talking to it. "Configure" with python. "Script" with python. "Code" with python. "Debug" with python.
The twisted philosophy is that all of these tasks are inherently the same, and are all complex enough to warrant a full-powered programming language. Event-based dynamic modification of a stateful system is what object orientation was originally intended to do.
<Tv> I see something beautiful there, but.. <Tv> noone has yet explained persistence to me well enough to convince me. <Acapnotic> but what happens when you upgrade? heh. <Tv> What if a buggy program corrupts its state? <Tv> Then you lose your config file also.
This assumes you don't back up your original config file. Why would you throw it away? If it were complex, I'd keep mine in CVS :)
<Tv> There needs to be something else. <Tv> Something that is less volatile. <DavidC_> what do you mean? both config and state are the same thing. <Tv> No. <Tv> config changes when *I* say it changes.
Config is *initial* state. e.g. the state that you start the server up with; so it changes whenever the program changes. Or does anything. You can always stop the server and start from scratch again, but that doesn't mean that bugs won't affect you.
<itamar|home> nah, config is parsed and then "rendered" into state <Tv> program can change it's state as it wants. <Tv> Any bug in program must never change the port to bind to.
If the bug would change the persistent port, then it would change the runtime port. That seems like it would be pretty bad too :)
<Tv> :) <Tv> I see config as sort of a factory for initial program states, if you want to put it that way.
If one were to believe there is something beyond this that describes "configuration", then they would be confused :-). The *state* of the server changes whenever it executes an action that causes state to change; the difference between Twisted and other systems is that Twisted *can* remember these changes to state automatically. This state will not replace your initial state, nor will it prevent you from writing scripts which create the initial state each time that the server is run.
The reason I suggest the pickle-the-state approach is because if you make a change to the "configuration" as the server is running, you can change it *live*, test it, etc, and know that when the server comes up next time, it'll start the same way, rather than having to change your "factory", shut down your server and start it again, or make the change to your factory and your runtime state concurrently and take the risk that they'll be different.
<itamar|home> lets say by mistake someone does "del serverInstace. port" <itamar|home> and then shuts down the server <itamar|home> the pickled state is now wrong
The pickled state that the server saved *when it shut down* is now wrong. The initial state (what *I* consider the "configuration") is still correct; but you now have the corrupt state to study as well (so you can figure out what happened with information beyond the log.)
<DavidC_> well that would be a bug. <itamar|home> the point is that if you have an external, version controleld config file <itamar|home> these bugs don't bite you, or even if they do at least they don't effect the configuration
In conclusion, "so do that then" :).
I still don't understand how a pickled file isn't "external". It's not like it gets mmap'd when you start the Python process and then automatically overwritten after every action that happens :)
______ __ __ _____ _ _ | ____ | \_/ |_____] |_____| |_____| |_____ | | | | @ t w i s t e d m a t r i x . c o m http://twistedmatrix.com/users/glyph