Re: [Twisted-Python] Twisted Plugins - Implementation Discussion

On Apr 7, 2011, at 1:08 AM, Tim Allen wrote:
If you need a non-Turing-complete config language and rule out .ini and XML, I'm not sure what's left. JSON, perhaps.
I bet a lot of people have a deja-vu feeling about a config file syntax debate so I'll propose an alternative approach: RDF. Perhaps most people in this community will not like it, yet some might find it more fun that revamping their 2002 arguments about merits and pitfalls of various syntaxes. One of the reasons why I like RDF so much is that I can focus on what I need to express and let people pick the serialization syntax that better suits their mood, habits, tools and use-cases. I know that the use-case that's being discussed is slightly different (config files for the plugin system as opposed to config files for a specific plugin) but as an example: my twistd-plugin-driven webserver will gladly accept any of the attached configuration files, they are equivalent and there are commonly available tools to switch back and forth, including pure python ones. It could as well accept any other standard RDF serialization syntax, for example there are several other XML formats, a line-based grep-friendly syntax (NTriples) and a JSON format. The code that parses this and turns it into running twisted Services and web applications is about the same size of your average TAC file. If anybody wants to see it please email me privately, I'm not proud enough of other parts of my open source project containing it to advertise it on this list. Other than mentioning that RDF also comes with a standard query and update language and protocol (SPARQL), I won't enumerate other advantages here so let's see what some of drawbacks (and their counter-arguments) are: 1) it's not widely known yet (but so was XML in 2000 and JSON in 2002 and INI in 2011) 2) it would require to add a dependency for an RDF parser (people often argued the same way about XML, remember when libxml2 became a Gnome dependency? RDF is now becoming a requirement of Gnome and KDE...) 3) it's not python (yet the several python object-RDF-mapper libraries available seem to me much easier to use and way more simple than SQLAlchemy so I already switched from pickle to RDF whenever I want to serialize some object graph, BTW it's also safer and hand-editable) Sorry if I went too off-topic, ciao ste

On 2011-04-07, Stefano Debenedetti wrote:
On Apr 7, 2011, at 1:08 AM, Tim Allen wrote:
If you need a non-Turing-complete config language and rule out .ini and XML, I'm not sure what's left. JSON, perhaps.
I bet a lot of people have a deja-vu feeling about a config file syntax debate so I'll propose an alternative approach: RDF.
I am +1 on this idea. I like rdf. My question is now: is there an rdf parser lib that is available on python2.4+ which can either be gently embedded within twisted, or used as a dependency? We don't really need SparQL or anything complicated, just the ability to resolve some simple triples. I do not like angle brackets, but I have always had a fond affection for n3. -- Regards, Stephen Thorne Development Engineer Netbox Blue

On Apr 7, 2011, at 7:54 PM, Stephen Thorne wrote:
On 2011-04-07, Stefano Debenedetti wrote:
On Apr 7, 2011, at 1:08 AM, Tim Allen wrote:
If you need a non-Turing-complete config language and rule out .ini and XML, I'm not sure what's left. JSON, perhaps.
I bet a lot of people have a deja-vu feeling about a config file syntax debate so I'll propose an alternative approach: RDF.
I am +1 on this idea. I like rdf. My question is now: is there an rdf parser lib that is available on python2.4+ which can either be gently embedded within twisted, or used as a dependency?
We don't really need SparQL or anything complicated, just the ability to resolve some simple triples.
I do not like angle brackets, but I have always had a fond affection for n3.
You're welcome to try and do this; I'm not particularly interested in blocking it or holding it up, but I don't think that changing the input format actually solves any real problems. I guess I will hold it up if you can't convince me that I'm wrong about that, and demonstrate an actual problem that it solves :-). You still have to define all the same classes in order to get a plugin, unless we change some of that too - which has nothing to do with the metadata format at all. I think the way to avoid caching issues in general is to generate the packaging metadata from the source earlier in advance (i.e. at development time, and check it in with the source code, like you would do with a Cython-generated C file or something), not to just mess around with it in a text editor. I think that there is a benefit to sticking with a format that people very much dislike editing. Having separately manually-edited metadata introduces an opportunity for the metadata to diverge from the reality of the code. Making this easy to edit manually means making it more likely that people will think that they need to introduce some manual tweaks. If it's a huge pain to actually generate the metadata without running a tool that inspects the code, it's less likely that someone will feel the need to get clever. -glyph

On 12:09 am, glyph@twistedmatrix.com wrote:
On Apr 7, 2011, at 7:54 PM, Stephen Thorne wrote:
On 2011-04-07, Stefano Debenedetti wrote:
On Apr 7, 2011, at 1:08 AM, Tim Allen wrote:
If you need a non-Turing-complete config language and rule out .ini and XML, I'm not sure what's left. JSON, perhaps.
I bet a lot of people have a deja-vu feeling about a config file syntax debate so I'll propose an alternative approach: RDF.
I am +1 on this idea. I like rdf. My question is now: is there an rdf parser lib that is available on python2.4+ which can either be gently embedded within twisted, or used as a dependency?
We don't really need SparQL or anything complicated, just the ability to resolve some simple triples.
I do not like angle brackets, but I have always had a fond affection for n3.
You're welcome to try and do this; I'm not particularly interested in blocking it or holding it up, but I don't think that changing the input format actually solves any real problems. I guess I will hold it up if you can't convince me that I'm wrong about that, and demonstrate an actual problem that it solves :-). You still have to define all the same classes in order to get a plugin, unless we change some of that too - which has nothing to do with the metadata format at all.
I agree. I don't see how this addresses any of the problems with the current system which have been raised so far. Jean-Paul

On 2011-04-07, Glyph Lefkowitz wrote:
I am +1 on this idea. I like rdf. My question is now: is there an rdf parser lib that is available on python2.4+ which can either be gently embedded within twisted, or used as a dependency?
You're welcome to try and do this; I'm not particularly interested in blocking it or holding it up, but I don't think that changing the input format actually solves any real problems. I guess I will hold it up if you can't convince me that I'm wrong about that, and demonstrate an actual problem that it solves :-). You still have to define all the same classes in order to get a plugin, unless we change some of that too - which has nothing to do with the metadata format at all.
I think the way to avoid caching issues in general is to generate the packaging metadata from the source earlier in advance (i.e. at development time, and check it in with the source code, like you would do with a Cython-generated C file or something), not to just mess around with it in a text editor.
I think that there is a benefit to sticking with a format that people very much dislike editing. Having separately manually-edited metadata introduces an opportunity for the metadata to diverge from the reality of the code. Making this easy to edit manually means making it more likely that people will think that they need to introduce some manual tweaks. If it's a huge pain to actually generate the metadata without running a tool that inspects the code, it's less likely that someone will feel the need to get clever.
I was just thinking about this. It would be very easy to write a single twisted/plugins/rdf_plugins.py file that scans for non-python metadata defined plugins and creates them. That way twisted doesn't need to depend on an RDF lib, and this can be a 'third party' outside-of-twisted package that if you want to use, you just specify it as a dependancy along with the rest of the things that your project depends on. ... I like this idea for a variety of reasons. -- Regards, Stephen Thorne Development Engineer Netbox Blue
participants (4)
-
exarkun@twistedmatrix.com
-
Glyph Lefkowitz
-
Stefano Debenedetti
-
Stephen Thorne