Mailman 3 September 2002 - Twisted

[Twisted-Python] Re: [XML-SIG] Re: Can anyone recommend a sensible XML parser for Python?
by Uche Ogbuji Sept. 10, 2002

Sept. 10, 2002

> > Accordin to the DOM Level 2 spec: "And, cloning Document, DocumentType, > > Entity, and Notation nodes is implementation dependent." > > This is why standards compliance is not terribly important to me. I would > rather have a useful XML API than a standardized one. Well, what do you think is the most useful behavior of cloning a document? Is it the one I posted in response to thread? If so, don't you think the element of surprise is too great (I'd be surprised … [View More]myself at that behavior)? Wouldn't it be better for Python/XML to offer a *separate*, specialized function for cloning nodes, rather than doing weird things with cloneNode? > > Can you expand a bit more on the actual use case that makes you think you want > > to clone a document node? > > I have a template "frame" document. I want to clone the document, populate it > with information lifted from other XML files, and then write the resultant > (cloned) document out. This is the very first use-case I ever had working with > XML and it is still the most common. I see. It sounds as if a general document duplication function would be of use to you. I agree that this would be useful. I'm willing to write one and add it to xml.dom.ext. But I don't think this is a use case for node.cloneNode. > > We choose not to allow it. Perfectly legal, and I think this is the right > > choice. > > Yes, but the point remains that this *used* to work, and now it *doesn't*. I don't remember. What did it do when it "worked"? > This is functionality I found useful. While I can't comment on the intrinsic > sense or nonsense of cloning document nodes in DOM, I do know that it's > difficult to keep track of when features like this appear and disappear in the > various different XML solutions for Python. Was it ever documented? Every software module has undocumented "features" that you use at your peril. I don't think it's fair to complain when these appear and disappear. Then again, the poor state of PyXML documentation in general weakens that point of mine, doesn't it? Ah well. > Maybe this is the only feature that has done this; I don't know. It just > happens that it's a very commonly-used one for me. > > This is just another instance of my general complaint that tracking versioning > dependencies is not worth the effort for my degenerately simple use-cases for > XML. > > > You mean you can't require, say PyXML 0.8.1? Tough crowd you develop for? > > :-) > > There are still some parties interested in Twisted who are upset that it > requires Python 2.1; in fact, I felt guilty doing 2.1 support because I am > likely going to have to backport portions of it to 1.5.2 for some people. We > can all thank Red Hat for this inane persistence of ancient python versions, > but it is sadly the world I live in. I sympethize. It's largely because of Red Hat that it took us so long to drop 1.5 support in 4Suite. But a couple of months ago we decided it is not worth the developemtn and support overhead and ditched support for all versions before 2.1. I sleep better since then :-) > > > My main frustration is with packaging. > > > Here you have a point. Python, PyXML, and a lot of the related packages move > > very quickly,. and so quickly that they cause all manner of packaging > > problems. > > This is my main point, and this is the one that the PyXML community can do the > least to address. Buggy and idiosyncratic implementations are already in the > wild, and some apps will depend on those particular bugs and idiosyncrasies. > If twisted depends on a new or different set of bugs and quirks, I make it > incompatible with whatever other XML-using applications are out there today. > > Given that XML is an integration technology this is certainly less than > desirable. > > > There is no easy solution to this. > > Having a project that is precipitously approaching 1.0 myself, I can > sympathize. As much as this sort of dependency and compatibility problem has > bothered me, I *know* there will be people that write apps for Twisted and will > curse my name when I enhance some functionality later on :-). > > > I have had it in mind to suggest a PyXML-in-a-tie type effort in the Python > > Business Forum once the effort on Python itself starts to gain legs. I guess > > I can count on you to at least help cheerlead? :-) > > Cheerleading, certainly :-). Although I'm less interested in seeing PyXML > prepared for "business" clients and more interested in just seeing the level of > QA on the volunteer work go up. If I *had* any spare "scarce resources" to > commit beyond my own projects, I would certainly help getting the unit tests > unified and automated. > > > > or produce what amounts to my own `implementation' of an XML parser. > > > > If you try going this route, I guarantee you'll still be trying to get the > > most basic things right six months from now. > > ... > > > > For the applications that I'm intending to write, just doing my own parser and > > > API is both more appealing and more rewarding. > > > > Really? Color me deep skeptical. I have not seen an application on earth > > where implementing one's own parser is a good idea, and precious few where > > implementing one's own API is a good idea. I have a lot of colleagues who > > have tried. > > While it is *possible* that I'm smarter than you think I am, it is certain that > I'm more stubborn. I think you take the wrong gloss on my words. I think Linus Torvalds himself would take years to write a complete and correct XML parser. It's the nature of the beast (XML), not the programmer. I certainly do not consider myself smart enough to take on that dragon. I'm just glad to lean on folk like Clark (and Drake, Evans and co), Garshol and Viellard. > My sophomoric attempt at an XML parser is now in Twisted > CVS. Interesting. So how did you test it? > I've had this objection raised over writing yet another a web server, yet > another remote procedure call protocol, yet another asynchronous socket server > and yet another database interface. It seems like at least some of these ideas > were good ones, so I went ahead and wrote an XML parser and representation > anyway :-). I would rather write a Web server, another RPC, another async socket server *and* another DBMS interface all in a row than just take on the single task of writing an XML parser. And I think I can speak authoritatively, because I *have* implemented all four of those things. > As a data point for this hypothesis, writing the parser and the node tree took > me less than half as much time as writing these posts to various mailing lists > about XML tools (not counting this post, which has been the most > time-consuming): it took less than a quarter as much time as attempting (and > failing) to track down bugs in PyXML, not counting the time I spent trying to > figure out how to turn off undesired features in a way that would work on more > than one version. My two main existing PyXML-using applications are already > ported to this, changing barely any of their code. As I said, I am very skeptical of the result. I'll be impressed when you tell me your home-brew XML parser passes the OASIS conformance suite. Anyway, this is all moot argument. It looks as if you've satisfied yourself for now. Good luck. -- Uche Ogbuji Fourthought, Inc. http://uche.ogbuji.net http://4Suite.org http://fourthought.com Track chair, XML/Web Services One Boston: http://www.xmlconference.com/ Basic XML and RDF techniques for knowledge management, Part 7 - http://www-106.ibm.com/developerworks/xml/library/x-think12.html Keeping pace with James Clark - http://www-106.ibm.com/developerworks/xml/libra ry/x-jclark.html Python and XML development using 4Suite, Part 3: 4RDF - http://www-105.ibm.com/developerworks/education.nsf/xml-onlinecourse-bytitl… 1EA5A2CF4621C386256BBB006F4CEC [View Less]

3 2

[Twisted-Python] Twisted Enterprise ROW
by Sean Riley Sept. 10, 2002

Sept. 10, 2002

A major update to Twisted Enterprise Row is in! The new features include: - support for relationships between database tables. rows from child tables can be loaded automatically when a row from a parent table is loaded. - improved interface for intializing reflectors - refactoring of the DBReflector into Reflector and SQLReflector to allow easier implementation of new types of reflectors - an XML Reflector that uses twisted.persisted.marmalade. - added a ReflectorAuthorizer to dbcred to … [View More]

1 0

Re: [XML-SIG] Re: [Twisted-Python] Can anyone recommend a sensible XML parser for Python?
by Uche Ogbuji Sept. 9, 2002

Sept. 9, 2002

> uche wrote: > > > > For the applications that I'm intending to write, just doing my own parser and > > > API is both more appealing and more rewarding. > > > > Really? Color me deep skeptical. I have not seen an application on earth > > where implementing one's own parser is a good idea, and precious few where > > implementing one's own API is a good idea. > > on the other hand, virtually every commercial XML python user > I know … [View More]

1 0

Re: [XML-SIG] Re: [Twisted-Python] Can anyone recommend a sensible XML parser for Python?
by Uche Ogbuji Sept. 9, 2002

Sept. 9, 2002

> ----Security_Multipart(Fri_Sep__6_22:44:42_2002_475)-- > Content-Type: Text/Plain; charset=us-ascii > Content-Transfer-Encoding: 7bit > > > On Fri, 06 Sep 2002 17:31:51 -0600, Uche Ogbuji <uche.ogbuji(a)fourthought.com> wrote: > > > BTW, from what you're describing above, you are *not* parsing XHTML. If it > > violates the DTD, it is not XHTML. Period. > > > Just say you're parsing "HTML" and don't mention a version. That's the only … [View More]> > way to say it correctly ;-) > > OK, "XML which browsers will render". I am not parsing HTML, in that I won't > accept XML that is not well-formed. I suppose I could try to wrap HtmlParser > with minidom... yuck. Gross, but probably a good idea, come to think of it :) I can't imagine why this would be gross. IMO, it's illustrates very admirable technique, and one of the strengths of Python/XML. The parsing mechanism and the generated representation are independent of each other, so you can mix them and match them in order to take advantage of the most needed features of either. We put a lot of work into making this possible, and I find it very elegant. C++ folks took ages before they cottonned on to such an approach (in the STL), and now it has them in raptures (generic programming is all the rage). Of course, old strait-jacket Java can't touch this. Too bad for them. > The reason I mentioned the cloneNode bug is because it is the most reliable and > the most trivial to demonstrate. Like I said; at some point, I will clean up > my complaints and submit some bug reports. Here's a "bare test case" of that > particular spurious accusation: > > glyph@zelda:~% python > Python 2.2.1 (#1, Aug 30 2002, 09:36:47) > [GCC 2.95.4 20011002 (Debian prerelease)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from xml.dom.minidom import parseString > >>> parseString("<hello_world/>").cloneNode(1) > Traceback (most recent call last): > File "<stdin>", line 1, in ? > File "/usr/lib/python2.2/site-packages/_xmlplus/dom/minidom.py", line 186, in cloneNode > clone = _clone_node(self, deep, self.ownerDocument or self) > File "/usr/lib/python2.2/site-packages/_xmlplus/dom/minidom.py", line 1248, in _clone_node > elif node.nodeType == PROCESSING_INSTRUCTION_NODE: > NameError: global name 'PROCESSING_INSTRUCTION_NODE' is not defined You see, this is why reporting such "bugs" early is helpful. I could have told you ages ago that it is a *bad* idea to call cloneNode on a Document object. Accordin to the DOM Level 2 spec: "And, cloning Document, DocumentType, Entity, and Notation nodes is implementation dependent." IOW, yer gets what yer gets and can't really complain :-) Can you expand a bit more on the actual use case that makes you think you want to clone a document node? I do agree that the confused error message is a glitch. Current PyXML CVS gives a more straightforward "sod off" :-) >>> from xml.dom.minidom import parseString >>> parseString("<hello_world/>").cloneNode(1) Traceback (most recent call last): File "<stdin>", line 1, in ? File "/home/uogbuji/lib/python2.2/site-packages/_xmlplus/dom/minidom.py", line 198, in cloneNode clone = _clone_node(self, deep, self.ownerDocument or self) File "/home/uogbuji/lib/python2.2/site-packages/_xmlplus/dom/minidom.py", line 1454, in _clone_node raise Exception("Cannot clone node %s" % repr(node)) Exception: Cannot clone node <xml.dom.minidom.Document instance at 0x82e9cbc> We choose not to allow it. Perfectly legal, and I think this is the right choice. > In order to do the work I want to do, though, those bug reports aren't going to > help. Even if you resolved every bug report that I submitted within a week, I > would be stuck in the same place I am now: I have to work around the bugs in a > bunch of old versions of PyXML You mean you can't require, say PyXML 0.8.1? Tough crowd you develop for? :-) > or produce what amounts to my own `implementation' of an XML parser. If you try going this route, I guarantee you'll still be trying to get the most basic things right six months from now. > Granted, if I packaged a newer, fixed-up > version of PyXML with Twisted, I wouldn't have to be mucking about with bits > and bytes -- but I *would* have to understand the entire ontology of confusion > associated with cross-language XML APIs. > > My main frustration is with packaging. If all the world were running Debian > unstable, I'd be fine: I'd just say Depends: python2.2-xml >= 0.9. However, > with lots of users in Windows, and many more on other linux platforms with less > pleasant package management, every new package that Twisted requires is another > fifteen minutes that the software takes to get running. It's already confusing > enough to understand it when it *works*; I want the process of getting it > running to be as seamless as possible :). Here you have a point. Python, PyXML, and a lot of the related packages move very quickly,. and so quickly that they cause all manner of packaging problems. There is no easy solution to this. Python is much more of a volunteer community than, say JAva. People work on Python and PyXML mostly to scratch their itches, which means they have less incentive to worry about the packaging mess they leave behind. This is the impetus for the Python-in-a-tie effort for Python proper. I do think we'd make a lot more friends if there were a matching PyXML-in-a-tie. It would mean companies would have to commit scarce resources to freezing interfaces and then testing and packaging to oblivion. I have had it in mind to suggest a PyXML-in-a-tie type effort in the Python Business Forum once the effort on Python itself starts to gain legs. I guess I can count on you to at least help cheerlead? :-) > For the applications that I'm intending to write, just doing my own parser and > API is both more appealing and more rewarding. Really? Color me deep skeptical. I have not seen an application on earth where implementing one's own parser is a good idea, and precious few where implementing one's own API is a good idea. I have a lot of colleagues who have tried. By all means, if you'd like to try, go ahead. > Neither DOM nor SAX will > present an API which allows me to get network XML events in quite the way I > want, so I'm going to have to do some wrapping. I have learned through my own bitter experience that you do not want network interfaces to have *anything* to do with the lexical XML layer (or even Infoset). It is best to design network interactions around *application* level semantics. Basically sending around chunks of XML text is far less hazardous than what I think you mean. > (I do wish pyRXP were > event-based... it's very close, in spirit, to what I want.) If the general > quality of XML parsers in Python were really high, I would regard this impulse > as contrary and counterproductive -- why write my own library for doing this > when perfectly good ones already exist and and are deployed all over the place? Well, as I said, I don't see any evidence that the quality of XML parsers in Python is not high. You pointed out one problem in cloneNode which, from what I gather, was mostly because you're abusing DOM. This had nothing to do with parsing. Are you speaking generically? > So maybe I'm just rationalizing what I would have done anyway. Nevertheless, > it is easier to write my own XML parser than to even properly report the bugs > that I have thus far discovered. I find this claim ludicrous on its face. Writing an XML parser with the compliance level and quality of any of the ones in PyXML takes years. Yes. Years. Feel free to re-learn this fact the hard way, if you wish. > For more information my perception of XML, and why my requirements are as > stripped-down as they are, look at the presentation here: > > http://xmlsucks.org/but_you_have_to_use_it_anyway/ > > (Yes, it's a real URL, and it's not mine.) Yes. I'd guess we've all seen that link. <shrug> So what useful technology doesn't suck? XML works for me. Your mileage may vary. -- Uche Ogbuji Fourthought, Inc. http://uche.ogbuji.net http://4Suite.org http://fourthought.com Track chair, XML/Web Services One Boston: http://www.xmlconference.com/ Basic XML and RDF techniques for knowledge management, Part 7 - http://www-106.ibm.com/developerworks/xml/library/x-think12.html Keeping pace with James Clark - http://www-106.ibm.com/developerworks/xml/libra ry/x-jclark.html Python and XML development using 4Suite, Part 3: 4RDF - http://www-105.ibm.com/developerworks/education.nsf/xml-onlinecourse-bytitl… 1EA5A2CF4621C386256BBB006F4CEC [View Less]

4 3

[Twisted-Python] typed mktap usage.Options parameters
by Kevin Turner Sept. 9, 2002

Sept. 9, 2002

just jotting down this note so it gets saved someplace: There are some very common types of parameters mktap uses -- remote host or IP, port number, local interface to bind to, password. We should have some sort of typed optParameters for this, so that any tap constructor that takes a port can just say "wordsPort isa port" and get back a number instead of having to int() the string and/or look it up in /etc/services, etc. Another idea being that password-type options should be prompted for … [View More]

4 3

[Twisted-Python] caching of row objects
by Sean Riley Sept. 8, 2002

Sept. 8, 2002

Currently, if an application loads a RowObject for a particular row twice, it gets two instances that correspond to the same database row. this has the potential to cause data errors as these instances don't know about each other. One solution is make the reflector know about all the rowObjects loaded (they all have unique key identifiers from the db) and act as a factory - passing out either new objects or already loaded objects instead of duplicates. This is pretty easy to do, but I'm not … [View More]

2 1

[Twisted-Python] relationships in t.e.row
by Sean Riley Sept. 8, 2002

Sept. 8, 2002

a natural extension of twisted.enterprise.row is to add support for relationships between tables. anyone got any good examples of tools or apps that do this well? Some initial thoughts.. Currently, the reflector is initialized with the "stubs" structure that consists of: stubClass, tablename, keycolumns for each (database table / row class). This information could be extended to include some relationship data in the form of foreign keys. This foreign key data would allow relationships … [View More]between tables in the database to be mirrored into the reflector. For example, in the existing row example code the stubs structure is: stubs = [ (RoomRow, "testrooms", [("roomId","int4")]) ] If there was another table "furniture" and a foreign key in that table that connects it to the room table, then that could be specified when initializing the reflector: stubs = [ (RoomRow, "testrooms", [("roomId","int4")]), [] ), (FurnitureRow, "furniture", [("furnId","int4")]), [("roomId", "int4", "rooms", "roomId")] )] where the forth element in each entry is a foreign key of : (local_column_name, column_type, foreign_table_name, foreign_column_name) (could specify the foreign class also, but the reflector should already know which class corresponds to the foreign table.) This foreign key data would then be used when loading row objects from the database. Any time a "RoomRow" object is loaded, it would load all of the FurnitureRows for that room. Although the foreign key data is specified for the child table (furniture) it is when objects of the parent table (testrooms) are loaded that the significant work happens. Allowing an option for child rows to be loaded when loading a row is probably a good idea. This probably requires some changes to t.e.row to work. Currently the optional "factoryMethod" is passed into Reflector::loadObjectsFrom. The factoryMethod will need to be available for children classes also, so it probably should be added to the stubs structure. The "data" passed to loadObjectsFrom is also an issue, maybe the data passed into the highest level loadObjectsFrom is passed down to children, or the parent Row instance that the rows are being loaded for is passed as the data...? Maybe loadObjectsFrom takes a parent object every time it is called - this may even be required. The relationship between rows once they are loaded is another issue. Where are children row objects put once they are loaded? In the above example, maybe the RoomRow object has a list member variable called "furniture" or "children" added to it and the FurnitureRow objects are put in that list? This becomes a type of containment system with a heirarchy of containers... So, loading row objects is doable. I'm not sure of all the implications for other SQL operations. Assuming that all changes to the database are going through the row interface makes things easier, but it requires more thought... A good goal is to be able to operate on these row objects in memory and have the changes propagate into the database in controllable way. ---------------- "If he's so smart, then how come he's dead?", Homer Simspon Sean Riley sean(a)ninjaneering.com [View Less]

2 2

[Twisted-Python] container elements in domtemplate
by Kevin Turner Sept. 7, 2002

Sept. 7, 2002

webquotes.html in the domtemplate howto doesn't cover this case. template: {{{ <foo> stuff <bar> things </bar> </foo> }}} How do I replace "stuff" in the template without clearing the entire foo node, thus destroyng bar and depriving me of the opportunity to insert "things"?

2 1

Re: [XML-SIG] Re: [Twisted-Python] Can anyone recommend a sensible XML parser for Python?
by Uche Ogbuji Sept. 7, 2002

Sept. 7, 2002

> > Uche Ogbuji writes: > > You see, this is why reporting such "bugs" early is helpful. I could have > > told you ages ago that it is a *bad* idea to call cloneNode on a Document > > object. > > > > Accordin to the DOM Level 2 spec: > > > > "And, cloning Document, DocumentType, Entity, and Notation nodes is > > implementation dependent." > > That's no reason to think its a bad idea to implement it or need it, > just … [View More]that you can't rely on it being supported by an arbitrary DOM > implementation. OK. So what should it mean to clone any of these node types? I can't hardly imagine anything that doesn't run into circular madness. I think the DOM WG refused to specify this for good reason. > > I do agree that the confused error message is a glitch. Current PyXML CVS > > gives a more straightforward "sod off" :-) > > Not quite; the previous message would have been raised calling > cloneNode() on a processing instruction as well. Or calling it with > deep=1 on a portion of the tree that contained a processing > instruction. That was a real bug, and not an arbitrary limitation. OK. Glad it's fixed, then. > > We choose not to allow it. Perfectly legal, and I think this is the right > > choice. > > Honestly, I think we should implement cloneNode() for Document, simply > because not doing so seems an unnecessary limitation. It is not for > the library to decide what is right for the application. It's not arbitrary at all. cloneNode is not supposed to alter the ownerDocument: that is for importNode to do. So if you clone a document node, what happens? Do you Create a new document (and thus docType) node and then effectively call importNode on the childNodes? That's the only approach that makes sense to me. Yet it's quite arbitrary and magical. I would rather force people to be clear about what they're doing by manually creating another document and then calling importNode on all the childNodes of the original. So I do not agree that we should support cloneNode for the 4 unspecified node types. > I agree that not supporting it is legal. The exception that is raised > is wrong: it should be xml.dom.NotSupportedErr. There is no stipulation that such an exception should be thrown. The behavior is impl dependent, and I don't see why that doesn't mean the implementation can choose to throw whatever exception it wishes. However, I certainly do not object to a change to throwing xml.dom.NotSupportedErr. I just don't think it's a bug that right now it doesn't. > > This is the impetus for the Python-in-a-tie effort for Python > > proper. I do think we'd make a lot more friends if there were a > > matching PyXML-in-a-tie. It would mean companies would have to > > That would be nice to have. First task: improve & integrate all the > random piles of tests out there! They should all be run when I type > "make check" at the top level, not just a handful. Yes. Build and test farms would be the main engine of such an effort. -- Uche Ogbuji Fourthought, Inc. http://uche.ogbuji.net http://4Suite.org http://fourthought.com Track chair, XML/Web Services One Boston: http://www.xmlconference.com/ Basic XML and RDF techniques for knowledge management, Part 7 - http://www-106.ibm.com/developerworks/xml/library/x-think12.html Keeping pace with James Clark - http://www-106.ibm.com/developerworks/xml/libra ry/x-jclark.html Python and XML development using 4Suite, Part 3: 4RDF - http://www-105.ibm.com/developerworks/education.nsf/xml-onlinecourse-bytitl… 1EA5A2CF4621C386256BBB006F4CEC [View Less]

1 0

[Twisted-Python] 2.3 and an unstable chroot
by Moshe Zadka Sept. 7, 2002

Sept. 7, 2002

Dear root@pyramid, Python2.3's problems with distutils are fixed, so I would like to turn Python2.3 support on for Twisted 0.99.1. We have two options: 1. I can build in the unstable chroot on moshez.org: that will mean wasted upload and download time. 2. We can create an ssh-able unstable chroot on pyramid. Here are the steps to do so: 1. Pick a directory 2. Run debootstrap *woody* 3. Enter the chroot, change apt/sources.list to point to sid and dist-upgrade 4. Install ssh, and configure … [View More]

1 0