Mailman 3 September 2002 - Twisted

Re: [XML-SIG] Re: [Twisted-Python] Can anyone recommend a sensible XML parser for Python?
by Uche Ogbuji 07 Sep '02

07 Sep '02

> Hmm, I know that minidom has had some problems recently, but it has also > seen some good improvements. It sounds like you need more robust DOM > support--have you tried 4DOM? It's not as fast, but it does adhere to > the spec the best. cDomlette's cloneNode does work. If minidom's doesn't, a bug report would be nice. > Maybe (when you have time) if you let us know what > you expect to accomplish we can help out--the people in XML-SIG are some > of the sharpest in the community. Perhaps TREX or RELAX-NG would be more > suitable. I guess the only comforting thing I can say is that every > development community is experiencing growing pains when it comes to an > XML strategy. > > Good luck, > > Eron > > On Fri, 2002-09-06 at 02:39, Glyph Lefkowitz wrote: > > > > On 05 Sep 2002 17:53:09 -0400, Eron Lloyd <elloyd(a)lancaster.lib.pa.us> wrote: > > > Are you referring to PyXML? I know xml.* in the Standard Library is > > > pretty weak by far (but getting better!). > > > > Yes. In fact, PyXML is a big part of the problem. Its "minidom" module, for > > example, is *far* buggier than the one found in the standard library. (As an > > example of that, try to figure out how to make cloneNode work on a Document > > object.) What version of PyXML? > > I could deal with one set of potential problems and pitfalls using XML in > > Python and work around then, but I have to work around every combination of > > versions to make a useful app that doesn't have very stringent installation > > requirements: in pracitice this means 4 environments: python2.1 with pyxml, > > python2.1 standalone, python2.2 with pyxml, python2.2 standalone. > > > > I don't want a plethora of XML parsers with rich features, all of which are > > broken. I want *one* XML parser that can *reliably* transform a stream of > > bytes into a stream of nodes, and a text file into a tree of nodes. You haven't given any evidence to the effect that PyXML does not have this. A bug in cloneNode has nothing to do with parsing. > > You > > mentioned validatation in your post and I explicitly said that validation is > > worse than useless to me; in most cases I want to parse XHTML, which means > > dealing with lots of potentially DTD-violating stuff which is still "valid" as > > far as I'm concerned. Doesn't HtmlParser do the trick? If not, you could try dom.ext.readers.HtmlReader with a minidom implementation used to override the default. BTW, from what you're describing above, you are *not* parsing XHTML. If it violates the DTD, it is not XHTML. Period. Just say you're parsing "HTML" and don't mention a version. That's the only way to say it correctly ;-) > > Eventually I'll clean up the problem cases I'm having and submit them as bug > > reports, but right now it's not worth my time, because I really don't want to > > deal with the fragility of the PyXML or python-standard-library xml.* stuff. Well, no one can tell you what to do with your time, but such general comments are not very useful. It's not as if you posted 10 bug reports, then threw up your hands and said "I'm blowing this joint". You made one vague mention of a cloneNode bug, without even a bare test case. No one gets paid to develop PyXML, but if you come our way a bit, we're quite willing to help. -- Uche Ogbuji Fourthought, Inc. http://uche.ogbuji.net http://4Suite.org http://fourthought.com Track chair, XML/Web Services One Boston: http://www.xmlconference.com/ Basic XML and RDF techniques for knowledge management, Part 7 - http://www-106.ibm.com/developerworks/xml/library/x-think12.html Keeping pace with James Clark - http://www-106.ibm.com/developerworks/xml/libra ry/x-jclark.html Python and XML development using 4Suite, Part 3: 4RDF - http://www-105.ibm.com/developerworks/education.nsf/xml-onlinecourse-bytitl… 1EA5A2CF4621C386256BBB006F4CEC

3 3

[Twisted-Python] Re: Helps if I could type ... there is no reportlacb.com of course.
by Glyph Lefkowitz 06 Sep '02

06 Sep '02

Hi, Laura, Andy, &c :), On Tue, 3 Sep 2002 13:09:28 +0100, "Andy Robinson" <andy(a)reportlab.com> wrote: > Hello Glyph, > > You might be interested to know of our pyRXP > parser: > > http://www.reportlab.com/xml/pyrxp.html Thanks for the reference. I think pyRXP looks pretty neat -- using existing language data structures rather than inventing an API (like DOM) certainly has its advantages. Personally I would have preferred something mutable, but I could see reasons to go either way. The one thing I don't see PyRXP doing is this: > * adhere to a subset of both DOM and SAX APIs for both event-based and > synchronous processing of XML data I shouldn't have phrased this as API adherence but more as a semantic requirement. I want to do parsing and routing of XML such as jabber's "XML streams", as a network protocol and not as a document parser. Of course I also want a document parser, for other purposes. at least passing DOM adherence means that I can rewrite much less of the existing Twisted/XML code :-) -- | <`'> | Glyph Lefkowitz: Traveling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com |

2 2

[Twisted-Python] Web MVC hello world difficulties
by Jonathan Lange 06 Sep '02

06 Sep '02

Hello, I am trying to use Web MVC to create a simple page that has a form with a single text field. When something is entered into the text field, 'Hello World!' should be displayed in the greeting section. I cannot get it to work, the greeting field remains unchanged, and the value in the text field keeps being reset. Below is a pretty simple example that demonstrates my problem. Thanks, jml  <html> <head> <title class="title">Title will go here</title> </head> <body> <h1 class="title"> Title will go here </h1> <p class="greeting">greeting</p> <form action=""> <input type="TEXT" id="name"/> </form> </body> </html> # hello.rpy import helloresource model = helloresource.MHello() resource = helloresource.CHello(model) # helloresource.py from twisted.web import wmvc from twisted.web import domwidgets from twisted.web import domhandlers from twisted.web import domtemplate from twisted.python import domhelpers class MHello(wmvc.WModel): def __init__(self): wmvc.WModel.__init__(self) self.title = "Mezzanine" self.name = "World" self.greeting = "" def updateGreeting(self): self.greeting = 'Hello, %s!' % self.name class VHello(wmvc.WView): templateFile = 'index.xhtml' def setUp(self, request, document): self.model.updateGreeting() def factory_title(self, request, node): domhelpers.clearNode(node) return domwidgets.Text(self.model) def factory_greeting(self, request, node): domhelpers.clearNode(node) return domwidgets.Text(self.model) def factory_name(self, request, node): domhelpers.clearNode(node) return domwidgets.InputText(self.model) class NameHandler(domhandlers.SingleValue): def check(self, request, data): if data: return 1 def commit(self, request, node, data): domhandlers.SingleValue.commit(self, request, node, data) print "data: %s" % data class CHello(wmvc.WController): def factory_name(self, model): return NameHandler(model) wmvc.registerViewForModel(VHello, MHello) wmvc.registerControllerForModel(CHello, MHello)

1 1

[Twisted-Python] Can anyone recommend a sensible XML parser for Python?
by Glyph Lefkowitz 06 Sep '02

06 Sep '02

So, I'm really pretty discouraged and disgusted with the state of XML tools that ship with Python today. Mainly, they do surprising and insecure things when I try to parse XML, and I don't understand how to tell what will and won't work between various versions of them. I think my requirements of an XML parser are pretty simple. Here are the basics of what I want it to do: * adhere to a subset of both DOM and SAX APIs for both event-based and synchronous processing of XML data * allow creation of DOM trees from fragments of an XML stream so that discrete "packets" can be processed, a-la jabber "xml streams" * perform relatively well (optional) More importantly, here are the things I *don't* want an XML parser to do: * validate in any way, ever, at all * fetch DTDs or otherwise do helpful things like eval()ing python code found in random attributes in the node tree * break necessary extensions to SAX/DOM and subtleties of API compatibility between versions, making my code do lots of checks * look for Unicode flag characters * pay attention to !DOCTYPE and ?xml directives * split Text nodes into multiple pieces on newlines or whitespace * pay attention specially to any attribute, like "xmlns" * dump core Does anybody know of an XML parser that meets these requirements or am I going to have to write my own? -- | <`'> | Glyph Lefkowitz: Travelling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com |

2 3

[Twisted-Python] Re: [Twisted-commits] allow for the possibility that a bot's perspective already exists
by Glyph Lefkowitz 05 Sep '02

05 Sep '02

On Thu, 05 Sep 2002 14:49:05 -0500, acapnotic CVS <acapnotic(a)twistedmatrix.com> wrote: > Modified files: > Twisted/twisted/words/service.py 1.63 1.64 > > Log message: > allow for the possibility that a bot's perspective already exists I think this *should* fail if the perspective already exists; otherwise you'd kick a user off if you tried to create a bot with their nick. -- | <`'> | Glyph Lefkowitz: Traveling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com |

2 1

[Twisted-Python] [Fwd: Re: Twisted docs (was Re: Python IPC options)]
by Itamar Shtull-Trauring 02 Sep '02

02 Sep '02

-------- Original Message -------- Subject: Re: Twisted docs (was Re: Python IPC options) Date: 2 Sep 2002 02:51:06 -0700 From: hwlgw(a)hotmail.com (Will Stuyvesant) Organization: http://groups.google.com/ Newsgroups: comp.lang.python References: <1030829572.943495(a)irys.nyx.net> <3D715160.2020109(a)itamarst.org_NOSPAM> <cb035744.0209011308.7b4b3ae2(a)posting.google.com> <3D728386.2060908(a)itamarst.org_NOSPAM> Itamar Shtull-Trauring <itamar(a)itamarst.org_NOSPAM> wrote: > I take it you're going to volunteer some time to write > docs? We'd really appreciate it! ;) Okay I will try, but my TODO list is already very long. So it will be a weekend thingy and could take a while to produce something useful, besides I will have to learn Twisted first! What I propose is to show how to develop a medium sized application in the Twisted framework that uses a lot of Twisted's capabilities. It should have asynchronous *and* synchronous communication, *mobile* components and because of that a dynamic network. It should also show how to exchange data with a server / database via its URL on the internet. Also to be fun it needs a GUI so I will use Tkinter, I believe there are examples already how to combine Twisted with Tkinter. Tkinter seems to be the best choice since it is one of Python's included batteries and it is portable and I have some experience with it. Now what kind of application would that be? My idea for the application is a trader agent and its environment. It could be something like a simulation or a game where you have to program an automated trader agent that buys and sells stocks for you. The client should also have an interactive mode so you can do things manually too. There is a central market / database / server online all the time. First I need to set up a website that simulates a stock market, or maybe I can just use an existing one, have to see how stable the data are they produce. www.nyse.com? Does this kind of application show enough of Twisted's features? You have more ideas? Do you think some things are too hard or impossible? Demonstrated features --------------------- * Asynchronous communication: Updates from the market to the client (if it is online) in a send_and_forget fashion. * Synchronous communication: Buying and selling transactions * Mobility: The clients should be able to connect from anywhere and it should maybe be possible to relocate the market. * Dynamicity: See mobility. Also there will be an unknown number of clients at any time. * RPC: client <-> market, but maybe also perhaps client <-> client for forming trade groups? * GUI integration To show what a client does and to be able to do manual trading. * Remote debugging? Remote market profiling? Market logging? Have to go back to my real work now. What surprises me is that the Twisted people did not come up with a demo apllication like this already, the more I think about it the more fun it seems. I did see a MUD-something-application mentioned on the Twisted site, but could not find it. The trader application looks a bit more ``serious'' perhaps, better for public relations? Also make absolutely sure people using Windows can install everything they need with just doubleclicking an .exe or whatever. You can do it, Twisted already has that for the modules. Writing software for installing things and writing documentation is generally considered *boring* and uncool work among programmers. I know. Hope this helps inspiring you to do it and to look at it in another way? For now good luck with writing good docs about things I do not know about yet: Twisted deserves it. So one day it can become one of Python's included batteries too! :: A child of five could understand this! Fetch me a child of five. -- Itamar Shtull-Trauring http://itamarst.org/ Available for Python, Twisted, Zope and Java consulting

1 0

[Twisted-Python] ReconnectingClientFactory & stopFactory
by Andrew Bennetts 02 Sep '02

02 Sep '02

I think there might be a slight issue with the current client API. If the API is fine, the docs need improving! :) Say you've subclasses ReconnectingClientFactory to connect to a database in startFactory, disconnect in stopFactory. If takes several connection attempts, then multiple connections to the database will be opened and closed (assuming this factory is only constructing one client, which is the usual case, I suspect). The problem appears to be that BaseConnector (in twisted.internet.default) will always call stopFactory from connectionLost/connectionFailed if it's state is 'disconnected', even if it is between retries, because the connector knows nothing of the retry logic in the Factory. Does this sound like a geniune flaw, or am I misunderstanding something? :) I'm wondering if perhaps the solution would be to add a method (or methods) to IConnector that let clientConnectionLost and clientConnectionFailed signal to the connector that it isn't done using that connector yet. -Andrew.

3 4

[Twisted-Python] random security thoughts
by Glyph Lefkowitz 01 Sep '02

01 Sep '02

Hi Twistedistanians, Here is some internal rambling I've been doing about fixing up PB to be more bulletproof. Apologies if it's not coherent to anyone else but I figured I'd better mention the idea on the list and see of someone smarter than me can fill in a gap or two :). So, I think I understand what the potential security problem is with PB -- specifically, it's in Jelly, because the way that types are registered with the serializer is a very primitive ACL. This violates POLA for all PB method calls in a subtle (and usually harmless) way -- the Broker object, which deserializes Jelly types, can effectively receive arbitrary messages from any object on the other end of the connection. pb.Broker is the deputy which can become confused. The trouble is that this is so damn *handy*. If we want to enforce correct authority boundaries here, some form of IDL is necessary, since a method can only (return values, pass arguments) of appropriately declared types, because those types are methods sent to the serializer. Is there some middle ground? Currently there's quite a bit of room for malicious confusion of networked code, since arbitrary types may be passed to arbitrary methods, and there is only one serialization/deserialization registry. However, full IDL would make publishing remote methods and new copyable types rather tedious (and more fragile across different versions of network interface for a given program). You'd also need a full replication of the server-side class hierarchy on the client in order to allow any kind of polymorphism at all. So I would view IDL as correct and complete for POLA purposes, but too tedious to be practical. I figure there's got to be some equivalent here that would be more like UNIX's half-hearted ACLs and less like a superuser bit for various serializeable types. This wouldn't be perfect, but given how little deserialization methods typically do it's probably OK. Unfortunately I can't think of what form such a compromise would take. Easy enough to understand, but still possible to ensure against obvious abuses. Any suggestions? -- | <`'> | Glyph Lefkowitz: Travelling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com |

1 0

[Twisted-Python] Re: [Twisted-commits] fixed connectionLost calls to accept Failures. Read on, I've got gripes.
by Glyph Lefkowitz 01 Sep '02

01 Sep '02

On Sat, 31 Aug 2002 11:54:08 -0500, radix CVS <radix(a)twistedmatrix.com> wrote: > I think all the methods that are returning CONNECTION_* should be _raising_ > them. "Return victorious or not at all". This has the benefit of useful > tracebacks in the Failure objects. Well, it was originally that way. However, it turns out that creating exception objects in Python, generating tracebacks et. al. is rather expensive. Losing connections happens a LOT. Since connectionLost is in the "positive path" of most applications, like, for example, HTTP, raising them creates an even higher overhead for creating/destroying connections. At the time I made this change, I benchmarked twisted.web with a small static page. IIRC there was a 2-3x speedup with doing these as "return" rather than "raise". Perhaps newer releases of Python don't have such a wide disparity, and the HTTP parser is a lot more robust (and attendantly complex) now, so it's possible that this slowdown is now marginal. Still, let's try to avoid using exceptions as a general-purpose control structure. They're for handling errors, not just unusual conditions. (The connection closing is only an error if you don't expect it to close.) -- | <`'> | Glyph Lefkowitz: Travelling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com |

1 0