Mailman 3 October 2004 - Twisted

[Twisted-Python] How to stuff new events into the core event loop?
by Ames Andreas (MPA/DF) 16 Oct '04

16 Oct '04

Hello all, I'm pretty new to twisted activity (just some doc and code reading so far). So please be forbearing if the answers to the following questions are obvious. I'm wrapping some C library which implements some networked client to use it from python, i.e. from a twisted app. Within the guts of this lib there is a thread created (i.e. it is normally never seen by python). This thread manages some heartbeat mechanism. I, as a user of this API, can register a callback with the lib. This callback is called _within the context of this underground thread_, when the connection to the server is lost (as recognised through the heartbeat). Now my first question is, how you would stuff that event (loss of server connection) into the core event loop? One possibility that I see would be to let my registered C-callback signal some sort of a synchronisation item (like mutex or so) whenever the connection to the server is lost. Then I'm using a threaded deferred to block on this synchronisation item, triggering the deferred's callback mechanism when the item gets signaled. But this approach seems very wasteful to me, as I'm using a deferred and a thread for an event that will never happen (my own application is a long running process either). Actually I'm wrapping not only one but two such C libraries. Both implement protocols that are not (yet?) available for python/twisted and that are not trivial (just to illustrate that there are not many alternatives to using these C libs). I'm planning to use both of them through twisted's thread pool mechanism. The only thing is that I would like to have fine grained control over the number of threads used for each of them. Is it possible to create custom thread pools besides the default one? If so, is there anything preventing me from creating a thread pool with only one thread? TIA, andreas

2 1

[Twisted-Python] Making database-code Twisted-friendly
by Thomas Weholt 15 Oct '04

15 Oct '04

Hi, Attached is a database-module I've created and used quite a lot and so far been very happy with. The problem is that it's not Twisted-compatible at all. It blocks and I need something light in my current project to merge a lot of data from one database into another and I'd like to use this library to do it. Perhpas somebody can have use for it as well?? My ideal form of working with databases is shown in the README.txt file. I'd like to be able to something similar using Twisted in the lower layers, not blocking. In short: What I'm looking for is comments on the concept and code in general, but more importantly a simple example on how to write similar code in a Twisted-friendly manner so it doesn't block, or if you're so inclined, feel free to add a Twisted-compatible Connection-object that doesn't block and return data in the same manner as the implemented objects do :-). NB! If my attachment causes problems I'm terribly sorry, but my current webserver is unavailable at the moment so I have no other place to post the code ( except making an entire new project at sourceforge, which seemed like overkill ). Best regards, Thomas

2 2

[Twisted-Python] Twisted in Python STDLIB?
by Ed Suominen 15 Oct '04

15 Oct '04

I'd like to start a discussion on whether Twisted ought to be a part of Python's standard library. I understand some of the developers think it should not, but I personally don't like seeing far inferior networking code sitting there in stdlib. Inclusion in the stdlib carries an implied endorsement, and that endorsement should be going to Twisted, in my view. Has Guido et al. ever expressed any viewpoint on Twisted? It is certainly very Pythonic, and though I have only limited Python and OOP experience to express this viewpoint, it is the best-written Python code I've ever seen. (Now, if only I could say the same for the commenting...) Ed Suominen Registered Patent Agent Open-Source Software Author (yes, both...) Web Site: http://www.eepatents.com

7 10

[Twisted-Python] Merging databases the Twisted way
by Thomas Weholt 13 Oct '04

13 Oct '04

Hi, I got a server running a SQLite-database. Sometimes I need to merge data from another SQLite database into the master. So far I've done blocking old-style read-some from source, write-some to destination stuff using the standard pysqlite-database module, but as mentioned, it blocks the server. AFAIK SQLite only allows one connection to the database as well so that might make it harder to implement a non-blocking solution. If anybody has tips/hints or examples directly related to this or any documentation on DB-operations using Twisted that's a bit more detailed and rich than the one on www.twistedmatrix.com I'd be very grateful. I'm having a hard time getting the hang of how to use twisted and databases in a non-blocking way. Everything seems overly complicated when done the Twisted way. Perhaps it's just me ... NB! I mostly do web-server stuff using SQLite as database. -- Mvh/Best regards, Thomas Weholt http://www.weholt.org

3 2

[Twisted-Python] Atop questions
by Guy Robinson 13 Oct '04

13 Oct '04

I posted this to Divmod mailing list, but perhaps that's not the right place for these sort of questions? Hello, I've just started to play around with Atop. I'm wanting to store data on what could get up to 500,000 unique products. The structure of the data for each item is the same or similar. There will be references between data in different products. Frequent querying of products will occur from the client application. Should I use nested Pools or, SubStores and Pools/items within each substore? From the documentation I can find, the advantages of Pools are easier referencing and indexing, while substores present better security features and scalability(less memory)? The documentation also says substores are used more frequently in most applications. Why? Are there other issues I should consider? Another question :-) A number of the atop examples define Interface classes even though they have no methods. Is this to make upgrading of object schema easier? Why do this? TIA, Guy

2 1

[Twisted-Python] changing password remotely with twisted.conch
by jean-marc.pouchoulon＠ac-montpellier.fr 13 Oct '04

13 Oct '04

Hi, Due to specificity of passwd command, I am not sure it's a goof idea to do that but I try to change the root password on remote machine as I can do it with pexpect The program seems to wait for a password's confirmation. 2004/10/12 13:05 CEST [SimpleTransport,client] starting service ssh-connection 2004/10/12 13:05 CEST [SimpleTransport,client] opening channel 0 with 65536 32768 2004/10/12 13:05 CEST [SimpleTransport,client] sending request for channel 0, request exec 2004/10/12 13:05 CEST [SimpleTransport,client] got extended data 1 'New UNIX password: ' 2004/10/12 13:05 CEST [SimpleTransport,client] got extended data 1 'Retype new UNIX password: ' here it is an extract of my CatChannel(channel.SSHChannel) class. CMD = 'passwd' def channelOpen(self, ignoredData): self.data = '' d = self.conn.sendRequest(self, 'exec', common.NS(CMD), wantReply = 1) d.addCallback(self._cbRequest) def _cbRequest(self, ignored): self.write('myrootpassword') self.conn.sendEOF(self) def dataReceived(self, data): self.write('myrootpassword') self.conn.sendEOF(self) Did anyone done it ? Thanks for you answers. ( and thanks to support my basic english ...)

3 3

[Twisted-Python] Re: [twisted+wxPython] widgets do not work?
by exarkun＠divmod.com 13 Oct '04

13 Oct '04

On Tue, 12 Oct 2004 19:58:38 +0200, Taki Jeden <bartekgorny(a)interia.pl> wrote: > > Hi > > Anybody used wxPython with twisted? > > I started putting together a Twisted-based app with wx GUI, and the widgets > just don't work - some controls do not show up etc. - at least on my > system. > > For example, if I just take the wxdemo.py shipped with Twisted, and add to > it a static text, it doesn't show - while if I change > reactor.registerWxApp(app) > reactor.run(0) > into > app.MainLoop(0) > the text is there. What the heck is wrong with this? Or am I doing something > wrong? Does the same work at yours? > > Bartek > There are several known problems with Twisted/wxPython integration, but this doesn't sound like one of them. Could you include your modified wxdemo.py? Jp

2 1

RE: [Twisted-Python] Twisted in Python STDLIB?
by Sergio Trejo 12 Oct '04

12 Oct '04

Interesting idea, especially when event-based programming is starting to get more attention in the Java world. See this article posted by Cnet today: http://news.com.com/A+new+event+in+programming/2100-1007_3-5406253.html Serg >From: Ed Suominen <general(a)eepatents.com> >Reply-To: Twisted general discussion <twisted-python(a)twistedmatrix.com> >To: twisted-python(a)twistedmatrix.com >Subject: [Twisted-Python] Twisted in Python STDLIB? >Date: Mon, 11 Oct 2004 11:34:34 -0700 > >I'd like to start a discussion on whether Twisted ought to be a part of >Python's standard library. I understand some of the developers think it >should not, but I personally don't like seeing far inferior networking >code sitting there in stdlib. Inclusion in the stdlib carries an >implied endorsement, and that endorsement should be going to Twisted, >in my view. > >Has Guido et al. ever expressed any viewpoint on Twisted? It is >certainly very Pythonic, and though I have only limited Python and OOP >experience to express this viewpoint, it is the best-written Python >code I've ever seen. (Now, if only I could say the same for the >commenting...) > >Ed Suominen >Registered Patent Agent >Open-Source Software Author (yes, both...) >Web Site: http://www.eepatents.com > >_______________________________________________ >Twisted-Python mailing list >Twisted-Python(a)twistedmatrix.com >http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python _________________________________________________________________ Don't just search. Find. Check out the new MSN Search! http://search.msn.com/

4 3

[Twisted-Python] The Anything Box
by William la Forge Jr 12 Oct '04

12 Oct '04

Woops! I forgot to mention the URL: http://compstrm.sourceforge.net Also, here's a snippet from that page describing the project a little better: The Anything Box This is a new kind of database. You can put anything into it, and make any kind of query to access its contents. It works with structures and complexes of structures using very scalable technology. And you can view current state, past state, and changes over time. It is an extensible system that begins with a rich set of metadata for describing data and operations, with operations and data properties grouped into capabilities, which in turn are used to construct data types. The database is implemented over BSDDB, a transactional store that is also used by MySQL, using 14 b-tree files to support a wide range of queries. The basic storage element is (time, fileid, propertyid, oldValue, newValue), but with additional support for document storage. The property/value pairs assigned to a file for a given time are then computed based on a reverse-chronological extraction process. This Anything Box is fully integrated with Twisted, a fast asynchronous server that supports both a web server and a very wide range of additional protocols. The Anything Box itself is implemented using coroutines, making the code more readable/maintainable than most asynchronous styles. It is this coroutine technology, implemented as a computational stream, that the CompStrm project name came from. User Process Management All user processes are coroutines, implemented for fast response rather than throughput. Process management is handled through flow control, with processes automatically terminated when no more output is desired. Any process can interact with its user; loosely coupled (background) processes will be implemented to pend when user input is requested. Persistent Garbage Collection Without some form of garbage collection, applications become much more complex--especially when dealing with complex structures. Unfortunately, garbage collection normally applies only to in-memory (non-persistent) structures. The Anything Box implements persistent garbage collection by using two different kinds of organization. First, data is organized into directories based on data type, each directory having an associated descriptor defining the type and its capabilities. Some data types are then identified as well-known (protected) and are not subject to garbage collection. A second organization is by groups. Data elements with the member capability are always tagged with the id of the id of the lead data element for their group. References to a data element (from another element) are identified as either weak or strong (non-weak). And when a strong reference to a data element from another in the same group is destroyed, then the previously referenced data element is a potential candidate for automatic garbage collection--as all members of a group must be strongly referenced by another element in the same group. (This of course places a design constraint on applications, that weak references be used to avoid any strong cyclic references.) The TKI Shell While the base-level operations are implemented in Python coroutines, the Anything Box includes the TKI shell. TKI commands are operations defined by the Anything Box, which in turn may be implemented as either Python coroutines or TKI scripts. The shell itself is quite simple, supporting argument passing to scripts, the assignment of property values to script variables, and environment variables for navigating space and time and for generating unique names. The richness of the TKI shell comes from the numerous operations defined by the Anything Box, which includes support for set values, lists and dictionaries, and for invoking (mapping) an operation over the contents of a set, list or dictionary.

1 0

[Twisted-Python] The Anything Box
by William la Forge Jr 12 Oct '04

12 Oct '04

A quick update on my twisted project: This is a new kind of database. You can put anything into it, and make any kind of query to access its contents. And you can view current state, past state, and changes over time. This project builds on BSDDB and is fully integrated with Twisted. Thankyou, Bill la Forge

1 0