Mailman 3 December 2010 - Twisted

Re: [Twisted-Python] [Twisted-web] render_GET and memory consumption
by Glyph Lefkowitz Dec. 24, 2010

Dec. 24, 2010

On Dec 24, 2010, at 11:47 AM, James Y Knight wrote: > On Dec 24, 2010, at 1:13 AM, Glyph Lefkowitz wrote: >> Or maybe we could wrap writev() instead of copying stuff at all? > > What an awesome idea! > > http://twistedmatrix.com/trac/changeset/15847/branches/--omg-optimized-2#fi… Doesn't look like that branch has gotten some love in quite a while :). Are each of the features there (crecv, iovec, cdefer) represented by a different ticket already?

1 0

Re: [Twisted-Python] [Twisted-web] render_GET and memory consumption
by Glyph Lefkowitz Dec. 24, 2010

Dec. 24, 2010

Cross-posting to Twisted core list because this is starting to get into general non-web stuff. Future replies about this fork of the thread should go there. On Dec 23, 2010, at 11:22 AM, exarkun(a)twistedmatrix.com wrote: > On 22 Dec, 06:51 am, glyph(a)twistedmatrix.com wrote: >> >> On Dec 21, 2010, at 2:24 PM, exarkun(a)twistedmatrix.com wrote: >>> Instead, you have to go all the way to producers/consumers, and only >>> write more data to the transport buffer when it has finished dealing >>> with what you previously gave it. >> >> While everybody should of course use producers and consumers, I feel >> like there should be a twisted core ticket for this behavior of >> transport buffering, and a twisted web ticket for this behavior of the >> request buffering. The naive implementation _could_ be much cheaper >> memory-wise; at the very least, twisted.web.static.Data ought to do the >> smart thing. > > Fixing Data sounds like a good idea. I don't know what improvement to > the transport buffering you're thinking of, though. It doesn't seem > like there is an obvious, generally correct fix. Right now, FileDescriptor.write appends its input data directly to _tempDataBuffer. So far, so good: no string mangling. So let's say we do fd.write(header); fd.write(veryBigBody). Then we start writing. FileDescriptor.doWrite comes along and notices that the dataBuffer is empty. The first thing it does in this case: ''.join(_tempDataBuffer); which copies the entire veryBigBody. FileDescriptor is kinda sorta trying to avoid this problem by maintaining an 'offset' so it doesn't need to re-copy dataBuffer; it could use a similar tactic and do write()s out of individual chunks which are greater than SEND_LIMIT directly, rather than coalescing them together. Or maybe we could wrap writev() instead of copying stuff at all?

1 0

[Twisted-Python] Python 2.6 with twisted on OSX
by Eric Dennison Dec. 23, 2010

Dec. 23, 2010

I am planning to teach a Python/Pygame workshop in which students will be (mostly) using Apple hardware and OSX in various flavors. The installation of Python 2.6 and Pygame runs smoothly but I hit a wall when trying to download/install Twisted because no dmg installers are being built now that Twisted is pre-installed on the system Python. Since I don't wish to use the (relatively old) system Python, I would like to figure out how to install Twisted into my Python 2.6 installs. I have had success with building and installing Twisted from sources, but this is not a viable solution for installing to 20 different machines. I have also had success with simply copying Twisted/Zope/SSL directories from the system Python to Python2.6, but I can't help feeling like this is going to come back to haunt me later :) Does anyone have experience with this dilemma or any suggestions for me? I am not experienced with OSX, so there may be some obvious things that I simply don't know about. TIA -Eric

3 3

[Twisted-Python] process output modulation
by David Dec. 21, 2010

Dec. 21, 2010

We are streaming a large amount of program output to the browser via a twisted app, and we are seeing huge memory consumption. We have a database process that generates large amounts of data to stdout, and we are streaming that to the browser through a twisted web2 app. We are using web2 because it supports upload streaming as well. Our code looks like: env = {} input = stream.MemoryStream('') SQLDUMP = '/usr/bin/dump' pstream = stream.ProcessStreamer(input, SQLDUMP, [SQLDUMP,'--format=p','--clean', '--host='+cfg.SOCKET, '--username='+self.role,dbname], env) pstream.run() outstream = WatchedStream(pstream.outStream) response = http.Response( headers=headers, stream=outstream) class WatchedStream(object): def __init__(self,stream): self.stream = stream def split(self, point): ... some implementation def close(self): ... some implementation def read(self): d = self.stream.read() bufSize = sum( [len(b) for b in self.stream.buffer if b]) log.msg('buffer size: %s'%bufSize) return d Watching the log shows us that the stream (a web2.ProducerStream) buffer is growing continuously to hundreds of MB. Doesn't a stream object have a bufferSize attribute and the ability to throttle the flow of data based on buffer fullness? Does that throttling behavior have to be triggered explicitly? Yes, I know that web2 is deprecated, but I don't know that the problem is in the web2 components. The reactor.spawnProcess documentation does not seem to address the matter of modulating the read speed. Any assistance will be appreciated. -- dkeeney(a)travelbyroad.net Rdbhost -> SQL databases as a webservice [www.rdbhost.com]

2 2

[Twisted-Python] Job server
by Umut Aydin Dec. 21, 2010

Dec. 21, 2010

Hello everybody My job server is on progress and maybe you want to check it out. Distribution part is ready but I still work on remote calls. As soon as I get it done I ll write an article about building a job server with twisted framework. (I wrote an article about Apache Mina framework while designing jobbar http://umut.mobi/blog/2010/11/11/how-to-build-a-distributed-job-server-with…) I m pretty busy these days it'll take almost a week more. Here is website of the project http://jobbarserver.org/ By the way, my goal with this project is building a distributed job server and useful extensions for needs of any kind of projects. Some extensions in my mind: Memory Cache, Document Storage, Multiple Job Handler for parallel processing, Log Manager, Data Traffic Analyser If you have any idea please share with me. Thanks -Umut

2 2

[Twisted-Python] ANN: antioch 1.0 developer preview released
by Phil Christensen Dec. 20, 2010

Dec. 20, 2010

Hey all, So, I've recently made big strides with an old project that I've been kicking around ever since working on Divunal back in the Twisted Reality days. Since it uses several Twisted-community projects to solve some problems I've been tinkering with for years, the first place I hoped to recruit interest was among the Twisted community. My goal has been to create a MUD-like environment, particularly one derived from the philosophies of LambdaMOO. However, I also wanted to use modern web technologies to create something that could be more than a text-based environment, though still probably less suited for an immersive 3D environment or some such. I wanted to allow "anyone" to safely develop in-game code using Python, without fear of accidentally creating an infinite loop and taking down the system. Finally, I wanted to ensure that if someone did create a virally popular game, it would be possible to scale the system in ways LambdaMOO never could. Today I released what is best described as a developer preview of the 1.0 release of this software, antioch. It comes somewhat from the lineage of a predecessor named txSpace, but is pretty much a complete rewrite. antioch approaches things with entirely different goals than Twisted Reality/Imaginary did. Like LambdaMOO, it makes very few assumptions about the attributes of your world, assuming it will all be implemented using game verb mechanics. It's primarily concerned with individual user rights, object ownership, and code isolation. Using Twisted, community libraries (such as Nevow, txAMQP and ampoule), PostgreSQL and RabbitMQ, antioch implements a high-performance, scalable virtual world. There's more details on the features below (sorry for the execu-speak, a habit), but to jump right in, head over to: http://philchristensen.github.com/antioch Or sign up for the discussion group/mailing list at: http://groups.google.com/group/antioch-dev I've done my best to create tons of documentation and installation guides, but I'm always adding to the collection. Any and all feedback is always appreciated, - phil antioch feature set -------------------- * **Customizable AJAX-Enabled Client** — The antioch web client provides a standards-compliant interface based on the Nevow Athena AJAX framework, and is supported by all modern browsers. Although the default database provided is geared towards text-based games, the flexibility of the client interface and the power of the plugin system allows many different kinds of games to be developed on the antioch platform. * **Graphical Object Editors** — Programmers and authors alike will find the built-in authoring tools expedite world creation and verb programming. * **Plugin System** — antioch 'modules' allow developers to create new client functionality. The module system was used to implement the built-in object and verb editors, demonstrating the creation of new verb-environment functions, Web resources, and client-side JavaScript. * **Isolated Verb Execution** — Database code is run in individual subprocesses, allowing for memory and runtime limits to be applied at the UNIX process level, and allowing for verb code to be written in a synchronous style. * **Highly Scalable Messaging Layer** — antioch uses the RabbitMQ server to implement message passing support, allowing support of thousands of active clients with minimal latency. * **Powerful Natural-Language Parser** — The built-in interpreter parses a large number of different imperative statements, making user-world interaction a more immersive experience. * **Internal Scripting** — A Pure-Python scripting layer integrates tightly with the framework code, allowing a high degree of customization. * **Robust Networking** — antioch relies on the Twisted Python and Nevow frameworks to implement client-server communication. This robust asynchronous networking library allows client counts to scale easily to hundreds of connections.

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Dec. 19, 2010

Dec. 19, 2010

1 0

[Twisted-Python] py2exe cannot find zope.interface -- fixed
by Ray Cote Dec. 18, 2010

Dec. 18, 2010

We're in the process of upgrading to all the latest Twisted goodness on Windows. All went smoothly until we tried to build Twisted with py2exe. The final executable gave an error that zope.interface could not be found. When we tried adding 'zope.interface' to the includes in setup.py, py2exe threw an error that 'zope' could not be found during the build process. The trick is go to into the installed zope module and add an empty __init__.py file in the zope directory. The build process now recognizes that as a valid module and all is happy. We found a number of people asking about this problem (not necessarily in regards to Twisted) but did not find a good thread with an answer. So, we're posting this in the hope it will save someone an hour or two of problem solving. --Ray -- Ray Cote, President Appropriate Solutions, Inc. We Build Software 603.924.6079

1 0

[Twisted-Python] twisted.words.protocols.jabber and XEP-0138 stream compression
by BURGUIERE Thomas Dec. 16, 2010

Dec. 16, 2010

Hey all, I've search (in Twisted 10.2.0) for a reference to XEP-0138 (stream compression) in twisted.words.protocols.jabber but I don't see anything referring to it. Is there a way to activate stream compression ? (I have a server which have a parameter for zlib stream compression) And if so what do i need to do ? Thanks,

2 2

[Twisted-Python] easy_install and twisted problems?
by Ken MacDonald Dec. 14, 2010

Dec. 14, 2010

I'm trying to use easy_install to load twisted, but I keep getting the following error about not being able to find "vcvarsall.bat". As you can see, I have the .bat and it's in the PATH. I have Visual Studio C++ Express installed, and googled a bunch of references to similar easy_install problems; several suggested installing MingW32's compiler. I tried that, which got rid of the vcvarsall message, but came up with a stack of fatal compile errors. Any suggestions as to what I might be doing wrong with the installation most welcome. Thanks, Ken ================================================================================ C:\Python26\Lib\distutils>vcvarsall.bat Setting environment for using Microsoft Visual Studio 2010 x86 tools. C:\Python26\Lib\distutils>easy_install twisted install_dir c:\python26\Lib\site-packages\ Searching for twisted Reading http://pypi.python.org/simple/twisted/ Reading http://twistedmatrix.com/ Reading http://tmrc.mit.edu/mirror/twisted/Twisted/10.0/ Reading http://tmrc.mit.edu/mirror/twisted/Twisted/9.0/ Reading http://www.twistedmatrix.com Reading http://twistedmatrix.com/products/download Reading http://twistedmatrix.com/projects/core/ Reading http://tmrc.mit.edu/mirror/twisted/Twisted/8.2/ Reading http://tmrc.mit.edu/mirror/twisted/Twisted/8.1/ Best match: Twisted 10.2.0 Downloading http://tmrc.mit.edu/mirror/twisted/Twisted/10.2/Twisted-10.2.0.tar.b z2#md5=73da62c793269eade8121da336b01ba5 Processing Twisted-10.2.0.tar.bz2 Running Twisted-10.2.0\setup.py -q bdist_egg --dist-dir c:\users\kmacdo~1\appdat a\local\temp\easy_install-dip2sj\Twisted-10.2.0\egg-dist-tmp-ctzrhl *error: Setup script exited with error: Unable to find vcvarsall.bat * C:\Python26\Lib\distutils>

2 2