Mailman 3 January 2010 - Twisted

[Twisted-Python] Lore to Sphinx Conversion Progress Report 4
by Kevin Horn 22 Jan '10

22 Jan '10

This time I think I'm gonna skip saying how I haven't gotten as much done as I would like...oh darn. Anyways, time for another gripping installment... Progress: - tables are now handled (mostly) properly, thanks to Zeth at http://commandline.org.uk/ - blockquote tags handled - much improved whitespace/indentation handling - some nicer styling thanks to Michael Thompson - I've managed to convert the docs for the 3 Divmod projects with Lore docs, though I've yet to put them up anywhere. Oustanding issues: - two files in the Lore source are not yet being converted, but it looks like one of them is about to be removed (http://twistedmatrix.com/trac/ticket/4188), and it's not really a Lore doc anyways. - due to ReST's insistence on "inline markup" being surrounded by whitespace or certain special characters, there are a lot of places where such inline markup gets jacked up, by not including whitespace in front of it. If I put whitespace in front of everything though, my indentation handling gets jacked up and about 400+ Sphinx build warning result. Not sure if I should spend the time to make whitespace handling really smart or if these should just be fixed manually post-conversion. - cite tags still need handling...not hard, just haven't decided the best way to do it yet. - Themeing/styling: still mostly a TODO, though new styling looks a lot better than the default to my eyes. I'm starting to think that eventually we might want to have 2 themes/styles...one to match the trac-based website, and one for bundled docs (docs tarballs, CHM files, etc.) - auto-generated toctree directives are currently generated in alphabetical order, which makes the "prev" and "next" links mostly make no sense - some of the Lore source files have nested "inline markup", which ReST disallows. This can be handled by: - fix the markup in the Lore source - figure out some kind of supersmart auto-conversion for every possible combination of nesting - just handle the outside level of nesting (what I'm doing now) and fix any problems manually post-conversion. - xhtml entities are not currently resolved...mostly because it makes the build take a LOOOONG time. They can be though. This shouldn't be a problem. - xhtml comments still need to be handled - <code class="API"> tags need something better..right now they are just the same as <code> tags...Sphinx has an upcoming feature coming in 1.0 that would make this nice and maintainable in the long run, but I don't know that I want to wait for it. I may try to "backport" the extension or just come up with a separate solution. - some of the generated links need fixing (e.g. links to directories, .py files) In other news: - Foolscap 0.5 was released today, which made me wonder what they use for docs...and it's Lore. I brought this up on IRC, and it was suggested by many that Lore should stick around even after the conversion according to the standard Twisted compatibility policy, to give anyone who still uses it time to migrate. This sounds like a fine idea to me. Any thoughts? As always, the lore2sphinx code is here: http://bitbucket.org/khorn/lore2sphinx/ And the sample output of the conversion process is here: http://twistedsphinx.funsize.net/ Cheers, Kevin Horn

3 4

Re: [Twisted-Python] LDAP server based on python
by Marten Lehmann 22 Jan '10

22 Jan '10

Hi, I'm refering to an older post of Xavier Dutoit on this list (in early 2008), in which he asked about the state of the ldap-server (ldaptor) plug-in for Twisted. What he needed this for was virtually the same as the way we intend to use it: Our users shall be able to integrate all contacts of their collaboration account in Outlook, Thunderbird, Evolution etc. by LDAP. For easy configuration and security purposes we want to provide a short LDAP tree which returns all the contacts of the user, so he can use his contacts everywhere, not just in the web application of the groupware. The backend is not 100% clear yet, it may be a SQL call to a database, it may be a special search within a less secured master LDAP address book, it may be a REST call returning JSON data or whatever. Since we are doing most of our current work in python, we don't want to manually extend OpenLDAP's sql backend, but rather help ldaptor's ldap server to reach the production level (in case it hasn't done so already). Does someone know about the current state of the ldaptor's ldap server plug-in? Kind regards Marten

1 0

[Twisted-Python] reactor start/stop frustrations in tests
by Brad Milne 21 Jan '10

21 Jan '10

Hi all I know this has been discussed numerous times, but I need some extra help. I have used nose.twistedtools' @deferred decorator to run the reactor within a test method to date. Now I am writing some performance tests for our application and need better control over the reactor. nose has threaded_reactor() and reactor_stop() to control things, but regardless, it still doesn't seem to shut down. I need to shut down the reactor between tests so that requests are dropped (ie get a real representation of completed tasks within test time). Would you recommend trial instead? Many thanks Brad

2 1

[Twisted-Python] HOTDOT , Love the code, I'm curious about your thought's on it's goals and some other solutions with orbited
by adamjamesdrew same 21 Jan '10

21 Jan '10

I c that you have used the embedded morbiq with the restq interface. To replace the morbidQ restq.py would I just create an activeMQ.py factory class? Since twisted is hosting both django and orbited you have access to the session data. If you had an architecture like so what are some strategies for handling security of orbited apache->mod_wsgi->django javascript->orbited->twisted->activemq or rabbitmq Here are some potential use cases. Case #1 Everyone has access to read any Q Write's to the Q are done via a secret account on the server side code can be called an authenticated via standard ajax Case #2 Only logged in users have access to a private Q Write's to the Q are done via a secret account on the server side code can be called an authenticated via standard ajax Thanks a lot

1 0

[Twisted-Python] WSGI Question
by adamjamesdrew same 21 Jan '10

21 Jan '10

I deploy code. Then I modify the code on my file system. The code does not automatically change on the server. It does this on the django dev server. How can I make this happen? I start the server as follows twistd -ny server.py # Django and static file server: root_resource = get_root_resource() root_resource.putChild("static", static.File("static")) http_factory = server.Site(root_resource, logPath="http.log") internet.TCPServer(STATIC_PORT, http_factory, interface=INTERFACE).setServiceParent(serviceCollection)

5 8

[Twisted-Python] WSGI Production ready
by adamjamesdrew same 21 Jan '10

21 Jan '10

Is twisted WSGI production ready as compared to apache and mod_wsgi For a simple server with 4 cores would you simple start 4 instances and load balance between them? How would you recommend a production deploy of twisted wsgi?

3 2

[Twisted-Python] deferring result to PB a callRemote method
by Chris Laws 20 Jan '10

20 Jan '10

> >* > Google is my friend but I could not find examples of this usage. Is *> >* there *> >* > any references to this usage in the twisted docs? *> >* *> >* *> >* Huh, surprisingly not in the primary PB docs. The Twisted “finger” *> >* tutorial *> >* does do this, though, if you read it carefully enough. See finger21.tac in *> >* < http://twistedmatrix.com/documents/current/core/howto/tutorial/pb.html>; *> >* the *> >* remote_* methods of PerspectiveFingerFromService delegate to self.service, *> >* which *> >* is FingerService, which returns Deferreds from its getUser and getUsers *> >* methods. *> >* *> >* *> >* *> It sems like this should be discussed in the PB docs. Maybe someone should > file a ticket.. > > (hint, hint) Done, #4228. Thanks for the informative response to my question. Regards, Chris

1 0

[Twisted-Python] [ANN] Foolscap-0.5.0 released
by Brian Warner 20 Jan '10

20 Jan '10

I'm pleased to announce the release of Foolscap-0.5.0 . This is a fairly minor release: the biggest difference from the previous 0.4.2 is a change to the preferred main entry point: application code should use e.g. "from foolscap.api import Tub" instead of "from foolscap import Tub". A DeprecationWarning will be raised for code that continues to import from the main "foolscap" module, which will become an error in a future foolscap-0.6.0. (this was done to avoid annoying circular dependencies within foolscap itself) In addition, many tools received minor improvements, and the "sets is deprecated in python2.6" warning was fixed. Bugs, docs, discussion: http://foolscap.lothar.com/trac Download: PyPI: http://pypi.python.org/pypi/foolscap/0.5.0 home page: http://foolscap.lothar.com/releases/foolscap-0.5.0.tar.gz Releases are signed with my GPG key 0x1514A7BD, recently renewed (expiration date extended to jan-2011, please re-download from keyservers) Foolscap is a Twisted-friendly remote object protocol, a descendant of Perspective Broker, with improved security properties, third-party references, adaptable serialization, remote logging, and other useful features. Please visit http://foolscap.lothar.com/trac for more details. have a importable day, -Brian * Release 0.5.0 (18 Jan 2010) ** Compatibility The wire format remains the same as in earlier releases. The preferred API import path has changed, see below. ** API changes: import statements, foolscap.api To reduce circular dependencies in Foolscap's internal code, a new "foolscap.api" module has been created. Applications should use: from foolscap.api import Tub instead of e.g. "from foolscap import Tub". Deprecation warnings will be raised for code which imports symbols directly from the "foolscap" module. These warnings will turn into errors in the 0.6.0 release. (see ticket #122 for details) The nearly-useless getRemoteURL_TCP() function was removed. ** setup.py is more windows-friendly The main setup.py script has been modified to use setuptools "entry_points=" on windows, which should help create runnable executables of "flogtool" and "flappserver", with proper extensions. Entry-point scripts are not used on non-windows platforms, but setuptools still creates fairly opaque executable scripts (which makes it hard to figure out that e.g. /usr/bin/flogtool wants to import the "foolscap" module). To get non-opaque scripts, install with "setup.py install --single-version-externally-managed". (#109) ** tool changes *** flappserver "flappserver create" now records the umask value from its environment, and uses it later when the server is started (since normally twistd resets the umask to a very restrictive value). A new --umask argument was added to override this. The server's base directory is chmod go-rwx to protect the private key from other users. The "flappserver start" command uses twisted.scripts.twistd.run(), instead of spawning an intermediate "twistd" process with os.execvp(). This should make things work better in environments where Twisted is not fully installed (especially on windows) and correctly launching "twistd" is non-trivial, such as when some other package is installing it as a setuptools dependency. "flappclient upload-file ~/foo.txt" will use os.path.expanduser() on the filename, even if your shell does not. This should make it easier to use from e.g. buildbot upload commands. (#134) *** logging The "flogtool dump" and "flogtool web-viewer" commands now have a --timestamps argument, which controls how timestamps are expressed (UTC vs localtime, ISO-9601, etc). The web-viewer HTML pages now have more timestamp and sorting options, and hyperlinks to select each. (#100) "flogtool web-viewer --open" will tell your local web browser to open to the correct page, using the Python stdlib "webbrowser" module. "flogtool dump" now emits a better error when told to open a missing file. *** examples Examples of running the Git version-control-system over a flappserver-based secure connection have been added to doc/examples/ . This enables remote-update authority to be expressed as a FURL with no other shell privileges. To accomplish the same with ssh "authorized_keys" command restrictions is annoying and error-prone. See doc/examples/git-proxy-flappclient for setup instructions. This will probably be simplified to a single "git-furl" executable in a later release. The xfer-client/xfer-server/command-client examples have been removed, obsoleted by the flappserver/flappclient tools. ** Other changes The DeprecationWarning for the obsolete "sets" module is now removed on python2.6 (#124) When a getReference() call fails because the remote Tub does not recognize the FURL, it now only emits the first two letters of the secret swissnum in the exception, instead of the whole thing. This reduces information leakage into e.g. stderr logs from a "flappclient --furlfile=X upload-file" command. DeadReferenceError now includes the remote tubid, interfacename, and remote method name of the message that was being sent when the lost connection was discovered, so log.err() calls which record a DeadReferenceError should include this information. This may make it easier to locate the code that provoked the error.

1 0

[Twisted-Python] deferring result to PB a callRemote method
by Chris Laws 20 Jan '10

20 Jan '10

I am writing a tool for work that will run various software development aids such as message capturing and diagnostic control of system processes. My current design strategy is to implement these aids as plugins (not twisted plugins) to a generic plugin runner rather than stand alone applications. The plugins are spawned as separate processes by the plugin-runner and communicate using Perspective Broker during the plugin installation and shutdown phases. Some plugins may take a while to shutdown as they need to close sockets, finalize files, move large files, etc. I would like to instruct the plugin to shutdown and be told when it is finally ready to be shut down. To date all my callRemote methods have effectively returned immediately. For example: def remote_get_name(self): return self.name My question to the list was going to be: Is there a pattern/example I could follow where I could call plugin.callRemote("shutdown") on a plugin and not return a result to the plugin-runner until the plugin has completed all it's, potentially, long running activities? However, while I was trying to write a small code snippet that would demonstrate what I was wanting, I think I got it working. I think the simple answer to my question is to just return a deferred as the result to the callRemote("shutdown") method and trigger it as normal. Google is my friend but I could not find examples of this usage. Is there any references to this usage in the twisted docs? Below (and attached in case the formatting goes screwy) is a short example which emulates a long running shutdown activity performed by the plugin prior to shutdown. It seems to delay the processing of the callRemote("shutdown") result until the plugin has completed it's long running activity. I have omitted the separate process stuff as it didn't seem relevant for this snippet. Is the following code snippet the standard/normal way to defer the return result of a callRemote method call? If this is the normal way, how does triggering the deferred on the plugin (client) side also trigger the same/copy deferred returned to the plugin-runner (server)? Is this PB magic, somehow managing deferreds across the PB interface? Regards, Chris from twisted.internet import reactor, defer from twisted.spread import pb import datetime class PluginClient(pb.Referenceable): """ Plugin client interface exposed to PluginServer's """ def __init__(self, shutdownCallback): self.shutdownHandler = shutdownCallback def remote_shutdown(self): """ Instruct Plugin to shutdown """ print "plugin instructed to shutdown" d = defer.Deferred() self.shutdownHandler(d) return d class PluginServer(pb.Root): """ Plugin server interface exposed to PluginClient's """ def remote_set_client_perspective(self, pluginRef): print "plugin-runner got client reference" reactor.callLater(1.0, self.shutdown_plugin, pluginRef) def shutdown_plugin(self, pluginRef): def pluginShutdownCompleted(result, startTime): endTime = datetime.datetime.now() print "Plugin shutdown took %s to complete." % (endTime - startTime) return result print "plugin-runner asking plugin to shutdown" d = pluginRef.callRemote("shutdown") d.addCallback(pluginShutdownCompleted, datetime.datetime.now()) d.addCallback(self.shutdown) def shutdown(self, _): reactor.stop() def startPluginServer(port): """ Start a plugin communications server """ print "starting server" reactor.listenTCP(port=port, factory=pb.PBServerFactory(PluginServer()), interface='localhost') def startPluginClient(port, shutdownHandler): """ Start a plugin communications client """ def gotServerPerspective(serverPerspective, pluginPerspective): """ Give the plugin-runner this client's perspective """ serverPerspective.callRemote("set_client_perspective", pluginPerspective) return serverPerspective print "starting plugin" client = PluginClient(shutdownHandler) factory = pb.PBClientFactory() reactor.connectTCP(host='localhost', port=port, factory=factory) return factory.getRootObject().addCallback(gotServerPerspective, client) if __name__ == "__main__": port = 42155 def longRunningAction(d, countDown=10): """ Emulate long running shutdown activities """ print "shuting down in %i seconds" % countDown if countDown == 0: d.callback(True) else: countDown -= 1 reactor.callLater(1, longRunningAction, d, countDown) # start plugin-runner reactor.callWhenRunning(startPluginServer, port) # start plugin reactor.callLater(2.0, startPluginClient, port, longRunningAction) reactor.run()

3 2

[Twisted-Python] Proxy support through urllib2?
by Gabriel Rossetti 19 Jan '10

19 Jan '10

Hello everyone, Is it possible to use a proxy with Twisted? I can do this with urllib2 : proxy = urllib2.ProxyHandler({"http": "http://my.proxy.com:8080"}) opener = urllib2.build_opener(proxy) urllib2.install_opener(opener) will that suffice? I'd need SOCKSv5 and HTTP proxy support. Thanks, Gabriel

3 4