Mailman 3 August 2009 - Twisted

Re: [Twisted-Python] Deferred documentation rewrite
by Andrew Francis 05 Aug '09

05 Aug '09

Hi Edward: Message: 2 Date: Mon, 03 Aug 2009 18:00:07 -0400 From: "Edward Z. Yang" <ezyang(a)MIT.EDU> Subject: Re: [Twisted-Python] Deferred documentation rewrite To: Twisted general discussion <twisted-python(a)twistedmatrix.com> Message-ID: <1249336297-sup-7263@javelin> Content-Type: text/plain; charset=UTF-8 >Why callbacks? > - Asynchronous interaction to synchronous interaction > - Delocalized execution (the parser example) > - High level functions in Python review I don't know if this is an aid but here goes: When I was preparing my Pycon 2008 talk, amongst the many papers I read, was the Doug Schmitt's concurrency design pattern papers including the original Reactor Pattern paper. I also read the Microsoft "Cooperative Task Management without Manual Stack Management Event-driven Programming is Not the Opposite of Threaded Programming" paper. In the light of those papers, this is the way I see things: 1) Reactors provide a portable form of non-preemptive multitasking. By implication, reactors are schedulers. 2) If you buy Reactor as scheduler, then Deferreds can be viewed as representing a thread (or chain) of execution, each callback analogous to a continuation - the next address to resume execution when a result is ready. 3) One of the main differences between asychronous and synchronous processing becomes who is responsible for setting up shared state between the links in the execution chain. >Quite frankly, I'm stumped on "defining synchronous and asynchronous." Simple definition: In a synchronous call, the caller blocks until a result is ready. Upon return the next statement is executed (barring something like an exception). In an asynchronous call, the caller does not wait for a result and continues. > I just don't know what direction people are coming from. I would suggest most people are trying to solve simple problems and want the least surprise. Unfortunately asynchronous programming has lots of surprises. In case of terminology, try looking up how terms like 'asynchronous' or 'synchronous' as used in a few of the more popular network programming books. Cheers, Andrew

2 1

[Twisted-Python] OS for Twisted Framework to run on
by vitaly＠synapticvision.com 04 Aug '09

04 Aug '09

hi, is there any significant reason/advantage to run the Twisted Framework on Linux or on FreeBSD? Appreciate the help.

2 1

[Twisted-Python] Twisted.web.client.getPage with proxy?
by disappearedng 04 Aug '09

04 Aug '09

Hello world, 1) I have a proxy server running on my computer on port 8888. It's listening on localhost. I am wondering whether Twisted.web.client.getPage has some sort of proxy kwargs that can do this for me. ( I checked the source and I really doubt it has something like that) What are my alternatives? I have looked at low-level alternatives, and ProxyClient doesn't really seem to fit my situation. 2) I sometimes have 2 proxy servers running. I know urllib has a method which allows you to build opener, then you can access webpages with opener.open('site').read(). Does twisted have something like this which allows me to keep multiple "proxy space" to access different websites at the same time on different proxy servers? Thx

1 0

[Twisted-Python] Questions about adding documentation
by Jessica McKellar 03 Aug '09

03 Aug '09

Hi everyone, I would like to help Twisted by adding documentation or reviewing existing documentation. However, I don't see many (any?) unassigned tickets regarding documentation of specific items, and because I am quite new to Twisted this makes it hard for me to determine where you wish I would focus my attention. Where should I focus my attention? Want to open some tickets for me to claim? Is adding to API docs more important than updating the examples and howtos? It looks like to add to the API docs you just update the doc strings for functions and let someone let pydoctor do its magic later. Is that true? Is there a special Twisted pydoctor incant to see how they'd look on the web before doing any hasty patch-submitting? I'd like to update http://twistedmatrix.com/trac/wiki/ContributingToTwistedLabs to include some of this information as well as help I got earlier for updating xhtml documentation. Thanks, -Jessica

15 40

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 02 Aug '09

02 Aug '09

1 0

[Twisted-Python] ANNOUNCING Tahoe, the Lofty-Atmospheric Filesystem, v1.5
by Zooko Wilcox-O'Hearn 02 Aug '09

02 Aug '09

Dear people of the twisted mailing list: This release announcement is relevant to this list because Tahoe-LAFS is built on Twisted's networking and concurrency framework. Also, we strive to emulate the UQDS. Regards, Zooko The Tahoe-LAFS team is pleased to announce the immediate availability of version 1.5 of Tahoe, the Lofty Atmospheric File System. Tahoe-LAFS is the first cloud storage technology which offers security and privacy in the sense that the cloud storage service provider itself can't read or alter your data. Here is the one-page explanation of its unique security and fault-tolerance properties: http://allmydata.org/source/tahoe/trunk/docs/about.html This release is the successor to v1.4.1, which was released April 13, 2009 [1]. This is a major new release, improving the user interface and performance and fixing a few bugs, and adding ports to OpenBSD, NetBSD, ArchLinux, NixOS, and embedded systems built on ARM CPUs. See the NEWS file [2] for more information. In addition to the functionality of Tahoe-LAFS itself, a crop of related projects have sprung up to extend it and to integrate it into operating systems and applications. These include frontends for Windows, Macintosh, JavaScript, and iPhone, and plugins for duplicity, bzr, Hadoop, and TiddlyWiki, and more. See the Related Projects page on the wiki [3]. COMPATIBILITY Version 1.5 is fully compatible with the version 1 series of Tahoe-LAFS. Files written by v1.5 clients can be read by clients of all versions back to v1.0. v1.5 clients can read files produced by clients of all versions since v1.0. v1.5 servers can serve clients of all versions back to v1.0 and v1.5 clients can use servers of all versions back to v1.0. This is the sixth release in the version 1 series. The version 1 series of Tahoe-LAFS will be actively supported and maintained for the forseeable future, and future versions of Tahoe-LAFS will retain the ability to read and write files compatible with Tahoe-LAFS v1. The version 1 series of Tahoe-LAFS is the basis of the consumer backup product from Allmydata, Inc. -- http://allmydata.com . WHAT IS IT GOOD FOR? With Tahoe-LAFS, you can distribute your filesystem across a set of servers, such that if some of them fail or even turn out to be malicious, the entire filesystem continues to be available. You can share your files with other users, using a simple and flexible access control scheme. We believe that the combination of erasure coding, strong encryption, Free/Open Source Software and careful engineering make Tahoe-LAFS safer than RAID, removable drive, tape, on-line backup or other Cloud storage systems. This software comes with extensive tests, and there are no known security flaws which would compromise confidentiality or data integrity in typical use. (For all currently known issues please see the known_issues.txt file [4].) LICENCE You may use this package under the GNU General Public License, version 2 or, at your option, any later version. See the file "COPYING.GPL" [5] for the terms of the GNU General Public License, version 2. You may use this package under the Transitive Grace Period Public Licence, version 1 or, at your option, any later version. (The Transitive Grace Period Public Licence has requirements similar to the GPL except that it allows you to wait for up to twelve months after you redistribute a derived work before releasing the source code of your derived work.) See the file "COPYING.TGPPL.html" [6] for the terms of the Transitive Grace Period Public Licence, version 1. (You may choose to use this package under the terms of either licence, at your option.) INSTALLATION Tahoe-LAFS works on Linux, Mac OS X, Windows, Cygwin, Solaris, *BSD, and probably most other systems. Start with "docs/install.html" [7]. HACKING AND COMMUNITY Please join us on the mailing list [8]. Patches are gratefully accepted -- the RoadMap page [9] shows the next improvements that we plan to make and CREDITS [10] lists the names of people who've contributed to the project. The Dev page [11] contains resources for hackers. SPONSORSHIP Tahoe-LAFS was originally developed thanks to the sponsorship of Allmydata, Inc. [12], a provider of commercial backup services. Allmydata, Inc. created the Tahoe-LAFS project and contributed hardware, software, ideas, bug reports, suggestions, demands, and money (employing several Tahoe-LAFS hackers and instructing them to spend part of their work time on this Free Software project). Also they awarded customized t-shirts to hackers who found security flaws in Tahoe-LAFS (see http://hacktahoe.org ). After discontinuing funding of Tahoe-LAFS R&D in early 2009, Allmydata, Inc. has continued to provide servers, co-lo space and bandwidth to the open source project. Thank you to Allmydata, Inc. for their generous and public-spirited support. This is the second release of Tahoe-LAFS which was created solely as a labor of love by volunteers; developer time is no longer funded by allmydata.com (see [13] for details). Zooko Wilcox-O'Hearn on behalf of the Tahoe-LAFS team Special acknowledgment goes to Brian Warner, whose superb engineering skills and dedication are primarily responsible for the Tahoe implementation, and significantly responsible for the Tahoe design as well, not to mention most of the docs and tests. Tahoe-LAFS wouldn't exist without him. August 1, 2009 Boulder, Colorado, USA P.S. Just kidding about that acronym. "LAFS" actually stands for "Lightweight Authorization File System". Or possibly for "Least-Authority File System". There is no truth to the rumour that it actually stands for "Long-lived Axe-tolerant File System". [1] http://allmydata.org/trac/tahoe/browser/relnotes.txt?rev=3853 [2] http://allmydata.org/trac/tahoe/browser/NEWS?rev=4033 [3] http://allmydata.org/trac/tahoe/wiki/RelatedProjects [4] http://allmydata.org/trac/tahoe/browser/docs/known_issues.txt [5] http://allmydata.org/trac/tahoe/browser/COPYING.GPL [6] http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html [7] http://allmydata.org/source/tahoe/trunk/docs/install.html [8] http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev [9] http://allmydata.org/trac/tahoe/roadmap [10] http://allmydata.org/trac/tahoe/browser/CREDITS?rev=4035 [11] http://allmydata.org/trac/tahoe/wiki/Dev [12] http://allmydata.com [13] http://allmydata.org/pipermail/tahoe-dev/2009-March/001461.html

1 0

[Twisted-Python] what's a workaround for Ticket #3472 (sendmail has no timeout)?
by Dave Britton 02 Aug '09

02 Aug '09

Trying to implement a custom mail service I have run into the problem that this ticket describes: ==== Ticket #3472 (new defect ) Opened 10 months ago Last modified 10 months ago twisted.mail.smtp sendmail() should [have] parameters to be passed for the retry and timeout logic supported by SMTPClientFactory: The method: def sendmail() spawns SMTPSenderFactories without a default timeout value. These cause factories to wait forever if there is no response from the remote server. On prolonged periods of time, stale SMTPSenderFactories accumulates and will cause file descriptors to ran out (Couldn't bind: 24: Too many open files.) ==== What would be the best way for me to work around this? Specifically, when I use twisted.mail.smtp sendmail() for a lengthy list, I fairly quickly get the "too many open files" message. I tried using a coiterator to send small enough batches of emails, waiting for each batch to be done before sending the next, because the "too many files" error seemed to be caused by creating too many deferreds when a too large batch of emails was sent at once. This worked in a test simulation, but with real emails the system never completes the batch, it just hangs waiting for sendmail to return. Is there a best timeout mechanism I should use to force sendmail's return? What I can think of so far are these: option 1: Call setTimeout() on the deferred returned by sendmail despite this warning in the source for Deferred.setTimeout(): "warnings.warn("Deferred.setTimeout is deprecated. Look for timeout support specific to the API you are using instead.", DeprecationWarning, stacklevel=2). or option2: Patch a timeout argument into sendmail to have it passed to SMTPSenderFactory(from_addr, to_addrs, msg, d)? Would this work? I can't figure out how or if SMTPSenderFactory would handle a timeout argument. Sorry for the long message. Thanks for your advice! -Dave

2 1

[Twisted-Python] AssertionError: '' for <module '__init__' from 'nfscraper\__init__.pyc'> not in import cache
by Konrads Smelkovs 02 Aug '09

02 Aug '09

Hello, I am running latest Twisted Python release 8.2.0 on Windows / Python 2.6. When running trial on my unittests, I get an AssertionError not in import cache: C:\Documents and Settings\Konrads\My Documents\nfscraper\src>tree /f Folder PATH listing Volume serial number is A899-B4C8 C:. └───nfscraper │ httpclient.py │ httpclient.pyc │ interfaces.py │ interfaces.pyc │ scraper.py │ scraper.pyc │ settings.py │ settings.pyc │ util.py │ util.pyc │ __init__.py │ __init__.pyc │ └───test test_scrapers.py test_scrapers.pyc __init__.py __init__.pyc C:\Documents and Settings\Konrads\My Documents\nfscraper\src>set PYTHONPATH="C:\ Documents and Settings\Konrads\My Documents\nfscraper\src" C:\Documents and Settings\Konrads\My Documents\nfscraper\src>trial nfscraper Traceback (most recent call last): File "C:\Python26\Scripts\trial.py", line 24, in <module> run() File "C:\Python26\lib\site-packages\twisted\scripts\trial.py", line 357, in ru n suite = _getSuite(config) File "C:\Python26\lib\site-packages\twisted\scripts\trial.py", line 315, in _g etSuite return loader.loadByNames(config['tests'], recurse) File "C:\Python26\lib\site-packages\twisted\trial\runner.py", line 647, in loa dByNames for thing in set(things)] File "C:\Python26\lib\site-packages\twisted\trial\runner.py", line 604, in loa dAnything return self.loadPackage(thing, recurse) File "C:\Python26\lib\site-packages\twisted\trial\runner.py", line 543, in loa dPackage pkgobj = modules.getModule(package.__name__) File "C:\Python26\lib\site-packages\twisted\python\modules.py", line 741, in g etModule return theSystemPath[moduleName] File "C:\Python26\lib\site-packages\twisted\python\modules.py", line 677, in _ _getitem__ self._findEntryPathString(moduleObject)), File "C:\Python26\lib\site-packages\twisted\python\modules.py", line 630, in _ findEntryPathString rval, modobj, pformat(self.importerCache)) AssertionError: '' for <module '__init__' from 'nfscraper\__init__.pyc'> not in import cache {'"C:\\Documents and Settings\\Konrads\\My Documents\\nfscraper\\sr c"': <imp.NullImporter object at 0x00A8F028>, 'C:\\Documents and Settings\\Konrads\\My Documents\\nfscraper\\src\\"C:\\Docume nts and Settings\\Konrads\\My Documents\\nfscraper\\src"': <imp.NullImporter obj ect at 0x00A8F488>, 'C:\\Documents and Settings\\Konrads\\My Documents\\nfscraper\\src\\"C:\\Docume nts and Settings\\Konrads\\My Documents\\nfscraper\\src"\\twisted\\plugins': <im p.NullImporter object at 0x00A8F528>, 'C:\\Python26': None, 'C:\\Python26\\DLLs': None, 'C:\\Python26\\DLLs\\twisted\\plugins': <imp.NullImporter object at 0x00A8F538> , 'C:\\Python26\\Scripts': None, 'C:\\Python26\\Scripts\\trial.py': <imp.NullImporter object at 0x00A8F498>, 'C:\\Python26\\Scripts\\twisted\\plugins': <imp.NullImporter object at 0x00A8F5 00>, 'C:\\Python26\\lib': None, 'C:\\Python26\\lib\\ctypes': None, 'C:\\Python26\\lib\\distutils': None, 'C:\\Python26\\lib\\encodings': None, 'C:\\Python26\\lib\\lib-tk': None, 'C:\\Python26\\lib\\lib-tk\\twisted\\plugins': <imp.NullImporter object at 0x00 A8F550>, 'C:\\Python26\\lib\\logging': None, 'C:\\Python26\\lib\\plat-win': <imp.NullImporter object at 0x00A8F490>, 'C:\\Python26\\lib\\plat-win\\twisted\\plugins': <imp.NullImporter object at 0x 00A8F548>, 'C:\\Python26\\lib\\site-packages': None, 'C:\\Python26\\lib\\site-packages\\PIL': None, 'C:\\Python26\\lib\\site-packages\\PIL\\twisted\\plugins': <imp.NullImporter ob ject at 0x00A8F560>, 'C:\\Python26\\lib\\site-packages\\Pythonwin': None, 'C:\\Python26\\lib\\site-packages\\Pythonwin\\twisted\\plugins': <imp.NullImpor ter object at 0x00A8F578>, 'C:\\Python26\\lib\\site-packages\\beautifulsoup-3.1.0.1-py2.6.egg': <zipimport er object "C:\Python26\lib\site-packages\beautifulsoup-3.1.0.1-py2.6.egg">, 'C:\\Python26\\lib\\site-packages\\beautifulsoup-3.1.0.1-py2.6.egg\\twisted\\pl ugins': <zipimporter object "C:\Python26\lib\site-packages\beautifulsoup-3.1.0.1 -py2.6.egg\twisted\plugins\">, 'C:\\Python26\\lib\\site-packages\\django-1.1_beta_1-py2.6.egg': None, 'C:\\Python26\\lib\\site-packages\\django-1.1_beta_1-py2.6.egg\\twisted\\plugin s': <imp.NullImporter object at 0x00A8F508>, 'C:\\Python26\\lib\\site-packages\\django_authopenid-1.0.1-py2.6.egg': None, 'C:\\Python26\\lib\\site-packages\\django_authopenid-1.0.1-py2.6.egg\\twisted\\ plugins': <imp.NullImporter object at 0x00A8F510>, 'C:\\Python26\\lib\\site-packages\\django_registration-0.7-py2.6.egg': <zipimpo rter object "C:\Python26\lib\site-packages\django_registration-0.7-py2.6.egg">, 'C:\\Python26\\lib\\site-packages\\django_registration-0.7-py2.6.egg\\twisted\\ plugins': <zipimporter object "C:\Python26\lib\site-packages\django_registration -0.7-py2.6.egg\twisted\plugins\">, 'C:\\Python26\\lib\\site-packages\\docutils-0.5-py2.6.egg': None, 'C:\\Python26\\lib\\site-packages\\docutils-0.5-py2.6.egg\\twisted\\plugins': < imp.NullImporter object at 0x00A8F520>, 'C:\\Python26\\lib\\site-packages\\epydoc-3.0.1-py2.6-win32.egg': None, 'C:\\Python26\\lib\\site-packages\\epydoc-3.0.1-py2.6-win32.egg\\twisted\\plugi ns': <imp.NullImporter object at 0x00A8F518>, 'C:\\Python26\\lib\\site-packages\\python_openid-2.2.4-py2.6.egg': <zipimporter object "C:\Python26\lib\site-packages\python_openid-2.2.4-py2.6.egg">, 'C:\\Python26\\lib\\site-packages\\python_openid-2.2.4-py2.6.egg\\twisted\\plug ins': <zipimporter object "C:\Python26\lib\site-packages\python_openid-2.2.4-py2 .6.egg\twisted\plugins\">, 'C:\\Python26\\lib\\site-packages\\setuptools-0.6c9-py2.6.egg': <zipimporter ob ject "C:\Python26\lib\site-packages\setuptools-0.6c9-py2.6.egg">, 'C:\\Python26\\lib\\site-packages\\setuptools-0.6c9-py2.6.egg\\twisted\\plugins ': <zipimporter object "C:\Python26\lib\site-packages\setuptools-0.6c9-py2.6.egg \twisted\plugins\">, 'C:\\Python26\\lib\\site-packages\\twisted': None, 'C:\\Python26\\lib\\site-packages\\twisted\\application': None, 'C:\\Python26\\lib\\site-packages\\twisted\\cred': None, 'C:\\Python26\\lib\\site-packages\\twisted\\internet': None, 'C:\\Python26\\lib\\site-packages\\twisted\\lore': None, 'C:\\Python26\\lib\\site-packages\\twisted\\lore\\scripts': None, 'C:\\Python26\\lib\\site-packages\\twisted\\persisted': None, 'C:\\Python26\\lib\\site-packages\\twisted\\plugins': None, 'C:\\Python26\\lib\\site-packages\\twisted\\python': None, 'C:\\Python26\\lib\\site-packages\\twisted\\scripts': None, 'C:\\Python26\\lib\\site-packages\\twisted\\trial': None, 'C:\\Python26\\lib\\site-packages\\twisted\\web': None, 'C:\\Python26\\lib\\site-packages\\twisted\\web2': None, 'C:\\Python26\\lib\\site-packages\\twisted\\words': None, 'C:\\Python26\\lib\\site-packages\\win32': None, 'C:\\Python26\\lib\\site-packages\\win32\\lib': None, 'C:\\Python26\\lib\\site-packages\\win32\\lib\\twisted\\plugins': <imp.NullImpo rter object at 0x00A8F570>, 'C:\\Python26\\lib\\site-packages\\win32\\twisted\\plugins': <imp.NullImporter object at 0x00A8F568>, 'C:\\Python26\\lib\\site-packages\\zope': None, 'C:\\Python26\\lib\\site-packages\\zope\\interface': None, 'C:\\Python26\\lib\\twisted\\plugins': <imp.NullImporter object at 0x00A8F540>, 'C:\\Python26\\twisted\\plugins': <imp.NullImporter object at 0x00A8F558>, 'C:\\WINDOWS\\system32\\python26.zip': <imp.NullImporter object at 0x00A8F030>, 'C:\\WINDOWS\\system32\\python26.zip\\twisted\\plugins': <imp.NullImporter obje ct at 0x00A8F530>} C:\Documents and Settings\Konrads\My Documents\nfscraper\src> -- Konrads Smelkovs Applied IT sorcery.

1 1

[Twisted-Python] getPage using ssl proxy
by Konrads Smelkovs 01 Aug '09

01 Aug '09

Hello, I am writing some scraper scripts and need to pass them through an intercepting proxy. getPage does not support a proxy argument and this code I found on internet won't work with SSL proxy (stalls indefinitely): def getPage(url, contextFactory=None, *args, **kwargs): scheme, host, port, path = _parse(url) factory = HTTPClientFactory(url, *args, **kwargs) if 0: # use a proxy host, port = 'localhost', 8080 factory.path = url if scheme == 'https': from twisted.internet import ssl if contextFactory is None: contextFactory = ssl.ClientContextFactory() reactor.connectSSL(host, port, factory, contextFactory) else: reactor.connectTCP(host, port, factory) return factory.deferred Plain http proxying works. My guess is that there is an issue with self-signed or otherwise invalid certificate the http proxy supplies. Any clues? -- Konrads Smelkovs Applied IT sorcery.

1 1

Re: [Twisted-Python] [Fwd: Re: Deferred documentation rewrite]
by Edward Z. Yang 01 Aug '09

01 Aug '09

Excerpts from Itamar Shtull-Trauring's message of Fri Jul 31 22:26:29 -0400 2009: > The problem with this is that it perpetuates the misunderstanding the > Deferreds *make* things asynchronous, even with the intro that says > otherwise. I think it's better to assume already asynchronous code, > handling the transition from synchronous to async in an intro event loop > howto. Either way, the function that the first segment of the new docs do belong somewhere. The documentation that traditionally served this purpose has been removed. As for perpetuating the misunderstanding of Deferreds making things asynchronous, I completely agree! However, I think this is something that can be fixed by spelling out the distinction between "writing asynchronous code" and "interacting with asynchronous code", and not just omitting the important paradigm shift that comes with sync->async. > A better comparative exposition might be with normal callbacks, e.g.: > > "def foo(x, gotResultCallback): pass" vs. "def foo(x): # return > Deferred". > > At the very least having that async but callbacky version in the middle > helps understanding. I briefly gloss on this, but I agree that this is an important point that could be further expanded. We could have implemented asynchronous mechanisms using normal callbacks, but we decided to use Deferreds instead. It's not clear to me if the common case of confusion of Deferreds occurs in people who "know callbacks" but "don't know Deferreds." As an incoming developer who was familiar with asynchronous programming, my primary problem was the ill-defined behavior of callback chains (which I resolved by hunkering down and reading the source code) rather than any fundamental misunderstanding of what Deferreds were supposed to do. > It also omits half the story: how you *create* Deferreds. There should > be a section on that as well. I agree. In fact, it might be worth making the document a little longer to address this point, because I realize now that even if you're not writing asynchronous code, you'll often need to baton Deferreds to make the execution flow work the way you want them to. > An example involving a parser, where you just wave your hands about who > pushes data in to the parser exactly (so no need to go into event loop > details), may work well. In particular, the object that wants the result > of the parsing wants to get parse errors, *not* whoever pushes data in. > Often it's same object, but not always. Deferreds help with that. That's a good distinction. I'll see how I can work that in. Cheers, Edward

2 1