I'm attempting to get some web pages using the following code which I did not write. While it seems to work (except for this, so far) I have no idea if this is a reasonable way to do this (get simple web pages) at all:

def getPage(url, contextFactory=None, *args, **kwargs): """ Download a web page as a string.

Download a page. Return a deferred, which will callback with a page (as a string) or errback with a description of the error.

See HTTPClientFactory to see what extra args can be passed. """ scheme, host, port, path = parse_url(url) factory = HTTPClientFactory(url, *args, **kwargs) if scheme == 'https': from twisted.internet import ssl if contextFactory is None: contextFactory = ssl.ClientContextFactory() reactor.connectSSL(host, port, factory, contextFactory) else: reactor.connectTCP(host, port, factory)

return factory.deferred

The code then adds a bunch of callbacks to the returned deferred to do various things to the data and everything's swell.

Until the url shown below occurs. The deferred never calls any of the callbacks and just never seems to finish.

I haven't found any way to dump the actual headers from within Twisted as this occurs so the header values shown below are from firefox calling into the same URL. I will put tcpdump in the way if I need to to figure this out but I'm thinking this is something simple (or wrong with the method used in the code above).

Can anyone tell me what it is about this particular transaction that's not allowing the deferred to fire its callbacks which I presume is because it never finishes getting the stuff it's looking for. This particular URL returns a .vcf file.

Also, what is the proper intervention? I'd like not to download the .vcf as it's completely useless for my purpose but I'm not familiar enough with twisted.web to know where to intervene.

Thanks,

S

http://www.integrateddevcorp.com/index.php?option=com_contact&task=vcard...

GET /index.php? option=com_contact&task=vcard&contact_id=1&format=raw&tmpl=component HTTP/1.1 Host: www.integrateddevcorp.com User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv: 1.9.1.3) Gecko/20090824 Firefox/3.5.3 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive

HTTP/1.x 200 OK Date: Thu, 08 Oct 2009 21:14:37 GMT Server: Apache X-Powered-By: PHP/5.2.8 Set-Cookie: ff70eb7218d444fa639af7ae7e66e82f=488606e54b7fdd9affb0b0725a2a6607; path=/ P3P: CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM" Content-Disposition: attachment; filename=Integrated_Development_Corporation.vcf Content-Length: 1020 Connection: close Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre- check=0 Pragma: no-cache Expires: Mon, 1 Jan 2001 00:00:00 GMT Last-Modified: Thu, 08 Oct 2009 21:14:37 GMT Content-Type: text/html; charset=utf-8 ----------------------------------------------------------

On Fri, Oct 9, 2009 at 10:53 AM, Reza Lotun rlotun@gmail.com wrote:

won't return returns. What you have to do is create your own slightly

I meant won't return headers ;-)

Reza

On Oct 10, 2009, at 1:34 AM, Steve Steiner (listsin) wrote:

Along the way, I ran into:

http://python.net/crew/mwh/apidocs/twisted.web.http.Request.html

Just realized that that was off-twistedmatrix, same doc at:

http://twistedmatrix.com/documents/8.2.0/api/twisted.web.http.Request.html

so it's not some off-site weirdness...

S

On Oct 10, 2009, at 3:18 AM, Glyph Lefkowitz wrote:

On Sat, Oct 10, 2009 at 2:22 AM, Steve Steiner (listsin) <listsin@integrateddevcorp.com

...

wrote:

Just realized that that was off-twistedmatrix, same doc at:

   http://twistedmatrix.com/documents/8.2.0/api/twisted.web.http.Request.html

so it's not some off-site weirdness...

What "weirdness" are you referring to?

I was just referring to the fact that the original document was not on the Twisted site so I was referring to it not being something someone else did with Twisted.

One of the things I was, and still am having some trouble with in Twisted is remembering the right names of things.

Much of the other code I'm using follows PEP-8 naming_conventions for methods and such whereas Twisted follows the Twisted Coding Standard which is a more javaCamelCase style convention.

That particular documentation refers to 'backwards compatible' functions using PEP-8 style attribute names while preferring the newer Twisted Coding Standard flavoured names.

S

On Oct 10, 2009, at 9:07 PM, Glyph Lefkowitz wrote:

Much of the other code I'm using follows PEP-8 naming_conventions for methods and such whereas Twisted follows the Twisted Coding Standard which is a more javaCamelCase style convention.

For what it's worth, the Twisted coding standard pre-dates PEP-8 (In fact, I think it might even predate the whole PEP process).

Also, Java did not invent the convention of camel-case names. My use of the convention in the early Twisted coding standard was a deliberate aping of Smalltalk's coding convention. I realize a lot more people have seen Java written this way than ST, but nevertheless ST was my inspiration for many of Twisted's conventions, coding-standard and otherwise.

I understand. I refer to it as "Java" since that's more current. I think of it more as ST as well; I was there "back then." It's one of the things I really like about Twisted. I just kind of "get" it 'cause it makes sense from for a long time <sic>.

I don't particularly agree with, or like some things in PEP-8.

For example, I like to line things up "right" when organization can be indicated by spacing like (which will be mutilated by e-mail, but you know what I mean):

foo = ( ("this value is long", "default"), ("short", "other"), )

If it's a table, it should be laid out like a table, not all crooked and stuff. CamelCase is another issue. Why it should be OK in ClassNames but not OK in variable_names, since you have to hit the shift key to get the flippin' underscore anyway, has 'logic' that escapes me. "Inserting an extra, meaningless extra character, which requires using the Shift key, instead of the next letter, also with the shift key saves *thing*."

That particular documentation refers to 'backwards compatible' functions using PEP-8 style attribute names while preferring the newer Twisted Coding Standard flavoured names.

Yes, before code review was consistently applied throughout the codebase, we did have a few inconsistent names slip through. Since then we've tried to update them where we've found them so everything is consistently in one style.

Also understood. That's the thing about writing stuff other people use; you've got to keep it working even when you're slapping your forehead about something you decided then that's now so _obviously_ better done another way.

Making the right things show up in the right order in our API documentation is a constant challenge. If you have any ideas for emphasizing the correct (i.e. non-"backwards-compatibility") names, you might want to submit a patch to pydoctor and/or Twisted's docstrings.

I actually took a quick look at pydoctor for the first time tonight. I'm wrestling with getting a doc set together from some code that's EpyDoc, some that's Sphinx, and some that's not there at all.

What I'd like is to use Sphinx, using all of EpyDoc's nice extensions to reStructuredText (e.g. the nice :Parameters: block stuff) so I'm looking to see if anyone's already done that work for Sphinx.

Meanwhile, vis-à-vis the Twisted docs, has there been any recent discussion about moving the Twisted docs to Sphinx? I found some doc discussions way back to 2002, nothing later than 2007/8-ish, but, as far as I know, there's not particular effort underway.

If Twisted could get documented with Sphinx, while adding anything Twisted needs that Sphinx doesn't have already, it would sure be nice...and would remove one more Twisted specific tool to maintain in the Twisted toolchain, while enhancing Sphinx at the same time.

I'm having to do the pydoctor vs. Sphinx vs. EpyDoc research anyway...I'll post it somewhere public and send a link/message to the list.

S

...

I'm having to do the pydoctor vs. Sphinx vs. EpyDoc research anyway...I'll post it somewhere public and send a link/message to the list.

This was raised in July (in a thread you posted to ;).

http://twistedmatrix.com/pipermail/twisted-python/2009-July/ 020102.html

I knew I'd thought about it before! I've been slammed with other work since then and haven't had to face the "doc monster" again until now. I'm pulling docs together on several new modules/packages that have been factored out of a large code base and we have a complete mish- mosh. Not as big as Twisted, but much less organized.

Twisted, and probably almost every project of similar size and, more particularly, age, has some level of doc fung. Back in the 'old days' the tools were sparse (or bad) enough that the Twisted project made some of its own; of course. So did lots of other people. Also, as time went on, more options (like wikis) came about and parts of the docs found their way into the mix.

From reading back over that previous discussion, I realize that my main thought was to build a database of what is on hand, from all the various sources, then pulling them together into some sort of system where they could all be organized, filtered, and finally sent to *something* to come out done on the other side.

As far as I know, there is no good answer available off the shelf.

Maybe someone else knows of something?

It could be that it's just something that has to be 'done.' And, done in a comprehensive (and/or extensible) enough way that it will serve the needs of other projects, too.

I'm not sure (as I wasn't in July) what the next step is, but it might be as simple as writing a simple DocCollector app that pulls everything into one database, cross references and indexes it, and gets it into some sort of visual presentation that we can start to make sense of the whole thing with.

Maybe the first step would be to pull together a list of "everywhere we know Twisted docs are known to live" and start pulling it in to see where that leads...

Where would be the best place to put the "collector" page where everyone would know to look for it and would be able to contribute to it (some wiki, but which one?).

Thanks,

S

On Oct 10, 2009, at 7:34 AM, Reza Lotun wrote:

...

I'm not quite sure what you mean. Are you talking about web.client or

web.server?

Web client. I was having trouble with a particular URL which happened to be a download URL and wanted to see what the request and response objects looked like and was trying to find where I could examine them. In the code I posted before, we were creating an HTTPClientFactory, then snagging the deferred. I've simplified that by just using client.getPage(). The advantage of the HTTPClientFactory approach is that you can get at the response_headers.

There is a new twisted.web client implementation in the works which will hopefully be ready for trunk in the near future, and which should provide nearly the same functionality offered by urllib2 (or more).

Where might that be hiding? I'd love to have something that does deferreds like Twisted and also implements all of urllib2. I find the info() from urlopen() particularly helpful when debugging issues like this.

...

I finally figured out that it was a stupid string formatting bug in the code that was being echoed by Twisted but swallowed by the "Grand Catcher of all Things Exception" in Twisted..

Once I found and fixed that, everything works fine -- until the next time.

Exceptions shouldn't be swallowed by twisted. If you simply attach errBacks then you'll get the exception and full traceback. It really isn't that bad at all - I thought it looked crazy when I first started, but all the information is there to debug, etc.

Yes, the problem was in my mis-reading of the info in my errback. Unfortunately, the way the logging is set up in this particular code, logging.debug() returns 10 zillion times too much repetitive information and it makes it hard to wade through to find the relevant info (like the errback's output I was missing).

Thanks again for your help with this and I'd love to know more about this new urllib2 type module for Twisted.

Thanks,

S

On Oct 10, 2009, at 5:15 PM, exarkun@twistedmatrix.com wrote:

...

Where might that be hiding? I'd love to have something that does deferreds like Twisted and also implements all of urllib2. I find the info() from urlopen() particularly helpful when debugging issues like this.

http://twistedmatrix.com/trac/ticket/886 http://twistedmatrix.com/trac/ticket/3987

As I understand the "info" method, it gives you access to the response headers. The new API definitely exposes that information.

Yes, pretty much everything known about the round-trip is attached as attributes of the return value for easy inspection with a couple of little convenience methods for getting at them.

I just had a major debugging nightmare where the client, a web services company, insisted that "I must be sending some weird headers" and it wasn't that easy to prove I wasn't without being able to just access that info. I finally ended up using tcpdump. I had forgotten how much "fun" setting up tcpdump filters is.

You'll notice that neither of these tickets is closed. #3987 is presently up for review. If you want to help out, you could take a look at the code and point out any problems you notice.

Hopefully I can just fire up a virtualenv, set this as my Twisted, and run a few of the things I'm working on through it.

For some of them, I've been doing tcpdump pushups to find out what's going on and it'd be great to have that right in the Python code instead.

Please excuse my ignorance of how Twisted branches are used (there sure do seem to be a lot of them!), but is this branch supposed to be trunk + branch mods or latest release + branch mods?

Thanks!

S

On 29 Oct, 11:34 pm, listsin@integrateddevcorp.com wrote:

On Oct 10, 2009, at 5:15 PM, exarkun@twistedmatrix.com wrote:

...

...

Where might that be hiding? I'd love to have something that does deferreds like Twisted and also implements all of urllib2. I find the info() from urlopen() particularly helpful when debugging issues like this.

http://twistedmatrix.com/trac/ticket/886 http://twistedmatrix.com/trac/ticket/3987

As I understand the "info" method, it gives you access to the response headers. The new API definitely exposes that information.

You'll notice that neither of these tickets is closed. #3987 is presently up for review. If you want to help out, you could take a look at the code and point out any problems you notice.

Jean-Paul

Well...I've now spent some actual time with 886 (-2,-3,and -4), and also with 3987.

While there is some very nice stuff in there, I'm really not sure how one would actually put it to use since the back-end and the front-end seem to have been split into separate branches.

They're not really split. The 3987 branch is complete and usable on its own. You don't need a checkout of the 886 branch.

Jean-Paul

On Oct 29, 2009, at 11:05 PM, exarkun@twistedmatrix.com wrote:

...

Well...I've now spent some actual time with 886 (-2,-3,and -4), and also with 3987.

While there is some very nice stuff in there, I'm really not sure how one would actually put it to use since the back-end and the front-end seem to have been split into separate branches.

They're not really split. The 3987 branch is complete and usable on its own. You don't need a checkout of the 886 branch.

Ok, I have re-checked out just the 3987 branch without messing with 886.

Using it as a direct substitution in a Django application using Twisted's WSGI, it completely breaks my application in ways that look suspiciously like the 8.2.0 release's WSGI problems that have since been fixed in Twisted trunk.

Is 3987 based on 8.2.0 and, if so, what is the procedure for merging the trunk back to this branch (or visa versa, if that would be better).

If 3987 is supposed to be in sync with trunk, I'll have to hunt down the WSGI problems I'm having because it's definitely not working in an application that works just fine with Twisted trunk.

S

On Nov 2, 2009, at 6:07 PM, exarkun@twistedmatrix.com wrote:

On 03:20 pm, ssteinerx@gmail.com wrote:

...

On Oct 29, 2009, at 11:05 PM, exarkun@twistedmatrix.com wrote:

...

...

Well...I've now spent some actual time with 886 (-2,-3,and -4), and also with 3987.

While there is some very nice stuff in there, I'm really not sure how one would actually put it to use since the back-end and the front- end seem to have been split into separate branches.

They're not really split. The 3987 branch is complete and usable on its own. You don't need a checkout of the 886 branch.

Ok, I have re-checked out just the 3987 branch without messing with 886.

Using it as a direct substitution in a Django application using Twisted's WSGI, it completely breaks my application in ways that look suspiciously like the 8.2.0 release's WSGI problems that have since been fixed in Twisted trunk.

Is 3987 based on 8.2.0 and, if so, what is the procedure for merging the trunk back to this branch (or visa versa, if that would be better).

If 3987 is supposed to be in sync with trunk, I'll have to hunt down the WSGI problems I'm having because it's definitely not working in an application that works just fine with Twisted trunk.

It's not based on 8.2.0, but neither is it in "sync" with trunk. It is a copy of trunk from r27271. You may indeed find that there are a number of bug fixes or feature additions which are present in trunk@HEAD which are not present in the #3987 branch.

If you want everything from trunk@HEAD and the #3987 branch, then you may now be happy you did all that stuff with git. Merge the #3987 branch into trunk@HEAD and you'll get everything from trunk@HEAD and everything from the branch (that's tautological, sorry, I hope you know what I mean).

If you're not sure what to do with git, then here's what to do with svn:

BRANCHES=svn://svn.twistedmatrix.com/svn/Twisted/branches EXPRESSIVE=$BRANCHES/expressive-http-client-886-4 HIGHLEVEL=$BRANCHES/high-level-web-client-3987/ cd path/to/Twisted/trunk svn merge $EXPRESSIVE@27272 EXPRESSIVE@HEAD svn merge $HIGHLEVEL@27275 $HIGHLEVEL@HEAD

Or using Combinator:

chbranch Twisted expressive-http-client-886-4 unbranch Twisted chbranch Twisted high-level-web-client-3987 unbranch Twisted --force

Wow. That's quite a lot of stuff.

Now I have an @HEAD.ache ;-) .

I'll see if I can get this whacked together in any meaningful way and maybe publish my resultant branch to Github or something.

Meanwhile...