Hi, I have just been learning twisted and been Googling for an example of something that is simple to do with pycurl or even straight python but I can't figure out for the life of me how to do with twisted. I want to be able to do a POST and debug a login to a system. Ultimately, what I am trying to do (and twisted seems to be the best choice for this) is to make the login process automatic for the user so after he or she is logged in he or she will interact with the remote system as if he or she had logged in his or herself. The proxy class seems perfect for the second part and getPage should allow me to do the first, but I don't seem to understand something. What I have done is to set up a simple login/password page on my own server with login fred and password 1234. I am using Python 2.6 on Linux 2.6. I will describe what I expect to happen, then what I did, then what happened that I did not expect and if anyone could set me straight I would be eternally grateful AND because of the good Karma you will find parking spaces easily all during 2012. If I telnet to localhost port 80 and copy and paste the following: === BEGINNING OF WORKING INPUT TO SERVER === POST /locked/checklogin.php HTTP/1.1 Host: www.mysite.com User-Agent: Mozilla/4.0 Content-Length: 31 Content-Type: application/x-www-form-urlencoded myusername=john&mypassword=1234 ===== END OF WORKING INPUT TO SERVER ==== I get just what I would expect. On successful login it goes to the page "login_success.php"and I print out the _POST array to make sure the variables are getting passed: ==== BEGINNING OF CORRECT OUTPUT FROM SERVER === Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. POST /locked/checklogin.php HTTP/1.1 Host: www.mysite.com User-Agent: Mozilla/4.0 Content-Length: 31 Content-Type: application/x-www-form-urlencoded myusername=john&mypassword=1234 HTTP/1.1 302 Found Date: Sat, 31 Dec 2011 18:42:28 GMT Server: Apache/2.2.16 (Debian) X-Powered-By: PHP/5.3.3-7+squeeze3 Set-Cookie: PHPSESSID=tueqdd71kha040d18d1qabg424; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Location: login_success.php Vary: Accept-Encoding Content-Length: 57 Content-Type: text/html IN CHECKLOGIN<BR> myusername: john, mypassword: 1234<BR> Connection closed by foreign host. === END OF CORRECT OUTPUT FROM SERVER === I took a getPage example that is in the documentation: === START CODE === from pprint import pformat from twisted.internet import reactor from twisted.internet.defer import Deferred from twisted.internet.protocol import Protocol from twisted.web.client import Agent from twisted.web.http_headers import Headers class BeginningPrinter(Protocol): def __init__(self, finished): self.finished = finished self.remaining = 1024 * 10 def dataReceived(self, bytes): if self.remaining: display = bytes[:self.remaining] print 'Some data received:' print display self.remaining -= len(display) def connectionLost(self, reason): print 'Finished receiving body:', reason.getErrorMessage() self.finished.callback(None) agent = Agent(reactor) d = agent.request( 'GET', 'http://localhost/locked/checklogin.php', Headers({'User-Agent': ['Twisted Web Client Example']}), None) def cbRequest(response): print 'Response version:', response.version print 'Response code:', response.code print 'Response phrase:', response.phrase print 'Response headers:' print pformat(list(response.headers.getAllRawHeaders())) finished = Deferred() response.deliverBody(BeginningPrinter(finished)) return finished d.addCallback(cbRequest) def cbShutdown(ignored): reactor.stop() d.addBoth(cbShutdown) reactor.run() ==== END CODE === The output was just what I expected: === OUTPUT FROM RUN OF CODE == Response version: ('HTTP', 1, 1) Response code: 200 Response phrase: OK Response headers: [('Date', ['Sat, 31 Dec 2011 18:55:17 GMT']), ('Content-Type', ['text/html']), ('X-Powered-By', ['PHP/5.3.3-7+squeeze3']), ('Vary', ['Accept-Encoding']), ('Server', ['Apache/2.2.16 (Debian)'])] Some data received: IN CHECKLOGIN<BR> myusername: , mypassword: <BR> Wrong Username or Password: , Finished receiving body: Response body fully received shutdown called ===== END OF OUTPUT FROM RUN OF CODE -- I modified the example by changing the GET to a POST and including StringProducer and changing the None to a body and the header to multipart/form-data. Here: ==== START OF CODE MODIFIED WITH POST === from pprint import pformat from twisted.internet import reactor from twisted.internet.defer import Deferred from twisted.internet.protocol import Protocol from twisted.web.client import Agent from twisted.web.http_headers import Headers from twisted.web.iweb import IBodyProducer from zope.interface import implements, Interface from stringprod import StringProducer class BeginningPrinter(Protocol): def __init__(self, finished): self.finished = finished self.remaining = 1024 * 10 def dataReceived(self, bytes): if self.remaining: display = bytes[:self.remaining] print 'Some data received:' print display self.remaining -= len(display) def connectionLost(self, reason): print 'Finished receiving body:', reason.getErrorMessage() self.finished.callback(None) creds = "myusername=john&mypassword=1234\r\n" body = StringProducer(creds) agent = Agent(reactor) d = agent.request( 'POST', 'http://localhost/locked/checklogin.php', Headers({'User-Agent': ['Mozilla 5.0'], 'Content-Type': ['multipart/form-data; charset=utf-8']}), body) def cbRequest(response): print 'Response version:', response.version print 'Response code:', response.code print 'Response phrase:', response.phrase print 'Response headers:' print pformat(list(response.headers.getAllRawHeaders())) finished = Deferred() response.deliverBody(BeginningPrinter(finished)) return finished d.addCallback(cbRequest) def cbShutdown(ignored): print "shutdown called" reactor.stop() d.addBoth(cbShutdown) reactor.run() === END CODE MODIFIED WITH POST === When I run this, I do not get the output I expect, for some reason, it does not seem to be sending the body. ==== BEGINNING OF RUN OF MODIFIED CODE ==== Response version: ('HTTP', 1, 1) Response code: 200 Response phrase: OK Response headers: [('Date', ['Sat, 31 Dec 2011 19:06:39 GMT']), ('Content-Type', ['text/html']), ('X-Powered-By', ['PHP/5.3.3-7+squeeze3']), ('Vary', ['Accept-Encoding']), ('Server', ['Apache/2.2.16 (Debian)'])] Some data received: IN CHECKLOGIN<BR> myusername: , mypassword: <BR> Wrong Username or Password: , Finished receiving body: Response body fully received shutdown called ==== END OF RUN OF MODIFIED CODE === What am I missing? The body does not seem to be being send. I would like to have a way to easily be able to debug logins to remote systems. This is NOT for spamming. I am trying to develop a system for higher security where our users would log locally (VPN) with their own favorite passwords and that would map to awful username and password that no one would ever use or remember and would be unguessable to log them into external cloud based systems. In some cases there are libraries and APIs to help me, but I want a general solution. Mechanic seems to do part of this and pycurl, but the proxy and networking part seem much better in twisted. HELP!!! Thanks, Donald