[Twisted-Python] treq POST abborting with: err: ('Could not adapt', '{"....", "..."} <InterfaceClass twisted.web.iweb.IBodyProducer>)
hi, sending a simple post request with a body (cookies) results in a error here. im using python 3.x with the latest twisted and treq installed via pip in a virtualenv. the code gets a chatroom login page, extracts two values, gets the cookies and logs a user into the chat. (incomplete) thing is for being logged in i need to send my current phpsessionid in the post request... glyph (without seeing the the code or backtrace) suggested that: "I think the issue might be that it doesn't support bytes() on py3, it's registered against str() or something" here is my code: from myhttp import getLoginPage, login from twisted.internet import reactor def myPrint(result): print("result:", result) def myError(failure): print("failure:", failure) def lala(): #d = getLoginPage() d = login() d.addCallback(myPrint) d.addErrback(myError) if __name__ == '__main__': lala() reactor.run() from myhttp import getLoginPage, login from twisted.internet import reactor def myPrint(result): print("result:", result) def myError(failure): print("failure:", failure) def lala(): #d = getLoginPage() d = login() d.addCallback(myPrint) d.addErrback(myError) if __name__ == '__main__': lala() reactor.run() (venv) julius@t560:~/code/twisted/webchat-client$ cat myhttp.py from bs4 import BeautifulSoup import random from twisted.internet import reactor, defer from twisted.web.client import getPage import treq import urllib url = b'http://chatroom2000.de' useragent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36".encode('utf-8') cookies = {} @defer.inlineCallbacks def getLoginPage(): global cookies response = yield treq.get(url) final_page = yield response.content() cookies = response.cookies() defer.returnValue(final_page) @defer.inlineCallbacks def login(): response = yield getLoginPage() print(cookies) print("rsponse:", response) soup = BeautifulSoup(response, "lxml") hmac = soup.input["name"] key = soup.input["value"] print("hmac:key", hmac,key) username_list = ["Demetrius", "Tyrone", "Marshawn"] username = random.choice(username_list) body = urllib.parse.urlencode({hmac: key, 'username': username, 'pw':'', 'gender':'m', 'aaaa':''}) target_url = url + b"/?CheckUserName" headers = {'Content-Type': 'application/x-www-form- urlencoded','cache-control': ['no-store, no-cache, must-revalidate, pre-check=0, post-check=0, max-age=0']} import json try: print("cookies2:", cookies) somepage = yield treq.post(target_url, json.dumps({hmac: key, "username": username, 'pw':'', 'gender':'m', 'aaaa':''}), headers=headers, cookies=cookies ) except Exception as err: print("err:", err) response = yield somepage.content() defer.returnValue(response) if __name__ == '__main__': #url = 'http://chatroom2000.de' getPage(url) reactor.run() and the error: cookies2: <RequestsCookieJar[<Cookie PHPSESSID=jevr00m25j0clfv6aibbgbro26 for chatroom2000.de/>, <Cookie db1_cookie_test=1483729313 for chatroom2000.de/>]> err: ('Could not adapt', '{"pw": "", "aaaa": "", "username": "Demetrius", "cd5a6e4774acfd7e090467c9ed67f7e6": "1483729313", "gender": "m"}', <InterfaceClass twisted.web.iweb.IBodyProducer>) failure: [Failure instance: Traceback: <class 'UnboundLocalError'>: local variable 'somepage' referenced before assignment /home/julius/code/twisted/webchat-client/venv/lib/python3.5/site- packages/twisted/internet/defer.py:457:callback /home/julius/code/twisted/webchat-client/venv/lib/python3.5/site- packages/twisted/internet/defer.py:565:_startRunCallbacks /home/julius/code/twisted/webchat-client/venv/lib/python3.5/site- packages/twisted/internet/defer.py:651:_runCallbacks /home/julius/code/twisted/webchat-client/venv/lib/python3.5/site- packages/twisted/internet/defer.py:1355:gotResult --- <exception caught here> --- /home/julius/code/twisted/webchat-client/venv/lib/python3.5/site- packages/twisted/internet/defer.py:1299:_inlineCallbacks /home/julius/code/twisted/webchat-client/myhttp.py:54:login i followd some files in the traceback, but got nowhere :(
On Fri, 6 Jan 2017 at 21:21 steven meiers <commercials24@yahoo.de> wrote:
glyph (without seeing the the code or backtrace) suggested that: "I think the issue might be that it doesn't support bytes() on py3, it's registered against str() or something"
Actually, the problem is the reverse of this; only bytes is supported, but you're passing str (unicode) in.
somepage = yield treq.post(target_url, json.dumps({hmac: key, "username": username, 'pw':'', 'gender':'m', 'aaaa':''}),
json.dumps returns str (ie. unicode) on py3; throwing in a .encode('utf-8') here is probably what you want.
1
On Jan 6, 2017, at 3:36 PM, Tristan Seligmann <mithrandi@mithrandi.net> wrote:
On Fri, 6 Jan 2017 at 21:21 steven meiers <commercials24@yahoo.de <mailto:commercials24@yahoo.de>> wrote: glyph (without seeing the the code or backtrace) suggested that: "I think the issue might be that it doesn't support bytes() on py3, it's registered against str() or something"
Actually, the problem is the reverse of this; only bytes is supported, but you're passing str (unicode) in.
Aah, thanks for pointing this out.
somepage = yield treq.post(target_url, json.dumps({hmac: key, "username": username, 'pw':'', 'gender':'m', 'aaaa':''}),
json.dumps returns str (ie. unicode) on py3; throwing in a .encode('utf-8') here is probably what you want.
Maybe we should support unicode for the body as well. We can set the charset in the mime-type and everything so that it will be properly intelligible by the server, which doesn't happen if the user manually encodes like this. Even if we do this though - is there any way to convince json.dumps to behave consistently between 2/3 for the purposes of examples? -glyph
On Sat, 7 Jan 2017 at 03:23 Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Even if we do this though - is there any way to convince json.dumps to behave consistently between 2/3 for the purposes of examples?
As far as I know, the only way to handle this is with an if PY3 or if isinstance(…). I don't know of a way to make json.dumps on Python 2 consistently return unicode, and I don't know of a way to make json.dumps on Python 3 return bytes.
On Sat, 7 Jan 2017 at 03:23 Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Maybe we should support unicode for the body as well. We can set the charset in the mime-type and everything so that it will be properly intelligible by the server, which doesn't happen if the user manually encodes like this.
Oh, forgot to comment on this point; in the specific case of JSON, it isn't necessary to specify UTF-8 in Content-Type[1], but for HTML or XML it's a pretty good idea. However, I'm not sure if it's possible to modify Content-Type in a generic fashion to make this sort of thing work; for example, "Content-Type: application/octet-stream; charset=UTF-8" is nonsense. I'll defer to some HTTP experts here ;) [1] https://tools.ietf.org/html/rfc7159#section-8.1
On 7 Jan 2017, at 02:18, Tristan Seligmann <mithrandi@mithrandi.net> wrote:
On Sat, 7 Jan 2017 at 03:23 Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Maybe we should support unicode for the body as well. We can set the charset in the mime-type and everything so that it will be properly intelligible by the server, which doesn't happen if the user manually encodes like this.
Oh, forgot to comment on this point; in the specific case of JSON, it isn't necessary to specify UTF-8 in Content-Type[1], but for HTML or XML it's a pretty good idea. However, I'm not sure if it's possible to modify Content-Type in a generic fashion to make this sort of thing work; for example, "Content-Type: application/octet-stream; charset=UTF-8" is nonsense. I'll defer to some HTTP experts here ;)
This is really not simple, for the reason that many MIME types do not define a charset extension. In the case of JSON, it’s not just not necessary to specify UTF-8 in Content-Type, but the standard explicitly does not define charset for the JSON content type[0]:
Note: No "charset" parameter is defined for this registration. Adding one really has no effect on compliant recipients.
Strictly a completely compliant implementation would not emit charset details for content types that have no charset registration. Such a thing is pretty tricky to do. Knowing that, it’s probably best to YOLO your way though, or forbid unicode in bodies. Cory [0]: https://tools.ietf.org/html/rfc7159#section-11
thanks for the quick replys guys, you were right. a .encode('utf-8') helps. actually the json.dumps part in the code is wrong. it has to be: somepage = yield treq.post(target_url, body.encode('utf-8'), headers=headers, cookies=cookies ) Am Samstag, den 07.01.2017, 12:00 +0000 schrieb Cory Benfield:
On 7 Jan 2017, at 02:18, Tristan Seligmann <mithrandi@mithrandi.net
wrote:
On Sat, 7 Jan 2017 at 03:23 Glyph Lefkowitz <glyph@twistedmatrix.co m> wrote:
Maybe we should support unicode for the body as well. We can set the charset in the mime-type and everything so that it will be properly intelligible by the server, which doesn't happen if the user manually encodes like this.
Oh, forgot to comment on this point; in the specific case of JSON, it isn't necessary to specify UTF-8 in Content-Type[1], but for HTML or XML it's a pretty good idea. However, I'm not sure if it's possible to modify Content-Type in a generic fashion to make this sort of thing work; for example, "Content-Type: application/octet- stream; charset=UTF-8" is nonsense. I'll defer to some HTTP experts here ;)
This is really not simple, for the reason that many MIME types do not define a charset extension. In the case of JSON, it’s not just not necessary to specify UTF-8 in Content-Type, but the standard explicitly does not define charset for the JSON content type[0]:
Note: No "charset" parameter is defined for this registration. Adding one really has no effect on compliant recipients.
Strictly a completely compliant implementation would not emit charset details for content types that have no charset registration. Such a thing is pretty tricky to do. Knowing that, it’s probably best to YOLO your way though, or forbid unicode in bodies.
Cory
[0]: https://tools.ietf.org/html/rfc7159#section-11 _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Jan 7, 2017, at 4:00 AM, Cory Benfield <cory@lukasa.co.uk> wrote:
On 7 Jan 2017, at 02:18, Tristan Seligmann <mithrandi@mithrandi.net> wrote:
On Sat, 7 Jan 2017 at 03:23 Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Maybe we should support unicode for the body as well. We can set the charset in the mime-type and everything so that it will be properly intelligible by the server, which doesn't happen if the user manually encodes like this.
Oh, forgot to comment on this point; in the specific case of JSON, it isn't necessary to specify UTF-8 in Content-Type[1], but for HTML or XML it's a pretty good idea. However, I'm not sure if it's possible to modify Content-Type in a generic fashion to make this sort of thing work; for example, "Content-Type: application/octet-stream; charset=UTF-8" is nonsense. I'll defer to some HTTP experts here ;)
This is really not simple, for the reason that many MIME types do not define a charset extension. In the case of JSON, it’s not just not necessary to specify UTF-8 in Content-Type, but the standard explicitly does not define charset for the JSON content type[0]:
I see you your standards pedantry, and raise you! MIME defines content-type to always have a charset: https://tools.ietf.org/html/rfc1521#section-4 <https://tools.ietf.org/html/rfc1521#section-4>
Among the defined parameters is a "charset" parameter by which the character set used in the body may be declared.
and from the spec you're citing,
Adding one really has no effect on compliant recipients
Given that we'd always choose utf-8 anyway it would be fine.
Note: No "charset" parameter is defined for this registration. Adding one really has no effect on compliant recipients.
Strictly a completely compliant implementation would not emit charset details for content types that have no charset registration. Such a thing is pretty tricky to do. Knowing that, it’s probably best to YOLO your way though, or forbid unicode in bodies.
A bigger part of the problem here is that treq has no way of knowing that the characters you're sending are JSON at all, let alone what encoding you want them in. So if we actually want to do MIME stuff, we'd probably want to have treq be the layer to call treq.dumps on the dict anyway, so that it knows what it's dealing with. Automatic treatment of unicode would likely set the type to text/plain;charset=utf8. -glyph
On Sun, 8 Jan 2017 at 01:57 Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
I see you your standards pedantry, and raise you!
MIME defines content-type to always have a charset:
*https://tools.ietf.org/html/rfc1521#section-4 <https://tools.ietf.org/html/rfc1521#section-4>*
RFC 1521 is obsoleted by RFC 2045 which has more nuanced semantics: https://tools.ietf.org/html/rfc2045#section-5 But actually, HTTP is not MIME; so we should be looking here instead: https://tools.ietf.org/html/rfc7231#section-3.1.1.1
On Jan 6, 2017, at 11:03 AM, steven meiers <commercials24@yahoo.de> wrote:
hi,
sending a simple post request with a body (cookies) results in a error here.
im using python 3.x with the latest twisted and treq installed via pip in a virtualenv.
Something that we discussed on IRC but did not come up in the mailing list thread may be the one issue that *should* be fixed in treq: Because of this issue, this example - http://treq.readthedocs.io/en/latest/#post <http://treq.readthedocs.io/en/latest/#post> - which the OP was following, doesn't work on Python 3. We should fix the docs so that it does. -glyph
hi, i use twisted to send a web request over a proxy, this works over treq.get(url, agent=myagent) without problems. but in the process (started out with agent.request) i could not figure out from what line of twisted code this error: [Failure instance: Traceback (failure with no frames): <class 'twisted.web._newclient.RequestGenerationFailed'>: [<twisted.python.failure.Failure builtins.TypeError: sequence item 0: expected a bytes-like object, str found>] came from. btw, is there a way to have a debugger attached to twisted code that shows what code is currently executed and has a step forward option? the error does come up when you give agent.request a "GET" instead of a b"GET". heres the test code: from twisted.python.log import err from twisted.web.client import ProxyAgent from twisted.internet import reactor, defer from twisted.internet.endpoints import TCP4ClientEndpoint import treq def display(response): print("Received response") print(response.content) def err(failure): print(failure) def main(): endpoint = TCP4ClientEndpoint(reactor, "223.25.102.186", 8080) agent = ProxyAgent(endpoint) d = agent.request("GET", bytes("http://somedomain.de", 'utf-8')) d.addCallbacks(display) d.addErrback(err) if __name__ == "__main__": main() reactor.run() this code will produce this error: [Failure instance: Traceback (failure with no frames): <class 'twisted.web._newclient.RequestGenerationFailed'>: [<twisted.python.failure.Failure builtins.TypeError: sequence item 0: expected a bytes-like object, str found>] and some .py files with line numbers. none of which really are the problem. as it turns out, the "GET" as a string gets problematic here: https://github.com/twisted/twisted/blob/twisted-16.6.0/src/twisted/web/ _newclient.py#L651 requestLines.append(b' '.join([self.method, self.uri, b'HTTP/1.1\r\n'])) wouldnt it be better to have something like this: try: requestLines.append(b' '.join([self.method, self.uri, b'HTTP/1.1\r\n'])) except TypeError as e: raise TypeError("could not join: " + str(self.method) + " " + str(self.uri) + ' ' + 'HTTP/1.1\r\n' ) which will produce: <class 'twisted.python.failure.Failure'> [Failure instance: Traceback: <class 'TypeError'>: could not join: GET b'http://somedomain.de' HTTP/1.1 to give the user a chance to find out what he did wrong? ...when he uses agent instead of treq for whatever reason.
On Jan 9, 2017, at 5:20 PM, steven meiers <commercials24@yahoo.de> wrote:
the error does come up when you give agent.request a "GET" instead of a b"GET".
That's expected; you do have to pass the method as `bytes`, as documented here: https://twistedmatrix.com/documents/16.6.0/api/twisted.web.iweb.IAgent.html#... <https://twistedmatrix.com/documents/16.6.0/api/twisted.web.iweb.IAgent.html#...> -g
Am Montag, den 09.01.2017, 23:12 -0800 schrieb Glyph Lefkowitz:
On Jan 9, 2017, at 5:20 PM, steven meiers <commercials24@yahoo.de> wrote:
the error does come up when you give agent.request a "GET" instead of a b"GET".
That's expected; you do have to pass the method as `bytes`, as documented here: https://twistedmatrix.com/documents/16.6.0/api/twist ed.web.iweb.IAgent.html#request
True, but since python 2.x is still shipped with debian testing for example there will be some people running into this issue. Just to make twisted more approachable this could he handled.
On Jan 10, 2017, at 3:19 PM, steven meiers <commercials24@yahoo.de> wrote:
Am Montag, den 09.01.2017, 23:12 -0800 schrieb Glyph Lefkowitz:
On Jan 9, 2017, at 5:20 PM, steven meiers <commercials24@yahoo.de> wrote:
the error does come up when you give agent.request a "GET" instead of a b"GET".
That's expected; you do have to pass the method as `bytes`, as documented here: https://twistedmatrix.com/documents/16.6.0/api/twist ed.web.iweb.IAgent.html#request
True, but since python 2.x is still shipped with debian testing for example there will be some people running into this issue. Just to make twisted more approachable this could he handled.
I think that Tristan's suggestion is the best way to address this. -glyph
Am Montag, den 09.01.2017, 23:12 -0800 schrieb Glyph Lefkowitz:
On Jan 9, 2017, at 5:20 PM, steven meiers <commercials24@yahoo.de> wrote:
the error does come up when you give agent.request a "GET" instead of a b"GET".
That's expected; you do have to pass the method as `bytes`, as documented here: https://twistedmatrix.com/documents/16.6.0/api/twist ed.web.iweb.IAgent.html#request
True, but since python 2.x is still shipped with debian testing for example there will be some people running into this issue. Just to make twisted more approachable this could he handled.
On Wed, 11 Jan 2017 at 02:26 steven meiers <commercials24@yahoo.de> wrote:
True, but since python 2.x is still shipped with debian testing for example there will be some people running into this issue. Just to make twisted more approachable this could he handled.
The b'GET' syntax for byte strings works on Python 2.7 (you get a str) as well as Python 3, so I would suggest we use it everywhere in the documentation / examples to make it easier on programmers using both Python 2 and 3.
On 01/08/2017 06:49 PM, Glyph Lefkowitz wrote:
On Jan 6, 2017, at 11:03 AM, steven meiers <commercials24@yahoo.de <mailto:commercials24@yahoo.de>> wrote:
hi,
sending a simple post request with a body (cookies) results in a error here.
im using python 3.x with the latest twisted and treq installed via pip in a virtualenv.
Something that we discussed on IRC but did not come up in the mailing list thread may be the one issue that *should* be fixed in treq:
Because of this issue, this example - http://treq.readthedocs.io/en/latest/#post - which the OP was following, doesn't work on Python 3.
We should fix the docs so that it does.
I submitted a PR to fix this a while ago: https://github.com/twisted/treq/pull/152 Also, the docs on RTD are still out of date. Who owns them? I'd be happy to help. ---Tom
On Jan 10, 2017, at 6:43 PM, Tom Most <tommost@gmail.com> wrote:
On 01/08/2017 06:49 PM, Glyph Lefkowitz wrote:
On Jan 6, 2017, at 11:03 AM, steven meiers < <mailto:commercials24@yahoo.de>commercials24@yahoo.de <mailto:commercials24@yahoo.de>> wrote:
hi,
sending a simple post request with a body (cookies) results in a error here.
im using python 3.x with the latest twisted and treq installed via pip in a virtualenv.
Something that we discussed on IRC but did not come up in the mailing list thread may be the one issue that *should* be fixed in treq:
Because of this issue, this example - <http://treq.readthedocs.io/en/latest/#post>http://treq.readthedocs.io/en/latest/#post <http://treq.readthedocs.io/en/latest/#post> - which the OP was following, doesn't work on Python 3.
We should fix the docs so that it does.
I submitted a PR to fix this a while ago: https://github.com/twisted/treq/pull/152 <https://github.com/twisted/treq/pull/152>
Also, the docs on RTD are still out of date. Who owns them? I'd be happy to help.
Thanks, hopefully the next time my periodic review comes around I'll catch that one. Looks like I own the docs. Builds are apparently failing - should I add you to the project? -glyph
Sure, I'd be happy to help (but won't get to it until the weekend). Username is twm. Thanks, Tom On 01/11/2017 04:44 PM, Glyph Lefkowitz wrote:
On Jan 10, 2017, at 6:43 PM, Tom Most <tommost@gmail.com <mailto:tommost@gmail.com>> wrote:
On 01/08/2017 06:49 PM, Glyph Lefkowitz wrote:
On Jan 6, 2017, at 11:03 AM, steven meiers <commercials24@yahoo.de> wrote:
hi,
sending a simple post request with a body (cookies) results in a error here.
im using python 3.x with the latest twisted and treq installed via pip in a virtualenv.
Something that we discussed on IRC but did not come up in the mailing list thread may be the one issue that *should* be fixed in treq:
Because of this issue, this example - http://treq.readthedocs.io/en/latest/#post - which the OP was following, doesn't work on Python 3.
We should fix the docs so that it does.
I submitted a PR to fix this a while ago: https://github.com/twisted/treq/pull/152
Also, the docs on RTD are still out of date. Who owns them? I'd be happy to help.
Thanks, hopefully the next time my periodic review comes around I'll catch that one.
Looks like I own the docs. Builds are apparently failing - should I add you to the project?
-glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Done. -g
On Jan 11, 2017, at 10:05 PM, Tom Most <tommost@gmail.com> wrote:
Sure, I'd be happy to help (but won't get to it until the weekend). Username is twm.
Thanks, Tom On 01/11/2017 04:44 PM, Glyph Lefkowitz wrote:
On Jan 10, 2017, at 6:43 PM, Tom Most < <mailto:tommost@gmail.com>tommost@gmail.com <mailto:tommost@gmail.com>> wrote:
On 01/08/2017 06:49 PM, Glyph Lefkowitz wrote:
On Jan 6, 2017, at 11:03 AM, steven meiers <commercials24@yahoo.de <mailto:commercials24@yahoo.de>> wrote:
hi,
sending a simple post request with a body (cookies) results in a error here.
im using python 3.x with the latest twisted and treq installed via pip in a virtualenv.
Something that we discussed on IRC but did not come up in the mailing list thread may be the one issue that *should* be fixed in treq:
Because of this issue, this example - <http://treq.readthedocs.io/en/latest/#post>http://treq.readthedocs.io/en/latest/#post <http://treq.readthedocs.io/en/latest/#post> - which the OP was following, doesn't work on Python 3.
We should fix the docs so that it does.
I submitted a PR to fix this a while ago: <https://github.com/twisted/treq/pull/152>https://github.com/twisted/treq/pull/152 <https://github.com/twisted/treq/pull/152>
Also, the docs on RTD are still out of date. Who owns them? I'd be happy to help.
Thanks, hopefully the next time my periodic review comes around I'll catch that one.
Looks like I own the docs. Builds are apparently failing - should I add you to the project?
-glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python <http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (5)
-
Cory Benfield -
Glyph Lefkowitz -
steven meiers -
Tom Most -
Tristan Seligmann