how to handle response data that is streaming and chunked?
MRAB
python at mrabarnett.plus.com
Thu Mar 21 17:21:49 EDT 2019
On 2019-03-21 20:12, Artie Ziff wrote:
> Hello,
>
> I am trying to learn how to do use response library in the least confusing
> way possible. My data source is streaming.
>
> The sample I share below looks more complicated that it needs to be. I do
> not have enough experience with this to know better. Hence why I came here
> for guidance and direction. Comments are very welcome as well as pointers
> to reading material. I want to learn this.
>
> I expected that I was going to be able to use the JSON methods such as load
> or loads. After all, this is JSON and should be able to slurp into builtin
> python objects and data structures, I'd imagine.
>
> Does this mailing list use a paste site or is better to paste code in-line
> for future?
>
> Thanks for reading!
> --Art
>
>
> import os, sys, json, requests, signal
> from requests_oauthlib import OAuth1
>
> api_key = 'HIDDEN'
> api_secret = 'HIDDEN'
> user_token_key = 'HIDDEN'
> user_token_secret = 'HIDDEN'
>
> authorization = OAuth1(api_key, api_secret, user_token_key,
> user_token_secret)
> session = requests.Session()
> session.auth = authorization
> finis = 0
>
>
> def handler(signum, frame):
> global finis
> finis = 1
>
>
> signal.signal(signal.SIGINT, handler)
>
> url = "https://stream.tradeking.com/v1/market/quotes.json?symbols=AAPL,DJT"
> resp = session.get(url, stream=True)
> lines = ''
> for chunk in resp.iter_content(None, decode_unicode=True):
> if finis:
> break
> lines += chunk.decode('utf-8')
> while lines.find('}}') > 0:
> line, lines = lines.split('}}', 1)
> print(line)
>
As far I can tell from the docs, calling '.iter_content' with
'decode_unicode=True' will make it decode, so you shouldn't be decoding
it again.
Also, string indexes start at 0, and '.find' will return -1 if the
string is not found.
As you don't need the index itself, you might as well use 'in' instead:
while '}}' in lines:
On the other hand, you're splitting the string, so maybe it would be
better to use '.partition':
before, sep, after = lines.partition('}}'):
while sep:
print(before + sep)
lines = after
before, sep, after = lines.partition('}}'):
Note that when you split or partition on '}}', you'll lose the '}}'
itself if you're not careful, hence the 'before + sep' above.
More information about the Python-list
mailing list