Sorry I had a typo in twisted program @defer.inlinecallbacks def long_computation(rec_type, data): # some long computation *defer.returnValue(recs)* @defer.inlinecallbacks def fetch_data(user_id): r = yieldjson.loads(requests.get('url/to/fetch/%s'%user_id).text) defer.returnValue(r) @defer.inlinecallbacks def fetch_recs(user_id): data = yield fetch_data(user_id) recs = {} for stype in similar_types: *d = defer.ToThread(long_computation, *(stype, data)) // typo was here* rec = yield d recs[stype] = rec defer.returnValue(recs) On Tue, Jun 25, 2019 at 11:48 PM Waqar Khan <wk80333@gmail.com> wrote:
Hello folks, I recently stumbled upon twisted and was wondering if it could suit my needs. On one hand, I want to use python but on another hand there are all these scalability concerns with this language so, I though I would pick the brains of the community. So.. a flask based app would look something like this.
similar_types = ['foo', 'bar', 'baz']
def long_computation(rec_type): # some long computation return recs
@app.route('/fetch_similar_users/<user_id>' def fetch_similar_users(user_id) r = json.loads(requests.get('url/to/fetch/%s'%user_id).text) recs = {} for stype in similar_types: recs[stype] = long_computation(rec_type) return recs
Now, I tried to "twistify" but it failed.
@defer.inlinecallbacks
def long_computation(rec_type): # some long computation *defer.returnValue(recs)*
@defer.inlinecallbacks def fetch_data(user_id): r = yieldjson.loads(requests.get('url/to/fetch/%s'%user_id).text) defer.returnValue(r)
@defer.inlinecallbacks def fetch_recs(user_id): data = yield fetch_data(user_id) recs = {} for stype in similar_types: d = defer.ToThread(fetch_data, *(stype)) rec = yield d recs[stype] = rec defer.returnValue(recs)
I wrapped all the above in twisted render_Get method.. but then I did a load test with locust ( https://docs.locust.io/en/latest/what-is-locust.html) framework. It choked. As the time progressed, the response time increased. I am guessing, things are still blocking.
Can you please help me look into the right place. Why exactly am I seeing increase in response time as the time progresses. I am guessing things are still working in "blocking" fashion but i thought the above should run things in async. Thanks