Hi,
  Thank you all for your kind response.

So, I am trying to use treq library

import treq
@defer.inlinecallbacks
def long_computation(rec_type, data):
# some long computation
defer.returnValue(recs)
@defer.inlinecallbacks
def fetch_data(user_id):
        r = yield treq.get('url/to/fetch/%s'%user_id)
    text = yield r.text()
     defer.returnValue(text)


@defer.inlinecallbacks
def fetch_recs(user_id):
       data = yield fetch_data(user_id)
        recs = {}
       for stype in similar_types:
            d = defer.ToThread(long_computation, *(stype, data)) // typo was here

Now, I do believe that the call is happening asyncronously. So.. yay..

But then, I feel like I have a misconception on how the yield works.
data = yield fetch_data(user_id)
I was hoping data here was actual data.. But it is a deferred.. Which makes sense.
And then.. this deferred is being passed on instead of the actual data...

My couple of questions are:
1) What is the difference between data = yield fetch_data(user_id) and data = fetch_data(user_id) (without yield). How does twisted handle these two ?
2) How do I actually send the data to long computation rather than a deferred.

Appreciate all the help.
Thanks


On Sat, Jul 13, 2019 at 1:57 AM Tom Most <twm@freecog.net> wrote:
On Thu, Jul 11, 2019, at 1:46 AM, Scott, Barry wrote:
> On Tuesday, 9 July 2019 22:04:11 BST Tom Most wrote:
>
> ...snip...
>
> > The reactor's own thread pool is really for DNS resolution.
>
> Is that still true in the default case? We are use the twisted code that talks
> to DNS servers as the threaded resolver adds too much latency.

As far as I know, yes. The higher-level APIs use getaddrinfo() at least.

https://twistedmatrix.com/documents/current/api/twisted.internet._resolver.GAIResolver.html
https://github.com/twisted/twisted/blob/c0776850e756adfcdc179a7fd9e4c8f5cbc4838d/src/twisted/internet/base.py#L1007

TCP6ClientEndpoint also invoke getaddrinfo() directly.

twisted.names is certainly more performance but it's missing some system integration features that make it unsuitable as a default:

* No support for the domain or search resolv.conf directives
* No NSS lookups (e.g., systemd integration)

This is all on Linux, YMMV on other platforms.

---Tom

_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python