new library: sniffio – Sniff out which async library your code is running under
Hi all, A number of people are working on packages that support multiple async backends (e.g., asyncio + trio, or trio + curio, or trio + twisted, ...). So then the question arises... how can I figure out which async library my user is actually using? Answer: install sniffio, and then call sniffio.current_async_library(), and it tells you. Well, right now it only works for trio and asyncio, but if you maintain an async library and you want to make it easier for packages to detect you, then it's easy to add support – see the manual. We considered various clever things, but ultimately decided that the best approach was to use a ContextVar and make it the coroutine runner's responsibility to advertise which async flavor it uses. In particular, this approach works even for hybrid programs that are using multiple coroutine runners in the same loop, like a Twisted program with asyncio-flavored and twisted-flavored coroutines in the same thread, or a Trio program using trio-asyncio to run both asyncio-flavored and trio-flavored coroutines in the same thread. Github: https://github.com/python-trio/sniffio Manual: https://sniffio.readthedocs.io/ PyPI: https://pypi.org/p/sniffio -n -- Nathaniel J. Smith -- https://vorpus.org
Neat! On Thu, Aug 16, 2018 at 9:02 AM Nathaniel Smith <njs@pobox.com> wrote:
Hi all,
A number of people are working on packages that support multiple async backends (e.g., asyncio + trio, or trio + curio, or trio + twisted, ...). So then the question arises... how can I figure out which async library my user is actually using?
Answer: install sniffio, and then call sniffio.current_async_library(), and it tells you.
Well, right now it only works for trio and asyncio, but if you maintain an async library and you want to make it easier for packages to detect you, then it's easy to add support – see the manual. We considered various clever things, but ultimately decided that the best approach was to use a ContextVar and make it the coroutine runner's responsibility to advertise which async flavor it uses. In particular, this approach works even for hybrid programs that are using multiple coroutine runners in the same loop, like a Twisted program with asyncio-flavored and twisted-flavored coroutines in the same thread, or a Trio program using trio-asyncio to run both asyncio-flavored and trio-flavored coroutines in the same thread.
Github: https://github.com/python-trio/sniffio Manual: https://sniffio.readthedocs.io/ PyPI: https://pypi.org/p/sniffio
-n
-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
-- --Guido (mobile)
This was my approach: def _detect_running_asynclib() -> str: if 'trio' in sys.modules: from trio.hazmat import current_trio_token try: current_trio_token() except RuntimeError: pass else: return 'trio' if 'curio' in sys.modules: from curio.meta import curio_running if curio_running(): return 'curio' if 'asyncio' in sys.modules: from .backends.asyncio import get_running_loop if get_running_loop() is not None: return 'asyncio' raise LookupError('Cannot find any running async event loop') Is there something wrong with this? to, 2018-08-16 kello 00:01 -0700, Nathaniel Smith kirjoitti:
Hi all, A number of people are working on packages that support multiple asyncbackends (e.g., asyncio + trio, or trio + curio, or trio + twisted,...). So then the question arises... how can I figure out which asynclibrary my user is actually using? Answer: install sniffio, and then callsniffio.current_async_library(), and it tells you. Well, right now it only works for trio and asyncio, but if youmaintain an async library and you want to make it easier for packagesto detect you, then it's easy to add support – see the manual. Weconsidered various clever things, but ultimately decided that the bestapproach was to use a ContextVar and make it the coroutine runner'sresponsibility to advertise which async flavor it uses. In particular,this approach works even for hybrid programs that are using multiplecoroutine runners in the same loop, like a Twisted program withasyncio-flavored and twisted-flavored coroutines in the same thread,or a Trio program using trio-asyncio to run both asyncio- flavored andtrio-flavored coroutines in the same thread. Github: https://github.com/python-trio/sniffioManual: https://sniffio.readthedocs.io/PyPI: https://pypi.org/p/sniffio -n
Importation does not equate to execution. I.e. since I could have multiple event loops running at once that means what's in sys.modules can't tell me what event loop I'm currently interacting with. On Fri, 17 Aug 2018 at 09:09 Alex Grönholm <alex.gronholm@nextday.fi> wrote:
This was my approach:
def _detect_running_asynclib() -> str: if 'trio' in sys.modules: from trio.hazmat import current_trio_token try: current_trio_token() except RuntimeError: pass else: return 'trio'
if 'curio' in sys.modules: from curio.meta import curio_running if curio_running(): return 'curio'
if 'asyncio' in sys.modules: from .backends.asyncio import get_running_loop if get_running_loop() is not None: return 'asyncio'
raise LookupError('Cannot find any running async event loop')
Is there something wrong with this?
to, 2018-08-16 kello 00:01 -0700, Nathaniel Smith kirjoitti:
Hi all,
A number of people are working on packages that support multiple async
backends (e.g., asyncio + trio, or trio + curio, or trio + twisted,
...). So then the question arises... how can I figure out which async
library my user is actually using?
Answer: install sniffio, and then call
sniffio.current_async_library(), and it tells you.
Well, right now it only works for trio and asyncio, but if you
maintain an async library and you want to make it easier for packages
to detect you, then it's easy to add support – see the manual. We
considered various clever things, but ultimately decided that the best
approach was to use a ContextVar and make it the coroutine runner's
responsibility to advertise which async flavor it uses. In particular,
this approach works even for hybrid programs that are using multiple
coroutine runners in the same loop, like a Twisted program with
asyncio-flavored and twisted-flavored coroutines in the same thread,
or a Trio program using trio-asyncio to run both asyncio-flavored and
trio-flavored coroutines in the same thread.
Github: https://github.com/python-trio/sniffio
Manual: https://sniffio.readthedocs.io/
PyPI: https://pypi.org/p/sniffio
-n
_______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
If you look at the code more carefully, you'll see that I'm not merely checking what's been imported. In each case I'm asking the relevant framework if they're running an event loop *in the current thread*. pe, 2018-08-17 kello 09:26 -0700, Brett Cannon kirjoitti:
Importation does not equate to execution. I.e. since I could have multiple event loops running at once that means what's in sys.modules can't tell me what event loop I'm currently interacting with.
On Fri, 17 Aug 2018 at 09:09 Alex Grönholm <alex.gronholm@nextday.fi> wrote:
This was my approach:
def _detect_running_asynclib() -> str: if 'trio' in sys.modules: from trio.hazmat import current_trio_token try: current_trio_token() except RuntimeError: pass else: return 'trio'
if 'curio' in sys.modules: from curio.meta import curio_running if curio_running(): return 'curio'
if 'asyncio' in sys.modules: from .backends.asyncio import get_running_loop if get_running_loop() is not None: return 'asyncio'
raise LookupError('Cannot find any running async event loop')
Is there something wrong with this? to, 2018-08-16 kello 00:01 -0700, Nathaniel Smith kirjoitti:
Hi all, A number of people are working on packages that support multiple asyncbackends (e.g., asyncio + trio, or trio + curio, or trio + twisted,...). So then the question arises... how can I figure out which asynclibrary my user is actually using? Answer: install sniffio, and then callsniffio.current_async_library(), and it tells you. Well, right now it only works for trio and asyncio, but if youmaintain an async library and you want to make it easier for packagesto detect you, then it's easy to add support – see the manual. Weconsidered various clever things, but ultimately decided that the bestapproach was to use a ContextVar and make it the coroutine runner'sresponsibility to advertise which async flavor it uses. In particular,this approach works even for hybrid programs that are using multiplecoroutine runners in the same loop, like a Twisted program withasyncio-flavored and twisted- flavored coroutines in the same thread,or a Trio program using trio-asyncio to run both asyncio-flavored andtrio-flavored coroutines in the same thread. Github: https://github.com/python-trio/sniffioManual: https://sniffio.readthedocs.io/PyPI: https://pypi.org/p/sniffio -n
_______________________________________________
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/
On Fri, Aug 17, 2018, 09:09 Alex Grönholm <alex.gronholm@nextday.fi> wrote:
This was my approach:
def _detect_running_asynclib() -> str: if 'trio' in sys.modules: from trio.hazmat import current_trio_token try: current_trio_token() except RuntimeError: pass else: return 'trio'
if 'curio' in sys.modules: from curio.meta import curio_running if curio_running(): return 'curio'
if 'asyncio' in sys.modules: from .backends.asyncio import get_running_loop if get_running_loop() is not None: return 'asyncio'
raise LookupError('Cannot find any running async event loop')
Is there something wrong with this?
If you're using trio-asyncio, then you can have both trio-flavored coroutines and asyncio-flavored coroutines running in the same thread. And in particular, the trio and asyncio tests you do above will both return true at the same time, even though at any given moment only you can only 'await' one kind of async function or the other. Twisted running on the asyncio reactor has a similar situation. -n
If I'm reading the docs correctly, it looks like an async library has to depend on sniffio in order to be detected by sniffio: https://sniffio.readthedocs.io/en/latest/#adding-support-to-a-new-async-libr... Did you also think about whether it would be possible for a library to advertise itself without having to depend on a third-party library (e.g. using some sort of convention)? That would permit a less "centralized" approach. --Chris On Thu, Aug 16, 2018 at 12:01 AM, Nathaniel Smith <njs@pobox.com> wrote:
Hi all,
A number of people are working on packages that support multiple async backends (e.g., asyncio + trio, or trio + curio, or trio + twisted, ...). So then the question arises... how can I figure out which async library my user is actually using?
Answer: install sniffio, and then call sniffio.current_async_library(), and it tells you.
Well, right now it only works for trio and asyncio, but if you maintain an async library and you want to make it easier for packages to detect you, then it's easy to add support – see the manual. We considered various clever things, but ultimately decided that the best approach was to use a ContextVar and make it the coroutine runner's responsibility to advertise which async flavor it uses. In particular, this approach works even for hybrid programs that are using multiple coroutine runners in the same loop, like a Twisted program with asyncio-flavored and twisted-flavored coroutines in the same thread, or a Trio program using trio-asyncio to run both asyncio-flavored and trio-flavored coroutines in the same thread.
Github: https://github.com/python-trio/sniffio Manual: https://sniffio.readthedocs.io/ PyPI: https://pypi.org/p/sniffio
-n
-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
On Fri, Aug 17, 2018, 12:12 Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
If I'm reading the docs correctly, it looks like an async library has to depend on sniffio in order to be detected by sniffio:
https://sniffio.readthedocs.io/en/latest/#adding-support-to-a-new-async-libr...
If you don't want to depend then you can do the traditional try: import sniffio except ImportError: have_sniffio = False else: have_sniffio = True dance. For trio I was lazy and just made it a dependency because the sniffio wheel is a whopping 4 kilobytes. But there's no problem with doing that if you want.
Did you also think about whether it would be possible for a library to advertise itself without having to depend on a third-party library (e.g. using some sort of convention)? That would permit a less "centralized" approach.
What kind of convention do you have in mind? The problem with a convention AFAICT is that you need some shared agreement about where to rendezvous. That's basically all the sniffio library is: a shared, neutral place for libraries to advertise themselves. (Plus a fallback for detecting asyncio, because stdlib libraries have special constraints.) -n
On Fri, Aug 17, 2018 at 12:50 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Aug 17, 2018, 12:12 Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
Did you also think about whether it would be possible for a library to advertise itself without having to depend on a third-party library (e.g. using some sort of convention)? That would permit a less "centralized" approach.
What kind of convention do you have in mind?
Good question. I don't claim to know the answer which is why I asked if you had thought about it. The *kind* of thing I had in mind was to set a variable with an agreed-upon name and value on an agreed-upon module in the standard library -- though I agree that seems hacky as stated. It does seem to me like something that should (already?) have a general solution. What other ways does Python let things register or "announce" themselves? --Chris
The problem with a convention AFAICT is that you need some shared agreement about where to rendezvous. That's basically all the sniffio library is: a shared, neutral place for libraries to advertise themselves. (Plus a fallback for detecting asyncio, because stdlib libraries have special constraints.)
-n
On Fri, Aug 17, 2018 at 11:44 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Fri, Aug 17, 2018 at 12:50 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Aug 17, 2018, 12:12 Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
Did you also think about whether it would be possible for a library to advertise itself without having to depend on a third-party library (e.g. using some sort of convention)? That would permit a less "centralized" approach.
What kind of convention do you have in mind?
Good question. I don't claim to know the answer which is why I asked if you had thought about it. The *kind* of thing I had in mind was to set a variable with an agreed-upon name and value on an agreed-upon module in the standard library -- though I agree that seems hacky as stated.
It does seem to me like something that should (already?) have a general solution. What other ways does Python let things register or "announce" themselves?
Well, you could register an entry in an some global dict under an agreed-on key, like, say, sys.modules["sniffio"]. Of course, whenever you're mutating a global object like this you should worry about name collisions, but fortunately that particular dict has a good convention for reserving names. In fact there's a whole web service called "PyPI" devoted to managing those registrations! And then you might as well upload the code for accessing that variable to the web service, so everyone doesn't have to copy/paste it into their programs... ;-) Now that packaging works reliably, it's a pretty good solution for this kind of thing IMHO. -n -- Nathaniel J. Smith -- https://vorpus.org
On Sat, Aug 18, 2018 at 12:20 AM, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Aug 17, 2018 at 11:44 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Fri, Aug 17, 2018 at 12:50 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Aug 17, 2018, 12:12 Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
Did you also think about whether it would be possible for a library to advertise itself without having to depend on a third-party library (e.g. using some sort of convention)? That would permit a less "centralized" approach.
What kind of convention do you have in mind?
Good question. I don't claim to know the answer which is why I asked if you had thought about it. The *kind* of thing I had in mind was to set a variable with an agreed-upon name and value on an agreed-upon module in the standard library -- though I agree that seems hacky as stated.
It does seem to me like something that should (already?) have a general solution. What other ways does Python let things register or "announce" themselves?
Well, you could register an entry in an some global dict under an agreed-on key, like, say, sys.modules["sniffio"]. Of course, whenever you're mutating a global object like this you should worry about name collisions, but fortunately that particular dict has a good convention for reserving names. In fact there's a whole web service called "PyPI" devoted to managing those registrations! And then you might as well upload the code for accessing that variable to the web service, so everyone doesn't have to copy/paste it into their programs... ;-)
Yes, I know. My original question was whether it would be possible _without_ a third-party library. It just feels to me as overly heavy-weight to rely on all of that infrastructure just to pick a key name. You have to rely on convention for the "identifier string" anyways to prevent name collisions. (It can still be misused.) And the sniffio docs already have code you have to copy-and-paste anyways (not including the "have_sniffio = False" code above). --Chris
Now that packaging works reliably, it's a pretty good solution for this kind of thing IMHO.
-n
-- Nathaniel J. Smith -- https://vorpus.org
Also, just to be clear, I think the idea of a library to sniff this information is great. It's just the location of where this information is being stored that I'm focusing in on and asking about. It seems like it should be in a "neutral" location, and in particular different from / decoupled from the library above. One advantage of decoupling the sniffing library from the registry location is that it would allow for someone to write an improved library in the future (say "detectio") without forever locking in / requiring the old library to be installed. --Chris On Sat, Aug 18, 2018 at 12:50 AM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Sat, Aug 18, 2018 at 12:20 AM, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Aug 17, 2018 at 11:44 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Fri, Aug 17, 2018 at 12:50 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Aug 17, 2018, 12:12 Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
Did you also think about whether it would be possible for a library to advertise itself without having to depend on a third-party library (e.g. using some sort of convention)? That would permit a less "centralized" approach.
What kind of convention do you have in mind?
Good question. I don't claim to know the answer which is why I asked if you had thought about it. The *kind* of thing I had in mind was to set a variable with an agreed-upon name and value on an agreed-upon module in the standard library -- though I agree that seems hacky as stated.
It does seem to me like something that should (already?) have a general solution. What other ways does Python let things register or "announce" themselves?
Well, you could register an entry in an some global dict under an agreed-on key, like, say, sys.modules["sniffio"]. Of course, whenever you're mutating a global object like this you should worry about name collisions, but fortunately that particular dict has a good convention for reserving names. In fact there's a whole web service called "PyPI" devoted to managing those registrations! And then you might as well upload the code for accessing that variable to the web service, so everyone doesn't have to copy/paste it into their programs... ;-)
Yes, I know. My original question was whether it would be possible _without_ a third-party library.
It just feels to me as overly heavy-weight to rely on all of that infrastructure just to pick a key name. You have to rely on convention for the "identifier string" anyways to prevent name collisions. (It can still be misused.) And the sniffio docs already have code you have to copy-and-paste anyways (not including the "have_sniffio = False" code above).
--Chris
Now that packaging works reliably, it's a pretty good solution for this kind of thing IMHO.
-n
-- Nathaniel J. Smith -- https://vorpus.org
On Sat, Aug 18, 2018, 01:22 Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
Also, just to be clear, I think the idea of a library to sniff this information is great.
It's just the location of where this information is being stored that I'm focusing in on and asking about. It seems like it should be in a "neutral" location, and in particular different from / decoupled from the library above.
One advantage of decoupling the sniffing library from the registry location is that it would allow for someone to write an improved library in the future (say "detectio") without forever locking in / requiring the old library to be installed.
Yeah, we discussed this some when initially designing it, because we were worried about the lock in issue too. Obviously if you want to do anything you need to make some decisions, but to reduce the risk here we intentionally kept sniffio as minimal and unopinionated as possible: https://github.com/python-trio/sniffio/issues/1#issuecomment-408812146 Have you seen the code? It's not *entirely* trivial – certainly big enough to contain a bug or two – but it's tiny and only contains what it absolutely needs to work: a contextvar (native on 3.7, or via Yury's backport library on earlier pythons), and a fallback for detecting asyncio (since as a stdlib module it can't really work any other way). https://github.com/python-trio/sniffio/blob/master/sniffio/_impl.py -n
On Sat, Aug 18, 2018 at 2:13 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Sat, Aug 18, 2018, 01:22 Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
Also, just to be clear, I think the idea of a library to sniff this information is great.
It's just the location of where this information is being stored that I'm focusing in on and asking about. It seems like it should be in a "neutral" location, and in particular different from / decoupled from the library above.
One advantage of decoupling the sniffing library from the registry location is that it would allow for someone to write an improved library in the future (say "detectio") without forever locking in / requiring the old library to be installed.
Yeah, we discussed this some when initially designing it, because we were worried about the lock in issue too. Obviously if you want to do anything you need to make some decisions, but to reduce the risk here we intentionally kept sniffio as minimal and unopinionated as possible: https://github.com/python-trio/sniffio/issues/1#issuecomment-408812146
Have you seen the code? It's not *entirely* trivial – certainly big enough to contain a bug or two – but it's tiny and only contains what it absolutely needs to work: a contextvar (native on 3.7, or via Yury's backport library on earlier pythons), and a fallback for detecting asyncio (since as a stdlib module it can't really work any other way).
https://github.com/python-trio/sniffio/blob/master/sniffio/_impl.py
Yes, I had looked at the code a few times and agree it's simple and useful. (But it could always grow.) I looked in the linked issue and didn't see any discussion of the registry location though, which is the part I was interested in. The kind of alternative I had in mind for a neutral location is setting an attribute with an agreed upon name on a module in the standard lib, perhaps something like `contextvars.current_async_library_cvar` to use your naming. This is analogous to agreeing on a file name to store information of a certain kind in a repository root, like .travis.yml, package.json, or pyproject.toml. It's light-weight and doesn't require any infrastructure or tying to a particular package on PyPI. --Chris
On Sat, Aug 18, 2018 at 2:44 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
The kind of alternative I had in mind for a neutral location is setting an attribute with an agreed upon name on a module in the standard lib, perhaps something like `contextvars.current_async_library_cvar` to use your naming. This is analogous to agreeing on a file name to store information of a certain kind in a repository root, like .travis.yml, package.json, or pyproject.toml. It's light-weight and doesn't require any infrastructure or tying to a particular package on PyPI.
Yeah, it'd be possible. I guess it just didn't seem worth the extra complication. -n -- Nathaniel J. Smith -- https://vorpus.org
participants (5)
-
Alex Grönholm -
Brett Cannon -
Chris Jerdonek -
Guido van Rossum -
Nathaniel Smith