
I mainly did up this patch to see how hard it would be, and now it's turned out to be fairly simple, I'm curious as to whether it would actually be useful to people. At the point where a global/builtin name lookup is about to raise NameError, first try calling a Python function, along the same lines as __getattr__. If that function hasn't been defined, raise NameError as normal; but if it has, let it either raise NameError or return some object, which is then used as if the name had been bound to it. Patch is here: http://bugs.python.org/issue23126 The idea is to allow convenient interactive use; auto-importing modules is easy, and importing names from modules ("exp" --> math.exp) can be done easily enough too, given a list of modules to try. It's probably not a good idea to use this in application code, and I definitely wouldn't encourage it in library code, but it has its uses interactively. Thoughts? ChrisA

+1 from me. The interactive uses would be great, and aren't really nice to implement without a hook like this. Top-posted from my Windows Phone ________________________________ From: Chris Angelico<mailto:rosuav@gmail.com> Sent: 12/28/2014 10:42 To: python-ideas<mailto:python-ideas@python.org> Subject: [Python-ideas] Python hook just before NameError I mainly did up this patch to see how hard it would be, and now it's turned out to be fairly simple, I'm curious as to whether it would actually be useful to people. At the point where a global/builtin name lookup is about to raise NameError, first try calling a Python function, along the same lines as __getattr__. If that function hasn't been defined, raise NameError as normal; but if it has, let it either raise NameError or return some object, which is then used as if the name had been bound to it. Patch is here: http://bugs.python.org/issue23126 The idea is to allow convenient interactive use; auto-importing modules is easy, and importing names from modules ("exp" --> math.exp) can be done easily enough too, given a list of modules to try. It's probably not a good idea to use this in application code, and I definitely wouldn't encourage it in library code, but it has its uses interactively. Thoughts? ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Sun, Dec 28, 2014 at 4:42 PM, Chris Angelico <rosuav@gmail.com> wrote:
I find it weird that it's *really truly* global, instead of being scoped to a single global namespace. Random thoughts: http://bugs.python.org/issue22986 enables a similar feature for module attribute lookups, which is one substantial source of "global" lookups. (Also it's been waiting for a review for some time, *coughcough* ;-)). If evaluating a specific piece of code, you can already pass an arbitrary dict-like object as the globals argument to exec, with whatever __getitem__ you want. Of course the built-in REPL doesn't provide any way to set the interactive namespace's globals object, but it could, or it'd be easy to implement in a REPL like IPython. You could achieve a similar effect by assigning a custom object to the interactive namespace's __builtins__ variable. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org

On Mon, Dec 29, 2014 at 03:42:16AM +1100, Chris Angelico wrote:
An interesting idea, but I don't actually think much of it for interactive use. Having modules magically import themselves without an import is a bad habit for beginners to learn, and less useful for experienced users who know to import things. If I've understood it correctly, it's also process-wide global, rather than limited to a single module. That makes it much less useful, as it risks disguising bugs in library code. -- Steven

On Sun, Dec 28, 2014 at 6:37 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Though if you are using interactive mode to test library code, you might have bigger problems on your hands than implicit imports... This topic came up in a completely unrelated thread in c.l.py wherein I apologized for an automatic import of the csv module in some interactive copy-n-paste. Chris asked about it, and I posted the module I use (which I don't think was original with me) as a convenience in interactive mode. Then he looked into things and came up with this patch. S

On 28/12/2014 16:42, Chris Angelico wrote:
On Mon, Dec 29, 2014 at 6:46 PM, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote:
+1 from me as I'm always forgetting the "obvious" imports such as sys and os when trying things interactively.
On Mon, Dec 29, 2014 at 11:37 AM, Steven D'Aprano <steve@pearwood.info> wrote:
This is why the feature is not "auto-import anything", but "give Python code the chance to deal with NameError". Open to bikeshedding about whether it's better per-module or global or what, but the point is that you, as the programmer, get the flexibility. Maybe you want to white-list potential imports, which would make this like pre-importing those names but lazily. Maybe you want to allow a half-dozen common "from" imports, but not just anything. Maybe you'd like to have some completely different magic - like REXX mode:
Or something more suitable for floating point work:
The power is in your hands. And by default, nothing is any different: $ python3 Python 3.5.0a0 (default:1c51f1650c42+, Dec 29 2014, 02:29:06) [GCC 4.7.2] on linux Type "help", "copyright", "credits" or "license" for more information.
Don't like auto-importing? Don't use it. That simple. :) Incidentally, I fully expect that the dunder name __getglobal__ will be rejected. But the proposal works equally well with any name. ChrisA

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 29.12.2014 09:00, Chris Angelico wrote:
Or with sympy, have undefined names be a sympy.core.symbol.Symbol automatically. This is something which cannot be accomplished with the PYTHONSTARTUP-script I already have for calculator work in python (importing sympy and defining a few more constants and such). regards, jwi p.s.: Sorry for the messed up quote, didn’t manage to figure out how to trick my client into not re-flowing the interactive prompt. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCgAGBQJUo968AAoJEMBiAyWXYliK2tAQAJfqwGn5vBhck1bdzbPVFMED Vu65jOAssvCk7reCTmtKtpY4PYDHde+XfDawRShagTPnMFpVvADAAxfOcLb11blr 6vrcCTitAPoy7NbOU44TjHDeAsrDkibF6HDtDRMy3Z6Vu+srRe/+teHQTjSZaR1W zFhBh0lCVYnL/I67ADEJbU7AzDK4na/3CrwgSrcTVCXzgdaKeVixtKypdli5AHis IiXURVy1Zy52xDpzbfhUgEMWMuMANOnX9wXjKaS2JlaHIT+YJ5MCCbqE7tlVElJb HAKz1nLwaN6QE8tXtb1EWNV/3Lnsr8dcd0BBNU0v6YU5JvZGK69rcpGwUxJJzjNK 5HT64Z/LQB64yRwlf6butIPppzsvMWCAzGmlZoarvwnSh+gnbuj8M9syVnbqhcTS 8tHyuOqvRU+VPGY5TdXOR4Qq56nSPyEsDHB3hPeXsRh0+1sn4AQEMUhhKrLfRdPP nyTUYvnMWMcZr4b3CR2nNujz3gq5ON/IbINPrrQwU4fSGxqTHkofi4fOfjB56nNw Fzbuv6wwXzLoFPmknTU9EiZD1EsIzgvW3bKzN5qzgKBsN6nN23UBroTU3vz2/zM9 jkgQhQ7lgp9+x/myRwxc7LAefJ1DqFiXubcS95Y9tzGh6HFeSc4t1cBLEOVRQ3xy PKzIobEoSFL5M3QakFDU =6GR9 -----END PGP SIGNATURE-----

On 29 Dec 2014 07:46, "Mark Lawrence" <breamoreboy@yahoo.co.uk> wrote:
os when trying things interactively. I just set up a PYTHONSTARTUP script to pre-import the obvious things and that fixed this problem for me. You'd still have to make a startup script to set up the hook, and it's not like 'import os, sys' will appreciably affect startup time. -n

On Mon, Dec 29, 2014 at 10:43 PM, Nathaniel Smith <njs@pobox.com> wrote:
Maybe those two won't, but I work with a number of Python students who are using heavier libraries like psycopg2 and numpy, so I'd really rather not auto-import those into everything. On the other hand, the course is currently written for Python 2.7, so I can't actually make use of this there... yet. Another reason to nudge the course writers about updating to 3.x! ChrisA

On 29 December 2014 at 21:50, Chris Angelico <rosuav@gmail.com> wrote:
FWIW, making it easier to create CPython variants that pre-populate __main__ differently (ala pyp or the --pylab option to ipython) is one of the reasons I'd like to eventually implement the interpreter initialisation changes proposed in PEP 432 (or see it implemented). However, even today, it's not particularly difficult to create custom launch scripts that initialise __main__ with a few different modules based on what you plan to work on and then set os.environ["PYTHONINSPECT"] = "1" to drop into the interactive interpreter: $ python3 -c "import sys, os; os.environ['PYTHONINSPECT'] = '1'"
sys, os (<module 'sys' (built-in)>, <module 'os' from '/usr/lib64/python3.4/os.py'>)
Making existing pre-initialisation mechanisms easier to discover and/or use seems like a better path forward than adding yet more tools that are even more cryptic and hard to follow. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Dec 29, 2014 at 11:46 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Once again, that's fine for sys and os, which won't take long to import. For this to work with _every_ module on the system, you'd need to couple it with a lazy import mechanism. We had some proposals along those lines recently... what happened to them? ChrisA

On 29 December 2014 at 23:02, Chris Angelico <rosuav@gmail.com> wrote:
The infrastructure landed for 3.5 already, so the custom startup script can also register a lazy loader if it wants to do so: https://docs.python.org/dev/library/importlib.html#importlib.util.LazyLoader (Lazy loading is already possible, since it's just a particular way of using the existing import customisation mechanisms that have been around for over a decade, 3.5 just makes it easier by providing more of the infrastructure to do it directly in the standard library) You can also fairly easily have different startup scripts for different scenarios, and then use explicit imports to bring in any extra pieces you decide you want for a given session. One trick I'll sometimes use myself is to do my playing around in IPython Notebooks, where any common setup code can just go in the first cell. (I'm not sure if IPython Notebook has a "clone existing notebook" feature yet, but that would be a very nice way to have template environments with different setup cells) New language features, or even default interpreter features, are generally an absolute last resort for solving problems - the vast majority of problems can be better solved externally. Incurring the long term maintenance costs of standardisation only makes sense when the perceived pay-off is deemed likely to justify that investment. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, Dec 30, 2014 at 12:02:32AM +1100, Chris Angelico wrote:
I just threw this lazy import proxy object together. I haven't tested it extensively, but it seems to work: from types import ModuleType class LazyImporter(ModuleType): def __init__(self, name): self._module = None self.__name__ = name self.__package__ = None def __getattr__(self, name): if self._module is None: module = self._module = __import__(self.__name__) self.__package__ = module.__package__ else: module = self._module return getattr(module, name) def lazy_import(name): from sys import modules if name in modules: return modules[name] else: return LazyImporter(name) Usage: decimal = lazy_import('decimal') decimal.Decimal If the module has already been cached, you get the module itself, otherwise you get a proxy. This was so obvious and simple, I cannot believe I am the first to have thought of it. Perhaps there is something similar that has been extensively used and tested in the real-world that the standard library could steal? -- Steven

On 29/12/2014 11:43, Nathaniel Smith wrote:
I should really get around to that, it's been at the back of my mind to put it on the TODO list for about 12 years :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence

On Mon, Dec 29, 2014 at 07:46:02AM +0000, Mark Lawrence wrote:
+1 from me as I'm always forgetting the "obvious" imports such as sys and os when trying things interactively.
This is why I have a startup.py file that contains (among other things): import sys, os I'm not sure how you set environment variables under Windows, but under Linux I put this in my .bashrc file: export PYTHONSTARTUP=/home/steve/python/utilities/startup.py and whenever I start an interactive session, I get the common imports that I want, a custom prompt (I use 'py>' rather than '>>>') and assorted other goodies. -- Steven

+1 from me. The interactive uses would be great, and aren't really nice to implement without a hook like this. Top-posted from my Windows Phone ________________________________ From: Chris Angelico<mailto:rosuav@gmail.com> Sent: 12/28/2014 10:42 To: python-ideas<mailto:python-ideas@python.org> Subject: [Python-ideas] Python hook just before NameError I mainly did up this patch to see how hard it would be, and now it's turned out to be fairly simple, I'm curious as to whether it would actually be useful to people. At the point where a global/builtin name lookup is about to raise NameError, first try calling a Python function, along the same lines as __getattr__. If that function hasn't been defined, raise NameError as normal; but if it has, let it either raise NameError or return some object, which is then used as if the name had been bound to it. Patch is here: http://bugs.python.org/issue23126 The idea is to allow convenient interactive use; auto-importing modules is easy, and importing names from modules ("exp" --> math.exp) can be done easily enough too, given a list of modules to try. It's probably not a good idea to use this in application code, and I definitely wouldn't encourage it in library code, but it has its uses interactively. Thoughts? ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Sun, Dec 28, 2014 at 4:42 PM, Chris Angelico <rosuav@gmail.com> wrote:
I find it weird that it's *really truly* global, instead of being scoped to a single global namespace. Random thoughts: http://bugs.python.org/issue22986 enables a similar feature for module attribute lookups, which is one substantial source of "global" lookups. (Also it's been waiting for a review for some time, *coughcough* ;-)). If evaluating a specific piece of code, you can already pass an arbitrary dict-like object as the globals argument to exec, with whatever __getitem__ you want. Of course the built-in REPL doesn't provide any way to set the interactive namespace's globals object, but it could, or it'd be easy to implement in a REPL like IPython. You could achieve a similar effect by assigning a custom object to the interactive namespace's __builtins__ variable. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org

On Mon, Dec 29, 2014 at 03:42:16AM +1100, Chris Angelico wrote:
An interesting idea, but I don't actually think much of it for interactive use. Having modules magically import themselves without an import is a bad habit for beginners to learn, and less useful for experienced users who know to import things. If I've understood it correctly, it's also process-wide global, rather than limited to a single module. That makes it much less useful, as it risks disguising bugs in library code. -- Steven

On Sun, Dec 28, 2014 at 6:37 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Though if you are using interactive mode to test library code, you might have bigger problems on your hands than implicit imports... This topic came up in a completely unrelated thread in c.l.py wherein I apologized for an automatic import of the csv module in some interactive copy-n-paste. Chris asked about it, and I posted the module I use (which I don't think was original with me) as a convenience in interactive mode. Then he looked into things and came up with this patch. S

On 28/12/2014 16:42, Chris Angelico wrote:
On Mon, Dec 29, 2014 at 6:46 PM, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote:
+1 from me as I'm always forgetting the "obvious" imports such as sys and os when trying things interactively.
On Mon, Dec 29, 2014 at 11:37 AM, Steven D'Aprano <steve@pearwood.info> wrote:
This is why the feature is not "auto-import anything", but "give Python code the chance to deal with NameError". Open to bikeshedding about whether it's better per-module or global or what, but the point is that you, as the programmer, get the flexibility. Maybe you want to white-list potential imports, which would make this like pre-importing those names but lazily. Maybe you want to allow a half-dozen common "from" imports, but not just anything. Maybe you'd like to have some completely different magic - like REXX mode:
Or something more suitable for floating point work:
The power is in your hands. And by default, nothing is any different: $ python3 Python 3.5.0a0 (default:1c51f1650c42+, Dec 29 2014, 02:29:06) [GCC 4.7.2] on linux Type "help", "copyright", "credits" or "license" for more information.
Don't like auto-importing? Don't use it. That simple. :) Incidentally, I fully expect that the dunder name __getglobal__ will be rejected. But the proposal works equally well with any name. ChrisA

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 29.12.2014 09:00, Chris Angelico wrote:
Or with sympy, have undefined names be a sympy.core.symbol.Symbol automatically. This is something which cannot be accomplished with the PYTHONSTARTUP-script I already have for calculator work in python (importing sympy and defining a few more constants and such). regards, jwi p.s.: Sorry for the messed up quote, didn’t manage to figure out how to trick my client into not re-flowing the interactive prompt. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCgAGBQJUo968AAoJEMBiAyWXYliK2tAQAJfqwGn5vBhck1bdzbPVFMED Vu65jOAssvCk7reCTmtKtpY4PYDHde+XfDawRShagTPnMFpVvADAAxfOcLb11blr 6vrcCTitAPoy7NbOU44TjHDeAsrDkibF6HDtDRMy3Z6Vu+srRe/+teHQTjSZaR1W zFhBh0lCVYnL/I67ADEJbU7AzDK4na/3CrwgSrcTVCXzgdaKeVixtKypdli5AHis IiXURVy1Zy52xDpzbfhUgEMWMuMANOnX9wXjKaS2JlaHIT+YJ5MCCbqE7tlVElJb HAKz1nLwaN6QE8tXtb1EWNV/3Lnsr8dcd0BBNU0v6YU5JvZGK69rcpGwUxJJzjNK 5HT64Z/LQB64yRwlf6butIPppzsvMWCAzGmlZoarvwnSh+gnbuj8M9syVnbqhcTS 8tHyuOqvRU+VPGY5TdXOR4Qq56nSPyEsDHB3hPeXsRh0+1sn4AQEMUhhKrLfRdPP nyTUYvnMWMcZr4b3CR2nNujz3gq5ON/IbINPrrQwU4fSGxqTHkofi4fOfjB56nNw Fzbuv6wwXzLoFPmknTU9EiZD1EsIzgvW3bKzN5qzgKBsN6nN23UBroTU3vz2/zM9 jkgQhQ7lgp9+x/myRwxc7LAefJ1DqFiXubcS95Y9tzGh6HFeSc4t1cBLEOVRQ3xy PKzIobEoSFL5M3QakFDU =6GR9 -----END PGP SIGNATURE-----

On 29 Dec 2014 07:46, "Mark Lawrence" <breamoreboy@yahoo.co.uk> wrote:
os when trying things interactively. I just set up a PYTHONSTARTUP script to pre-import the obvious things and that fixed this problem for me. You'd still have to make a startup script to set up the hook, and it's not like 'import os, sys' will appreciably affect startup time. -n

On Mon, Dec 29, 2014 at 10:43 PM, Nathaniel Smith <njs@pobox.com> wrote:
Maybe those two won't, but I work with a number of Python students who are using heavier libraries like psycopg2 and numpy, so I'd really rather not auto-import those into everything. On the other hand, the course is currently written for Python 2.7, so I can't actually make use of this there... yet. Another reason to nudge the course writers about updating to 3.x! ChrisA

On 29 December 2014 at 21:50, Chris Angelico <rosuav@gmail.com> wrote:
FWIW, making it easier to create CPython variants that pre-populate __main__ differently (ala pyp or the --pylab option to ipython) is one of the reasons I'd like to eventually implement the interpreter initialisation changes proposed in PEP 432 (or see it implemented). However, even today, it's not particularly difficult to create custom launch scripts that initialise __main__ with a few different modules based on what you plan to work on and then set os.environ["PYTHONINSPECT"] = "1" to drop into the interactive interpreter: $ python3 -c "import sys, os; os.environ['PYTHONINSPECT'] = '1'"
sys, os (<module 'sys' (built-in)>, <module 'os' from '/usr/lib64/python3.4/os.py'>)
Making existing pre-initialisation mechanisms easier to discover and/or use seems like a better path forward than adding yet more tools that are even more cryptic and hard to follow. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Dec 29, 2014 at 11:46 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Once again, that's fine for sys and os, which won't take long to import. For this to work with _every_ module on the system, you'd need to couple it with a lazy import mechanism. We had some proposals along those lines recently... what happened to them? ChrisA

On 29 December 2014 at 23:02, Chris Angelico <rosuav@gmail.com> wrote:
The infrastructure landed for 3.5 already, so the custom startup script can also register a lazy loader if it wants to do so: https://docs.python.org/dev/library/importlib.html#importlib.util.LazyLoader (Lazy loading is already possible, since it's just a particular way of using the existing import customisation mechanisms that have been around for over a decade, 3.5 just makes it easier by providing more of the infrastructure to do it directly in the standard library) You can also fairly easily have different startup scripts for different scenarios, and then use explicit imports to bring in any extra pieces you decide you want for a given session. One trick I'll sometimes use myself is to do my playing around in IPython Notebooks, where any common setup code can just go in the first cell. (I'm not sure if IPython Notebook has a "clone existing notebook" feature yet, but that would be a very nice way to have template environments with different setup cells) New language features, or even default interpreter features, are generally an absolute last resort for solving problems - the vast majority of problems can be better solved externally. Incurring the long term maintenance costs of standardisation only makes sense when the perceived pay-off is deemed likely to justify that investment. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, Dec 30, 2014 at 12:02:32AM +1100, Chris Angelico wrote:
I just threw this lazy import proxy object together. I haven't tested it extensively, but it seems to work: from types import ModuleType class LazyImporter(ModuleType): def __init__(self, name): self._module = None self.__name__ = name self.__package__ = None def __getattr__(self, name): if self._module is None: module = self._module = __import__(self.__name__) self.__package__ = module.__package__ else: module = self._module return getattr(module, name) def lazy_import(name): from sys import modules if name in modules: return modules[name] else: return LazyImporter(name) Usage: decimal = lazy_import('decimal') decimal.Decimal If the module has already been cached, you get the module itself, otherwise you get a proxy. This was so obvious and simple, I cannot believe I am the first to have thought of it. Perhaps there is something similar that has been extensively used and tested in the real-world that the standard library could steal? -- Steven

On 29/12/2014 11:43, Nathaniel Smith wrote:
I should really get around to that, it's been at the back of my mind to put it on the TODO list for about 12 years :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence

On Mon, Dec 29, 2014 at 07:46:02AM +0000, Mark Lawrence wrote:
+1 from me as I'm always forgetting the "obvious" imports such as sys and os when trying things interactively.
This is why I have a startup.py file that contains (among other things): import sys, os I'm not sure how you set environment variables under Windows, but under Linux I put this in my .bashrc file: export PYTHONSTARTUP=/home/steve/python/utilities/startup.py and whenever I start an interactive session, I get the common imports that I want, a custom prompt (I use 'py>' rather than '>>>') and assorted other goodies. -- Steven
participants (8)
-
Chris Angelico
-
Jonas Wielicki
-
Mark Lawrence
-
Nathaniel Smith
-
Nick Coghlan
-
Skip Montanaro
-
Steve Dower
-
Steven D'Aprano