Python Environment Initialization Scripts
I posted a question on c.l.py[1] looking for a feature and I suspect it doesn't exist. Some quick searches on -ideas and -dev didn't reveal anything either. Has the idea of [python] environment initialization been pitched/rejected before or does it already exist? Simply, I'm looking for a mechanism to automatically load some meta_path hooks after site initialization. Without modifying libpython, I can achieve what I'm trying to do by: * Mangling the site.py file to run the code. (in my case, "import c.lib; c.lib.install()") * Overriding the python3 executable (not desirable; consider embedding) * Provide an alternate executable. (not desirable; again embedding and lack of availability in scripts pointing directly to python3) * LD_PRELOAD overrides? (portability problems abound, I imagine. mangling seems preferable) * Perform installation in the dependent (duplication of installation in each dependent) [1] https://groups.google.com/forum/?fromgroups#!topic/comp.lang.python/CCHe8_WH...
On Sat, Jul 28, 2012 at 6:47 PM, James William Pye <x@jwp.io> wrote:
No need to modify site.py. Either create a sitecustomize.py in your site-packages. Here are some references: http://docs.python.org/library/site.html http://www.python.org/dev/peps/pep-0370 Nothing new is needed. As to metapath hooks, one alternative is to make a helper function that puts the hooks in place (or otherwise manages your hooks). Then you can call that API in each script that needs those hooks. -eric
On Jul 28, 2012, at 5:56 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
No need to modify site.py. Either create a sitecustomize.py in your site-packages.
I had recently forgotten about that. Thanks. =) However, from the documentation: "After these path manipulations, an attempt is made to import a module named site customize," "which can perform arbitrary site-specific customizations. It is typically created by a system "administrator in the site-packages directory." The "It is typically created by a system administrator" part is what worries me about using sitecustomize.py. In order for this to be automatic, the package containing the meta_path hook would have to install/setup sitecustomize.py upon `pip install metapathawesomeness` which may already be in use by the system administrator. That is, based on the above documentation, it doesn't seem appropriate for a package to presume that it can use sitecustomize.py in such a way.
Yeah, I referred to that in my list of alternatives. My gripe with it is the duplication across the set of dependent scripts/packages. However, it would seem to be the most appropriate solution ATM. =\
On Jul 28, 2012, at 8:17 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
But it's not desirable for it to be limited one user... The situation is that the meta_path hook is provided in a package that is depended on by other packages that contain modules that are importable by the hook. It doesn't seem particularly appropriate to me for a given *package* to be fiddling with the system administrator's sitecustomize or the user's sitecustomize. No? Now, that's not to say that I'm unwilling to be inappropriate. >8)
On Sun, Jul 29, 2012 at 3:22 PM, James William Pye <x@jwp.io> wrote:
The short answer is that, no, a coherent proposal for application specific preinitialisation hasn't been put forward as a PEP. The slightly longer answer is that there are related problems in terms of initialising debuggers, warnings, logging, faulthandler, coverage, profiling, etc, during development where it would be incredibly convenient to be able to execute a short snippet of Python code prior to execution. See http://bugs.python.org/issue14803 (I'm nervous about the security implications of a PYTHONRUNFIRST environment variable that is enabled by default, but the idea of a -C option doesn't pose any such concerns) In general though, messing with process global state (including the import system) implicitly is just *not cool*. It leads to applications that have a dependency on a particular tool being present, but don't actually make sure that it is available. Thus you end up with manual instructions that say "before running this script, make sure infrastructure X is in place to automagically take care of things I should be doing myself". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jul 28, 2012, at 11:57 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
The short answer is that, no, a coherent proposal for application specific preinitialisation hasn't been put forward as a PEP.
=)
Yeah, that's been painful.
Rightly so.
but the idea of a -C option doesn't pose any such concerns)
Cool. I implemented something like that for "-m postgresql.bin.pg_python"[1] Never ended up using it much. But I could see myself using it with the Python executable: alias py='python3 -C …' Tucking that into a sh source file that sets up my dev context.
Even in the case of some designated mechanism where there is some attempt to expose/record changes to process global state so that it can be easily referenced by a dependency, is there a win over making an explicit attempt to validate that the state change occurred by the dependency? I think in this particular case involving meta_path, there would be a small win with regards to being able to perform the check at package installation time as opposed to every time the package is imported, given a "designated mechanism" of sorts. Essentially, the package being installed requests assurance that a state change *will* occur upon Python startup. [1] https://github.com/jwp/py-postgresql/blob/v1.1/postgresql/python/command.py
On Sat, Jul 28, 2012 at 6:47 PM, James William Pye <x@jwp.io> wrote:
No need to modify site.py. Either create a sitecustomize.py in your site-packages. Here are some references: http://docs.python.org/library/site.html http://www.python.org/dev/peps/pep-0370 Nothing new is needed. As to metapath hooks, one alternative is to make a helper function that puts the hooks in place (or otherwise manages your hooks). Then you can call that API in each script that needs those hooks. -eric
On Jul 28, 2012, at 5:56 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
No need to modify site.py. Either create a sitecustomize.py in your site-packages.
I had recently forgotten about that. Thanks. =) However, from the documentation: "After these path manipulations, an attempt is made to import a module named site customize," "which can perform arbitrary site-specific customizations. It is typically created by a system "administrator in the site-packages directory." The "It is typically created by a system administrator" part is what worries me about using sitecustomize.py. In order for this to be automatic, the package containing the meta_path hook would have to install/setup sitecustomize.py upon `pip install metapathawesomeness` which may already be in use by the system administrator. That is, based on the above documentation, it doesn't seem appropriate for a package to presume that it can use sitecustomize.py in such a way.
Yeah, I referred to that in my list of alternatives. My gripe with it is the duplication across the set of dependent scripts/packages. However, it would seem to be the most appropriate solution ATM. =\
On Jul 28, 2012, at 8:17 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
But it's not desirable for it to be limited one user... The situation is that the meta_path hook is provided in a package that is depended on by other packages that contain modules that are importable by the hook. It doesn't seem particularly appropriate to me for a given *package* to be fiddling with the system administrator's sitecustomize or the user's sitecustomize. No? Now, that's not to say that I'm unwilling to be inappropriate. >8)
On Sun, Jul 29, 2012 at 3:22 PM, James William Pye <x@jwp.io> wrote:
The short answer is that, no, a coherent proposal for application specific preinitialisation hasn't been put forward as a PEP. The slightly longer answer is that there are related problems in terms of initialising debuggers, warnings, logging, faulthandler, coverage, profiling, etc, during development where it would be incredibly convenient to be able to execute a short snippet of Python code prior to execution. See http://bugs.python.org/issue14803 (I'm nervous about the security implications of a PYTHONRUNFIRST environment variable that is enabled by default, but the idea of a -C option doesn't pose any such concerns) In general though, messing with process global state (including the import system) implicitly is just *not cool*. It leads to applications that have a dependency on a particular tool being present, but don't actually make sure that it is available. Thus you end up with manual instructions that say "before running this script, make sure infrastructure X is in place to automagically take care of things I should be doing myself". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jul 28, 2012, at 11:57 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
The short answer is that, no, a coherent proposal for application specific preinitialisation hasn't been put forward as a PEP.
=)
Yeah, that's been painful.
Rightly so.
but the idea of a -C option doesn't pose any such concerns)
Cool. I implemented something like that for "-m postgresql.bin.pg_python"[1] Never ended up using it much. But I could see myself using it with the Python executable: alias py='python3 -C …' Tucking that into a sh source file that sets up my dev context.
Even in the case of some designated mechanism where there is some attempt to expose/record changes to process global state so that it can be easily referenced by a dependency, is there a win over making an explicit attempt to validate that the state change occurred by the dependency? I think in this particular case involving meta_path, there would be a small win with regards to being able to perform the check at package installation time as opposed to every time the package is imported, given a "designated mechanism" of sorts. Essentially, the package being installed requests assurance that a state change *will* occur upon Python startup. [1] https://github.com/jwp/py-postgresql/blob/v1.1/postgresql/python/command.py
participants (3)
-
Eric Snow -
James William Pye -
Nick Coghlan