Re: [Python-Dev] Question about sys.path and sys.argv and how packaging (may) affects default values

On 2016-03-02 18:45, Thomas Wouters wrote:

On Wed, Mar 2, 2016 at 3:50 AM, Michael Felt wrote:

...

Hello all,

1) There are many lists to choose from - if this is the wrong one for questions about packaging - please forgive me, and point me in the right direction.

It's hard to say where this belongs best, but python-list would probably have done as well.

...

2) Normally, I have just packaged python, and then moved on. However, recently I have been asked to help with packaging an 'easier to install' python by people using cloud-init, and more recently people wanting to use salt-stack (on AIX).

FYI: I have been posting about my complete failure to build 2.7.11 ( http://bugs.python.org/issue26466) - so, what I am testing is based on 2.7.10 - which built easily for me.

Going through the 'base documentation' I saw a reference to both sys.argv and sys.path. atm, I am looking for an easy way to get the program name (e.g., /opt/bin/python, versus ./python). I have my reasons (basically, looking for a compiled-in library search path to help with http://bugs.python.org/issue26439)

I think the only way to get at the compiled-in search path is to recreate it based on the compiled-in prefix, which you can get through distutils. Python purposely only uses the compiled-in path as the last resort. Instead, it searches for its home relative to the executable and adds a set of directories relative to its home (if they exist).

It's not clear to me why you're focusing on these differences, as (as I describe below) they are immaterial.

...

Looking on two platforms (AIX, my build, and debian for power) I am surprised that sys.argv is empty in both cases, and sys.path returns /opt/lib/python27.zip with AIX, but not with debian.

When you run python interactively, sys.argv[0] will be '', yes. Since you're not launching a program, there's nothing else to set it to. 'python' (or the path to the executable) wouldn't be the right thing to set it to, because python itself isn't a Python program :)

The actual python executable is sys.executable, not sys.argv[0], but you shouldn't usually care about that, either. If you want to know where to install things, distutils is the thing to use. If you want to know where Python thinks it's installed (for debugging purposes only, really), sys.prefix will tell you.

...

root@x064:[/data/prj/aixtools/python/python-2.7.10]/opt/bin/python Python 2.7.10 (default, Nov 3 2015, 14:36:51) [C] on aix5 Type "help", "copyright", "credits" or "license" for more information.

...

...

...

import sys sys.argv [''] sys.path ['', '/opt/lib/python27.zip', '/opt/lib/python2.7', '/opt/lib/python2.7/plat-aix5', '/opt/lib/python2.7/lib-tk', '/opt/lib/python2.7/lib-old', '/opt/lib/python2.7/lib-dynload', '/opt/lib/python2.7/site-packages']

michael@ipv4:~$ python Python 2.7.9 (default, Mar 1 2015, 13:01:00) [GCC 4.9.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.

...

...

...

import sys sys.argv [''] sys.path ['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-powerpc-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7']

In sys.path, you're seeing the difference between a vanilla Python and Debian's patched Python. Vanilla Python adds $prefix/lib/python27.zip to sys.path unconditionally, whereas Debian removes it when it doesn't exist. Likewise, the dist-packages directory is a local modification by Debian; in vanilla Python it's called 'site-packages' instead. The subdirectories in dist-packages that you see in the Debian case are added by .pth files installed in $prefix -- third-party packages, in other words, adding their own directories to the module search path.

...

And I guess I would be interested in getting '/opt/lib/python2.7/dist-packages' in there as well (or learn a way to later add it for pre-compiled packages such as cloud-init AND that those would also look 'first' in /opt/lib/python2.7/dist-packages/cloud-init for modules added to support cloud-init - should I so choose (mainly in case of compatibility issues between say cloud-init and salt-stack that have common modules BUT may have conflicts) - Hopefully never needed for that reason, but it might also simplify packaging applications that depend on python.

A vanilla Python (or non-Debian-built python, even) has no business looking in dist-packages. It should just use site-packages. Third-party packages shouldn't care whether they're installed in site-packages or dist-packages, and instead should use distutils one way or another (if not by having an actual setup.py that uses distutils or setuptools, then at least by querying distutils for the installation directory the way python-config does).

...

Many thanks for your time and pointers into the documentation, It is a bit daunting :)

Michael _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/thomas%40python.org

Many Thanks Thomas for the extensive answer. My question is to help me understand where to look for default libraries in order to work on a patch. That has come a long way, but it is stuck now with/by something else I need to learn to debug (to find where the non-zero exit status comes from during a build). As a packager I hope to be as 'vanilla' as possible from the python perspective. Another time (another list!) I shall ask about what goes into python.zip Michael