Hello, A while ago I've proposed to refactor the APIs that provides access to the installation paths and configuration variables at runtime into a single module called "sysconfig", and make it easier for all implementations to work with them. I've started a branch and worked on it, and I'd like to ask here for some feedback. And in particular from Jython and IronPython people because they would probably need to work in that file for their implementation and/or propose things to add. My understanding is that we have people like Phillip (Jenvey), Michael F., Frank W. in this list so they can comment directly and I don't need to cross-post this mail elsewhere. == Installation schemes == First, the module contains the installation schemes for each platform CPython uses. An install scheme is a mapping where the key is the "code" name for a directory, and the value the path of that directory, with some $variable that can be expanded. Install schemes are stored in a private mapping, where the keys are usually the value of os.name, and the value, the mapping I've mentionned earlier. So, for example, the paths for win32 are: _INSTALL_SCHEMES = { ... 'nt': { 'stdlib': '$base/Lib', 'platstdlib': '$base/Lib', 'purelib': '$base/Lib/site-packages', 'platlib': '$base/Lib/site-packages', 'include': '$base/include', 'platinclude': '$base/include', 'scripts': '$base/Scripts', 'data' : '$base', }, ... } where each key corresponds to a directory that contains some Python files: - stdlib : root of the standard library - platstdlib: root of platform-specific elements of the standard library - purelib: the site-packages directory for pure python modules - platlib: the site-packages directory for platform-specific modules - include: the include dir - platinclude: the include dir for platform-specific files - scripts: the directory where scripts are added - data: the directory where data file are added All these directory are read and used by: - distutils when a package is installed, so the install command can dispatch files in the right place - site.py, when Python is initialized IOW, any part of the stdlib can use these paths to locate and work with Python files. The public APIs are: * get_path_names() : returns a list of the path names ("stdlib", "platstdlib", etc.) * get_paths(scheme, vars) : Returns a mapping containing an install scheme. - "scheme" is the name of the scheme, if not provided will get the default scheme of the current platform - vars is an optonal mapping that can provide values for the various $variables. Notice that they all have default values, for example $base == sys.prefix. for example: get_paths('nt') * get_path(name, scheme, vars): Returns one path corresponding to the scheme. for example : get_paths('stdlib', 'nt') Those API are generic, but maybe we could add specific APIs like: * get_stdlib_path('nt') These API are basically a refactoring of what already exist in distutils/command/install.py == Configuration variables == distutils.sysconfig currently provides some APIs to read values from files like Makefile and pyconfig.h. These API have been placed in the global sysconfig module: * get_config_vars(): return a dictionary of all configuration variables relevant for the current platform. * get_config_var(name): Return the value of a single variable * get_platform(): Return a string that identifies the current platform. (this one is used by site.py for example) * get_python_version() : return the short python version (sys.version[:3]) -- this one could probably go away but is useful because that's the marker used by Python in some paths. == code, status, next steps == The code of the module can be viewed here, it's a revamp of distutils.sysconfig: http://svn.python.org/view/*checkout*/python/branches/tarek_sysconfig/Lib/sy... I've refactored distutils/ and site.py so they work with this new module, and added deprecation warnings in distutils.sysconfig. All tests pass in the branch, but note that the code is still using the .h and Makefile files. This will probably be removed later in favor of a static _sysconfig.py file generated when Python is built, containing the variables sysconfig reads. I'll do this second step after I get some feedback on the proposal. Regards Tarek -- Tarek Ziadé | http://ziade.org
On Sat, Dec 12, 2009 at 12:02, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Hello,
A while ago I've proposed to refactor the APIs that provides access to the installation paths and configuration variables at runtime into a single module called "sysconfig", and make it easier for all implementations to work with them.
I've started a branch and worked on it, and I'd like to ask here for some feedback. And in particular from Jython and IronPython people because they would probably need to work in that file for their implementation and/or propose things to add. My understanding is that we have people like Phillip (Jenvey), Michael F., Frank W. in this list so they can comment directly and I don't need to cross-post this mail elsewhere.
== Installation schemes ==
First, the module contains the installation schemes for each platform CPython uses. An install scheme is a mapping where the key is the "code" name for a directory, and the value the path of that directory, with some $variable that can be expanded.
Install schemes are stored in a private mapping, where the keys are usually the value of os.name, and the value, the mapping I've mentionned earlier.
So, for example, the paths for win32 are:
_INSTALL_SCHEMES = { ... 'nt': { 'stdlib': '$base/Lib', 'platstdlib': '$base/Lib', 'purelib': '$base/Lib/site-packages', 'platlib': '$base/Lib/site-packages', 'include': '$base/include', 'platinclude': '$base/include', 'scripts': '$base/Scripts', 'data' : '$base', }, ... }
Are you using string.Template because this code needs to run on installs older than 2.6? -Brett
where each key corresponds to a directory that contains some Python files:
- stdlib : root of the standard library - platstdlib: root of platform-specific elements of the standard library - purelib: the site-packages directory for pure python modules - platlib: the site-packages directory for platform-specific modules - include: the include dir - platinclude: the include dir for platform-specific files - scripts: the directory where scripts are added - data: the directory where data file are added
All these directory are read and used by: - distutils when a package is installed, so the install command can dispatch files in the right place - site.py, when Python is initialized
IOW, any part of the stdlib can use these paths to locate and work with Python files.
The public APIs are:
* get_path_names() : returns a list of the path names ("stdlib", "platstdlib", etc.)
* get_paths(scheme, vars) : Returns a mapping containing an install scheme. - "scheme" is the name of the scheme, if not provided will get the default scheme of the current platform - vars is an optonal mapping that can provide values for the various $variables. Notice that they all have default values, for example $base == sys.prefix.
for example: get_paths('nt')
* get_path(name, scheme, vars): Returns one path corresponding to the scheme.
for example : get_paths('stdlib', 'nt')
Those API are generic, but maybe we could add specific APIs like:
* get_stdlib_path('nt')
These API are basically a refactoring of what already exist in distutils/command/install.py
== Configuration variables ==
distutils.sysconfig currently provides some APIs to read values from files like Makefile and pyconfig.h.
These API have been placed in the global sysconfig module:
* get_config_vars(): return a dictionary of all configuration variables relevant for the current platform.
* get_config_var(name): Return the value of a single variable
* get_platform(): Return a string that identifies the current platform. (this one is used by site.py for example)
* get_python_version() : return the short python version (sys.version[:3]) -- this one could probably go away but is useful because that's the marker used by Python in some paths.
== code, status, next steps ==
The code of the module can be viewed here, it's a revamp of distutils.sysconfig:
http://svn.python.org/view/*checkout*/python/branches/tarek_sysconfig/Lib/sy...
I've refactored distutils/ and site.py so they work with this new module, and added deprecation warnings in distutils.sysconfig.
All tests pass in the branch, but note that the code is still using the .h and Makefile files. This will probably be removed later in favor of a static _sysconfig.py file generated when Python is built, containing the variables sysconfig reads. I'll do this second step after I get some feedback on the proposal.
Regards Tarek
-- Tarek Ziadé | http://ziade.org _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
On Sat, Dec 12, 2009 at 10:35 PM, Brett Cannon <brett@python.org> wrote: [...]
'nt': { 'stdlib': '$base/Lib', 'platstdlib': '$base/Lib', 'purelib': '$base/Lib/site-packages', 'platlib': '$base/Lib/site-packages', 'include': '$base/include', 'platinclude': '$base/include', 'scripts': '$base/Scripts', 'data' : '$base', }, ... }
Are you using string.Template because this code needs to run on installs older than 2.6? -Brett
Not really. That's mostly because I reused the existing implementation and I found them quite readable in that case. But a string.Formatter could work well here too I guess. Tarek
On Sat, Dec 12, 2009 at 14:13, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
On Sat, Dec 12, 2009 at 10:35 PM, Brett Cannon <brett@python.org> wrote: [...]
'nt': { 'stdlib': '$base/Lib', 'platstdlib': '$base/Lib', 'purelib': '$base/Lib/site-packages', 'platlib': '$base/Lib/site-packages', 'include': '$base/include', 'platinclude': '$base/include', 'scripts': '$base/Scripts', 'data' : '$base', }, ... }
Are you using string.Template because this code needs to run on installs older than 2.6? -Brett
Not really. That's mostly because I reused the existing implementation and I found them quite readable in that case. But a string.Formatter could work well here too I guess.
Just figured that with formatters the way of the future that "{base}/include" would work just as well and be "future-proof". -Brett
On Sat, Dec 12, 2009 at 11:18 PM, Brett Cannon <brett@python.org> wrote: [..]
Not really. That's mostly because I reused the existing implementation and I found them quite readable in that case. But a string.Formatter could work well here too I guess.
Just figured that with formatters the way of the future that "{base}/include" would work just as well and be "future-proof". -Brett
Sure, I'll change it that way, thanks for the tip Tarek
Tarek Ziadé wrote:
On Sat, Dec 12, 2009 at 10:35 PM, Brett Cannon <brett@python.org> wrote: [...]
'nt': { 'stdlib': '$base/Lib', 'platstdlib': '$base/Lib', 'purelib': '$base/Lib/site-packages', 'platlib': '$base/Lib/site-packages', 'include': '$base/include', 'platinclude': '$base/include', 'scripts': '$base/Scripts', 'data' : '$base', }, ... }
Are you using string.Template because this code needs to run on installs older than 2.6? -Brett
Not really. That's mostly because I reused the existing implementation and I found them quite readable in that case. But a string.Formatter could work well here too I guess.
I don't really understand the complete context, but I think you want str.format(), not string.Formatter here. Eric.
Tarek Ziadé wrote:
== code, status, next steps ==
The code of the module can be viewed here, it's a revamp of distutils.sysconfig:
http://svn.python.org/view/*checkout*/python/branches/tarek_sysconfig/Lib/sy...
I've refactored distutils/ and site.py so they work with this new module, and added deprecation warnings in distutils.sysconfig.
I think we really need to do something about these kinds of deprecations. IMHO, there is no need to deprecate an entry point if the code has just moved to some other location in the stdlib. A simple note in the documentation and the NEWS file should be enough to get programmers to use the new location for code that doesn't have to work with older Python versions. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 14 2009)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Mon, Dec 14, 2009 at 10:52 AM, M.-A. Lemburg <mal@egenix.com> wrote: [..]
I've refactored distutils/ and site.py so they work with this new module, and added deprecation warnings in distutils.sysconfig.
I think we really need to do something about these kinds of deprecations.
IMHO, there is no need to deprecate an entry point if the code has just moved to some other location in the stdlib.
A simple note in the documentation and the NEWS file should be enough to get programmers to use the new location for code that doesn't have to work with older Python versions.
There are both cases in fact: some APIs have just moved, and some have changed. So I guess I could keep a deprecation warning only for the latter. Regards Tarek
Tarek Ziadé wrote:
On Mon, Dec 14, 2009 at 10:52 AM, M.-A. Lemburg <mal@egenix.com> wrote: [..]
I've refactored distutils/ and site.py so they work with this new module, and added deprecation warnings in distutils.sysconfig.
I think we really need to do something about these kinds of deprecations.
IMHO, there is no need to deprecate an entry point if the code has just moved to some other location in the stdlib.
A simple note in the documentation and the NEWS file should be enough to get programmers to use the new location for code that doesn't have to work with older Python versions.
There are both cases in fact: some APIs have just moved, and some have changed. So I guess I could keep a deprecation warning only for the latter.
Sounds like a plan. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 15 2009)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Tarek wrote:
== Installation schemes ==
First, the module contains the installation schemes for each platform CPython uses. An install scheme is a mapping where the key is the "code" name for a directory, and the value the path of that directory, with some $variable that can be expanded.
Install schemes are stored in a private mapping, where the keys are usually the value of os.name, and the value, the mapping I've mentionned earlier.
So, for example, the paths for win32 are:
_INSTALL_SCHEMES = { ... 'nt': { 'stdlib': '$base/Lib', 'platstdlib': '$base/Lib', 'purelib': '$base/Lib/site-packages', 'platlib': '$base/Lib/site-packages', 'include': '$base/include', 'platinclude': '$base/include', 'scripts': '$base/Scripts', 'data' : '$base', }, ... }
Not mentioned here are the user schemes. Looking at the code it seems that _getuserbase is adding "Python" to the path on nt. Isn't that redundant? The paths in _INSTALL_SCHEMES already include "Python". Also w/ user dirs in general - I think there should be some implementation specific portion of the path as well. Right now I think IronPython and CPython will end up with the same user paths for the same versions which could cause problems if someone releases separate modules for IronPython and CPython for some reason (or if users just want different sets of things installed for different interpreters).
* get_platform(): Return a string that identifies the current platform. (this one is used by site.py for example)
I wonder if this would make more sense a built-in. Ultimately it seems like the interpreter implementation knows best about what aspects of it's underlying platform would require different libraries. With the existing code I think IronPython would return "cli" everywhere (because os.name == 'nt' on Windows but posix on Linux & Mac OS/X but we still don't have uname). I think Jython will return something like java1.6.0_17 because it's os.name is java - which might be more specific than is necessary. Also if the purpose of this is for platform specific build directories it might be interesting to have multiple return values. For example in IronPython the minimal thing we'd want I think is a "cli" directory for .NET assemblies. But there could also be native extensions which are platform specific so we'd want "cli-x86" and "cli-x64" depending upon the platform. Jython might want the same thing as they add ctypes support.
2009/12/14 Dino Viehland <dinov@microsoft.com>: [..]
Not mentioned here are the user schemes. Looking at the code it seems that _getuserbase is adding "Python" to the path on nt. Isn't that redundant? The paths in _INSTALL_SCHEMES already include "Python".
Right that's a small bug in the refactoring (there's an extra /) but there will be a bit of redundancy on "Python", at the end nevertheless: The base directory in win32 for the user scheme in is : APPDATA *or* ~/Python (under linux it's ~/.local) then various subdirectories are added in that directory, and some of them starts with "PythonXY", like: ~/Python/Python26/.. See http://www.python.org/dev/peps/pep-0370/#specification
Also w/ user dirs in general - I think there should be some implementation specific portion of the path as well. Right now I think IronPython and CPython will end up with the same user paths for the same versions which could cause problems if someone releases separate modules for IronPython and CPython for some reason (or if users just want different sets of things installed for different interpreters).
Yes that's one point someone raised (can't recall who) and the idea was to have a separate top directory for user dirs, that would start with the name of the implementation: so for Windows: ~/Python/Python26/.. ~/IronPython/../ ~/Jython/../ and for Linux and al, I am not sure but maybe a prefix for Jython/etc.. could be used for all paths. ~/.locale/lib/python/2.6/site-packages/... ~/.locale/jython/lib/python/2.6/site-packages/... (I didn't digg on how Jython organizes things yet, any hint would be appreciated)
* get_platform(): Return a string that identifies the current platform. (this one is used by site.py for example)
I wonder if this would make more sense a built-in. Ultimately it seems like the interpreter implementation knows best about what aspects of it's underlying platform would require different libraries.
With the existing code I think IronPython would return "cli" everywhere (because os.name == 'nt' on Windows but posix on Linux & Mac OS/X but we still don't have uname). I think Jython will return something like java1.6.0_17 because it's os.name is java - which might be more specific than is necessary.
Ok so it sounds like it would be easier to cook that value in a built-in in each implementation,
Also if the purpose of this is for platform specific build directories it might be interesting to have multiple return values. For example in IronPython the minimal thing we'd want I think is a "cli" directory for .NET assemblies. But there could also be native extensions which are platform specific so we'd want "cli-x86" and "cli-x64" depending upon the platform. Jython might want the same thing as they add ctypes support.
So like, having an architecture marker, that defaults to the current ? get_platform(architecture=platform.architecture) Regards Tarek -- Tarek Ziadé | http://ziade.org
Tarek wrote:
(I didn't digg on how Jython organizes things yet, any hint would be appreciated)
The installation directory looks like it's organized just like CPython but I have no clue how user directories would/should be arranged.
Also if the purpose of this is for platform specific build directories it might be interesting to have multiple return values. For example in IronPython the minimal thing we'd want I think is a "cli" directory for .NET assemblies. But there could also be native extensions which are platform specific so we'd want "cli-x86" and "cli-x64" depending upon the platform. Jython might want the same thing as they add ctypes support.
So like, having an architecture marker, that defaults to the current ?
get_platform(architecture=platform.architecture)
How would you know what other architectures would be valid to pass in here? Returning a list would let the implementation say that it knows a certain set of architectural binaries are valid.
On Tue, Dec 15, 2009 at 1:00 AM, Dino Viehland <dinov@microsoft.com> wrote: [..]
How would you know what other architectures would be valid to pass in here? Returning a list would let the implementation say that it knows a certain set of architectural binaries are valid.
How would you use it when a list is returned ? Can you provide a few examples where the code wants to know the default architecture for the current platform ? etc. Tarek
Tarek wrote:
How would you use it when a list is returned ? Can you provide a few examples where the code wants to know the default architecture for the current platform ? etc.
The consumer could enumerate over it and then do whatever they were doing w/ the platform multiple times. If an earlier one succeeds at what they're attempting to do then they're done. If there's no "success", lets say they're appending something to sys.path, then they'd do it for all elements in the sequence. It's not an absolute requirement or anything like that - it just jumped out at me because IronPython and Jython do have multiple platforms which could be relevant at one time.
On Tue, Dec 15, 2009 at 6:59 PM, Dino Viehland <dinov@microsoft.com> wrote:
Tarek wrote:
How would you use it when a list is returned ? Can you provide a few examples where the code wants to know the default architecture for the current platform ? etc.
The consumer could enumerate over it and then do whatever they were doing w/ the platform multiple times. If an earlier one succeeds at what they're attempting to do then they're done. If there's no "success", lets say they're appending something to sys.path, then they'd do it for all elements in the sequence.
get_platform is currently used 1/ by Distutils' build command to define the default platform name to build from (and there's an option to override it : --plat-name), 2/ by site.py to add the build dir of Python in case we are running from it. For 1/, build allows cross-platform building only if the user is under windows, otherwise it'll throw an error. IOW, this api is used to know what is the *current* platform, even if there are some (tiny) ways to work for "other" platforms. 2/ was meant to be dropped at some point (see http://bugs.python.org/issue586680) and very CPython specific I guess. I am not sure, but in Jython/IronPython, do you guys have a "default" platform ? if so, returning that default in get_platform() would work.
It's not an absolute requirement or anything like that - it just jumped out at me because IronPython and Jython do have multiple platforms which could be relevant at one time.
Sure, and that's why I have asked feedback from IrontPython/Jython people, Thanks for that. Regards Tarek -- Tarek Ziadé | http://ziade.org
On Mon, 14 Dec 2009 23:58:08 +0100, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Yes that's one point someone raised (can't recall who) and the idea was to have a separate top directory for user dirs, that would start with the name of the implementation:
so for Windows:
~/Python/Python26/.. ~/IronPython/../ ~/Jython/../
I follow your reasoning. But application developers (and traditional windows developers) have an upside-down view of that. They might think that python is just the language interpreter, and that it should just stay 'out-of-the-way'. For example, mercurial and many python based apps include the python as a sub-app to their own. Just neccessary to run the application. That is the way it is for commercial windows apps. We want the python interpretor installed, and then we want our apps (that we get paid money for) to sit at the top. Not the other way round, sitting beneath the language interpreter. This is pretty much the way it has been for windows for close on 15 years now. Regards David
On Mon, Dec 14, 2009 at 5:58 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
and for Linux and al, I am not sure but maybe a prefix for Jython/etc.. could be used for all paths.
~/.locale/lib/python/2.6/site-packages/... ~/.locale/jython/lib/python/2.6/site-packages/...
(I didn't digg on how Jython organizes things yet, any hint would be appreciated) Jython does not yet support user site-packages, but I think the above looks like a fine structure for us when we get to implementing it.
-Frank
On Dec 23, 2009, at 10:00 AM, Frank Wierzbicki wrote:
On Mon, Dec 14, 2009 at 5:58 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
and for Linux and al, I am not sure but maybe a prefix for Jython/etc.. could be used for all paths.
~/.locale/lib/python/2.6/site-packages/... ~/.locale/jython/lib/python/2.6/site-packages/...
(I didn't digg on how Jython organizes things yet, any hint would be appreciated) Jython does not yet support user site-packages, but I think the above looks like a fine structure for us when we get to implementing it.
Two minor points: 1. It's "~/.local", not "~/.locale" ;-) 2. I think it would be a better idea to do "~/.local/lib/jython/2.6/site-packages". The idea with ~/.local is that it's a mirror of the directory structure convention in /, /usr/, /opt/ or /usr/local/. In other words it's got "bin", "lib", "share", "etc", etc.. ~/.local/jython/lib suggests that the parallel scripts directory would be ~/.local/jython/bin, which means I need 2 entries on my $PATH instead of one, which means yet more setup for people who use both Python and Jython per-user installation.
2009/12/23 Glyph Lefkowitz <glyph@twistedmatrix.com>: [..]
2. I think it would be a better idea to do "~/.local/lib/jython/2.6/site-packages".
The idea with ~/.local is that it's a mirror of the directory structure convention in /, /usr/, /opt/ or /usr/local/. In other words it's got "bin", "lib", "share", "etc", etc.. ~/.local/jython/lib suggests that the parallel scripts directory would be ~/.local/jython/bin, which means I need 2 entries on my $PATH instead of one, which means yet more setup for people who use both Python and Jython per-user installation.
Right, good idea Tarek -- Tarek Ziadé | http://ziade.org
Dino Viehland <dinov <at> microsoft.com> writes:
* get_platform(): Return a string that identifies the current platform. (this one is used by site.py for example)
I wonder if this would make more sense a built-in. Ultimately it seems like the interpreter implementation knows best about what aspects of it's underlying platform would require different libraries.
I don't think it makes sense to make a builtin of such a little used, non-performance critical API. If we want to factor out all implementation-specific things, we could add a module dedicated to that (e.g. "_interpreter") and have other modules (os, platform, sysconfig...) draw from that. Regards Antoine.
Antoine wrote:
Dino Viehland <dinov <at> microsoft.com> writes:
* get_platform(): Return a string that identifies the current platform. (this one is used by site.py for example)
I wonder if this would make more sense a built-in. Ultimately it seems like the interpreter implementation knows best about what aspects of it's underlying platform would require different libraries.
I don't think it makes sense to make a builtin of such a little used, non-performance critical API. If we want to factor out all implementation-specific things, we could add a module dedicated to that (e.g. "_interpreter") and have other modules (os, platform, sysconfig...) draw from that.
Putting this in an _interpreter module is fine with me (or even putting it somewhere in sys works - e.g. there was a sys.implementation discussion not too long ago). The important thing is that the interpreter implementation really is the one that understands what binaries they would be compatible with it regardless of how often it gets used and how it performs. If it's not a built-in then I think we'd be in a world where either every implementation needs to patch this file in their distribution or contribute a change back to support their implementation and that just seems ugly. And it's already a very large function even w/o IronPython, Jython, PyPy and Unladen Swallow support.
Hi Tarek, Is there anything in this proposal for windows developers ? Just that I can't see anything that would help us... For me, the terminology isn't anything a windows developer could really understand. It presumes that the developer understands the python implementation. A developer might not understand all those details and might not be interested to learn. I accept that the terminology is good on linux.. but it's near meaningless on windows - for me - anyway. David On Sat, 12 Dec 2009 21:02:13 +0100, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Hello,
A while ago I've proposed to refactor the APIs that provides access to the installation paths and configuration variables at runtime into a single module called "sysconfig", and make it easier for all implementations to work with them.
I've started a branch and worked on it, and I'd like to ask here for some feedback. And in particular from Jython and IronPython people because they would probably need to work in that file for their implementation and/or propose things to add. My understanding is that we have people like Phillip (Jenvey), Michael F., Frank W. in this list so they can comment directly and I don't need to cross-post this mail elsewhere.
== Installation schemes ==
First, the module contains the installation schemes for each platform CPython uses. An install scheme is a mapping where the key is the "code" name for a directory, and the value the path of that directory, with some $variable that can be expanded.
Install schemes are stored in a private mapping, where the keys are usually the value of os.name, and the value, the mapping I've mentionned earlier.
So, for example, the paths for win32 are:
_INSTALL_SCHEMES = { ... 'nt': { 'stdlib': '$base/Lib', 'platstdlib': '$base/Lib', 'purelib': '$base/Lib/site-packages', 'platlib': '$base/Lib/site-packages', 'include': '$base/include', 'platinclude': '$base/include', 'scripts': '$base/Scripts', 'data' : '$base', }, ... }
where each key corresponds to a directory that contains some Python files:
- stdlib : root of the standard library - platstdlib: root of platform-specific elements of the standard library - purelib: the site-packages directory for pure python modules - platlib: the site-packages directory for platform-specific modules - include: the include dir - platinclude: the include dir for platform-specific files - scripts: the directory where scripts are added - data: the directory where data file are added
All these directory are read and used by: - distutils when a package is installed, so the install command can dispatch files in the right place - site.py, when Python is initialized
IOW, any part of the stdlib can use these paths to locate and work with Python files.
The public APIs are:
* get_path_names() : returns a list of the path names ("stdlib", "platstdlib", etc.)
* get_paths(scheme, vars) : Returns a mapping containing an install scheme. - "scheme" is the name of the scheme, if not provided will get the default scheme of the current platform - vars is an optonal mapping that can provide values for the various $variables. Notice that they all have default values, for example $base == sys.prefix.
for example: get_paths('nt')
* get_path(name, scheme, vars): Returns one path corresponding to the scheme.
for example : get_paths('stdlib', 'nt')
Those API are generic, but maybe we could add specific APIs like:
* get_stdlib_path('nt')
These API are basically a refactoring of what already exist in distutils/command/install.py
== Configuration variables ==
distutils.sysconfig currently provides some APIs to read values from files like Makefile and pyconfig.h.
These API have been placed in the global sysconfig module:
* get_config_vars(): return a dictionary of all configuration variables relevant for the current platform.
* get_config_var(name): Return the value of a single variable
* get_platform(): Return a string that identifies the current platform. (this one is used by site.py for example)
* get_python_version() : return the short python version (sys.version[:3]) -- this one could probably go away but is useful because that's the marker used by Python in some paths.
== code, status, next steps ==
The code of the module can be viewed here, it's a revamp of distutils.sysconfig:
http://svn.python.org/view/*checkout*/python/branches/tarek_sysconfig/Lib/sy...
I've refactored distutils/ and site.py so they work with this new module, and added deprecation warnings in distutils.sysconfig.
All tests pass in the branch, but note that the code is still using the .h and Makefile files. This will probably be removed later in favor of a static _sysconfig.py file generated when Python is built, containing the variables sysconfig reads. I'll do this second step after I get some feedback on the proposal.
Regards Tarek
On 15/12/2009 2:07 PM, David Lyon wrote:
Hi Tarek,
Is there anything in this proposal for windows developers ?
Just that I can't see anything that would help us...
So I understand - help doing what?
For me, the terminology isn't anything a windows developer could really understand. It presumes that the developer understands the python implementation. A developer might not understand all those details and might not be interested to learn.
That seems true for all operating systems, not just Windows. The vast majority of Python programmers will probably never need to use this feature, but those who do will need to understand enough of the python implementation to make sense of the values returned regardless of the OS.
I accept that the terminology is good on linux.. but it's near meaningless on windows - for me - anyway.
I think it is fine. If you are really looking for properties specific to the operating system (eg, the location of the start menu, desktop, appdata folders etc) I don't think they belong in this PEP; they are a property of the OS install/version, not of the specific Python install/version. Cheers, Mark
On Tue, 15 Dec 2009 14:40:56 +1100, Mark Hammond <skippy.hammond@gmail.com> wrote:
I think it is fine. If you are really looking for properties specific to the operating system (eg, the location of the start menu, desktop, appdata folders etc)
But under windows, an application developer might (as in probably would) like to install an application in \Program Files\someapp rather than hidden in the bowels of the python interpretor. They might like their data in "Application Data", which is where support people get trained to look for application data. Not down in \pythonX.Y\ ...
I don't think they belong in this PEP; they are a property of the OS install/version, not of the specific Python install/version.
Yes - exactly. My point. The operating system says that programs should be installed into "Program Files" and data into "Application Data". Why can not python respect the operating system directions for well behaved apps? David
On 15/12/2009 2:42 PM, David Lyon wrote:
On Tue, 15 Dec 2009 14:40:56 +1100, Mark Hammond<skippy.hammond@gmail.com> wrote:
I think it is fine. If you are really looking for properties specific to the operating system (eg, the location of the start menu, desktop, appdata folders etc)
But under windows, an application developer might (as in probably would) like to install an application in \Program Files\someapp rather than hidden in the bowels of the python interpretor.
I agree - but in that case you are talking about an application built with Python - that is a different set of requirements. But regardless of where Python itself lives the returned values will be correct; they will not reflect the operating system preference for various other folders, but will correctly return the paths they are documented as returning. IOW, this isn't designed for applications which happen to be written in Python. There might be a case for such a module to be created, but this PEP doesn't attempt to solve that particular problem.
They might like their data in "Application Data", which is where support people get trained to look for application data. Not down in \pythonX.Y\ ...
Nothing is stopping them from doing that - but this PEP isn't intended to provide that information.
I don't think they belong in this PEP; they are a property of the OS install/version, not of the specific Python install/version.
Yes - exactly. My point.
The operating system says that programs should be installed into "Program Files" and data into "Application Data". Why can not python respect the operating system directions for well behaved apps?
It does - many applications written in Python exist which do exactly that. If a user really wants to install Python itself under "\Program Files", sysconfig should correctly reflect that. Python doesn't care where it is installed, or even when it is (sensibly) bundled with an app which is designed to be "stand-alone". You are after operating system properties - I understand your need to know those paths (and there are already reasonable Windows specific ways to get them), but sysconfig isn't trying to solve that for you and I agree it should not attempt to. Mark
On Tue, 15 Dec 2009 15:05:18 +1100, Mark Hammond <mhammond@skippinet.com.au> wrote:
But under windows, an application developer might (as in probably would) like to install an application in \Program Files\someapp rather than hidden in the bowels of the python interpretor.
I agree - but in that case you are talking about an application built with Python - that is a different set of requirements.
Building an application with python.. that's right. Of course. Why not?
IOW, this isn't designed for applications which happen to be written in Python. There might be a case for such a module to be created, but this PEP doesn't attempt to solve that particular problem.
But programmers might want to write an application with python. It doesn't seem like such an edge-case thing to do.
They might like their data in "Application Data", which is where support people get trained to look for application data. Not down in \pythonX.Y\ ...
Nothing is stopping them from doing that - but this PEP isn't intended to provide that information.
Distutils is stopping them.
It does - many applications written in Python exist which do exactly that.
Yes. And they don't use any of the built in facilities, under windows.
If a user really wants to install Python itself under "\Program Files", sysconfig should correctly reflect that. Python doesn't care where it is installed, or even when it is (sensibly) bundled with an app which is designed to be "stand-alone".
No debate about that.
You are after operating system properties - I understand your need to know those paths (and there are already reasonable Windows specific ways to get them), but sysconfig isn't trying to solve that for you and I agree it should not attempt to.
So under windows, then, what is it trying to solve? Thats what I am asking. David
On 15/12/2009 3:09 PM, David Lyon wrote:
On Tue, 15 Dec 2009 15:05:18 +1100, Mark Hammond <mhammond@skippinet.com.au> wrote:
But under windows, an application developer might (as in probably would) like to install an application in \Program Files\someapp rather than hidden in the bowels of the python interpretor.
I agree - but in that case you are talking about an application built with Python - that is a different set of requirements.
Building an application with python.. that's right. Of course. Why not?
I'm missing your point - many applications exist written in Python.
IOW, this isn't designed for applications which happen to be written in Python. There might be a case for such a module to be created, but this PEP doesn't attempt to solve that particular problem.
But programmers might want to write an application with python. It doesn't seem like such an edge-case thing to do.
They can, and they have. So again your point is lost on me.
They might like their data in "Application Data", which is where support people get trained to look for application data. Not down in \pythonX.Y\ ...
Nothing is stopping them from doing that - but this PEP isn't intended to provide that information.
Distutils is stopping them.
I don't agree with that and I can present many applications as evidence. You yourself mentioned mercurial and it looks for mercurial.ini in the user's appdata directory. Regardless, this discussion isn't about distutils.
It does - many applications written in Python exist which do exactly that.
Yes. And they don't use any of the built in facilities, under windows.
To continue the mercurial example - mercurial will not use sysconfig to determine where to look for mercurial.ini on *any* operating system. sysconfig is not about solving that particular problem.
So under windows, then, what is it trying to solve? Thats what I am asking.
The same thing it is trying to solve for non-Windows users - various threads here have articulated this well. You needn't feel bad about not having such use-cases yourself - that simply means sysconfig isn't targetted at you - it isn't targetted at application developers on any operating system. Cheers, Mark
On Tue, 15 Dec 2009 15:24:36 +1100, Mark Hammond <mhammond@skippinet.com.au> wrote:
But under windows, an application developer might (as in probably would) like to install an application in \Program Files\someapp rather than hidden in the bowels of the python interpretor. ... I'm missing your point ....
The point is that if somebody writes an application in C, they will generally speaking not want (under say linux) for that application to live in the C compiler directory. Same goes for many other languages. The point is not controversial in other languages. And it shouldn't be here either.
Distutils is stopping them.
I don't agree with that and I can present many applications as evidence.
Please do - if you wish.
You yourself mentioned mercurial and it looks for mercurial.ini in the user's appdata directory.
Sure. There's growing support within the python interpretor for things like that. But Mercurial uses an external installer. Like NSIS, to overcome the deficiencies that I am pointing out.
.. it isn't targetted at application developers on any operating system.
I see. I get it now. Thanks. David
On Mon, Dec 14, 2009 at 11:39:02PM -0500, David Lyon wrote:
On Tue, 15 Dec 2009 15:24:36 +1100, Mark Hammond <mhammond@skippinet.com.au> wrote:
But under windows, an application developer might (as in probably would) like to install an application in \Program Files\someapp rather than hidden in the bowels of the python interpretor. ... I'm missing your point ....
The point is that if somebody writes an application in C, they will generally speaking not want (under say linux) for that application to live in the C compiler directory.
Same goes for many other languages.
The point is not controversial in other languages. And it shouldn't be here either.
If I write a shared library under C I am expected to install it under one of the default locations if I don't want to require people to have to tweak things before they can use it. I see no difference with python modules or packages. Any private modules or packages used by an application built using python don't have to be on the sys.path by default (in fact I would encourage them not to be).
Distutils is stopping them.
Distutils isn't perfect but solves the need of installing public modules and packages quite well. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org
On Tue, 15 Dec 2009 09:32:55 +0000, Floris Bruynooghe <floris.bruynooghe@gmail.com> wrote:
If I write a shared library under C I am expected to install it under one of the default locations if I don't want to require people to have to tweak things before they can use it. I see no difference with python modules or packages. Any private modules or packages used by an application built using python don't have to be on the sys.path by default (in fact I would encourage them not to be).
Hi Floris. Well Mark Hammond summed up for me - I'm satisfied with his answer. You're talking about modules and packages and I agree with you. I guess I was asking about was extending the set to of things that could easily be dealt with by python from just modules/packages to modules/packages + applications. What do I mean by an application?, well it's one or two steps up from the simple as 'helloworld.py'. There's lots of machines in the company, and lots of different apps. Not unlike a scientific area where there is lots of specialised equipment and each machine has slightly different requirements as to its function. Installing python is painless. That's all good. Installing all the python-packages has a learning curve and isn't very streamlined. In the end, it can be made to work. Installing the application (helloworld.py for the want of a better name), well, there isn't much in python to help there. I was thinking that perphaps sysconfig could help me get my helloworld.py application into a \Program Files\hello world directory and everything would be rosy. But not yet. So I will wait.
Distutils isn't perfect but solves the need of installing public modules and packages quite well.
If you say so - but compared to what ? CPAN? :-) Have a nice day David
David Lyon wrote:
I was thinking that perphaps sysconfig could help me get my helloworld.py application into a \Program Files\hello world directory and everything would be rosy.
But not yet. So I will wait.
No, we mostly leave that to the py2exe/py2app + native installer developers of the world. There are a *lot* of thorny issues in getting installers fully in accordance with OS developer guidelines, which is why we tend to shy away from it. That said, we did add the zip archive execution capability so that complex Python applications can be more easily executed without needing to install them at all. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Wed, 16 Dec 2009 21:21:01 +1000, Nick Coghlan <ncoghlan@gmail.com> wrote:
.. we mostly leave that to the py2exe/py2app + native installer developers of the world. There are a *lot* of thorny issues in getting installers fully in accordance with OS developer guidelines, which is why we tend to shy away from it.
And that is fair enough. In the commercial world, once the python gets compiled, you're mostly talking about some application where the source code needs protection. The bar is raised - to match the available budgets. However, I would like to point out a category of applications that live in source code form. Inside scientific or commercial organisations. These are apps that are never compiled - and just run in interpreted mode. Maybe they're on workstations, or maybe they're on web servers. It doesn't matter that much. The point is that the python Configurations exist over many machines. What I'd like to suggest is that python apps are becoming more network centric. To the point where it might at some time in the future it might well become a 'python-core' issue. I'm not suggesting writing a new SCM because so many already exist. And in python too - haha - how great is that. All that I'm doing is suggesting that the python of the future and the stdlib of the future will include celery or superpy or the mercurial or bzr interfaces, and it will be really easy to roll out the 'helloworld.py' app/web-app out to the desktop machines or django server or cluster or cloud of machines wherever they actually are. The machines will just have 'python' installed, and then from there everything will pretty easily take place (apps/packages get pushed to remote machines). I'm not star-gazing, because all these things are already needed and already being done to some degree in some organisations. There's already the libraries on pypi for most of this anyway. The practical advantages of some of us going in this direction is that it might be possible for us open sourcers to attract the attention of our commercial sponsors attention. Because they're always interested in getting new toys and utilising their resources in the most efficient way. If we do the above, incorporate tested packages from pypi, it's possible that the glow of CPAN might be tarnished somewhat. David
David Lyon writes:
I'm not star-gazing, because all these things are already needed and already being done to some degree in some organisations. There's already the libraries on pypi for most of this anyway.
Sure. But in a volunteer project, it's beg, buy, or build. Begging has not worked, and it's not because people don't understand what you're saying. Nobody is saying that want you want is stupid or impossible, either. It's just that they have created those libraries you mention, they have built PyPI, they have written distutils and setuptools and others. *These work well enough* ... except for you, apparently. I have no problem with that, and you're welcome to beg. But IMO at this point you're coming close to crossing the line from begging to whining. There clearly is no interest in going down the road you propose. Post a bounty or build it yourself (you were pretty much done with something last time around, weren't you?), and either way use the usual channels (eg, PyPI) to distribute the product and accumulate user interest and support for future attempts at logrolling to get it into the stdlib.
On Thu, 17 Dec 2009 12:18:00 +0900, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
.. because all these things are already needed and already being done to some degree in some organisations. There's already the libraries on pypi for most of this anyway.
... There clearly is no interest in going down the road you propose.
Hmm.. well.. projects like buildout, superpy and celery suggest that people are already working and thinking this way. Lots of companies use clusters of python computers and these needs will only increase over time. If it's only +1 from one person to make python more network centric with sysconfig in 2010, then so be it. Have a nice day. David
David Lyon <david.lyon@preisshare.net> writes:
On Tue, 15 Dec 2009 14:40:56 +1100, Mark Hammond <skippy.hammond@gmail.com> wrote:
I think it is fine. If you are really looking for properties specific to the operating system (eg, the location of the start menu, desktop, appdata folders etc)
But under windows, an application developer might (as in probably would) like to install an application in \Program Files\someapp rather than hidden in the bowels of the python interpretor.
They might like their data in "Application Data", which is where support people get trained to look for application data. Not down in \pythonX.Y\ ...
Similarly, GNU operating system distributions usually have documentation separate from programs, separate from configuration files, separate from static data, separate from mutable data.
I don't think they belong in this PEP; they are a property of the OS install/version, not of the specific Python install/version.
I think the “sysconfig” specification should allow for *extension*, without needing a modified specification each time a new distinct location is requested. That is, those developers and packagers who don't need a gamut of system locations can stick to a minimal set, while those who need many can use them, and those who need many can extend the set compatibly. -- \ “If you make people think they're thinking, they'll love you; | `\ but if you really make them think, they'll hate you.” —Donald | _o__) Robert Perry Marquis | Ben Finney
On Tue, Dec 15, 2009 at 5:12 AM, Ben Finney <ben+python@benfinney.id.au> wrote: [..]
I don't think they belong in this PEP; they are a property of the OS install/version, not of the specific Python install/version.
I think the “sysconfig” specification should allow for *extension*, without needing a modified specification each time a new distinct location is requested.
That is, those developers and packagers who don't need a gamut of system locations can stick to a minimal set, while those who need many can use them, and those who need many can extend the set compatibly.
That's part of the idea, get_path() is a generic API that would allow any implementation to add extra paths if needed. The keys I have provided are the existing keys we have in distutils and some extra keys that removes all the custo code that was building paths using os.path.join here and there in the stdlib. Now I didn't add an API to allow dynamic insertion of new paths because I don't see a use case for that : if a packager wants to work with an extra path, she can work at the project level by providing custom distutils commands to do the job. Unless you have another use case in mind ? Tarek
On Sat, Dec 12, 2009 at 9:02 PM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
Hello,
A while ago I've proposed to refactor the APIs that provides access to the installation paths and configuration variables at runtime into a single module called "sysconfig", and make it easier for all implementations to work with them.
I've applied the suggestions made in this thread. If no one objects, here's what I am going to do now: I'll merge this change in the trunk, (I'll drop the branch changesets in favor of one single, clean changeset) and start to work on the corresponding doc, so more people will be able to give some feedback on the APIs once the second 2.7 alpha is out. Then, in a second step, I'll work on the removal of the Makefile and pyconfig.h dependencies by adding a _synconfig.py file as suggested earlier, that will be created during the build process. Regards, Tarek
participants (15)
-
Antoine Pitrou -
Ben Finney -
Brett Cannon -
David Lyon -
Dino Viehland -
Eric Smith -
Floris Bruynooghe -
Frank Wierzbicki -
Glyph Lefkowitz -
M.-A. Lemburg -
Mark Hammond -
Mark Hammond -
Nick Coghlan -
Stephen J. Turnbull -
Tarek Ziadé