...

If the files are shared among all users then /usr/local/<something> seems more reasonable.

Oh, right, I forgot to think about system-wide config files. They have to be supported by another function in site. A lot of programs have similar-looking code to get a list of filenames and then process it with configparser or something else: find_config_files in Distutils, get_standard_config_files in Docutils, rcpath in Mercurial. There are different namings to support (rc, .cfg, .conf, etc.) and also config directories (for Mercurial), but it does not seem impossible to define a standard function that would benefit Python itself and other programs.

Choosing an arbitrary location we think is good on every system is fine and non risky I think, as long as Python let the various distribution change those paths though configuration.

Don’t you have a bootstrapping problem? How do you know where to look at the sysconfig file that tells where to look at config files?

In fact, that's one of the future goal of the sysconfig module I had in mind.

Seems great.

So let's:

- define all the default locations for each system - define a sysconfig.cfg file that contains all installation paths, for each target system - change sysconfig.py so it uses it, instead of the global dicts it currently has

I wonder if this is so complex that a wiki page or something else would be better suited than email to iron out the proposal. Or maybe it’s time for that sysconfig.cfg PEP you planned to do. Regards

On Fri, Aug 13, 2010 at 07:48:22AM +1000, Nick Coghlan wrote:

2010/8/12 Éric Araujo <merwok@netwok.org>:

...

...

Choosing an arbitrary location we think is good on every system is fine and non risky I think, as long as Python let the various distribution change those paths though configuration.

Don’t you have a bootstrapping problem? How do you know where to look at the sysconfig file that tells where to look at config files?

I'd hardcode a list of locations. [os.path.join(os.path.dirname(__file__), 'sysconfig.cfg'), os.path.join('/etc', 'sysconfig.cfg')] The distributor has a limited choice of options on where to look. A good alternative would be to make the config file overridable. That way you can have sysconfig.cfg next to sysconfig.py or in a known config directory relative to the python stdlib install but also let the distributions and individual sites override the defaults by making changes to /etc/python3/sysconfig.cfg for instance.

Personally, I'm not clear on what a separate syconfig.cfg file offers over clearly separating the directory configuration settings and continuing to have distributions patch sysconfig.py directly. The bootstrapping problem (which would encourage classifying synconfig.cfg as source code and placing it alongside syscongig.py) is a major part of that point of view.

Here's some advantages but some of them are of dubious worth: * Allows users/site-administrators to change paths and not have packaging systems overwrite the changes. * Makes it conceptually cleaner to make this overridable via user defined config files since it's now a matter of parsing several config files instead of having a hardcoded value in the file and overridable values outside of it. * Allows sites to add additional paths to the config file. * Makes it clear to distributions that the values in the config file are available for making changes to rather than having to look for it in code and not know the difference between thaat or say, the encoding parameter in python2. * Documents the format to use for overriding the paths if individual sites can override the defaults that are shipped in the system version of python. -Toshio

On Fri, Aug 13, 2010 at 03:15:28AM +0200, Éric Araujo wrote:

...

A good alternative would be to make the config file overridable. That way you can have sysconfig.cfg next to sysconfig.py or in a known config directory relative to the python stdlib install but also let the distributions and individual sites override the defaults by making changes to /etc/python3/sysconfig.cfg for instance.

Putting configuration files next to source code does not agree with the FHS.

That depends -- sometimes a config file is actually a data file. In following the FHS, a linux distribtuion and system administrators want to make sure a sysadmin never needs to modify a file outside of /etc for configuration. However, if you had a config file that is installed with python and not meant to be modified next to the code and the sysadmin is able to customize python's behaviour by copying lines from that file to a file in /etc/ that they do modify, that's perfectly fine. Where developers often run afoul of the FHS is creating a configuration option that does not live in a file in /etc/ (either because they think it's data but system administrators routinely want to modify.it or because they don't care about the FHS and just want to put everything in a single directory) and hte system admin has no means to set that option permanently but to modify the files outside of /etc/.

Your message gave me an idea though: Some function could be called in sitecustomize (symlinked from /etc on Debian e.g.) to set the path to sysconfig.cfg.

I hope that you mean that the actual file lives in /etc/ and then a symlink in the stdlib points to it. That sounds acceptable from an FHS view but it's a bad idea. Most of the config files in /etc/ are declarative. A few of the files in /etc/ are imperative (for instance, init scripts) and distributions frequently debate whether these are actually config files or files misplaced in /etc/ for historical reasons. Fedora, for instance, does not mark initscripts as config -- all configuration of initscripts are done by setting shell variables in a different file which is marked config. (Files marked config in a package manager generally are treated differently on upgrades. If the files have been modified since install, they are either not replaced or, when replaced, a backup is made so that the system administrator can merge their local changes.) The problem with having imperative constructs in /etc/ is that they're much harder (perhaps impossible) to fix up when changes have to be made at the system packaging level. Let's say that you hard code this into the sitecustomize.py when you ship it for the first time on Fedora: siteconfig_cfg = '/etc/siteconfig.cfg' The system administrator installs this package and then does some unrelated customization for his site in the file: if sys.version_info[:2] == '3.0': sys.path.append('/opt/python3.0') siteconfig_cfg = '/etc/siteconfig30.cfg' elif sys.version_info[:2] == '3.1': sys.path.append(os.environ['USER'] + '/opt/python3.1') siteconfig_cfg = '/etc/siteconfig31.cfg' elif sys.version_info[:2] == '3.2': sys.path.append(os.environ['USER'] + '/opt/python3.2') siteconfig_cfg = '/etc/siteconfig.cfg' # [...] Do other strange stuff because if it's possible, someone will. Now, when we update to python3.3 we decide to move siteconfig.cfg into /etc/python3.3/. But, because the system admin has modified that file so heavily, we don't update that file. This means the updated python is broken out of the box. If you're enamoured of symlinks, you can do this directly with the config file instead of using sitecustomize:: /etc/python3.2/sysconfig.cfg /usr/lib64/python3.2/sysconfig.cfg => /etc/python3.2/sysconfig.cfg as a third alternative. -Toshio

On 12/08/2010 08:26, Tarek Ziadé wrote:

[snip...] Choosing an arbitrary location we think is good on every system is fine and non risky I think, as long as Python let the various distribution change those paths though configuration.

In fact, that's one of the future goal of the sysconfig module I had in mind.

Instead of having it reading paths from python variables, I would like to introduce a configuration file global to Python containing all the installation paths, and the paths we are currently discussing.

That file could then be changed by distributions depending on their layouts.

Yes - this sounds good. +1 It also provides a single place for users who are unhappy with the defaults that we choose. Michael

It will avoid distributions to hack Python to change those paths. For instance, Ubuntu currently patches distutils to change the installation paths.

So let's:

- define all the default locations for each system - define a sysconfig.cfg file that contains all installation paths, for each target system - change sysconfig.py so it uses it, instead of the global dicts it currently has

Regards Tarek

-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 8/12/2010 5:50 AM, Tim Golden wrote:

On 11/08/2010 16:22, Éric Araujo wrote:

...

It would be nice to define one standard location for config files used by stdlib modules, and maybe also by third-party programs related closely to Python development (testing tools, static code checkers and the like), in a way that doesn’t clutter the user home directory with a dozen dotfiles while still being easily found.

(The Unix notions of dotfiles and home directory have to be adapted to use non-dotfiles in some standard place on various Windows. The Mac experts disagree on the right directory to use.)

The canonical place on Windows (per Microsoft [1]) is:

for roaming data: the Application Data folder (exposed as the APPDATA env var and as CSIDL_APPDATA pre-Vista and the FOLDERID_RoamingAppData Known Folder on Vista+).

for non-roaming data: the local app data (CSIDL_LOCAL_APPDATA or FOLDERID_LocalAppData)

The obvious difference is whether a user expects the data to be available for the same account on a different computer. At present, the user site packages are held under %APPDATA%\Python\Pythonxx, ie are considered roaming.

Unfortunately, the canonical place is not always the place most used. Especially since the convention under *nix is to place dotfile or dotdirs under $HOME. Windows doesn't, by default, have a $HOME so various locations are considered $HOME, including (but not limited to):

* the directory pointed to by %HOME% * the directory pointed to by %USERPROFILE% * the "My Documents" shell folder (often, but not always, %USERPROFILE%\My Documents) * c:\ (no: really) * the NT profile home directory (typically a mapped share)

Dotfiles are possible on Windows; the issue is that they can't be created within the shell (ie within Windows Explorer). They can be renamed, edited, etc without issue, so it's not much of a problem. They can be created by any program which can call CreateFile -- including Python, obviously.

TJG

[1] http://support.microsoft.com/kb/310294

Didn't we have this discussion when per-user libraries came up? Shouldn't we be using a subdirectory of that location? We ruin the risk of Python becoming distributed so finely it becomes impossible to change things, what with per-user libraries, per-version shared binaries, per-user-per-version library configuration files and the rest. One might make a case that all configuration data should be stored in a single SQLite database (with a suitable API to hide the relational nature of the store). Then at least there would only be one file no matter how many versions of Python were installed. Seriously. We are already spending enough time doing stat calls on non-existent directories. regards Steve PS: Then we can start putting the libraries themselves in the database. That works really well. -- Steve Holden +1 571 484 6266 +1 800 494 3119 DjangoCon US September 7-9, 2010 http://djangocon.us/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/

On Aug 12, 2010, at 6:30 AM, Tim Golden wrote:

I don't care how many stats we're doing

You might not, but I certainly do. And I can guarantee you that the authors of command-line tools that have to start up in under ten seconds, for example 'bzr', care too.

Antoine Pitrou wrote:

On Thu, 12 Aug 2010 18:14:44 -0400 Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:

...

On Aug 12, 2010, at 6:30 AM, Tim Golden wrote:

...

I don't care how many stats we're doing

You might not, but I certainly do. And I can guarantee you that the authors of command-line tools that have to start up in under ten seconds, for example 'bzr', care too.

The idea that import time is dominated by stat() calls sounds rather undemonstrated (and unlikely) to me.

In the case of bzr startup, the exact breakdown varies depending on a range of factors like OS and whether the relevant parts of the filesystem are in the OS cache or not (i.e. is this the first time the user has run bzr since booting?). The short answer is that the number of stat calls isn't really the problem at the moment for bzr, at least not compared to the number of *directories* Python searches, and the amount of non-trivial work done at import time by many modules. But... Your Mileage May Vary. Here's the longer answer: I think some stats about this have been posted to this list before, but some points of interest from the top of my head: * the cost of trying and failing to open foomodule.so + foo.so + foo.pyc + foo.py in a directory isn't much greater than trying to open just one on them. Once the OS has cached the directory entries for that directory subsequent lookups in that directory are fast. The experiement is fairly easy: - strace -e open,stat64 -o py.strace python -c "something..." - by hand, create a .C file that repeats all the stat and open in py.strace (it's pretty easy to munge into valid C) - and also create one with only the *successful* stat and open calls - compare them (using /proc/sys/vm/drop_caches or whatever as appropriate) * each directory probed is a significant cost, especially in the "cold boot" case. So every in sys.path, and every subdirectory of a package. * that said, Windows seems much slower than Linux on equivalent hardware, perhaps attempting to open files is intrinsically more expensive there? Certainly it's not safe to assume conclusions drawn on Linux will apply equally well on Windows, or vice versa. * modules with many class/function definitions are measurably slower than smaller modules. * module-level re.compile calls and other non-trivial operations are to be avoided, but many modules you depend on will do that. This matters so much that bzr monkey-patches the re module to make re.compile lazy. Try grepping the stdlib to see how many modules do re.compile at import time (including as default values of keyword args)! * it's death by a thousand cuts: each module import probably imports a dozen others... by far the simplest way to reduce startup time is to just import less modules. Lazy module imports (bzrlib.lazy_import or hg's demandload or whatever) help a lot, and I wish they were a builtin feature of Python. * I haven't even mentioned NFS or other network filesystems, but you can bet they change the picture significantly. -Andrew.

On Fri, Aug 13, 2010 at 3:21 PM, John Arbash Meinel <john@arbash-meinel.com> wrote:

I don't know what the specific issue is here, but adding entries to sys.path makes startup time *significantly* slower.

I happen to use easy_install since Windows doesn't have its own package manager. Unfortunately the default of creating a new directory and adding it to easy_install.pth is actually pretty terrible.

Adding sys.path entries on Linux isn't free either. Fortunately, this isn't about adding anything to sys.path.   -Fred -- Fred L. Drake, Jr.    <fdrake at gmail.com> "A storm broke loose in my mind."  --Albert Einstein

On 12/08/2010 11:18, Steve Holden wrote:

On 8/12/2010 5:50 AM, Tim Golden wrote:

...

On 11/08/2010 16:22, Éric Araujo wrote:

...

It would be nice to define one standard location for config files used by stdlib modules, and maybe also by third-party programs related closely to Python development (testing tools, static code checkers and the like), in a way that doesn’t clutter the user home directory with a dozen dotfiles while still being easily found.

(The Unix notions of dotfiles and home directory have to be adapted to use non-dotfiles in some standard place on various Windows. The Mac experts disagree on the right directory to use.)

The canonical place on Windows (per Microsoft [1]) is:

for roaming data: the Application Data folder (exposed as the APPDATA env var and as CSIDL_APPDATA pre-Vista and the FOLDERID_RoamingAppData Known Folder on Vista+).

for non-roaming data: the local app data (CSIDL_LOCAL_APPDATA or FOLDERID_LocalAppData)

The obvious difference is whether a user expects the data to be available for the same account on a different computer. At present, the user site packages are held under %APPDATA%\Python\Pythonxx, ie are considered roaming.

Unfortunately, the canonical place is not always the place most used. Especially since the convention under *nix is to place dotfile or dotdirs under $HOME. Windows doesn't, by default, have a $HOME so various locations are considered $HOME, including (but not limited to):

* the directory pointed to by %HOME% * the directory pointed to by %USERPROFILE% * the "My Documents" shell folder (often, but not always, %USERPROFILE%\My Documents) * c:\ (no: really) * the NT profile home directory (typically a mapped share)

Dotfiles are possible on Windows; the issue is that they can't be created within the shell (ie within Windows Explorer). They can be renamed, edited, etc without issue, so it's not much of a problem. They can be created by any program which can call CreateFile -- including Python, obviously.

TJG

[1] http://support.microsoft.com/kb/310294

Didn't we have this discussion when per-user libraries came up? Shouldn't we be using a subdirectory of that location? We ruin the risk of Python becoming distributed so finely it becomes impossible to change things, what with per-user libraries, per-version shared binaries, per-user-per-version library configuration files and the rest.

User editable configuration files are very different from libraries. The per user site-packages folder *should* be hidden somewhere out of the way where you can get at them if you want them but won't stumble across them all the time. e.g. AppData on Windows. For files we expect the user to be able to edit we shouldn't go out of our way to make them hard or inconvenient to find.

One might make a case that all configuration data should be stored in a single SQLite database (with a suitable API to hide the relational nature of the store). Yay - let's recreate the Windows registry! A binary blob only editable with special tools, we know how much users love that.

Michael

Then at least there would only be one file no matter how many versions of Python were installed. Seriously. We are already spending enough time doing stat calls on non-existent directories.

regards Steve

PS: Then we can start putting the libraries themselves in the database. That works really well.

-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 12/08/2010 11:54, Tim Golden wrote:

On 12/08/2010 11:40, Michael Foord wrote:

...

User editable configuration files are very different from libraries. The per user site-packages folder *should* be hidden somewhere out of the way where you can get at them if you want them but won't stumble across them all the time. e.g. AppData on Windows.

For files we expect the user to be able to edit we shouldn't go out of our way to make them hard or inconvenient to find.

This really comes down -- as with the Mac discussion last week, I think -- to the tension between standard file locations and easy-to-get-to. The difference on Windows, though, is that while on Mac it's either: Apple standard /This/That/TheOther vs Unix standard $HOME/.something on Windows it's %APPDATA%\myprog vs whatever-I-thought-of-at-the-time.

APPDATA isn't actually that hard to get to on any version of Windows post 2000. (Just stick %APPDATA% into any Explorer window -- even IE, for goodness sake). Even from the command line it's not hard:

notepad "%APPDATA%\python\python26\idle.ini"

Ultimately, as a Windows user/developer, I'd vote for a memorable consistency, which for me would be %APPDATA%\python\... If everything's in one place, I'll just create a shortcut / hardlink / whatever to get me there when I want to change things.

How is ~/python not memorable or consistent? (And cross-platform memorability and consistency is valuable too.) Another issue is discoverability. Many users won't know about these config files unless they *see* them. Sure if you *know* where they are then editing them is easy - but how often as a user do you browse your AppData folder? (As a windows user the only times I went delving in this mysterious location was trying to diagnose obscure errors.) I'm not aware of any consistent pattern of putting *user editable* files in AppData, and I doubt this is where most users would look for them. In fact for Windows the *correct* thing to do is probably to use the registry and then provide a graphical tool for editing the values. If you're arguing for consistency why not argue for this? If we were to provide native GUI tools for editing values then the location suggested by Ronald for the Mac (which is the standard location for preferences files maintained by GUI tools) would also be the right location. All the best, Michael

TJG _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 12 August 2010 12:59, Tim Golden <mail@timgolden.me.uk> wrote:

re: using the Registry: To be honest, I was answering the literal question posed by Eric: where to put config files? Not the wider question: how should config data be stored? Where the answer to the latter question might be: the Registry -- much as I find it awkward to work with at times.

One very definite point in its favour as regards this discussion is that Python on Windows *already* defines a key in the Registry (and has done so since v2.0 at least) and it would make some sense to place things under there.

Ultimately, I don't feel very strongly about this subject. I'm more concerned that the chosen location (file or registry or whatever) be documented -- and documented from a Windows perspective as well, so you don't have to guess what "HOME" means in this context.

One potential solution would be to use something like os.path.expanduser("~/.python"), where os.path.expanduser has a defined behaviour on Windows (one that could do with a little clarification, yes I know, doc patch welcomed...), but also add an additional option, that os.path.expanduser will first look in the registry for the value HKLM\Software\Python\PythonCore\HOME and use that if present. Then users could set their own interpretation, in a Windows-standard manner. If that would be an acceptable option, I could supply a patch to os.path.expanduser. I will do a clarifying doc patch regardless. Paul. PS We could also support ~user by looking in HKEY_USERS\<user SID>\Software\Python\... but that's probably YAGNI :-)

Michael Foord writes:

How is ~/python not memorable or consistent? (And cross-platform memorability and consistency is valuable too.)

But what does "~" mean on Windows? Inside of Python you can have a consistent definition, but that doesn't help people whose installer gets mixed signals so Python itself is in the wrong place, or who are using external tools (uninstaller, editor, application, etc) that work on or with Python. I'm not arguing for use of AppData, but at least it's easy to explain, and easy to implement, for consistency.

Another issue is discoverability. Many users won't know about these config files unless they *see* them.

OK, maybe AppData's out.

In fact for Windows the *correct* thing to do is probably to use the registry and then provide a graphical tool for editing the values. If you're arguing for consistency why not argue for this?

Sounds reasonable to me. Except Python's standard GUI is probably not up to Windows standards of beauty....

On 13/08/2010 10:02 PM, Michael Foord wrote:

On 13/08/2010 06:39, Stephen J. Turnbull wrote:

...

Michael Foord writes:

...

How is ~/python not memorable or consistent? (And cross-platform memorability and consistency is valuable too.)

But what does "~" mean on Windows? There is a "user directory" in Windows directly analagous to ~, and this is the path returned by os.path.expanduser('~') in Python:

Well, see my post early on in the thread about the various senses of "user directory" under Windows. The addition of expanduser to include Windows caused a long debate at the time. (ISTR) I'm not really all that bothered for this purpose. There's something to be said for all the suggestions so far. However, as I've said elsewhere, I'm more concerned that whatever we end up choosing for location(s) be clearly documented as such. TJG

On Thu, Aug 12, 2010 at 6:18 AM, Steve Holden <steve@holdenweb.com> wrote:

Didn't we have this discussion when per-user libraries came up? Shouldn't we be using a subdirectory of that location? We ruin the risk of Python becoming distributed so finely it becomes impossible to change things, what with per-user libraries, per-version shared binaries, per-user-per-version library configuration files and the rest.

Yes, we did. That was in the context of some people's desire for a per-user site-packages directory, and the result is that we're now saddled with a per-user site-packages directory. The structure determined there was the same sort of half-way version-specific mess as so many others are: libraries are per-python-version, and scripts are installed cowboy-style (whoever got there last, wins). Perhaps user configuration belongs in ~/.local/, or ~/.local/python/ (with attendant Windows & Mac OS noises); I don't really care where it lands, because right now we just have a mess. Getting it "right" with respect to Window's "roaming" notion and how people expect to work with it is... unlikely. Picking a place is better than not, so we know where to find things. But that's all it buys us.   -Fred -- Fred L. Drake, Jr.    <fdrake at gmail.com> "A storm broke loose in my mind."  --Albert Einstein

On Fri, Aug 13, 2010 at 11:57 AM, Barry Warsaw <barry@python.org> wrote:

I've missed most of this discussion while on vacation, but if ~/.local is supposed to mirror /usr/local, then wouldn't a logical place for per-user configuration files be ~/.local/etc/whatever.cfg?

Maybe it is; I'd hope so. The fd.o spec that describes ~/.local/share/ doesn't say anything *else* about ~/.local/, though. So our expansion of ~/.local/ is very much our own. My own ~/.local/ contains a bunch of gnomish desktop stuff in share/ and Python stuff in lib/, and that's it. I'd be as happy as I'm going to be with configs in ~~/.local/etc/. (Still going to be a grouchy old fart.)   -Fred -- Fred L. Drake, Jr.    <fdrake at gmail.com> "A storm broke loose in my mind."  --Albert Einstein

On Thu, 12 Aug 2010 08:18:27 pm Steve Holden wrote:

One might make a case that all configuration data should be stored in a single SQLite database (with a suitable API to hide the relational nature of the store).

-1 Please don't even *consider* such a thing. Haven't we learned from Firefox? I for one am getting heartily sick and tired of having to "fix" (that is, throw away) corrupted Firefox databases. Please don't go down that path. -- Steven D'Aprano