On Windows and Mac OS, clearly these should go somewhere under sys.prefix: this is the directory for all things Python, including third-party module distributions. If Brian Hooper distributes a module "foo" that requires a data file containing character encoding data (yes, this is based on a true story), then the module belongs in (eg.) C:\Python and the data file in (?) C:\Python\Data. (Maybe C:\Python\Data\foo, but that's a minor wrinkle.)

Any disagreement so far?

A little. I dont think we need a new dump for arbitary files that no one can associate with their application. Why not put the data with the code? It is quite trivial for a Python package or module to find its own location, and this way we are not dependent on anything. Why assume packages are installed _under_ Python? Why not just assume the package is _reachable_ by Python. Once our package/module is being executed by Python, we know exactly where we are. On my machine, there is no "data" equivilent; the closest would be "python-cvs\pcbuild\data", and that certainly doesnt make sense. Why can't I just place it where I put all my other Python extensions, ensure it is on the PythonPath, and have it "just work"? It sounds a little complicated - do we provide an API for this magic location, or does everybody cut-and-paste a reference implementation for locating it? Either way sounds pretty bad - the API shouldnt be distutils dependent (I may not have installed this package via distutils), and really Python itself shouldnt care about this... So all in all, I dont think it is a problem we need to push up to this level - let each package author do whatever makes sense, and point out how trivial it would be if you assumed code and data in the same place/tree. [If the data is considered read/write, then you need a better answer anyway, as you can't assume "c:\python\data" is writable (when actually running the code) anymore than "c:\python\my_package" is] Mark.

[Guido writes]

Modifyable data needs to go in a per-user directory, even on Windows, outside the Python tree.

This seems to be the value of key "AppData" stored under in HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Filders Right? Thomas

Is there a reliable algorithm to find a "per-user" directory on any Win95/98/NT/2000 system?

Ahhh - where to start. SHGetFolderLocation offers the following alternatives: CSIDL_APPDATA Version 4.71. File system directory that serves as a common repository for application-specific data. A typical path is C:\Documents and Settings\username\Application Data CSIDL_COMMON_APPDATA Version 5.0. Application data for all users. A typical path is C:\Documents and Settings\All Users\Application Data. CSIDL_LOCAL_APPDATA Version 5.0. File system directory that serves as a data repository for local (non-roaming) applications. A typical path is C:\Documents and Settings\username\Local Settings\Application Data. CSIDL_PERSONAL File system directory that serves as a common repository for documents. A typical path is C:\Documents and Settings\username\My Documents. CSIDL_PERSONAL File system directory that serves as a common repository for documents. A typical path is C:\Documents and Settings\username\My Documents. Plus a few I didnt bother listing... <sigh> Mark.

On 26 May 2000, Gordon McMillan said:

Yeah. I tend to install stuff outside the sys.prefix tree and then use .pth files. I realize I'm, um, unique in this regard but I lost everything in some upgrade gone bad. (When a Windows de- install goes wrong, your only option is to do some manual directory and registry pruning.)

I think that's appropriate for Python "applications" -- in fact, now that Distutils can install scripts and miscellaneous data, about the only thing needed to properly support "applications" is an easy way for developers to say, "Please give me my own directory and create a .pth file". (Actually, the .pth file should only be one way to install an application: you might not want your app's Python library to muck up everybody else's Python path. An idea AMK and I cooked up yesterday would be an addition to the Distutils "build_scripts" command: along with frobbing the #! line to point to the right Python interpreter, add a second line: import sys ; sys.append(path-to-this-app's-python-lib) Or maybe "sys.insert(0, ...)". Anyways, that's neither here nor there. Except that applications that get their own directory should be free to put their (static) data files wherever they please, rather than having to put them in the app's Python library. I'm more concerned with the what the Distutils works best with now, though: module distributions. I think you guys have convinced me; static data should normally sit with the code. I think I'll make that the default (instead of prefix + "share"), but give developers a way to override it. So eg.: data_files = ["this.dat", "that.cfg"] will put the files in the same place as the code (which could be a bit tricky to figure out, what with the vagaries of package-ization and "extra" install dirs); data_files = [("share", ["this.dat"]), ("etc", ["that.cfg"])] would put the data file in (eg.) /usr/local/share and the config file in /usr/local/etc. This obviously makes the module writer's job harder: he has to grovel from sys.prefix looking for the files that he expects to have been installed with his modules. But if someone really wants to do this, they should be allowed to. Finally, you could also put absolute directories in 'data_files', although this would not be recommended.

(Hmm, I guess it is if you use rpms.)

All the smart Unix installers (RPM, Debian, FreeBSD, ...?) I know of have some sort of dependency mechanism, which works to varying degrees of "work". I'm only familar with RPM, and my usual response to a dependency warning is "dammit, I know what I'm doing", and then I rerun "rpm --nodeps" to ignore the dependency checking. (This usually arises because I build my own Perl and Python, and don't use Red Hat's -- I just make /usr/bin/{perl,python} symlinks to /usr/local/bin, which RPM tends to whine about.) But it's nice to know that someone is watching. ;-) Greg -- Greg Ward - software developer gward@mems-exchange.org MEMS Exchange / CNRI voice: +1-703-262-5376 Reston, Virginia, USA fax: +1-703-262-5367

Greg Ward wrote:

On 26 May 2000, Gordon McMillan said:

...

Yeah. I tend to install stuff outside the sys.prefix tree and then use .pth files. I realize I'm, um, unique in this regard but I lost everything in some upgrade gone bad. (When a Windows de- install goes wrong, your only option is to do some manual directory and registry pruning.)

I think that's appropriate for Python "applications" -- in fact, now that Distutils can install scripts and miscellaneous data, about the only thing needed to properly support "applications" is an easy way for developers to say, "Please give me my own directory and create a .pth file".

Hmm. I see an application as a module distribution that happens to have a script. (Or maybe I see a module distribution as a scriptless app ;-)). At any rate, I don't see the need to dignify <prefix>/share and friends with an official position.

(Actually, the .pth file should only be one way to install an application: you might not want your app's Python library to muck up everybody else's Python path. An idea AMK and I cooked up yesterday would be an addition to the Distutils "build_scripts" command: along with frobbing the #! line to point to the right Python interpreter, add a second line: import sys ; sys.append(path-to-this-app's-python-lib)

Or maybe "sys.insert(0, ...)".

$PYTHONSTARTUP ?? Never really had to deal with this. On my RH box, /usr/bin/python is my build. At a client site which had 1.4 installed, I built 1.5 into $HOME/bin with a hacked getpath.c.

I'm more concerned with the what the Distutils works best with now, though: module distributions. I think you guys have convinced me; static data should normally sit with the code. I think I'll make that the default (instead of prefix + "share"), but give developers a way to override it. So eg.:

data_files = ["this.dat", "that.cfg"]

will put the files in the same place as the code (which could be a bit tricky to figure out, what with the vagaries of package-ization and "extra" install dirs);

That's an artifact of your code ;-). If you figured it out once, you stand at least a 50% chance of getting the same answer a second time <.5 wink>. - Gordon

On Sun, 28 May 2000, Guido van Rossum wrote:

...

Idea: Wouldn't it be nice if the 'nt' and 'mac' versions of the 'os' module would provide 'os.environ["HOME"]' similar to the posix version? This would certainly simplify the task of application programmers intending to write portable applications.

This sounds like a nice idea...

Now that this idea has fermented for a few days, I'm inclined to not like it. It smells of making Unix-centric interface to something that isn't terribly portable as a concept. Perhaps there should be a function that does the "right thing", extracting os.environ["HOME"] if defined, and taking an alternate approach (os.getcwd() or whatever) otherwise. I don't think setting os.environ["HOME"] in the library is a good idea because that changes the environment that gets published to child processes beyond what the application does. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>

[Fred]

Now that this idea has fermented for a few days, I'm inclined to not like it. It smells of making Unix-centric interface to something that isn't terribly portable as a concept. Perhaps there should be a function that does the "right thing", extracting os.environ["HOME"] if defined, and taking an alternate approach (os.getcwd() or whatever) otherwise. I don't think setting os.environ["HOME"] in the library is a good idea because that changes the environment that gets published to child processes beyond what the application does.

The passing on to child processes doesn't sound like a big deal to me. Either these are Python programs, in which case they might appreciate that the work has already been done, or they aren't, in which case they probably don't look at $HOME at all (since apparently they worked before). I could see defining a new API, e.g. os.gethomedir(), but that doesn't help all the programs that currently use $HOME... Perhaps we could do both? (I.e. add os.gethomedir() *and* set $HOME.) --Guido van Rossum (home page: http://www.python.org/~guido/)

[Fred]

...

Now that this idea has fermented for a few days, I'm inclined to not like it. It smells of making Unix-centric interface to something that isn't terribly portable as a concept.

Yes. After thinking more carefully and after a closer look to what Jack Jansen finally figured out for MacOS (see <http://www.python.org/pipermail/pythonmac-sig/2000-May/003667.html> ) I agree with Fred. My initial idea to put something into 'os.environ["HOME"]' on those platforms was too simple minded.

...

Perhaps there should be a function that does the "right thing", extracting os.environ["HOME"] if defined, and taking an alternate approach (os.getcwd() or whatever) otherwise. [...]

Every serious (non trivial) application usually contains something like "user preferences" or other state information, which should --if possible-- survive the following kind of events: 1. An upgrade of the application to a newer version. This is often accomplished by removing the directory tree, in which this application lives and replacing it by unpacking or installing in archive containing the new version of the application. 2. Another colleague uses the application on the same computer and modifies settings to fit his personal taste. On several versions of WinXX and on MacOS prior to release 9.X (and due to stability problems with the multiuser capabilities even in MacOS 9) the second kind of event seems to be rather unimportant to the users of these platforms, since the OSes are considered as "single user" systems anyway. Or in other words: the users are already used to this situation. Only the first kind of event should be solved for all platforms: <FANTASY> Imagine you are using grail version 4.61 on a daily basis for WWW browsing and one day you decide to install the nifty upgrade grail 4.73 on your computer running WinXX or MacOS X.Y and after doing so you just discover that all your carefully sorted bookmarks are gone! That wouldn't be nice? </FANTASY> I see some similarities here to the standard library module 'tempfile', which supplies (or at least it tries ;-) to) a cross-platform portable strategy for all applications which have to store temporary data. My intentation was to have simple a cross-platform portable API to store and retrieve such user specific state information (examples: the bookmarks of a Web browser, themes, color settings, fonts... other GUI settings, and so... you get the picture). On unices applications usually use the idiom os.path.join(os.environ.get("HOME", "."), ".dotfoobar") or something similar. Do people remember 'grail'? I've just stolen the following code snippets from 'grail0.6/grailbase/utils.py' just to demonstrate, that this is still a very common programming problem: ---------------- snip --------------------- # XXX Unix specific stuff # XXX (Actually it limps along just fine for Macintosh, too) def getgraildir(): return getenv("GRAILDIR") or os.path.join(gethome(), ".grail") ----- snip ------ def gethome(): try: home = getenv("HOME") if not home: import pwd user = getenv("USER") or getenv("LOGNAME") if not user: pwent = pwd.getpwuid(os.getuid()) else: pwent = pwd.getpwnam(user) home = pwent[6] return home except (KeyError, ImportError): return os.curdir ---------------- snap --------------------- [...] [Guido van Rossum]:

I could see defining a new API, e.g. os.gethomedir(), but that doesn't help all the programs that currently use $HOME... Perhaps we could do both? (I.e. add os.gethomedir() *and* set $HOME.)

I'm not sure whether this is really generic enough for the OS module. May be we should introduce a new small standard library module called 'userprefs' or some such? A programmer with a MacOS or WinXX background will probably not know what to do with 'os.gethomedir()'. However for the time being this module would only contain one simple function returning a directory pathname, which is guaranteed to exist and to survive a deinstallation of an application. May be introducing a new module is overkill? What do you think? Regards, Peter -- Peter Funk, Oldenburger Str.86, D-27777 Ganderkesee, Germany, Fax:+49 4222950260 office: +49 421 20419-0 (ArtCom GmbH, Grazer Str.8, D-28359 Bremen)

Gordon writes,

But there's no $HOME as such.

There's HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\S hell Folders with around 16 subkeys, including AppData (which on my system has one entry installed by a program I've never used and didn't know I had). But MSOffice uses the Personal subkey. Others seem to use the Desktop subkey.

SHGetSpecialFolderPath(,,CSIDL_APPDATA,) would be the current 'MS preferred' method for this as it allows roaming (not that I've ever seen roaming work). If Unix code expects $HOME to be per machine (and so used to store, for example, window locations which are dependent on screen resolution) then CSIDL_LOCAL_APPDATA would be a better choice. To make these work on 9x and NT 4 Microsoft provides a redistributable Shfolder.dll. Fred writes,

Look at your $HOME on Unix box; most of the dotfiles are *files*, not directories, and that's all most applications need;

This may have been the case in the past and for people who understand Unix well enough to maintain it, but for us just-want-it-to-run folks, its no longer true. I formatted my Linux partition this week and installed Red Hat 6.2 and Gnome 1.2 and then used a few applications. The dot directories outnumber the dot files 18 to 16. Neil

Using a place on the local machine of the user makes more sense to me.

But excuse my ingorance: I've just 'grep'ed through the Python 1.6a2 sources and also through Mark Hammonds Win32 Python extension c++ sources (here on my Notebook running Linux) and found nothing called 'SHGetSpecialFolderPath'. So I believe, this API is currently not exposed to the Python level. Right?

Only through the Win32 Python extensions, I think:

...

...

from win32com.shell import shell from win32com.shell import shellcon shell.SHGetSpecialFolderPath(0, shellcon.CSIDL_APPDATA) u'G:\\Documents and Settings\\Neil1\\Application Data' shell.SHGetSpecialFolderPath(0, shellcon.CSIDL_LOCAL_APPDATA) Traceback (most recent call last): File "<stdin>", line 1, in ? AttributeError: CSIDL_LOCAL_APPDATA shell.SHGetSpecialFolderPath(0, 0x1c) u'G:\\Documents and Settings\\Neil1\\Local Settings\\Application Data'

Looks like CSIDL_LOCAL_APPDATA isn't included yet, but its value is 0x1c. Neil