[Python-Dev] [Python-3000] Reminder: last alphas next Wednesday 07-May-2008

[glyph at divmod.com]

On 01:55 am, guido at python.org wrote:
>On Thu, May 1, 2008 at 5:03 PM,  <glyph at divmod.com> wrote:

Hi everybody.  I apologize for writing yet another lengthy screed about 
a simple directory naming issue.  I feel strongly about it but I 
encourate anyone who doesn't to simply skip it.

First, some background: my strong feelings here are actually based on an 
experience I had a long time ago when helping someone with some C++ 
programming homework.  They were baffled because when I helped them the 
programs compiled, but then as soon as they tried it on their own it 
didn't.  The issue was that I had replicated my own autotools-friendly 
directory structure for them (at the time, "~/bin", "~/include", 
"~/lib", "~/etc", and so on managed with GNU stow) onto their machine 
and edited their shell setup to include them appropriately.  But, as 
soon as I was finished, they "cleaned up" the "mess" I had left behind, 
and thereby removed all of their build dependencies.  This was on a 
shared university build server, before the days of linux as a friendly, 
graphical operating system which encouraged you to look even more 
frequently at your home directory, so if anything I suspect the 
likelihood that this is a problem would be worse now.  Since cleaning up 
my own home directory, of course, I find that I appreciate the lack of 
visual noise in Nautilus et. al. as well.

Also, while I obviously think all tools should work this way, I think 
that Python in particular will attract an audience who is learning to 
program but not necessarily savvy with arcane nuances of filesystem 
layout, and it would be best if those details were abstracted.

My concern here is for the naive python developer reading installation 
instructions off of a wiki and trying to get started with Twisted 
development.  Seeing a directory created in your home directory (or, as 
the case may be, 3 directories, "bin", "lib", and "include") is a bit of 
a surprise.  They don't actually care where the files in their installed 
library are, as long as they're "installed", and they can import them. 
However, they may care that clicking on the little house icon now shows 
not "Pictures", "Movies", etc, but "lib" (what's a 'lib'?) "bin" (what's 
a bin?  is that like a box where I throw my stuff?) "share" (I put my 
stuff in "share", but it's not shared.  Wait, I'm supposed to put it in 
"Public"?).
>>  Briefly, "lib" is not the only directory participating in this 
>>convention;
>>you've also got the full complement of other stuff that might go into 
>>an
>>installation like /usr/local.  So, while "lib" might annoy me a 
>>little, "bin
>>etc games include lib lib32 man sbin share src" is going to get ugly 
>>pretty
>>fast, especially if this is what comes up in Finder or Nautilus or 
>>Explorer
>>every time I open a window.
>
>Unless I misread the PEP, there's only going to be a lib subdirectory.
>Python packages don't put stuff in other places AFAIK.

Python packages, at the very least, frequently put stuff in "bin" (or 
"scripts", I think, on Windows).  Not all Python packages are pure- 
Python packages either; setup.py boasts --install-platlib, --install- 
headers, --install-data, and --exec-prefix options, which suggests an 
"include", "bin", and "share" directory, at least.  I'm sure if I had 
more time to grovel around I'd find one that installed manpages. 
Twisted has some, but apparently setup.py doesn't do anything with them, 
we leave that to the OS packages...

Of course, very little of this is handled by the PEP.  But even the 
usage of the name "lib" implies that the PEP is taking some care to be 
compatible with an idiom that goes beyond Python itself here, or at 
least beyond simple Python packages.

Even assuming that no Python library ever wanted to install any of these 
things, there are many Python libraries which are simply wrappers around 
lower-level libraries, and if I want to perform a per-user install of 
one of those, I am going to ./configure --prefix=~/something (and by 
"something", I mean ".local" ;)) and it would be nice to have Python 
living in the same space.  For that matter it'd be nice to get autotools 
and Ruby and PHP and Perl and Emacs (ad nauseum) all looking at ~/.local 
as a mirror of /usr, so that I didn't have to write a bunch of shell 
bootstrap glue to get everything to behave consistently, or learn the 
new, special names for bits of configuration under "~" that are 
different from the ones under /usr/local or /etc.

I replicate a consistent Python development environment with a ton of 
bizarre dependencies across something like 15 different OS installations 
(not to mention a bevy of virtual machines I keep around just for fun), 
so I think about these issues a lot.  Most of these machines are macs 
and linux boxes, but I do my best on Windows too.  FWIW I don't have any 
idea what the right thing to do is on Windows; ".local" doesn't 
particularly make sense, but neither does "lib" in that context. 
There's no reasonable guess as to where to put scripts, or dependent 
shared libraries... but then, per-user installation is less of an issue 
on Windows.
>On the Mac, the default Finder window is not your home directory but
>your Desktop, which is a subdirectory thereof with a markedly public
>name. In fact, OS X has a whole bunch of reserved names in your home
>directory, and none of them start with a dot. The rule seems to be
>that if it contains stuff that the user cares about, it doesn't start
>with a dot.

Hmm.  On my Mac laptop, the default Finder window is definitely my home 
directory; this may be an artifact of many OS upgrades or some tweak 
that I performed a long time ago and forgot about, though.  Apologies if 
that is not the average user experience.

For what it's worth, Ubuntu also has some directories that it creates: 
Desktop, Pictures, Documents, Examples, Templates, Videos.  These are 
empty, and I typically delete the ones I don't use.
>>If it's going to be a visible directory on the
>>grounds that this is a Python- specific thing that is explicitly *not*
>>participating in a convention with other software, then please call it
>>"~/Python" or something.
>
>Much better than ~/.local/ IMO.

It depends how this is being perceived.  If this is Python mirroring the 
/usr/local layout convention for users, as the name "lib" implies, then 
this is worse.  However, if Python is just trying to select a location 
for its own library bookkeeping and not allow the installation of 
platform libraries or scripts using this mechanism... well, ~/.python.d 
would still be my preference ;-) but I could at least understand 
"Python" as mirroring the Mac, GNOME and KDE convention for a few very 
special directories.
>>  Am I the only guy who finds software that insists on visible, fixed 
>>files
>>in my home directory rude?  vmware, for example, wants a "~/vmware"
>>directory, but pretty much every other application I use is nice 
>>enough to
>>use dotfiles (even cedega, with a roughly-comparable-to- lib 
>>"applications
>>I've installed for you" folder).
>
>The distinction to my mind is that most dot files (with the exception
>of a few like .profile or .bashrc) are not managed by most users --
>the apps that manage them provide an APIs for manipulating their
>contents.  (Sort of like thw Windows registry.)  Non-dot files are for
>stuff that the user needs to be aware of.

My experience of modern Linux suggests that the usage you're describing 
is gradually being phased out - applications that want to manage some 
non-user-visible storage in something like the registry increasingly use 
gconf (or a database, in server-land).  Granted, gconf itself is stored 
in dotfiles, but it's just a few.

In my home directory I have, in version control, variously written by 
hand or databases maintained from externally downloaded stuff:

    ~/.asoundrc
    ~/.emacs
    ~/.vimrc
    ~/.vim
    ~/.Xresources
    ~/.fonts
    ~/.gnomerc
    ~/.inputrc
    ~/.bashrc
    ~/.bash_profile
    ~/.profile
    ~/.screenrc
    ~/.Xresources
    ~/.ssh/config
    ~/.ssh/authorized_keys
    ~/.ssh/known_hosts

I know about these dot files and I care about them and I maintain them, 
but they're there for the benefit of particular pieces of software, not 
me.  There are a lot of other dotfiles there, but I don't think that 
this set is "a few";   I am quite happy that I don't have to see every 
one of them every time I am looking at my home directory in a "save as" 
dialog.
>I'm not sure where Python packages fall, but ISTM that this is
>something a user must explicitly choose as the target of an installer.
>The user is also likely to have to dig through there to remove stuff,
>as Python package management doesn't have a way to remove packages.

I hope that users never have to explicitly choose this as the target of 
the installer; I was under the impression that the point of adding this 
feature was to allow the default behavior of distutils to work simply 
and automatically on UNIX-y platforms rather than puking about 
permissions, or requiring arcana like  "sudo" access or editing your 
shell's startup.  I am quietly agitating elsewhere to get ~/.local/bin 
added to $PATH by default, by the way ;-).  (~/.local/lib on 
$LD_LIBRARY_PATH is a hard sell, but that too...)

Once you have to know about it and explicitly choose it it's not much 
more work to set all the appropriate shell environment variables 
yourself.  And, for that matter, *I* already have, so I suppose 
regardless of the outcome of this discussion I'll still have a ~/.local 
:-).
>>  Put another way - it's trivial to make ~/.local/lib show up by 
>>symlinking
>>~/lib,
>
>That's not the same thing at all.

I'm not sure what you're saying it's not the same as.  All I'm saying is 
that if advanced users want to show it, they'll symlink it; if naive 
users want to hide it, they'll delete it and break python, possibly 
without knowing why ;).
>>but you can't make ~/lib disappear, and lots of software ends up
>>looking at ~.
>
>But what software cares about another file there? My home directory is
>mostly a switching point where I have quick access to everything I
>access regularly.

Nothing's going to break, if that's what you mean.  No software 
processes the list of ~ and does anything with it; but lots of stuff 
shows me that list.  In GNOME, on Ubuntu, when a "choose file" dialog 
comes up, 80% of the time it comes up by default in my home directory. 
When I open a terminal it opens in my home directory.  The default 
location for Emacs is my home directory.  I can quickly measure my 
cognitive load by looking at the contents of that directory.  Since my 
shell starts there, autocomplete starts there, and so common-letter real 
estate is scarce.  I have a directory called "Projects" that I currently 
autocomplete with 'p<tab>' and a directory called 'Linux' that I 
autocomplete with 'l<tab>'; either public-name proposal will have me 
typing an additional letter on these every day ;-).

In other words, I care about another file there.  I use my home 
directory as a sort of to-do list; it's mostly empty unless I have a lot 
going on, in which case it fills up with various objects I'm working on, 
and then I empty it out again.  There are a few exceptions to this rule; 
on every platform there are a few things the OS puts there, but they are 
generally things like "Pictures", "Desktop", and "Music"... where I put 
pictures, downloaded files, and music. The Mac's "Library" directory has 
never bothered me, since it's OS-provided and basically an alternate 
location for dotfiles.  ("Application Data" and friends are another 
story.)

In a way, I agree with you.  "everything I access regularly" is a good 
description of my home directory.  Except, this "lib" directory is not 
something I want to access regularly; very occasionally, maybe once 
every few weeks, I want to chuck some dependency in there and then 
forget about it for a year.