[Python-3000] [Python-Dev] Reminder: last alphas next Wednesday 07-May-2008

[Guido van Rossum]

I stand corrected on a few points. You've convinced me that ~/lib/ is
wrong. But I still don't like ~/.local/; not in the last place because
it's not any more local than any other dot files or directories. The
"symmetry" with /usr/local/ is pretty weak, and certainly won't help
beginning users.

As a compromise, I'm okay with ~/Python/. I would like to be able to
say that the user explicitly has to set an environment variable in
order to benefit from this feature, just like with $PYTHONPATH and
$PYTHONSTARTUP. But that might defeat the point of making this easy to
use for noobs.

On OS X I think we should put this somewhere under ~/Library/. Just
put it in a different place than where the Python framework puts its
stuff.

On Thu, May 1, 2008 at 8:25 PM,  <glyph at divmod.com> wrote:
> On 01:55 am, guido at python.org wrote:
>
> > On Thu, May 1, 2008 at 5:03 PM,  <glyph at divmod.com> wrote:
> >
>
>  Hi everybody.  I apologize for writing yet another lengthy screed about a
> simple directory naming issue.  I feel strongly about it but I encourate
> anyone who doesn't to simply skip it.
>
>  First, some background: my strong feelings here are actually based on an
> experience I had a long time ago when helping someone with some C++
> programming homework.  They were baffled because when I helped them the
> programs compiled, but then as soon as they tried it on their own it didn't.
> The issue was that I had replicated my own autotools-friendly directory
> structure for them (at the time, "~/bin", "~/include", "~/lib", "~/etc", and
> so on managed with GNU stow) onto their machine and edited their shell setup
> to include them appropriately.  But, as soon as I was finished, they
> "cleaned up" the "mess" I had left behind, and thereby removed all of their
> build dependencies.  This was on a shared university build server, before
> the days of linux as a friendly, graphical operating system which encouraged
> you to look even more frequently at your home directory, so if anything I
> suspect the likelihood that this is a problem would be worse now.  Since
> cleaning up my own home directory, of course, I find that I appreciate the
> lack of visual noise in Nautilus et. al. as well.
>
>  Also, while I obviously think all tools should work this way, I think that
> Python in particular will attract an audience who is learning to program but
> not necessarily savvy with arcane nuances of filesystem layout, and it would
> be best if those details were abstracted.
>
>  My concern here is for the naive python developer reading installation
> instructions off of a wiki and trying to get started with Twisted
> development.  Seeing a directory created in your home directory (or, as the
> case may be, 3 directories, "bin", "lib", and "include") is a bit of a
> surprise.  They don't actually care where the files in their installed
> library are, as long as they're "installed", and they can import them.
> However, they may care that clicking on the little house icon now shows not
> "Pictures", "Movies", etc, but "lib" (what's a 'lib'?) "bin" (what's a bin?
> is that like a box where I throw my stuff?) "share" (I put my stuff in
> "share", but it's not shared.  Wait, I'm supposed to put it in "Public"?).
>
>
> >
> > >  Briefly, "lib" is not the only directory participating in this
> convention;
> > > you've also got the full complement of other stuff that might go into an
> > > installation like /usr/local.  So, while "lib" might annoy me a little,
> "bin
> > > etc games include lib lib32 man sbin share src" is going to get ugly
> pretty
> > > fast, especially if this is what comes up in Finder or Nautilus or
> Explorer
> > > every time I open a window.
> > >
> >
> > Unless I misread the PEP, there's only going to be a lib subdirectory.
> > Python packages don't put stuff in other places AFAIK.
> >
>
>  Python packages, at the very least, frequently put stuff in "bin" (or
> "scripts", I think, on Windows).  Not all Python packages are pure- Python
> packages either; setup.py boasts --install-platlib, --install- headers,
> --install-data, and --exec-prefix options, which suggests an "include",
> "bin", and "share" directory, at least.  I'm sure if I had more time to
> grovel around I'd find one that installed manpages. Twisted has some, but
> apparently setup.py doesn't do anything with them, we leave that to the OS
> packages...
>
>  Of course, very little of this is handled by the PEP.  But even the usage
> of the name "lib" implies that the PEP is taking some care to be compatible
> with an idiom that goes beyond Python itself here, or at least beyond simple
> Python packages.
>
>  Even assuming that no Python library ever wanted to install any of these
> things, there are many Python libraries which are simply wrappers around
> lower-level libraries, and if I want to perform a per-user install of one of
> those, I am going to ./configure --prefix=~/something (and by "something", I
> mean ".local" ;)) and it would be nice to have Python living in the same
> space.  For that matter it'd be nice to get autotools and Ruby and PHP and
> Perl and Emacs (ad nauseum) all looking at ~/.local as a mirror of /usr, so
> that I didn't have to write a bunch of shell bootstrap glue to get
> everything to behave consistently, or learn the new, special names for bits
> of configuration under "~" that are different from the ones under /usr/local
> or /etc.
>
>  I replicate a consistent Python development environment with a ton of
> bizarre dependencies across something like 15 different OS installations
> (not to mention a bevy of virtual machines I keep around just for fun), so I
> think about these issues a lot.  Most of these machines are macs and linux
> boxes, but I do my best on Windows too.  FWIW I don't have any idea what the
> right thing to do is on Windows; ".local" doesn't particularly make sense,
> but neither does "lib" in that context. There's no reasonable guess as to
> where to put scripts, or dependent shared libraries... but then, per-user
> installation is less of an issue on Windows.
>
>
> > On the Mac, the default Finder window is not your home directory but
> > your Desktop, which is a subdirectory thereof with a markedly public
> > name. In fact, OS X has a whole bunch of reserved names in your home
> > directory, and none of them start with a dot. The rule seems to be
> > that if it contains stuff that the user cares about, it doesn't start
> > with a dot.
> >
>
>  Hmm.  On my Mac laptop, the default Finder window is definitely my home
> directory; this may be an artifact of many OS upgrades or some tweak that I
> performed a long time ago and forgot about, though.  Apologies if that is
> not the average user experience.
>
>  For what it's worth, Ubuntu also has some directories that it creates:
> Desktop, Pictures, Documents, Examples, Templates, Videos.  These are empty,
> and I typically delete the ones I don't use.
>
>
> >
> > > If it's going to be a visible directory on the
> > > grounds that this is a Python- specific thing that is explicitly *not*
> > > participating in a convention with other software, then please call it
> > > "~/Python" or something.
> > >
> >
> > Much better than ~/.local/ IMO.
> >
>
>  It depends how this is being perceived.  If this is Python mirroring the
> /usr/local layout convention for users, as the name "lib" implies, then this
> is worse.  However, if Python is just trying to select a location for its
> own library bookkeeping and not allow the installation of platform libraries
> or scripts using this mechanism... well, ~/.python.d would still be my
> preference ;-) but I could at least understand "Python" as mirroring the
> Mac, GNOME and KDE convention for a few very special directories.
>
>
> >
> > >  Am I the only guy who finds software that insists on visible, fixed
> files
> > > in my home directory rude?  vmware, for example, wants a "~/vmware"
> > > directory, but pretty much every other application I use is nice enough
> to
> > > use dotfiles (even cedega, with a roughly-comparable-to- lib
> "applications
> > > I've installed for you" folder).
> > >
> >
> > The distinction to my mind is that most dot files (with the exception
> > of a few like .profile or .bashrc) are not managed by most users --
> > the apps that manage them provide an APIs for manipulating their
> > contents.  (Sort of like thw Windows registry.)  Non-dot files are for
> > stuff that the user needs to be aware of.
> >
>
>  My experience of modern Linux suggests that the usage you're describing is
> gradually being phased out - applications that want to manage some
> non-user-visible storage in something like the registry increasingly use
> gconf (or a database, in server-land).  Granted, gconf itself is stored in
> dotfiles, but it's just a few.
>
>  In my home directory I have, in version control, variously written by hand
> or databases maintained from externally downloaded stuff:
>
>    ~/.asoundrc
>    ~/.emacs
>    ~/.vimrc
>    ~/.vim
>    ~/.Xresources
>    ~/.fonts
>    ~/.gnomerc
>    ~/.inputrc
>    ~/.bashrc
>    ~/.bash_profile
>    ~/.profile
>    ~/.screenrc
>    ~/.Xresources
>    ~/.ssh/config
>    ~/.ssh/authorized_keys
>    ~/.ssh/known_hosts
>
>  I know about these dot files and I care about them and I maintain them, but
> they're there for the benefit of particular pieces of software, not me.
> There are a lot of other dotfiles there, but I don't think that this set is
> "a few";   I am quite happy that I don't have to see every one of them every
> time I am looking at my home directory in a "save as" dialog.
>
>
> > I'm not sure where Python packages fall, but ISTM that this is
> > something a user must explicitly choose as the target of an installer.
> > The user is also likely to have to dig through there to remove stuff,
> > as Python package management doesn't have a way to remove packages.
> >
>
>  I hope that users never have to explicitly choose this as the target of the
> installer; I was under the impression that the point of adding this feature
> was to allow the default behavior of distutils to work simply and
> automatically on UNIX-y platforms rather than puking about permissions, or
> requiring arcana like  "sudo" access or editing your shell's startup.  I am
> quietly agitating elsewhere to get ~/.local/bin added to $PATH by default,
> by the way ;-).  (~/.local/lib on $LD_LIBRARY_PATH is a hard sell, but that
> too...)
>
>  Once you have to know about it and explicitly choose it it's not much more
> work to set all the appropriate shell environment variables yourself.  And,
> for that matter, *I* already have, so I suppose regardless of the outcome of
> this discussion I'll still have a ~/.local :-).
>
>
> >
> > >  Put another way - it's trivial to make ~/.local/lib show up by
> symlinking
> > > ~/lib,
> > >
> >
> > That's not the same thing at all.
> >
>
>  I'm not sure what you're saying it's not the same as.  All I'm saying is
> that if advanced users want to show it, they'll symlink it; if naive users
> want to hide it, they'll delete it and break python, possibly without
> knowing why ;).
>
>
> >
> > > but you can't make ~/lib disappear, and lots of software ends up
> > > looking at ~.
> > >
> >
> > But what software cares about another file there? My home directory is
> > mostly a switching point where I have quick access to everything I
> > access regularly.
> >
>
>  Nothing's going to break, if that's what you mean.  No software processes
> the list of ~ and does anything with it; but lots of stuff shows me that
> list.  In GNOME, on Ubuntu, when a "choose file" dialog comes up, 80% of the
> time it comes up by default in my home directory. When I open a terminal it
> opens in my home directory.  The default location for Emacs is my home
> directory.  I can quickly measure my cognitive load by looking at the
> contents of that directory.  Since my shell starts there, autocomplete
> starts there, and so common-letter real estate is scarce.  I have a
> directory called "Projects" that I currently autocomplete with 'p<tab>' and
> a directory called 'Linux' that I autocomplete with 'l<tab>'; either
> public-name proposal will have me typing an additional letter on these every
> day ;-).
>
>  In other words, I care about another file there.  I use my home directory
> as a sort of to-do list; it's mostly empty unless I have a lot going on, in
> which case it fills up with various objects I'm working on, and then I empty
> it out again.  There are a few exceptions to this rule; on every platform
> there are a few things the OS puts there, but they are generally things like
> "Pictures", "Desktop", and "Music"... where I put pictures, downloaded
> files, and music. The Mac's "Library" directory has never bothered me, since
> it's OS-provided and basically an alternate location for dotfiles.
> ("Application Data" and friends are another story.)
>
>  In a way, I agree with you.  "everything I access regularly" is a good
> description of my home directory.  Except, this "lib" directory is not
> something I want to access regularly; very occasionally, maybe once every
> few weeks, I want to chuck some dependency in there and then forget about it
> for a year.
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)