Why site-packages?

[Thomas Jollans]

On 07/03/2012 11:34 PM, Dan Stromberg wrote:
> On Tue, Jul 3, 2012 at 9:04 PM, Ian Kelly <ian.g.kelly at gmail.com
> <mailto:ian.g.kelly at gmail.com>> wrote:
> 
>     On Tue, Jul 3, 2012 at 2:40 PM, Dan Stromberg <drsalists at gmail.com
>     <mailto:drsalists at gmail.com>> wrote:
>     >
>     > Why is it that so much 3rd party python code gets installed to
>     > site-packages?
> 
>     Because that's what site-packages is for?
> 
> 
> Agh.  But -why- is it because that's what it's for?
> 
> "Who made this rule"?

It's (reasonably) consistent with the way things are usually done on UNIX.

Libraries go in /usr/lib. That's all libraries, not just core system
libraries.

Similarly, Python modules go in /usr/lib/python. That's all Python
modules, not just the standard library. That means that you can use them
in scripts without faffing around because the interpreter will find
them, even if you're in a completely random directory.

The point of site-packages then is to make sure that user-installed
packages are cleanly separated from standard and distribution-installed
packages, similarly to the way /usr/local/ is used in general (and can
be used for Python packages, of course)

>  
> 
>     > Even for things that are almost certainly going to be used by a single
>     > application?
>     >
>     > Even for things you might only use once?
>     >
>     > Even for things that might require one version for one app, and
>     another
>     > version for another app?
> 
>     For these types of uses, I suggest using virtualenv if you don't want
>     to pollute your site-packages.  Or Python 3.3 with the new built-in
>     virtual environment option.
> 
> Virtualenv is worth thinking about, but it kind of does the same thing,
> just less of it.
>  
> 
>     > Why not stash an application's python modules in
>     /usr/local/lib/[appname],
>     > and stash a little frontend in /usr/local/bin that adds
>     > /usr/local/lib/[appname] to sys.path?
> 
>     What if you then write an application that you find needs two of these
>     libraries?  You write yet another frontend that adds both of them to
>     sys.path?
> 
> If it was intended to be reusable code, I'd think it -should- go in
> site-packages.
> 
> But so much code in site-packages isn't intended to be reusable.
> 
> And even for stuff that's reusable, there are advantages to just
> duplicating them for the purposes of each application, because you never
> know when one of them is going to need a different version from another.
> 
> If we weren't stashing so much stuff in site-packages, there probably
> wouldn't have been a desire for something like virtualenv.
>  
> 
>     > Here's a thread on stackoverflow today asking why python starts up so
>     > slowly, and making it clear that this is because so much stuff
>     ends up in
>     > site-packages:
>     >
>     http://stackoverflow.com/questions/11318028/is-it-safe-to-use-pythons-s-option
> 
>     I think you may be misunderstanding that thread.  They're saying that
>     starting Python with the -S option (i.e. not automatically importing
>     the site module) significantly cuts down on Python's startup time.
> 
> Yes...
>  
> 
>     The site module has to process any .pth files in the site-packages,
>     but apart from that, I think the actual amount of stuff in
>     site-packages should be irrelevant.
> 
> Irrelevant to what?   More stuff in site slowing things down?  Are
> .pth's not correlated with more stuff in site-packages?  Aren't they
> actually a thing In site?
>  
> 
>      Nothing in site-packages is
>     actually imported or otherwise processed until the application
>     requests it.
> 
> Other than .pth's, apparently.
>  
> 
>      You could have a completely empty site-packages, or you
>     could have 20 gigs of third-party packages, and the time to import
>     site would be basically the same.
> 
> Well, I'd guess that a large directory would slow things down before
> causing filesystem complaints, but I see your point.
> 
> 
>