[Python-ideas] Policy for altering sys.path

[Thomas Güttler]

Am 06.05.2015 um 17:07 schrieb Paul Moore:
> On 6 May 2015 at 15:05, Thomas Güttler <guettliml at thomas-guettler.de> wrote:
>> I am missing a policy how sys.path should be altered.
> 
> Well, the docs say that applications can modify sys.path as needed.
> Generally, applications modify sys.path in place via sys.path[:] =
> whatever, but that's not mandated as far as I know.
> 
>> We run a custom sub class of list in sys.path. We set it in sitecustomize.py
> 
> Can you explain why? 

I forgot to explain the why I use a custom class. Sorry, here is the background.

I want sys.path to ordered:

 1. virtualenv
 2. /usr/local/
 3. /usr/lib

We use virtualenvs with system site-packages.

There are many places where sys.path gets altered.

The last time we had sys.path problems I tried to write a test
which checks that sys.path is the same for cron jobs and web requests.
I failed. Too many places,  I could not find all the places
and the conditions where sys.path got modified in a different way.

> It seems pretty risky to expect that no
> applications will replace sys.path. I understand that you're proposing
> that we say that applications shouldn't do that - but just saying so
> won't change the many applications already out there.

Of course I know that if we agree on a policy, it wont' change existing code
in one second. But if there is an official policy, you are able to
write bug reports like this "Please alter sys.path according to the docs. See http://www.python.org/...."

The next thing: If someone wants to add to sys.path, most of the
time the developer inserts its new entries in the front of the list.

This can break the ordering if you don't use a custom list class.




>> This instance get replace by a common list in lines like this:
>>
>> sys.path = glob.glob(os.path.join(WHEEL_DIR, "*.whl")) + sys.path
>>
>> The above line is from pip, it similar things happen in a lot of packages.
> 
> How does the fact that pip does that cause a problem? The sys.path
> modification is only in effect while pip is running, and no code in
> pip relies on sys.path being an instance of your custom class.

pip is a special case, since the pip authors say "we don't provide an API".
But they have handy methods which we want to use. We use "import pip"
and the class of sys.path of our application gets altered.


>> Before trying to solve this with code, I think the python community should
>> agree an a policy for altering sys.path.
> 
> I can't imagine that happening, and even if it does, it won't make any
> difference because a new policy won't change existing code. It won't
> even affect new code unless people know about it (which isn't certain
> - I doubt many people read the documentation that closely).

Code updates will happen step by step.
If someone has a problem, since his custom list class in sys.path gets
altered, he will write a bug report to the maintainer. A bug report
referencing official python docs has more weight. 

>> What can I do to this done?
> 
> I doubt you can.
> 
> A PR for pip that changes the above line to modify sys.path in place
> would probably get accepted (I can't see any reason why it wouldn't),
> and I guess you could do the same for any other code you find. But as
> for persuading the Python programming community not to replace
> sys.path in any code, that seems unlikely to happen.
> 
>> We use Python 2.7
> 
> If you were using 3.x, then it's (barely) conceivable that making
> sys.path read-only (so people could only modify it in-place) could be
> done as a new feature, but (a) it would be a major backward
> compatibility break, so there would have to be a strong justification,
> and (b) it would stop you from replacing sys.path with your custom
> class in the first place, so it wouldn't solve your issue.
> 
> Which also raises the question, why do you believe it's OK to forbid
> other people to replace sys.path when that's what you're doing in your
> sitecustomize code? That seems self-contradictory...

Yes, you are right this looks self-contradictory.
I am the one which is responsible for the set up of the environment.

Where is the best place during the interpreter initialization for
altering the class of sys.path? I guess it is sitecustomize. After
it was executed sys.path should be altered only in-place. 

Regards,
  Thomas Güttler


-- 
http://www.thomas-guettler.de/