[Python-ideas] Policy for altering sys.path

[Andrew Barnert]

On May 6, 2015, at 23:00, Thomas Güttler <guettliml at thomas-guettler.de> wrote:
> 
>> Am 06.05.2015 um 17:07 schrieb Paul Moore:
>>> On 6 May 2015 at 15:05, Thomas Güttler <guettliml at thomas-guettler.de> wrote:
>>> I am missing a policy how sys.path should be altered.
>> 
>> Well, the docs say that applications can modify sys.path as needed.
>> Generally, applications modify sys.path in place via sys.path[:] =
>> whatever, but that's not mandated as far as I know.
>> 
>>> We run a custom sub class of list in sys.path. We set it in sitecustomize.py
>> 
>> Can you explain why?
> 
> I forgot to explain the why I use a custom class. Sorry, here is the background.
> 
> I want sys.path to ordered:
> 
> 1. virtualenv
> 2. /usr/local/
> 3. /usr/lib

Can you instead just leave sys.path alone, and replace the module finder with a subclass that orders the directories in sys.path the way it wants to?

That's something a lot fewer packages are likely to screw with.

> We use virtualenvs with system site-packages.
> 
> There are many places where sys.path gets altered.
> 
> The last time we had sys.path problems I tried to write a test
> which checks that sys.path is the same for cron jobs and web requests.
> I failed. Too many places,  I could not find all the places
> and the conditions where sys.path got modified in a different way.
> 
>> It seems pretty risky to expect that no
>> applications will replace sys.path. I understand that you're proposing
>> that we say that applications shouldn't do that - but just saying so
>> won't change the many applications already out there.
> 
> Of course I know that if we agree on a policy, it wont' change existing code
> in one second. But if there is an official policy, you are able to
> write bug reports like this "Please alter sys.path according to the docs. See http://www.python.org/...."
> 
> The next thing: If someone wants to add to sys.path, most of the
> time the developer inserts its new entries in the front of the list.
> 
> This can break the ordering if you don't use a custom list class.
> 
> 
> 
> 
>>> This instance get replace by a common list in lines like this:
>>> 
>>> sys.path = glob.glob(os.path.join(WHEEL_DIR, "*.whl")) + sys.path
>>> 
>>> The above line is from pip, it similar things happen in a lot of packages.
>> 
>> How does the fact that pip does that cause a problem? The sys.path
>> modification is only in effect while pip is running, and no code in
>> pip relies on sys.path being an instance of your custom class.
> 
> pip is a special case, since the pip authors say "we don't provide an API".
> But they have handy methods which we want to use. We use "import pip"
> and the class of sys.path of our application gets altered.
> 
> 
>>> Before trying to solve this with code, I think the python community should
>>> agree an a policy for altering sys.path.
>> 
>> I can't imagine that happening, and even if it does, it won't make any
>> difference because a new policy won't change existing code. It won't
>> even affect new code unless people know about it (which isn't certain
>> - I doubt many people read the documentation that closely).
> 
> Code updates will happen step by step.
> If someone has a problem, since his custom list class in sys.path gets
> altered, he will write a bug report to the maintainer. A bug report
> referencing official python docs has more weight. 
> 
>>> What can I do to this done?
>> 
>> I doubt you can.
>> 
>> A PR for pip that changes the above line to modify sys.path in place
>> would probably get accepted (I can't see any reason why it wouldn't),
>> and I guess you could do the same for any other code you find. But as
>> for persuading the Python programming community not to replace
>> sys.path in any code, that seems unlikely to happen.
>> 
>>> We use Python 2.7
>> 
>> If you were using 3.x, then it's (barely) conceivable that making
>> sys.path read-only (so people could only modify it in-place) could be
>> done as a new feature, but (a) it would be a major backward
>> compatibility break, so there would have to be a strong justification,
>> and (b) it would stop you from replacing sys.path with your custom
>> class in the first place, so it wouldn't solve your issue.
>> 
>> Which also raises the question, why do you believe it's OK to forbid
>> other people to replace sys.path when that's what you're doing in your
>> sitecustomize code? That seems self-contradictory...
> 
> Yes, you are right this looks self-contradictory.
> I am the one which is responsible for the set up of the environment.
> 
> Where is the best place during the interpreter initialization for
> altering the class of sys.path? I guess it is sitecustomize. After
> it was executed sys.path should be altered only in-place. 
> 
> Regards,
>  Thomas Güttler
> 
> 
> -- 
> http://www.thomas-guettler.de/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/