[Distutils] Module import order issues.

[Brian Allen Vanderburg II]

I have devised the following hack which works for the time being.  I add
the following line to two new files called 000-fix-easyinstall.pth and
zzz-fix-easyinstall.pth:

import sys; sys.__egginsert = len(sys.path)

This causes the next path re-arrangement by an easy-install.pth to be
after all current paths.

Why two:

The .pth are sorted by site.py.  So 000-fix-easyinstall.pth will get
called first and set __egginsert after any current paths (such as the
core paths and the user site-packages path).  This way, when the user
easy-install.pth is processed if there is one, any path re-arrangements
will occur after them.

Now the problem with just one is, if there are any other user .pth files
that get processed after the user easy-install.pth, they will add paths
but not update the egginsert value.  So when the system easy-install.pth
gets invoked, those paths will be moved above the user paths that were
added after the user easy-install.pth.  The solution to this is the
second fix file zzz-fix-easyinstall.pth.  Since it is processed last, i
ensures that the egginsert is after any added paths.


To see it in action:

import sys
for i in sys.path: print(i)

Before:

/home/allen/.local/lib/python3.4/site-packages/setuptools
/usr/local/lib/python3.4/dist-packages/psutil-3.4.2-py3.4-linux-x86_64.egg
/usr/local/lib/python3.4/dist-packages/docker_py-1.7.0_rc2-py3.4.egg
/usr/local/lib/python3.4/dist-packages/raven-5.10.1-py3.4.egg
/usr/local/lib/python3.4/dist-packages/Jinja2-2.8-py3.4.egg
/usr/local/lib/python3.4/dist-packages/aiohttp-0.20.2-py3.4-linux-x86_64.egg
/usr/local/lib/python3.4/dist-packages/jsonschema-2.5.1-py3.4.egg
/usr/local/lib/python3.4/dist-packages/websocket_client-0.35.0-py3.4.egg
/usr/local/lib/python3.4/dist-packages/requests-2.9.1-py3.4.egg
/usr/local/lib/python3.4/dist-packages/MarkupSafe-0.23-py3.4-linux-x86_64.egg
/usr/lib/python3/dist-packages
/usr/local/lib/python3.4/dist-packages/paramiko-1.16.0-py3.4.egg
/usr/local/lib/python3.4/dist-packages/configobj-5.0.6-py3.4.egg
/usr/local/lib/python3.4/dist-packages/ecdsa-0.13-py3.4.egg
/usr/local/lib/python3.4/dist-packages/pycrypto-2.6.1-py3.4-linux-x86_64.egg
/usr/local/lib/python3.4/dist-packages/gns3_server-1.4.6-py3.4.egg
/usr/local/lib/python3.4/dist-packages/gns3_gui-1.4.6-py3.4.egg
/usr/local/lib/python3.4/dist-packages/gns3_net_converter-1.3.0-py3.4.egg
/usr/lib/python3.4
/usr/lib/python3.4/plat-x86_64-linux-gnu
/usr/lib/python3.4/lib-dynload
/home/allen/.local/lib/python3.4/site-packages
/usr/local/lib/python3.4/dist-packages

After:

/usr/lib/python3.4
/usr/lib/python3.4/plat-x86_64-linux-gnu
/usr/lib/python3.4/lib-dynload
/home/allen/.local/lib/python3.4/site-packages
/home/allen/.local/lib/python3.4/site-packages/setuptools
/usr/local/lib/python3.4/dist-packages/psutil-3.4.2-py3.4-linux-x86_64.egg
/usr/local/lib/python3.4/dist-packages/docker_py-1.7.0_rc2-py3.4.egg
/usr/local/lib/python3.4/dist-packages/raven-5.10.1-py3.4.egg
/usr/local/lib/python3.4/dist-packages/Jinja2-2.8-py3.4.egg
/usr/local/lib/python3.4/dist-packages/aiohttp-0.20.2-py3.4-linux-x86_64.egg
/usr/local/lib/python3.4/dist-packages/jsonschema-2.5.1-py3.4.egg
/usr/local/lib/python3.4/dist-packages/websocket_client-0.35.0-py3.4.egg
/usr/local/lib/python3.4/dist-packages/requests-2.9.1-py3.4.egg
/usr/local/lib/python3.4/dist-packages/MarkupSafe-0.23-py3.4-linux-x86_64.egg
/usr/lib/python3/dist-packages
/usr/local/lib/python3.4/dist-packages/paramiko-1.16.0-py3.4.egg
/usr/local/lib/python3.4/dist-packages/configobj-5.0.6-py3.4.egg
/usr/local/lib/python3.4/dist-packages/ecdsa-0.13-py3.4.egg
/usr/local/lib/python3.4/dist-packages/pycrypto-2.6.1-py3.4-linux-x86_64.egg
/usr/local/lib/python3.4/dist-packages/gns3_server-1.4.6-py3.4.egg
/usr/local/lib/python3.4/dist-packages/gns3_gui-1.4.6-py3.4.egg
/usr/local/lib/python3.4/dist-packages/gns3_net_converter-1.3.0-py3.4.egg
/usr/local/lib/python3.4/dist-packages

This keeps the core paths first, then the user paths, then the system paths



Brian Allen Vanderburg II

On 07/04/2016 04:59 AM, Piotr Dobrogost wrote:
> On Sun, Jul 3, 2016 at 7:05 PM, Brian Allen Vanderburg II via
> Distutils-SIG <distutils-sig at python.org> wrote:
>> The issue I'm having is when I install a test/local version of a package
>> in my user site-packages, but the same package is also installed in a
>> system path and has a .pth entry in the system easy-install.pth.  It
>> seems that from a path perspective, items under the user path should
>> automatically take precedence over the system path.  However, because of
>> the way the paths are rearranged in easy-install.pth, the system path
>> always takes precedence since the easy-install.pth line moves all added
>> items to the front of the path.
> You might want to take a look at issue "pip list reports wrong version
> of package installed both in system site and user site" at
> https://github.com/pypa/pip/issues/1018.
> One "solution" is not to install any packages in virtualenv with
> setuptools (which hacks easy-install.pth file as you described).
> Install only with pip instead.
>
> Regards,
> Piotr Dobrogost
>
>
>
>
>
>> So while it seems like the path should be something like:
>>
>> cwd:core-paths:user-site-paths:user-pth-paths:system-dist-paths:system-pth-paths
>>
>> The path turns out to be:
>>
>> cwd:user-pth-paths:system-pth-paths:core-paths:user-site-paths:system-dist-paths
>>
>> Is there a reason why the .pth files even contain the final line that
>> rearranges the paths for all the .eggs to be at the start of the system
>> path?
>>
>> For the time being, my work-around is to manually add a .pth file to my
>> user site-packages path for each item in site-packages as well, and add
>> the lines needed so that when the system easy-install.pth is processed,
>> it doesn't move all:
>>
>> import sys; sys.__plen = len(sys.path)
>> <my custom entries for each item in user site packages>
>> import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:];
>> p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert =
>> p+len(new)
>>
>> All of that is required, because without it, sys.__egginsert will still
>> be unset, and when site.py processes the system path, items in
>> easy-install.pth will be moved above any user .pth
>> items as well.
>>
>> However, ideally it seems like this path should automatically have
>> precedence over system packages, and it would if not for this line in
>> the various easy-install.pth files:
>>
>> import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:];
>> p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert =
>> p+len(new)
>>
>> I guess I just find myself wondering what is the purpose of the above
>> line in easy-install.pth files (for python3, but apparently not
>> python2), and why not just keep the path in the order that site.py would
>> add them.
>>
>> Thanks,
>>
>> Brian Allen Vanderburg II
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig