Hi I'm having trouble using pypi_whitelist on devpi-server 2.1.3, and I'm not even sure that I'm using it correctly. I have a local devpi server at http://pypi, with several indexes on it. root/pypi is a mirror of https://pypi.python.org/ and root/prod is the production server for my company. root/prod inherits from root/pypi, as well storing our internally developed Python apps and libraries. root/prod also stores pre-compiled Wheels for Windows packages where they are not available on the main PyPI repository (usually sourced from here: http://www.lfd.uci.edu/~gohlke/pythonlibs/), as I can't rely on C compilers being available on target Windows machines. On Linux, I can expect C compilers to be present, so I'm happy to install from the .tar.gz file available on PyPI. We run a mixed Windows and Linux environment, and I'd like to be able to install from devpi using the same interface: pip install --index-url http://pypi/root/prod --use-wheel <package-name> I have a problem however, in situations where root/prod holds a Windows Wheel, but not corresponding Linux package, I can't install that package on Linux. For example: root/prod/+simple: root/prod mercurial-3.2.4-cp27-none-win_amd64.whl root/prod mercurial-3.2.4-cp27-none-win32.whl root/pypi/+simple: root/pypi mercurial-3.3.tar.gz root/pypi mercurial-3.2.4.tar.gz root/pypi mercurial-3.2.3.tar.gz On Windows, this works as I would like: PS C:\Users\blaffoy\> pip install --index-url http://pypi/root/prod --use-wheel mercurial Collecting mercurial Downloading http://pypi/root/prod/+f/4a5/c7c65b8e4c6f3/mercurial-3.2.4-cp27-none-win32.w... (1.1MB) 100% |################################| 1.1MB 13.7MB/s ta 0:00:01 Installing collected packages: mercurial Successfully installed mercurial-3.2.4 However, on Linux: blaffoy@ubuntu2:~$ sudo pip install --index-url http://pypi/root/prod --use-wheel mercurial Downloading/unpacking mercurial Could not find any downloads that satisfy the requirement mercurial Cleaning up... No distributions at all found for mercurial Storing debug log for failure in /home/blaffoy/.pip/pip.log With the following in pip.log: ------------------------------------------------------------ /usr/local/bin/pip run on Tue Feb 3 09:43:01 2015 Downloading/unpacking mercurial Getting page http://pypi/root/prod/mercurial/ URLs to search for versions for mercurial: * http://pypi/root/prod/mercurial/ http://pypi/root/prod/mercurial/ uses an insecure transport scheme (http). Consider using https if pypi has it available Analyzing links from page http://pypi/root/prod/+simple/Mercurial Skipping http://pypi/root/prod/+f/1c9/46e79eba3324c/mercurial-3.2.4-cp27-none-win_amd... (from http://pypi/root/prod/+simple/Mercurial) because it is not compatible with this Python Skipping http://pypi/root/prod/+f/4a5/c7c65b8e4c6f3/mercurial-3.2.4-cp27-none-win32.w... (from http://pypi/root/prod/+simple/Mercurial) because it is not compatible with this Python Could not find any downloads that satisfy the requirement mercurial Cleaning up... Removing temporary dir /tmp/pip_build_root... No distributions at all found for mercurial Exception information: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg/pip/commands/install.py", line 278, in run requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle) File "/usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1177, in prepare_files url = finder.find_requirement(req_to_install, upgrade=self.upgrade) File "/usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg/pip/index.py", line 277, in find_requirement raise DistributionNotFound('No distributions at all found for %s' % req) DistributionNotFound: No distributions at all found for mercurial So, pip looks for mercurial under root/prod, finds two packages but identifies that neither of them are for this OS. At that point, pip gives up, but what I would like to have happen is for the Linux compatible root/pypi packages to appear alongside the Windows packages on root/prod. If I've understood the documentation (and I probably haven't), this scenario is what the pypi_whitelist option exists to address. But, I see this issue even when I have `mercurial` in the pypi_whitelist of root/prod: PS C:\Users\blaffoy\> devpi index http://pypi/root/prod: type=stage bases=root/pypi volatile=False uploadtrigger_jenkins=None acl_upload=blaffoy pypi_whitelist=wheel,setuptools,pip,hgtools,tornado,apscheduler,mercurial Have I misunderstood what pypi_whitelist is for, and if so, is there any way to achieve the behaviour I'm looking for? Thanks for any help Barry
Off the top of my head, I can think of two workarounds: 1) Manually push the tar.gz files from root/pypi to root/prod 2) Pass --extra-index-url http://pypi/root/pypi as an argument to all of my pip install calls. On Tuesday, 3 February 2015 09:58:28 UTC, Barry Laffoy wrote:
Hi
I'm having trouble using pypi_whitelist on devpi-server 2.1.3, and I'm not even sure that I'm using it correctly.
I have a local devpi server at http://pypi, with several indexes on it. root/pypi is a mirror of https://pypi.python.org/ and root/prod is the production server for my company. root/prod inherits from root/pypi, as well storing our internally developed Python apps and libraries.
root/prod also stores pre-compiled Wheels for Windows packages where they are not available on the main PyPI repository (usually sourced from here: http://www.lfd.uci.edu/~gohlke/pythonlibs/), as I can't rely on C compilers being available on target Windows machines. On Linux, I can expect C compilers to be present, so I'm happy to install from the .tar.gz file available on PyPI.
We run a mixed Windows and Linux environment, and I'd like to be able to install from devpi using the same interface: pip install --index-url http://pypi/root/prod --use-wheel <package-name>
I have a problem however, in situations where root/prod holds a Windows Wheel, but not corresponding Linux package, I can't install that package on Linux.
For example:
root/prod/+simple: root/prod mercurial-3.2.4-cp27-none-win_amd64.whl root/prod mercurial-3.2.4-cp27-none-win32.whl
root/pypi/+simple: root/pypi mercurial-3.3.tar.gz root/pypi mercurial-3.2.4.tar.gz root/pypi mercurial-3.2.3.tar.gz
On Windows, this works as I would like:
PS C:\Users\blaffoy\> pip install --index-url http://pypi/root/prod --use-wheel mercurial Collecting mercurial Downloading http://pypi/root/prod/+f/4a5/c7c65b8e4c6f3/mercurial-3.2.4-cp27-none-win32.w... (1.1MB) 100% |################################| 1.1MB 13.7MB/s ta 0:00:01 Installing collected packages: mercurial
Successfully installed mercurial-3.2.4
However, on Linux:
blaffoy@ubuntu2:~$ sudo pip install --index-url http://pypi/root/prod --use-wheel mercurial Downloading/unpacking mercurial Could not find any downloads that satisfy the requirement mercurial Cleaning up... No distributions at all found for mercurial Storing debug log for failure in /home/blaffoy/.pip/pip.log
With the following in pip.log:
------------------------------------------------------------ /usr/local/bin/pip run on Tue Feb 3 09:43:01 2015 Downloading/unpacking mercurial Getting page http://pypi/root/prod/mercurial/ URLs to search for versions for mercurial: * http://pypi/root/prod/mercurial/ http://pypi/root/prod/mercurial/ uses an insecure transport scheme (http). Consider using https if pypi has it available Analyzing links from page http://pypi/root/prod/+simple/Mercurial Skipping http://pypi/root/prod/+f/1c9/46e79eba3324c/mercurial-3.2.4-cp27-none-win_amd... (from http://pypi/root/prod/+simple/Mercurial) because it is not compatible with this Python Skipping http://pypi/root/prod/+f/4a5/c7c65b8e4c6f3/mercurial-3.2.4-cp27-none-win32.w... (from http://pypi/root/prod/+simple/Mercurial) because it is not compatible with this Python Could not find any downloads that satisfy the requirement mercurial Cleaning up... Removing temporary dir /tmp/pip_build_root... No distributions at all found for mercurial Exception information: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg/pip/commands/install.py", line 278, in run requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle) File "/usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1177, in prepare_files url = finder.find_requirement(req_to_install, upgrade=self.upgrade) File "/usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg/pip/index.py", line 277, in find_requirement raise DistributionNotFound('No distributions at all found for %s' % req) DistributionNotFound: No distributions at all found for mercurial
So, pip looks for mercurial under root/prod, finds two packages but identifies that neither of them are for this OS. At that point, pip gives up, but what I would like to have happen is for the Linux compatible root/pypi packages to appear alongside the Windows packages on root/prod. If I've understood the documentation (and I probably haven't), this scenario is what the pypi_whitelist option exists to address. But, I see this issue even when I have `mercurial` in the pypi_whitelist of root/prod:
PS C:\Users\blaffoy\> devpi index http://pypi/root/prod: type=stage bases=root/pypi volatile=False uploadtrigger_jenkins=None acl_upload=blaffoy pypi_whitelist=wheel,setuptools,pip,hgtools,tornado,apscheduler,mercurial
Have I misunderstood what pypi_whitelist is for, and if so, is there any way to achieve the behaviour I'm looking for?
Thanks for any help Barry
Ugh, okay. I figured it out. The whitelist is case sensitive, and the mercurial pypi package is actual called "Mercurial", not "mercurial". So the following fixed my problem: devpi index pypi_whitelist=wheel, setuptools,pip,hgtools,tornado,apscheduler,Mercurial Then I can pip install on Windows and Linux as I wanted. On Tuesday, 3 February 2015 10:00:15 UTC, Barry Laffoy wrote:
Off the top of my head, I can think of two workarounds:
1) Manually push the tar.gz files from root/pypi to root/prod 2) Pass --extra-index-url http://pypi/root/pypi as an argument to all of my pip install calls.
On Tuesday, 3 February 2015 09:58:28 UTC, Barry Laffoy wrote:
Hi
I'm having trouble using pypi_whitelist on devpi-server 2.1.3, and I'm not even sure that I'm using it correctly.
I have a local devpi server at http://pypi, with several indexes on it. root/pypi is a mirror of https://pypi.python.org/ and root/prod is the production server for my company. root/prod inherits from root/pypi, as well storing our internally developed Python apps and libraries.
root/prod also stores pre-compiled Wheels for Windows packages where they are not available on the main PyPI repository (usually sourced from here: http://www.lfd.uci.edu/~gohlke/pythonlibs/), as I can't rely on C compilers being available on target Windows machines. On Linux, I can expect C compilers to be present, so I'm happy to install from the .tar.gz file available on PyPI.
We run a mixed Windows and Linux environment, and I'd like to be able to install from devpi using the same interface: pip install --index-url http://pypi/root/prod --use-wheel <package-name>
I have a problem however, in situations where root/prod holds a Windows Wheel, but not corresponding Linux package, I can't install that package on Linux.
For example:
root/prod/+simple: root/prod mercurial-3.2.4-cp27-none-win_amd64.whl root/prod mercurial-3.2.4-cp27-none-win32.whl
root/pypi/+simple: root/pypi mercurial-3.3.tar.gz root/pypi mercurial-3.2.4.tar.gz root/pypi mercurial-3.2.3.tar.gz
On Windows, this works as I would like:
PS C:\Users\blaffoy\> pip install --index-url http://pypi/root/prod --use-wheel mercurial Collecting mercurial Downloading http://pypi/root/prod/+f/4a5/c7c65b8e4c6f3/mercurial-3.2.4-cp27-none-win32.w... (1.1MB) 100% |################################| 1.1MB 13.7MB/s ta 0:00:01 Installing collected packages: mercurial
Successfully installed mercurial-3.2.4
However, on Linux:
blaffoy@ubuntu2:~$ sudo pip install --index-url http://pypi/root/prod --use-wheel mercurial Downloading/unpacking mercurial Could not find any downloads that satisfy the requirement mercurial Cleaning up... No distributions at all found for mercurial Storing debug log for failure in /home/blaffoy/.pip/pip.log
With the following in pip.log:
------------------------------------------------------------ /usr/local/bin/pip run on Tue Feb 3 09:43:01 2015 Downloading/unpacking mercurial Getting page http://pypi/root/prod/mercurial/ URLs to search for versions for mercurial: * http://pypi/root/prod/mercurial/ http://pypi/root/prod/mercurial/ uses an insecure transport scheme (http). Consider using https if pypi has it available Analyzing links from page http://pypi/root/prod/+simple/Mercurial Skipping http://pypi/root/prod/+f/1c9/46e79eba3324c/mercurial-3.2.4-cp27-none-win_amd... (from http://pypi/root/prod/+simple/Mercurial) because it is not compatible with this Python Skipping http://pypi/root/prod/+f/4a5/c7c65b8e4c6f3/mercurial-3.2.4-cp27-none-win32.w... (from http://pypi/root/prod/+simple/Mercurial) because it is not compatible with this Python Could not find any downloads that satisfy the requirement mercurial Cleaning up... Removing temporary dir /tmp/pip_build_root... No distributions at all found for mercurial Exception information: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg/pip/commands/install.py", line 278, in run requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle) File "/usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1177, in prepare_files url = finder.find_requirement(req_to_install, upgrade=self.upgrade) File "/usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg/pip/index.py", line 277, in find_requirement raise DistributionNotFound('No distributions at all found for %s' % req) DistributionNotFound: No distributions at all found for mercurial
So, pip looks for mercurial under root/prod, finds two packages but identifies that neither of them are for this OS. At that point, pip gives up, but what I would like to have happen is for the Linux compatible root/pypi packages to appear alongside the Windows packages on root/prod. If I've understood the documentation (and I probably haven't), this scenario is what the pypi_whitelist option exists to address. But, I see this issue even when I have `mercurial` in the pypi_whitelist of root/prod:
PS C:\Users\blaffoy\> devpi index http://pypi/root/prod: type=stage bases=root/pypi volatile=False uploadtrigger_jenkins=None acl_upload=blaffoy
pypi_whitelist=wheel,setuptools,pip,hgtools,tornado,apscheduler,mercurial
Have I misunderstood what pypi_whitelist is for, and if so, is there any way to achieve the behaviour I'm looking for?
Thanks for any help Barry
participants (1)
-
Barry Laffoy