setuptools reading simple index takes 30sec
Hello, I have a python dev team that I support in our local jenkins/artifactory environment. Recently this team switched over from an old local pypi legacy repository to a current pypi remote/virtual of pypi.org. So we could get them updated libraries and such. Anyway, I am banging my head to explain the time taken for reading the simple index. This index is about 11MB, takes 1 second to download itself, but then it takes ~30 seconds for setuptools to scan this and find the package it's looking for. Multiple this by the ~170 libraries they have in requirements, and the build has gone from 30minutes to almost 3 hours. And 90% comes from reading the simple index for each dependency. Here's a sample output with timestamps 00:57:00.036 Searching for tenacity==4.8.0 00:57:00.037 Reading https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/ 00:57:27.650 Reading https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/tenacity/ 00:57:27.881 Downloading https://repo1.local/artifactory/api/pypi/py-pypi-virt/packages/packages/fc/e... 00:57:28.037 Best match: tenacity 4.8.0 00:57:28.037 Processing tenacity-4.8.0-py2.py3-none-any.whl 00:57:28.038 Installing tenacity-4.8.0-py2.py3-none-any.whl to /home/build/workspace/workflows-pr/workflows-extension/.eggs 00:57:28.069 writing requirements to /home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg/EGG-INFO/requires.txt 00:57:28.114 Installed /home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg Note the time on the first Reading. I'm hoping there is some obvious simple solution to make this faster. Any ideas or help would be greatly appreciated? Thanks
What command are you using to result in the output? The root page lists all available packages on that index, and with 11MB of data there must be literally millions of entries to parse, and would definitely take a long time. But it definitely don’t need to be done under normal circumstances, and pip/setuptools definitely does not do it under normal circumstances. Sent from Mail for Windows 10 From: chris.cralle@rsa.com Sent: 02 August 2019 05:06 To: distutils-sig@python.org Subject: [Distutils] setuptools reading simple index takes 30sec Hello, I have a python dev team that I support in our local jenkins/artifactory environment. Recently this team switched over from an old local pypi legacy repository to a current pypi remote/virtual of pypi.org. So we could get them updated libraries and such. Anyway, I am banging my head to explain the time taken for reading the simple index. This index is about 11MB, takes 1 second to download itself, but then it takes ~30 seconds for setuptools to scan this and find the package it's looking for. Multiple this by the ~170 libraries they have in requirements, and the build has gone from 30minutes to almost 3 hours. And 90% comes from reading the simple index for each dependency. Here's a sample output with timestamps 00:57:00.036 Searching for tenacity==4.8.0 00:57:00.037 Reading https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/ 00:57:27.650 Reading https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/tenacity/ 00:57:27.881 Downloading https://repo1.local/artifactory/api/pypi/py-pypi-virt/packages/packages/fc/e... 00:57:28.037 Best match: tenacity 4.8.0 00:57:28.037 Processing tenacity-4.8.0-py2.py3-none-any.whl 00:57:28.038 Installing tenacity-4.8.0-py2.py3-none-any.whl to /home/build/workspace/workflows-pr/workflows-extension/.eggs 00:57:28.069 writing requirements to /home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg/EGG-INFO/requires.txt 00:57:28.114 Installed /home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg Note the time on the first Reading. I'm hoping there is some obvious simple solution to make this faster. Any ideas or help would be greatly appreciated? Thanks -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/O7M27...
Well I think I found the culprit. I knew it must be something simple. And it was. Our pydisutils.cfg had both a index-url and find_links, both with the same simple url. When I removed the find_links, all the time sinks went away. What did take 3hr is now back to 24min. /facepalm fwiw, this is just a python setup.py test, and uses tox virtualenv setuptools. I also thought that setuptools just used pip, but now I see it's actually easy-install, which I didn't realize was a different thing from pip. Good luck on https://github.com/pypa/setuptools/issues/917
participants (3)
-
Chris C -
chris.cralle＠rsa.com
-
Tzu-ping Chung