python package management confusion

[dcs3spp]

On Sunday, 20 January 2019 20:38:30 UTC, Oscar Benjamin  wrote:
> On Sun, 20 Jan 2019 at 16:22, dcs3spp via Python-list
> <python-list at python.org> wrote:
> >
> > On Saturday, 19 January 2019 11:17:19 UTC, dcs3spp  wrote:
> > >
> > > My question is, can setuptools be configured to pull in child from a separate git repository when running python setup.py develop from parent folder? I have since found and tried this approach at https://stackoverflow.com/a/53706140/8325270
> > > It appears that later versions of setuptools can install via a PEP508 url. I currently trying to investigate this approach…..
> >
> > After trying PEP508 url approach my conclusions are as follows.
> >
> > A PEP508 url for a git repository can be used in *install_requires* of *setup.py*. An example is listed below.
> > ```
> > requires = [
> >     'parent',
> >     'kinto-http at git+https://github.com/Kinto/kinto-http.py',
> > ]
> > ...
> > install_requires=requires
> > ```
> > The package can then be installed with pip, using ```pip install -e . or pip install .```
> >
> > However, installation with setuptools is then broken, i.e. ```python setup.py develop``` and ```python setup.py install``` fails. setuptools looks for packages in pypi indexes. To install using setuptools a devpi index would have to be installed and configured or packages would have to installed from a paid for pypi repository in the cloud. Alternatively, developers could manually install each private package dependency individually, prior to running ```python setup.py develop``` for the source package. Unless, there are other alternative(s) such as zc.buildout with mr developer plugin etc.....
> >
> > If I want to have a Python private project, referencing other private project(s), available under source control and CI via gitlab.com, it seems that I can use the pip approach with PEP508 or use a requirements.txt file containing the git projects referenced as PEP508 urls, i.e. ```pip install -r requirements.txt```.
> >
> > Confusion, stems from the fact that pip and setuptools dependencies are then not synchronised, i.e. setuptools will break if PEP508 urls are listed for install_requires. Presumably the approach is to use either pip or setuptools but not both?
> 
> I'm not sure what you mean by pip and setuptools not being
> synchronised. Pip depends on setuptools and cannot be used without it.
> Both setuptools and pip are maintained under the PyPA. They are
> intended to work together. in fact if your setup.py uses distutils
> instead of setuptools then pip will "inject" setuptools into it to
> ensure that meets pip's needs.
> 
> You will need to be more specific about which commands you are using
> and what it is that becomes unsynchronised.
> 
> --
> Oscar

Hi, 

Have since done further testing and figured out how I can install from a setup.py file using both pip and python setup.py develop. Confusion was caused by trying to install using both pip and setuptools with PEP508 urls in install_requires, see note at end of post.... 

>From what I understand setuptools offers *dependency_links* as a list of dependency urls. In the example below, the *pyramid_core* package is a private dependency that I have written. 

I am currently using pip 18.1. pip has an option *--process-dependencies* that issues a deprecation warning. The following *setup.py* example works with both setuptools (python setup develop etc.) and pip (pip install -e . and pip install .).

The example *setup.py* below can be installed using both setuptools and pip as follows:
```
python setup.py develop
python setup.py install
pip install -e . --process-dependency-links
pip install .
```

**setup.py that is compatible with both setuptools and pip 18.1**
=================================================================
```
import os

from setuptools import setup, find_packages

here = os.path.abspath(os.path.dirname(__file__))
with open(os.path.join(here, 'README.md')) as f:
    README = f.read()
with open(os.path.join(here, 'CHANGES.md')) as f:
    CHANGES = f.read()

dependencies = [
    'git+ssh://git@gitlab.com/dcs3spp/plantoeducate_core.git#egg=pyramid_core-0',
]

requires = [
    'parent',
    'pyramid_core',
]

setup_requires = [
]

tests_require = [
    'pytest',
    'pytest-cov',
]

setup(name='parent',
      version='0.1',
      description='parent',
      long_description=README + '\n\n' + CHANGES,
      classifiers=[
          "Programming Language :: Python",
      ],
      author='dcs3spp',
      author_email='myemail at outlook.com',
      url='',
      keywords='setuptools',
      packages=find_packages('src'),
      package_dir={'': 'src'},
      include_package_data=True,
      zip_safe=False,
      extras_require={
          'testing': tests_require,
      },
      install_requires=requires,
      dependency_links=dependencies,
      setup_requires=setup_requires,
      tests_require=tests_require,
)
```

Pip 18.1 supports reading pep508 direct urls from install_requires. In future release there are plans to deprecate the --process-dependency-links pip install option:
- https://github.com/pypa/pip/issues/4187
- https://github.com/pypa/pip/pull/4175

Will setuptools provide ability to use direct pep508 urls in install_requires in the future also?