question re: local development workflow / egg-info / synced directories
Hi, I have a local development workflow question I'm looking for feedback or suggestions on. It relates to installing things in editable mode in conjunction with syncing source files to a VM. I'm developing a Python project that is deployed / runs inside a Docker container. I do my local development on a Mac, and I use the following script to "sync" my local source files to the Docker container: https://github.com/brikis98/docker-osx-dev At Docker build time, the project source files are installed in "editable" mode, which creates egg-info directories inside the source directories in the Docker image. When developing locally, however, the sync process mounts the synced directory "over" the directory that was initialized at Docker-build time. Thus the egg-info directories are missing from the synced version of the directory. Does anyone have any suggestions for handling this scenario? The three possibilities I can think of are-- 1. re-install the projects in the Docker container in editable mode after mounting the synced directory (which is a bit hacky because there is no obvious "hook" with the tools I'm using), 2. install the projects in editable mode locally on my Mac so the egg-info directories will also sync over (but I'm not sure if egg-info directories are sufficiently cross-platform), or 3. maybe there is a way to install things in "editable" mode that doesn't require egg-info directories to be written alongside the source files? Does anyone have any suggestions? And does anyone know if (3) is possible? Thanks a lot for any help, --Chris
On Thu, Mar 31, 2016 at 3:03 AM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
When developing locally, however, the sync process mounts the synced directory "over" the directory that was initialized at Docker-build time. Thus the egg-info directories are missing from the synced version of the directory.
Are you mounting the site-packages or something else?
2. install the projects in editable mode locally on my Mac so the egg-info directories will also sync over (but I'm not sure if egg-info directories are sufficiently cross-platform), or
Assuming you mount the site-packages this should be fine as long as the paths are the same, this is because easy-install.pth will contain absolute paths. When you do `setup.py develop` an entry is added in easy-install.pth and that egg-info dir is created. Note that the egg-info metadata is mostly used for entrypoint discovery (eg: you use console_scripts <https://pythonhosted.org/setuptools/setuptools.html#automatic-script-creation> or plugin system <https://pytest.org/latest/writing_plugins.html#making-your-plugin-installable-by-others> based on entrypoints). Also, develop installs allow overriding the develop command, so they might do something extra (like pytest-cov "installs" a pth file for the subprocess support). That means you can just symlink or use a pth <https://docs.python.org/2/library/site.html?highlight=pth> file if you don't care about any of those features. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Wed, Mar 30, 2016 at 10:56 PM, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
On Thu, Mar 31, 2016 at 3:03 AM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
When developing locally, however, the sync process mounts the synced directory "over" the directory that was initialized at Docker-build time. Thus the egg-info directories are missing from the synced version of the directory.
Are you mounting the site-packages or something else?
No, I'm only mounting the project / source files. The Docker build step creates a virtualenv for use inside the image, which is in a location different from the synced directory. I'm not syncing site-packages as I'm not installing the requirements on my Mac (and in particular the "downloadable" requirements). I'm only installing them inside the Docker image at Docker build time (and it's not clear to me whether packages pip-installed on my Mac would even be compatible with a different OS on the Docker image). It would be nice if pip-installing a project directory in editable mode didn't require writing to the project directory. Is there a fundamental reason it needs to? Thanks, --Chris
2. install the projects in editable mode locally on my Mac so the egg-info directories will also sync over (but I'm not sure if egg-info directories are sufficiently cross-platform), or
Assuming you mount the site-packages this should be fine as long as the paths are the same, this is because easy-install.pth will contain absolute paths. When you do `setup.py develop` an entry is added in easy-install.pth and that egg-info dir is created.
Note that the egg-info metadata is mostly used for entrypoint discovery (eg: you use console_scripts or plugin system based on entrypoints).
Also, develop installs allow overriding the develop command, so they might do something extra (like pytest-cov "installs" a pth file for the subprocess support).
That means you can just symlink or use a pth file if you don't care about any of those features.
Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On Thu, Mar 31, 2016 at 9:22 AM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
It would be nice if pip-installing a project directory in editable mode didn't require writing to the project directory. Is there a fundamental reason it needs to?
I'd speculate that no one was annoyed enough to fix it. You could manually create the egg-info (setup.py egg_info --egg-base=somewhere) and do the linking manually. I'm preeeety sure my suggestion will raise some eyebrows here :) Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On 31 March 2016 at 10:00, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
On Thu, Mar 31, 2016 at 9:22 AM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
It would be nice if pip-installing a project directory in editable mode didn't require writing to the project directory. Is there a fundamental reason it needs to?
I'd speculate that no one was annoyed enough to fix it.
Editable mode is basically managed by setuptools. So any fix would need to be in setuptools, not pip. It's quite possible that the reason editable mode doesn't get much love is that from a user's perspective it seems like a pip issue, but the pip devs see it as a setuptools problem they can't do much about. Not many users have the time or inclination to follow up on a bug report with the downstream project, so things tend to stay as they are. Paul
On Thu, Mar 31, 2016 at 2:12 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 31 March 2016 at 10:00, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
On Thu, Mar 31, 2016 at 9:22 AM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
It would be nice if pip-installing a project directory in editable mode didn't require writing to the project directory. Is there a fundamental reason it needs to?
I'd speculate that no one was annoyed enough to fix it.
Editable mode is basically managed by setuptools. So any fix would need to be in setuptools, not pip.
After your two comments, I searched setuptools's tracker and found the following issue ("develop does not support placing the egg outside of the source folder"): https://github.com/pypa/setuptools/issues/361 So it looks like the feature may already be present in setuptools, but just not working correctly (i.e. the feature has a bug). If this feature were working in setuptools, would pip need any changes to "install -e" to allow passing along this extra info to setuptools, or is there already a mechanism to allow pass-through of extra setuptools options? Thanks, --Chris
It's quite possible that the reason editable mode doesn't get much love is that from a user's perspective it seems like a pip issue, but the pip devs see it as a setuptools problem they can't do much about. Not many users have the time or inclination to follow up on a bug report with the downstream project, so things tend to stay as they are.
Paul
On Thu, Mar 31, 2016 at 1:20 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
If this feature were working in setuptools, would pip need any changes to "install -e" to allow passing along this extra info to setuptools, or is there already a mechanism to allow pass-through of extra setuptools options?
Pip install has the --install-option and --global-option arguments, try your luck with them - they are for fairly advanced usecases. Pip already passes other options so there's risk of breakages or weird issues if you pass conflicting options. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
On 31 March 2016 at 11:46, Ionel Cristian Mărieș <contact@ionelmc.ro> wrote:
On Thu, Mar 31, 2016 at 1:20 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
If this feature were working in setuptools, would pip need any changes to "install -e" to allow passing along this extra info to setuptools, or is there already a mechanism to allow pass-through of extra setuptools options?
Pip install has the --install-option and --global-option arguments, try your luck with them - they are for fairly advanced usecases. Pip already passes other options so there's risk of breakages or weird issues if you pass conflicting options.
Setuptools also supports the standard setup.cfg mechanism. That may be an easier alternative. Paul
participants (3)
-
Chris Jerdonek -
Ionel Cristian Mărieș -
Paul Moore