<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Feb 16, 2016, at 5:00 PM, Noah Kantrowitz <<a href="mailto:noah@coderanger.net" class="">noah@coderanger.net</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><br class=""><blockquote type="cite" class="">On Feb 16, 2016, at 4:46 PM, Glyph Lefkowitz <<a href="mailto:glyph@twistedmatrix.com" class="">glyph@twistedmatrix.com</a>> wrote:<br class=""><br class=""><blockquote type="cite" class=""><br class="">On Feb 16, 2016, at 4:33 PM, Noah Kantrowitz <<a href="mailto:noah@coderanger.net" class="">noah@coderanger.net</a>> wrote:<br class=""><br class=""><br class=""><blockquote type="cite" class="">On Feb 16, 2016, at 4:27 PM, Glyph Lefkowitz <<a href="mailto:glyph@twistedmatrix.com" class="">glyph@twistedmatrix.com</a>> wrote:<br class=""><br class=""><br class=""><blockquote type="cite" class="">On Feb 16, 2016, at 4:13 PM, Noah Kantrowitz <<a href="mailto:noah@coderanger.net" class="">noah@coderanger.net</a>> wrote:<br class=""><br class="">As someone that handles the tooling side, I don't care how it works as long as there is an override for tooling a la Chef/Puppet. For stuff like Supervisord, it is usually the least broken path to install the code globally.<br class=""></blockquote><br class="">I don't know if this is the right venue for this discussion, but I do think it would be super valuable to hash this out for good.<br class=""><br class="">Why does supervisord need to be installed in the global Python environment?<br class=""></blockquote><br class="">Where else would it go? I wouldn't want to assume virtualenv is installed unless absolutely needed.<br class=""></blockquote><br class="">This I can understand, but: in this case, it is needed ;).<br class=""><br class=""><blockquote type="cite" class="">Virtualenv is a project-centric view of the world which breaks down for stuff that is actually global like system command line tools.<br class=""></blockquote><br class="">[citation needed]. In what way does it "break down"? <a href="https://pypi.python.org/pypi/pipsi" class="">https://pypi.python.org/pypi/pipsi</a> is a nice proof-of-concept that dedicated virtualenvs are a better model for tooling than a big-ball-of-mud integrated system environment that may have multiple conflicting requirements. Unfortunately it doesn't directly address this use-case because it assumes that it is doing per-user installations and not a system-global one, but the same principle holds: what version of `ipaddress´ that supervisord wants to use is irrelevant to the tools that came with your operating system, and similarly irrelevant to your application.<br class=""><br class="">To be clear, what I'm proposing here is not "shove supervisord into a venv with the rest of your application", but rather, "each application should have its own venv". In supervisord's case, "python" is an implementation detail, and therefore the public interface is /usr/bin/supervisord and /usr/bin/supervisorctl, not 'import supervisord'; those should just be symlinks into /usr/lib/supervisord/environment/bin/<br class=""></blockquote><br class="">That isn't a thing that exists currently, I would have to make it myself and I wouldn't expect users to assume that is how I made it work. Given the various flavors of user expectations and standards that exist for deploying Python code, global does the least harm right now.<br class=""></div></div></blockquote><div><br class=""></div><div>I don't think users who <i class="">install</i> supervisord necessarily think they ought to be able to <i class="">import</i> supervisord. If they do expect that, they should probably revise their expectations.</div><div><br class=""></div><div>Here, I'll make it for you. Assuming virtualenv is installed:</div><div><br class=""></div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div><div>python -m virtualenv /usr/lib/supervisord/environment</div><div>/usr/lib/supervisord/environment/bin/pip install supervisord</div><div>ln -vs /usr/lib/supervisord/environment/bin/supervisor* /usr/bin</div></div></blockquote><div><br class=""></div><div>More tooling around this idiom would of course be nifty, but this is really all it takes.</div><div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><blockquote type="cite" class="">In fact, given that it is security-sensitive code that runs as root, it is extra important to isolate supervisord from your system environment for defense in depth, so that, for example, if, due to a bug, it can be coerced into importing an arbitrarily-named module, it has a restricted set and won't just load anything off the system.<br class=""></blockquote><br class="">Sounds cute but the threats that actually helps with seem really minor. If a user can install stuff as root, they can probably do whatever they want thanks to .pth files and other terrible things.<br class=""></div></div></blockquote><div><br class=""></div><div>Once <i class="">malicious</i> code is installed in a root-executable location it's game over; I didn't mean to imply otherwise. I'm saying that since supervisord might <i class="">potentially</i> import anything in its site-packages dir, this is just less code for you to worry about that might have security <i class="">bugs</i> in it.</div><div><br class=""></div><div>One specific example of how you might do this is by specifying a protocol-defined codec; if you ever do .decode(user_data) on a string you're doing an attacker-controlled dynamic import. This is a bug, of course, but a harmless one if you have a small set of modules with no surprises lurking in store. But, if the attacker can 'import qt' (whose default behavior was to abort() if it couldn't open $DISPLAY for many years, not sure if it still is) from the system, or anything like that, you have potential crashes on your hands.</div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Compare with `npm install -g grunt-cli`.<br class=""></blockquote><br class="">npm is different because npm doesn't create top-level script binaries unless you pass the -g option, so you need to install global tooling stuff with -g. virtualenv is different (and, at least in this case, better).<br class=""></blockquote><br class="">Pip also doesn't generate binstubs in /usr/bin unless you install globally so pretty much same difference.</div></div></blockquote><br class=""></div><div>Pip always generates binstubs into whatever prefix you're installing into, whereas npm sometimes doesn't generate binstubs at all; when it does generate them, it puts them in a package-specific directory and not in a common location. (I don't fully understand the specifics; npm generates local binstubs for coffeescript but not for grunt, for example.) It's fine to symlink pip's stubs. Is making the symlink really the sticking point?</div><div><br class=""></div><div>So far, in the "use virtualenv" column, I've got:</div><div><br class=""></div><div><ol class="MailOutline"><li class="">don't break tooling written in python in the host operating system by installing a conflicting dependency by accident</li><li class="">don't break the host operating system's package database by potentially overwriting packages installed by the package manager</li><li class="">don't break other pip-installed tools using the system or --user environments</li><li class="">don't let installing or upgrading any of <i class="">those</i> things accidentally break the tool (supervisord in this case) later</li><li class="">make provisioning possible by an unprivileged user, reducing the amount of code that needs to run as root (you can make /usr/lib/supervisord/environment writable by a dedicated user during the install process, to ensure that it doesn't provision anything outside of that directory). This is potentially useful because some setup.py scripts "helpfully" end up doing other weird stuff besides installing the package - there are a few who will remain nameless to protect the guilty which write a bunch of files into root's home directory, for example.</li><li class="">reduce the potential attack surface of any application with plugins by reducing the number of things that can get imported.</li></ol><div class=""><br class=""></div><div class="">and on the "use sudo pip install" side, I've got:</div><div class=""><br class=""></div><div class=""><ol class="MailOutline"><li class="">don't have to make a symlink</li><li class="">users expect applications to install importable modules</li></ol></div></div><div><br class=""></div><div>What am I missing?</div><div><br class=""></div><div>-glyph</div><div><br class=""></div></body></html>