[Twisted-Python] Python3 twistd daemon for Ubuntu 14.04 alternatives
Hi, so for my first post to the list I am going to ask a somewhat lame question. Apologies if this is not the right list -- clues welcome. I am doing a deploy of a server written in Python3 on Ubuntu 14.04. (14.04 for various reasons that are uninteresting to the list but DevOps has good/historical/hysterical reasons...) So my problem is that while my protocol stack spins up very nicely and trivially easy using the twistd daemon, the standard python3-twisted package for 14.04 is waaaay behind and doesn't include twistd (!), and I haven't found a good backport. I'd also like to avoid doing things that make life hard for DevOps, such as pip3 installing and risking a package manager fight. So some options: - Is there a reliable backport of the python3-twisted package for 14.04? Google is failing me on that one. - All I need is the most basic twistd functionality. Perhaps I should spin up my own by snarfing the code out of the twisted source repo. I can package this with my stuff temporarily and get on with life. Clues to the relevant parts of the repo welcome, I haven't poked around much in Twisted sources so need a road map. - Perhaps I should create my own backport for Ubuntu 14.04 of the current python3-twisted .deb. (This is probably not the right list to ask, but I'm happy to dereference a pointer.) This is a short-term problem for me, we are transitioning to 16.04 soon, but the process is driven by other parts of the software stack. So I don't want to over-invest. I'm looking for a reasonable band-aid. Thanks! -dave
On Feb 21, 2017, at 9:28 AM, Dave Curtis <dcurtis@savioke.com> wrote:
Hi, so for my first post to the list I am going to ask a somewhat lame question. Apologies if this is not the right list -- clues welcome.
This is totally the right list. All discussion of Twisted is welcome!
So my problem is that while my protocol stack spins up very nicely and trivially easy using the twistd daemon, the standard python3-twisted package for 14.04 is waaaay behind and doesn't include twistd (!), and I haven't found a good backport. I'd also like to avoid doing things that make life hard for DevOps, such as pip3 installing and risking a package manager fight.
The general solution to this problem is "don't use the system Python environment to deploy your applications". This is what virtualenv is for.
So some options: - Is there a reliable backport of the python3-twisted package for 14.04? Google is failing me on that one. [...] - Perhaps I should create my own backport for Ubuntu 14.04 of the current python3-twisted .deb. (This is probably not the right list to ask, but I'm happy to dereference a pointer.)
Both of these options are bad. Installing or creating a backport will break any system tools that depend on the python3-twisted package. This is better than `sudo pip3 install`, but only a little.
- All I need is the most basic twistd functionality. Perhaps I should spin up my own by snarfing the code out of the twisted source repo. I can package this with my stuff temporarily and get on with life. Clues to the relevant parts of the repo welcome, I haven't poked around much in Twisted sources so need a road map.
If you have the ability to ship parts of the Twisted stack, there's a tool that can automate the "snarfing": pip :-).
This is a short-term problem for me, we are transitioning to 16.04 soon, but the process is driven by other parts of the software stack. So I don't want to over-invest. I'm looking for a reasonable band-aid.
16.04 will also be out of date. I'm tempted to launch into a diatribe about namespacing, containers, and application isolation generally, but before I do - why is it that you want to use the system Python environment? Had you just not considered the option of virtual environments? -glyph
i could be totally wrong, and please correct me if i am, but try this: target: this will give you a current twisted / python installation and keep the packages seperate from the system ones # 1. install a package called virtualenv and pip on your system apt install virtualenv pip # 2. create a directory for your project, if you dont have one already, # to keep it seperate from other code mkdir yourproject # 3. create a virtualenv for your project, -p sets the python version # to use and "venv" is just the directory virtualenv installs the # python environment to virtualenv -p /usr/bin/python3 venv # 4. activate that environment (your command prompt will # change to: (venv)... source venv/bin/activate # 5. install twisted (while still being in the same terminal window # and directory with your activated virtual environment pip install twisted now you can code as usuall and run your code with python mycode.py, try python --version # 6. if you want to use trial for testing run: hash -r #to update the path settings, because trial is installed by the system twisted package as well and without the hash -r your env will still find the system trial first
On 22/02/17 11:12, Glyph Lefkowitz wrote:
I'm tempted to launch into a diatribe about namespacing, containers, and application isolation generally, but before I do - why is it that you /want/ to use the system Python environment? Had you just not considered the option of virtual environments?
Awesome though it is, virtualenv can be very tedious if you need to install hundreds of megabytes of compiler and -devel packages. System packages are attractive precisely because you can avoid this. I've had to do all sorts of tedious things with containers where I spin up a temporary container to build a bunch of .whl files, then actually install them in the final container - all to avoid bloating the container image with the build tools. It's a real shame that binary wheels on Linux/PyPI aren't a thing. I understand the reasons why, but TBH it substantially reduces the value of the Python ecosystem for me. Go is looking like a more and more attractive competitor to Python, frankly, having sidestepped this problem entirely.
I'm tempted to launch into a diatribe about namespacing, containers, and application isolation generally, but before I do - why is it that you /want/ to use the system Python environment? Had you just not considered the option of virtual environments? Awesome though it is, virtualenv can be very tedious if you need to install hundreds of megabytes of compiler and -devel packages. System packages are attractive precisely because you can avoid this.
That’s why you should use a build server and not ship build environments.
I've had to do all sorts of tedious things with containers where I spin up a temporary container to build a bunch of .whl files, then actually install them in the final container - all to avoid bloating the container image with the build tools.
I don’t see how that’s tedious since a compute does that for me. Although I don’t see any value at wheeling them (and some packages cannot be wheeled); my CI builds a venv and puts it into a container. There’s nothing tedious about it at all.
It's a real shame that binary wheels on Linux/PyPI aren't a thing.
This is incorrect.
On 22/02/17 12:49, Hynek Schlawack wrote:
That’s why you should use a build server and not ship build environments.
Which we do.
I don’t see how that’s tedious since a compute does that for me. Although I don’t see any value at wheeling them (and some packages cannot be wheeled); my CI builds a venv and puts it into a container. There’s nothing tedious about it at all.
I find the idea of running throwaway environments to generate a big blob of tarball'd python+libs, then copying said tarball to actual containers, a rather retrograde step by comparison with established package/build infrastructure tools.
It's a real shame that binary wheels on Linux/PyPI aren't a thing.
This is incorrect
Apparently so. The info JP provided was very useful.
I don’t see how that’s tedious since a compute does that for me. Although I don’t see any value at wheeling them (and some packages cannot be wheeled); my CI builds a venv and puts it into a container. There’s nothing tedious about it at all. I find the idea of running throwaway environments to generate a big blob of tarball'd python+libs, then copying said tarball to actual containers, a rather retrograde step by comparison with established package/build infrastructure tools.
I have to disagree here: I don’t want build tools of any kind in my final containers therefore I build my artifacts separately no matter what language. Of course you can just build the venv on your build server without wheeling up a temporary container and then package it using Docker or DEB or whatever. You should be separating building and running anyway so Python – as much as I’d like Go-style single binaries too – is in no way special here. The nice thing about temporary containers though is that I can do all of that on my Mac. —h
On 22/02/17 17:42, Hynek Schlawack wrote:
I have to disagree here: I don’t want build tools of any kind in my final containers therefore I build my artifacts separately no matter what language. Of course you can just build the venv on your build
Agreed, 100%. Apologies if I gave you the impression I was advocating otherwise.
server without wheeling up a temporary container and then package it using Docker or DEB or whatever. You should be separating building and running anyway so Python – as much as I’d like Go-style single binaries too – is in no way special here. The nice thing about temporary containers though is that I can do all of that on my Mac.
I agree that you need to separate building and installation, and I've got no particular beef with using a container, chroot, throwaway VM or whatever works for people in doing the build phase. (What people do with the resultant build output - and in particular whether there is a lot of ignoring of the hard-learned lessons of system package managers going on now - I will not comment on ;o) What I was trying to say - badly, apparently - was that the system python *could* be attractive to someone because many dependencies may exist in the OS package list in suitable form, but conversely may not exist in PyPI in binary form for Linux. As a very simple example: if you have a traditional (non-container) Linux system hosting a Python application in a virtualenv, and you deploy a Python app to a virtualenv e.g. using Puppet or Ansible, you either need to: 1. Use no C extensions 2. Hope there's a manylinux1 binary wheel 3. Use the OS package and --system-site-packages 4. Compile the C extensions and make them available to pip #2 seems useful now that I know about it but - correct me if I'm wrong - the manylinux1 permitted C dependencies are super-tiny, and would not permit e.g. cryptography or psycopg2? #4 is what you are advocating for I believe? But can we agree that for smaller projects, that might seem like a lot of repeated work if the package is already available in the OS repos? Wondering out loud, I guess it would be possible for OS-compiled python extensions to be somehow virtualenv or relocation-compatible. One could envisage something like: virtualenv t . t/bin/activate pip syspkg-install python-psycopg2 ...and this going off and grabbing the OS-provided dependency of that name, extracting it, and deploying it into the virtualenv, rather than the system Python. There are doubtless all sorts of reasons that is not practical. Anyway, to be clear - I'm not advocating using the system Python. I'm trying to explain why, based on the efforts we expend locally, it could seem attractive to smaller sites. Cheers, Phil
On 22 Feb 2017, at 18:16, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
#2 seems useful now that I know about it but - correct me if I'm wrong - the manylinux1 permitted C dependencies are super-tiny, and would not permit e.g. cryptography or psycopg2?
Depends. The permitted C dependencies *for dynamic linking* are tiny, sure. But those libraries may statically link as they please. The likely scenario for most such packages is that they statically link whatever they need. This is the approach taken by some that I ship. Cory
On 22/02/17 20:59, Cory Benfield wrote:
The permitted C dependencies *for dynamic linking* are tiny, sure. But those libraries may statically link as they please. The likely scenario for most such packages is that they statically link whatever they need. This is the approach taken by some that I ship.
Excellent point, I hadn't considered that. Tangentially I see there's also no-manylinux and associated github/pip issues w.r.t. people who *do* want to build local binary wheels on a local PyPI server... All in all, not a simple area.
On Feb 22, 2017, at 10:16 AM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
On 22/02/17 17:42, Hynek Schlawack wrote:
I have to disagree here: I don’t want build tools of any kind in my final containers therefore I build my artifacts separately no matter what language. Of course you can just build the venv on your build
Agreed, 100%. Apologies if I gave you the impression I was advocating otherwise.
server without wheeling up a temporary container and then package it using Docker or DEB or whatever. You should be separating building and running anyway so Python – as much as I’d like Go-style single binaries too – is in no way special here. The nice thing about temporary containers though is that I can do all of that on my Mac.
I agree that you need to separate building and installation, and I've got no particular beef with using a container, chroot, throwaway VM or whatever works for people in doing the build phase.
(What people do with the resultant build output - and in particular whether there is a lot of ignoring of the hard-learned lessons of system package managers going on now - I will not comment on ;o)
What I was trying to say - badly, apparently - was that the system python *could* be attractive to someone because many dependencies may exist in the OS package list in suitable form, but conversely may not exist in PyPI in binary form for Linux.
Yes, and building these binary artifacts is often harder than some people (cough, alpine, cough) seem to think. But there are better ways to square this circle than restricting yourself to the versions of python libraries that happen to be available in your distro.
As a very simple example: if you have a traditional (non-container) Linux system hosting a Python application in a virtualenv, and you deploy a Python app to a virtualenv e.g. using Puppet or Ansible, you either need to:
1. Use no C extensions 2. Hope there's a manylinux1 binary wheel 3. Use the OS package and --system-site-packages 4. Compile the C extensions and make them available to pip
#2 seems useful now that I know about it but - correct me if I'm wrong - the manylinux1 permitted C dependencies are super-tiny, and would not permit e.g. cryptography or psycopg2?
Cory already pointed this out tangentially, but I should emphasize: 'cryptography' and 'psycopg2' are things that you depend on at the Python level. The things you depend on at the C level are libssl, libcrypto, and libpq. If you want to build a manylinux wheel, you need to take this into account and statically link those C dependencies, which some projects are beginning to do. (Cryptography _could_ do this today, they already have the infrastructure for doing it on macOS and Windows, the reason they're not shipping manylinux1 wheels right now has to do with the political implications of auto-shipping a second copy of openssl to Linux distros that expect to manage security upgrades centrally).
#4 is what you are advocating for I believe? But can we agree that for smaller projects, that might seem like a lot of repeated work if the package is already available in the OS
If you're going to do #4 with dh_virtualenv, your .deb can depend on the relevant packages that contain the relevant C libraries, and build linux wheels, which are vendor-specific and can dynamically link to whatever you want (i.e. not manylinux wheels, which are vendor-neutral and must statically link everything). Manylinux wheels are required for uploading to PyPI, where you don't know who may be downloading - on your own infrastructure, where you are shipping inside an artifact (like a .deb) that specifically has metadata describing its dependencies, "linux" wheels are fine. Alone, hanging around on PyPI as .whl files rather than as .debs in your infrastructure, they'd be mystery meat, but that is not the case if they have proper dependency metadata. It might seem weird to use Python-specific tooling and per-application vendoring for Python dependencies, and yet use distro-global dynamic linking for C dependencies. But, this is actually a perfectly cromulent strategy, and I think this bears a more in-depth explanation. C, and particularly the ecosystem of weird dynamic linker ceremony around C, has an extremely robust side-by-side installation ecosystem, which distros leverage to great effect. For example, on the Ubuntu machine sitting next to me as I write this, I have libasan0 (4.8.5) libasan1 (4.9.4) libasan2 (5.4.1) *and* libasan3 (6.2.0) installed, and this isn't even a computer with a particularly significant amount of stuff going on! Nothing ever breaks and loads the wrong libasan.N. Python, by contrast, tried to do this in a C-ish way, but that attempt resulted in this mess: https://packaging.python.org/multi_version_install/ <https://packaging.python.org/multi_version_install/>, which almost nobody uses. Right at the top of that document, "For many use cases, virtual environments address this need without the complication ...". Even if you are 100%, completely bought into a distro-style way of life, no containers at all, everything has to be in a system package to get installed, virtualenvs still make more sense than trying to sync up the whole system's Python library versions. The reason nobody ever went back and tried to do multi-version installs "right" with Python is that the Python and C library ecosystems are fundamentally different in a bunch of important ways. For one thing, Python libraries have no such thing as an enforceable ABI, so coupling between libraries and applications is much closer than in C. For another, no SOVERSION. Also, many small C utilities (the ones that would be some of the smaller entries in requirements.txt in a Python app) are vendored in or statically linked in applications, so the "dependency management" happens prior to the container build, in the source repo of the upstream, where it is hidden. Python dependencies often have a far higher rate of churn than C dependencies because of the ease of development, which means both more divergence between required versions for different applications, and more benefits to being up-to-date for the applications that do rev faster. Finally, the build process for Python packages is much simpler, since they're usually treated as archives of files that move around, rather than elaborate pre-build steps that are often required for C libraries to make sure everything is smashed into the .so at build time. So think of your Python libraries as "vendored in" to your package for these reasons, rather than depended upon in the OS, and then participate in the broader distro (i.e. "C") ecosystem by building wheels that dynamically link whatever distro-level dependencies they need to.
Wondering out loud, I guess it would be possible for OS-compiled python extensions to be somehow virtualenv or relocation-compatible. One could envisage something like:
virtualenv t . t/bin/activate pip syspkg-install python-psycopg2
...and this going off and grabbing the OS-provided dependency of that name, extracting it, and deploying it into the virtualenv, rather than the system Python.
This is sort of what dh_virtualenv is. It doesn't set up the mapping for you automatically, but you can pretty quickly figure out that python-psycopg2 build-depends: libpq-dev.
There are doubtless all sorts of reasons that is not practical.
The main thing is just that you have to decide on a standard format for distro-specific metadata, and then go encode it everywhere. I'm pretty sure that distutils-sig would be open to codifying such an extension to the list of standardized metadata fields, so that tools can use it.
Anyway, to be clear - I'm not advocating using the system Python. I'm trying to explain why, based on the efforts we expend locally, it could seem attractive to smaller sites.
To be even clearer, using the system python is fine - it's using the global python environment that has the most significant problem. (Although, of course, the "system python" is probably CPython, and in most cases you want to be using PyPy, right? So yeah don't use the system Python.) I hope this explanation was helpful to those of you deploying with distro tooling! -glyph
On 23/02/17 09:11, Glyph Lefkowitz wrote:
Yes, and building these binary artifacts is often harder than some people (cough, alpine, cough) seem to think. But there are better ways to square this circle than restricting yourself to the versions of /python/ libraries that happen to be available in your distro.
I don't disagree. I'm not defending the practice. But it is *clearly* a thing people do, and understanding the reasons is, IMO, important.
Windows, the reason they're not shipping manylinux1 wheels right now has to do with the political implications of auto-shipping a second copy of openssl to Linux distros that expect to manage security upgrades centrally).
Understandable. I'm sure anyone who was around at the time remembers the zlib fiasco.
It might seem weird to use Python-specific tooling and per-application vendoring for Python dependencies, and yet use distro-global dynamic linking for C dependencies. But, this is actually a perfectly cromulent strategy, and I think this bears a more in-depth explanation.
It's not at all weird. It's exactly what we do, and I agree with your rationale.
On Feb 23, 2017, at 4:12 AM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
On 23/02/17 09:11, Glyph Lefkowitz wrote:
Yes, and building these binary artifacts is often harder than some people (cough, alpine, cough) seem to think. But there are better ways to square this circle than restricting yourself to the versions of /python/ libraries that happen to be available in your distro.
I don't disagree. I'm not defending the practice. But it is *clearly* a thing people do, and understanding the reasons is, IMO, important.
Oh, absolutely. What I'm driving at here - as I said perhaps more clearly in the other message I just sent - is that there are really good reasons to use distro infrastructure (C build toolchain complexity in large, critical dependencies) and not-so-good reasons (there just happen to be preinstalled python packages you could, easily, with zero build configuration, pip install) and the way that we give guidance to those folks should be informed by that.
Windows, the reason they're not shipping manylinux1 wheels right now has to do with the political implications of auto-shipping a second copy of openssl to Linux distros that expect to manage security upgrades centrally).
Understandable. I'm sure anyone who was around at the time remembers the zlib fiasco.
I remember enough zlib fiascos that I am not even sure which one you're referring to :).
It might seem weird to use Python-specific tooling and per-application vendoring for Python dependencies, and yet use distro-global dynamic linking for C dependencies. But, this is actually a perfectly cromulent strategy, and I think this bears a more in-depth explanation.
It's not at all weird. It's exactly what we do, and I agree with your rationale.
Cool. Glad to hear that this is being invented in parallel in various places :).
As a very simple example: if you have a traditional (non-container) Linux system hosting a Python application in a virtualenv, and you deploy a Python app to a virtualenv e.g. using Puppet or Ansible, you either need to:
1. Use no C extensions 2. Hope there's a manylinux1 binary wheel 3. Use the OS package and --system-site-packages 4. Compile the C extensions and make them available to pip
#2 seems useful now that I know about it but - correct me if I'm wrong - the manylinux1 permitted C dependencies are super-tiny, and would not permit e.g. cryptography or psycopg2?
#4 is what you are advocating for I believe? But can we agree that for smaller projects, that might seem like a lot of repeated work if the package is already available in the OS repos?
You always say “repeated work” but I hear “small Python script”. :) We have a few Python projects (not written/maintained by me :)) that did rely on system packages “because it’s easier”. Guess what! The other day Ubuntu upgraded PyCrypto (needed Ubuntu Trusty’s ancient Paramiko) under our butts, introduced a regressing and everything blew up: https://www.ubuntu.com/usn/usn-3199-2/ <https://www.ubuntu.com/usn/usn-3199-2/> . That particular project is a virtualenv now too. The Python build chain got so good on Linux by now that I really don’t even think about C extensions anymore. Windows is a different story entirely of course. :|
I'm trying to explain why, based on the efforts we expend locally, it could seem attractive to smaller sites.
Honestly, in the years I’ve been running Python services of different sizes, I have found that distro-provided system packages – unless you are writing software *for* a distribution – are loaded with so many downsides that they’re almost never worth it. They’re a shortcut and shortcuts usually bite back *eventually*. —h
Honestly, in the years I’ve been running Python services of different sizes, I have found that distro-provided system packages – unless you are writing software *for* a distribution – are loaded with so many downsides that they’re almost never worth it. They’re a shortcut and shortcuts usually bite back *eventually*.
Absolutely. Distro Python module packages are useless to dangerous most of the time. Eg Debian jessie is shipping Autobahn in a >3 years old version (0.5.14). From my perspective, Debian is hurting Autobahn's users this way - but we (upstream) cannot stop them distributing old outdated artifacts. The whole idea of having a "system wide" Python installation is technically wrong and bound to fail IMO. FWIW, I am in the Go/Rust camp: shipping single executables that are statically linked down to and including OpenSSL _and_ the C/C++ stdlibs. It's just awesome to "scp etcd" from a CentOS 6 to a Ubuntu 16 or whatever and it "just works". I have tried different approaches to get there with larger Python projects, but haven't found the equivalent to Go/Rust yet. Apart from that: Ubuntu has broken new ground with snapcraft - this is much better than debs .. it puts upstream into power gain. Let upstream talk directly to users, kicking out distro package "maintainers". Anway, just my 2cts /Tobias
—h
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Feb 22, 2017, at 9:42 AM, Hynek Schlawack <hs@ox.cx> wrote:
I don’t see how that’s tedious since a compute does that for me. Although I don’t see any value at wheeling them (and some packages cannot be wheeled); my CI builds a venv and puts it into a container. There’s nothing tedious about it at all. I find the idea of running throwaway environments to generate a big blob of tarball'd python+libs, then copying said tarball to actual containers, a rather retrograde step by comparison with established package/build infrastructure tools.
I have to disagree here: I don’t want build tools of any kind in my final containers therefore I build my artifacts separately no matter what language. Of course you can just build the venv on your build server without wheeling up a temporary container and then package it using Docker or DEB or whatever. You should be separating building and running anyway so Python – as much as I’d like Go-style single binaries too – is in no way special here. The nice thing about temporary containers though is that I can do all of that on my Mac.
It's worth pointing out that if you don't want a Go build toolchain in your container, you have exactly the same problem. And a Go build toolchain (read: dependency/vendoring management toolchain) actually requires _more_ care and feeding than the analogous Python in my experience; the reason it's superficially appealing is that you can "cheat" and copy the binary over from a development environment, but this is not a robust CI solution. Go's build toolchain has many features worth envying but most of its advantages have to do with deployments _outside_ of containers, where you have to ship to customer environments with fraught and unknown system configurations. If you have any level of control over your deployment target, Go and, say, Python with PEX are ~equivalent. -glyph
I don’t see how that’s tedious since a compute does that for me. Although I don’t see any value at wheeling them (and some packages cannot be wheeled); my CI builds a venv and puts it into a container. There’s nothing tedious about it at all. I find the idea of running throwaway environments to generate a big blob of tarball'd python+libs, then copying said tarball to actual containers, a rather retrograde step by comparison with established package/build infrastructure tools.
I have to disagree here: I don’t want build tools of any kind in my final containers therefore I build my artifacts separately no matter what language. Of course you can just build the venv on your build server without wheeling up a temporary container and then package it using Docker or DEB or whatever. You should be separating building and running anyway so Python – as much as I’d like Go-style single binaries too – is in no way special here. The nice thing about temporary containers though is that I can do all of that on my Mac.
It's worth pointing out that if you don't want a Go build toolchain in your container, you have exactly the same problem.
I thought I pointed that out in my very first sentence. :D What I don't quite understand is how people can be in love with Go’s static linking but complaining about Virtualenvs in deployments. Unwieldy as virtualenvs are: *for Python code* they are exactly that: statically linked build artifacts. The principles are very similar, the execution is arguably better for Go.
On 23/02/17 10:27, Hynek Schlawack wrote:
What I don't quite understand is how people can be in love with Go’s static linking but complaining about Virtualenvs in deployments. Unwieldy as virtualenvs are: *for Python code* they are exactly that: statically linked build artifacts. The principles are very similar, the execution is arguably better for Go.
Well, for one thing, Python virtualenv are problematic w.r.t. being relocatable. That alone makes them harder to use than a Go binary. But honestly I think we're talking at cross purposes. I'm not complaining about virtualenv. We use them quite successfully. I'm trying to point out that *some* might find the system Python attractive because they can use OS-supplied python packages in lieu of the effort of setting up a binary build infrastructure. But I'm clearly not getting my point across, so I'm going to stop now.
On Feb 23, 2017, at 2:51 AM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
On 23/02/17 10:27, Hynek Schlawack wrote:
What I don't quite understand is how people can be in love with Go’s static linking but complaining about Virtualenvs in deployments. Unwieldy as virtualenvs are: *for Python code* they are exactly that: statically linked build artifacts. The principles are very similar, the execution is arguably better for Go.
Well, for one thing, Python virtualenv are problematic w.r.t. being relocatable. That alone makes them harder to use than a Go binary.
Yeah, this is the big issue. Hence: pex ;-). Or, 'pile of wheels' and 'venv on the fly (without dev tools)' as the installation mechanism on the target environment.
But honestly I think we're talking at cross purposes. I'm not complaining about virtualenv. We use them quite successfully.
I think this has been a super useful discussion, not at cross purposes at all! There's been a little bit of back-and-forth and repeating certain things, but not useless.
I'm trying to point out that *some* might find the system Python attractive because they can use OS-supplied python packages in lieu of the effort of setting up a binary build infrastructure.
So, I think everyone has a different perspective here; Hynek seems to be saying "all containers all the time" ;-). But from my perspective, linux distros provide a super useful service by providing a gigantic integrated build environment for TONS of C code. Avoiding the time and cost associated with setting up, i.e., an ImageMagick dev environment is _WELL_ worth the complexity of distro packages. Multiply this out by dozens of dependencies that a large application or site ends up needing, and distro packaging pays for itself many times over. However, avoiding having python-dev and a C compiler installed in your build environment seems like a bit of a fool's errand to me. It's superficially appealing, but I think that our guidance to most people should be "always have a build environment so you can build extension modules, any non-trivial project will eventually need a few". The work required to have a _python_ dev environment is orders of magnitude below, say, OpenSSL or ImageMagick or Avahi or (et cetera).
But I'm clearly not getting my point across, so I'm going to stop now.
Really I think that this has been a great exchange of ideas, and though we're veering off topic a little bit, I want there to be more discussion among systems operators how Twisted can and should be deployed, operationalized, monitored, etc. Twisted even has weird little edge-cases of its own (like the plugin system) which can sometimes complicate things. So by all means please don't feel like you need to stop sharing your view. -glyph
I'm trying to point out that *some* might find the system Python attractive because they can use OS-supplied python packages in lieu of the effort of setting up a binary build infrastructure.
So, I think everyone has a different perspective here; Hynek seems to be saying "all containers all the time" ;-). But from my perspective, linux distros provide a super useful service by providing a gigantic integrated build environment for TONS of C code. Avoiding the time and cost associated with setting up, i.e., an ImageMagick dev environment is _WELL_ worth the complexity of distro packages. Multiply this out by dozens of dependencies that a large application or site ends up needing, and distro packaging pays for itself many times over.
I’m not sure what you mean by me meaning “all containers all the time” I feel I should rephrase my point: - You should statically link your Python application so you have always full control over your deployments. - Don’t ship build tools to prod servers. The way we can achieve that is by building virtualenvs on build servers. *How* you get those virtualenvs to the deployments target I don’t care. Whether you pex them or not, I don’t care either¹. I’ve used .debs with great success in the past and I’m using Docker containers now (and so far we’re not out of business :)). I consider Linux distributions very nice build environments for my own software. :) And there’s thing we totally don’t compile ourselves like OpenSSL. It just doesn’t make any goddam sense to rely on the limited and outdated offer of Python modules from your distribution. Even if they don’t upgrade them under your butt and introduce regressions. —h ¹ For me apps always go to the same location so relocability is not an issue. Contrary I’m not a super fan of having one opaque blob on server; the transparent structure of a virtualenv is something I learned to appreciate.
On 24 February 2017 at 10:25, Hynek Schlawack <hs@ox.cx> wrote:
... Contrary I’m not a super fan of having one opaque blob on server; the transparent structure of a virtualenv is something I learned to appreciate. ...
A zipfile containing a virtualenv (pex) isn't all that opaque -- with the right editor plugins, you can even *cough* edit the contained source inside one. Back when pex was a lot younger, this was handy for quick debugging as we were migrating from source-in-venvs-on-bare-metal-chroots to pexed-in-mesos-containers
... Contrary I’m not a super fan of having one opaque blob on server; the transparent structure of a virtualenv is something I learned to appreciate. ...
A zipfile containing a virtualenv (pex) isn't all that opaque -- with the right editor plugins, you can even *cough* edit the contained source inside one.
Back when pex was a lot younger, this was handy for quick debugging as we were migrating from source-in-venvs-on-bare-metal-chroots to pexed-in-mesos-containers
To be clear: I’m not trying to talk anybody out of using pex. :) I’m just saying there’s a slight downside (+ another tool) and no tangible upsides in Docker for me so it’s not a good fit for me. That said, I’m gonna try it out in my deb-based deployments (contrary to popular believe, you can’t/shouldn’t put *everything* into containers) because we run ZFS and apt-get + lots of files in a deb = omgiwannamurdersomeone.
Are you talking about building Docker containers on the fly? We use Docker extensively, but our build machine makes images that we push to Dockerhub (private repos). This has a lot of advantages: - Our images (on the hub) are effectively pinned at the version they were built - Our test and production servers (can, if we want) always get exactly the same image (even if we need to rebuild a server months later) - We test all our servers so we only have to manually pin packages (python or apt) if we run into regressions or other incompatibilities (i.e. an upgraded package that is no longer compatible with a manually pinned package) - Our build machine caches all the intermediate images (i.e. after each docker step). We intentionally sequence our images to place oft-changing items at the end. - Unless I change the list of apt packages, that layer is never rebuilt. - We have an extra step that uploads *just* the requirements files before pip installing - Our last step is the app code so changes to this layer are just a cached layer + PUT (i.e. seconds) - This optimization also makes our containers super efficient to upgrade because we only download the changed layers This sounds like it covers a lot of the PEX advantages plus the added benefits of containerization. Clayton Daley On Sat, Feb 25, 2017 at 3:19 AM, Hynek Schlawack <hs@ox.cx> wrote:
... Contrary I’m not a super fan of having one opaque blob on server; the
transparent structure of a virtualenv is something I learned to appreciate. ...
A zipfile containing a virtualenv (pex) isn't all that opaque -- with the right editor plugins, you can even *cough* edit the contained source inside one.
Back when pex was a lot younger, this was handy for quick debugging as we were migrating from source-in-venvs-on-bare-metal-chroots to pexed-in-mesos-containers
To be clear: I’m not trying to talk anybody out of using pex. :) I’m just saying there’s a slight downside (+ another tool) and no tangible upsides in Docker for me so it’s not a good fit for me.
That said, I’m gonna try it out in my deb-based deployments (contrary to popular believe, you can’t/shouldn’t put *everything* into containers) because we run ZFS and apt-get + lots of files in a deb = omgiwannamurdersomeone.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Feb 25, 2017, at 9:03 AM, Clayton Daley <clayton.daley@gmail.com> wrote:
Are you talking about building Docker containers on the fly?
Pretty sure Hynek has proper build-server/production separation.
We use Docker extensively, but our build machine makes images that we push to Dockerhub (private repos). This has a lot of advantages: Our images (on the hub) are effectively pinned at the version they were built Our test and production servers (can, if we want) always get exactly the same image (even if we need to rebuild a server months later) We test all our servers so we only have to manually pin packages (python or apt) if we run into regressions or other incompatibilities (i.e. an upgraded package that is no longer compatible with a manually pinned package) Our build machine caches all the intermediate images (i.e. after each docker step). We intentionally sequence our images to place oft-changing items at the end. Unless I change the list of apt packages, that layer is never rebuilt. We have an extra step that uploads *just* the requirements files before pip installing Our last step is the app code so changes to this layer are just a cached layer + PUT (i.e. seconds) This optimization also makes our containers super efficient to upgrade because we only download the changed layers This sounds like it covers a lot of the PEX advantages plus the added benefits of containerization.
Pex and containerization are completely orthogonal. You can use pex inside or outside of a container, and you can use a container with or without pex. Hynek lays out a good case against pex inside containers, but the blog post from Moshe that I linked to earlier lays out a good reason to use them together. -glyph
On Sat, Feb 25, 2017 at 5:50 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Feb 25, 2017, at 9:03 AM, Clayton Daley <clayton.daley@gmail.com> wrote:
Are you talking about building Docker containers on the fly?
Pretty sure Hynek has proper build-server/production separation.
Pretty sure Hynek can speak for himself.
On Feb 25, 2017, at 3:57 PM, Clayton Daley <clayton.daley@gmail.com> wrote:
On Sat, Feb 25, 2017 at 5:50 PM, Glyph Lefkowitz <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Feb 25, 2017, at 9:03 AM, Clayton Daley <clayton.daley@gmail.com <mailto:clayton.daley@gmail.com>> wrote:
Are you talking about building Docker containers on the fly?
Pretty sure Hynek has proper build-server/production separation.
Pretty sure Hynek can speak for himself.
I didn't mean to suggest he couldn't :). -g
Are you talking about building Docker containers on the fly?
I’m a bit baffled what gave you that idea after I’ve spent days arguing for strict build/runtime separation?
We use Docker extensively, but our build machine makes images that we push to Dockerhub (private repos). This has a lot of advantages: Our images (on the hub) are effectively pinned at the version they were built Our test and production servers (can, if we want) always get exactly the same image (even if we need to rebuild a server months later) We test all our servers so we only have to manually pin packages (python or apt) if we run into regressions or other incompatibilities (i.e. an upgraded package that is no longer compatible with a manually pinned package) Our build machine caches all the intermediate images (i.e. after each docker step). We intentionally sequence our images to place oft-changing items at the end. Unless I change the list of apt packages, that layer is never rebuilt. We have an extra step that uploads *just* the requirements files before pip installing Our last step is the app code so changes to this layer are just a cached layer + PUT (i.e. seconds) This optimization also makes our containers super efficient to upgrade because we only download the changed layers This sounds like it covers a lot of the PEX advantages plus the added benefits of containerization.
I don’t see anything that contradicts anything that I (or glyph) have written. At this point we were merely discussing what kind of isolated build artifact goes into the container/deb: a pex (= single file) or a vanilla venv (= directory structure).
On Sun, Feb 26, 2017 at 1:08 AM, Hynek Schlawack <hs@ox.cx> wrote:
Are you talking about building Docker containers on the fly?
I’m a bit baffled what gave you that idea after I’ve spent days arguing for strict build/runtime separation?
I was just trying to figure out where you were having this problem:
ZFS and apt-get + lots of files in a deb = omgiwannamurdersomeone
I heard the opposite of what you meant by this comment -- that you were deploying apt-gets + debs using docker and the combination had you wanting to murder someone. A really sharp dev wrote our software, but did a poor job sequencing Dockerfiles... and I had that exact same reaction. My question had nothing to do with build-runtime separation... just docker optimization. On a dev (not ops) list, it didn't seem like a crazy question. Obviously Glyph couldn't answer this question so his interjection came off as condescendingly unhelpful. Perhaps it's not an impression he deserves, but he got off to a horrible start when he responded to my first ever Twisted question (http://stackoverflow.com/questions/22896993/persisting- data-in-a-twisted-app) without bothering to add anything (sound familiar?). I rarely interject because I'm no developer... just an entrepreneurial jack-of-all-trades (masters in business, bachelors in business, minor in CS -- albeit from Carnegie Mellon) that sometimes contributes to our partly-Twisted product. However, you've succeeded in making my unusual perspective feel most unwelcome... so you'll be glad to know that you're never going to have to suffer me again.
Hi Clayton, On February 27, 2017 at 6:07:14 PM, Clayton Daley (clayton.daley@gmail.com) wrote: [SNIP] Obviously Glyph couldn't answer this question so his interjection came off as condescendingly unhelpful. Perhaps it's not an impression he deserves, but he got off to a horrible start when he responded to my first ever Twisted question (http://stackoverflow.com/questions/22896993/persisting-data-in-a-twisted-app) without bothering to add anything (sound familiar?). I rarely interject because I'm no developer... just an entrepreneurial jack-of-all-trades (masters in business, bachelors in business, minor in CS -- albeit from Carnegie Mellon) that sometimes contributes to our partly-Twisted product. However, you've succeeded in making my unusual perspective feel most unwelcome... so you'll be glad to know that you're never going to have to suffer me again. I think your characterization various parties involved in this thread is unfair, and that you are expressing an inappropriate sense of entitlement. Enumerating your academic degrees in no way enhances any point you are trying to make here, and strikes me as argument from authority. It is a shame that some miscommunication has occurred, and I think a more positive outcome could be achieved if you’d walk back a few of these remarks and re-engage. L. Daniel Burr
On Feb 27, 2017, at 4:05 PM, Clayton Daley <clayton.daley@gmail.com> wrote:
On Sun, Feb 26, 2017 at 1:08 AM, Hynek Schlawack <hs@ox.cx <mailto:hs@ox.cx>> wrote:
Are you talking about building Docker containers on the fly?
I’m a bit baffled what gave you that idea after I’ve spent days arguing for strict build/runtime separation?
I was just trying to figure out where you were having this problem:
ZFS and apt-get + lots of files in a deb = omgiwannamurdersomeone
I heard the opposite of what you meant by this comment -- that you were deploying apt-gets + debs using docker and the combination had you wanting to murder someone. A really sharp dev wrote our software, but did a poor job sequencing Dockerfiles... and I had that exact same reaction. My question had nothing to do with build-runtime separation... just docker optimization. On a dev (not ops) list, it didn't seem like a crazy question.
The question as such is not crazy at all, but rather, Hynek has been, both in general and repeatedly in this specific thread, a strong proponent of strict build & run container separation. What made the question confusing was that to ask it of him, in this way, implied that you weren't following his earlier suggestions, and it's not clear what you were reading into them if not "don't build containers on the fly".
Obviously Glyph couldn't answer this question so his interjection came off as condescendingly unhelpful.
It was not my intention to condescend, and I'm sorry that I did; my intention was to express a similar sort of bafflement that Hynek did. My intention was for it to be read more like "you must have read the same commitment to build and runtime separation that I did in Hynek's posts, so I'm not sure what you're referring to". (Plus, this response was a quip in a longer reply mainly about Pex.)
Perhaps it's not an impression he deserves, but he got off to a horrible start when he responded to my first ever Twisted question (http://stackoverflow.com/questions/22896993/persisting-data-in-a-twisted-app <http://stackoverflow.com/questions/22896993/persisting-data-in-a-twisted-app>) without bothering to add anything (sound familiar?).
The right thing to have done (with respect to the site's guidelines) would have been to vote to close (as "too broad" or perhaps "primarily opinion based") rather than to make that comment. However, I find that closing users' questions, especially relatively those asked by users without much experience on the site (i.e.: sub-1k reputation), is often interpreted as a hostile dismissal, so I ask for an SSCCE which can guide the asker to a more narrowly-constrained question that hews more closely to the guidelines set forth on the site. It really saddens me that a comment is taken as equally condescending; short of writing multiple paragraphs of apologetic preface for every piece of feedback, it's not clear what a regular answerer on the site ought to do. The fact that nobody else bothered to answer this question, comment on it, or even upvote it (one of the 2 upvotes was mine; I tend to upvote questions which are otherwise well-asked but need editing for specificity) in the intervening two years suggests that my "not appropriate for SO" assessment was accurate. What I was trying to tell you was "you aren't going to get any answers with a question phrased like this". One of the reasons I'm responding to this throwaway comment in such detail is that this category of misunderstanding - that attempting to be helpful in a volunteer endeavor like answering strangers' questions can be interpreted as arrogance, dismissal, or hostility (and the grudge nursed for years after the fact!), is a significant emotional cost for anyone involved in any form of volunteer support for open source. I'm in the middle of taking a year off from answering questions on SO for this exact reason. So, if you're reading this message, and you're ever mad at someone rejecting your question (or your PR or your bug report or whatever), please take a moment to consider the transaction from their perspective. They're commenting in an attempt to be helpful, and they've probably just answered 5 (or 10 or 50 or 100) similar questions, and so they may not be including the full context of their feedback. Ironically, the question you asked on SO would have been great mailing list fodder, it just doesn't fit the format stipulated by Stack Overflow's guidelines. The intersection of persistence and Twisted _is_ a subtle and non-obvious issue, one that I've personally been trying to help address recently (for example) by taking over maintenance of http://alchimia.readthedocs.io <http://alchimia.readthedocs.io/>.
I rarely interject because I'm no developer... just an entrepreneurial jack-of-all-trades (masters in business, bachelors in business, minor in CS -- albeit from Carnegie Mellon) that sometimes contributes to our partly-Twisted product. However, you've succeeded in making my unusual perspective feel most unwelcome... so you'll be glad to know that you're never going to have to suffer me again.
I'm very sorry to hear that's your impression of me, and sorrier still that it seems to extend to the community in general. Certainly, making people feel welcome is an explicit part of our mission here. Your perspective has been valuable over your substantial history with Twisted and I do hope you'll reconsider. -glyph
Are you talking about building Docker containers on the fly?
I’m a bit baffled what gave you that idea after I’ve spent days arguing for strict build/runtime separation?
I was just trying to figure out where you were having this problem:
ZFS and apt-get + lots of files in a deb = omgiwannamurdersomeone
I heard the opposite of what you meant by this comment -- that you were deploying apt-gets + debs using docker and the combination had you wanting to murder someone.
It was not clear to me you were referring to that line in particular. Especially because you were talking about Docker and not about debs. The problem is a long-standing “bug” (scare quotes because it’s more of a performance regression) in ZFS-on-Linux: https://github.com/zfsonlinux/zfs/issues/3857 <https://github.com/zfsonlinux/zfs/issues/3857> (I believe there are more related tickets but I don’t have the time to investigate). It’s related to excessive sync calls for every single file that accumulate very fast if you put a virtualenv with many files into a deb. Basically I have to run containers with Python debs in YOLO mode (sync=False) if I don’t want installs to take more than a minute while it takes almost no time on ext4 or XFS. That said, our Docker containers run on ext4 because our Docker daemon runs in an LXC and you cannot access the pool from within containers (yet, as I’ve hear). Which makes it run in VFS mode which is even slower (I’ve measured factor 3x vs overlay2 in our CI) because it means it has to create a physical copy of each layer. I suppose we were talking about different things because I don’t quite understand why you’re upset.
On 23 February 2017 at 08:36, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Go's build toolchain has many features worth envying but most of its advantages have to do with deployments _outside_ of containers, where you have to ship to customer environments with fraught and unknown system configurations. If you have any level of control over your deployment target, Go and, say, Python with PEX are ~equivalent .
I was going to chime in about pex earlier - but since you need to provide your own interpreter, I didn't think it solved his whole issue. Otherwise, pex is a great format for deploying python easily. At Twitter, we deployed the (Twisted) TweetDeck API using it for years, with a ton of dependencies, both c-libs and pure python. Really nice to be able to produce a deployable artefact, which can be hashed & stored for re-deploy later if needed (and ideally integration-tested & staged by passing different args on the cli).
On Feb 23, 2017, at 4:04 AM, James Broadhead <jamesbroadhead@gmail.com> wrote:
On 23 February 2017 at 08:36, Glyph Lefkowitz <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
Go's build toolchain has many features worth envying but most of its advantages have to do with deployments _outside_ of containers, where you have to ship to customer environments with fraught and unknown system configurations. If you have any level of control over your deployment target, Go and, say, Python with PEX are ~equivalent .
I was going to chime in about pex earlier - but since you need to provide your own interpreter, I didn't think it solved his whole issue.
Fully static binaries / integrated interpreters would be a good target for build tooling to hit, but I do think that (especially if you know your deployment platform) relying on _nothing_ but a Python interpreter, pex in any specific Linux environment is probably doable.
Otherwise, pex is a great format for deploying python easily.
Yep.
At Twitter, we deployed the (Twisted) TweetDeck API using it for years, with a ton of dependencies, both c-libs and pure python.
This is good to know!
Really nice to be able to produce a deployable artefact, which can be hashed & stored for re-deploy later if needed (and ideally integration-tested & staged by passing different args on the cli).
I originally felt like the lack of an interpreter made it useless, and I think I was very much mistaken. Moshe has been showing me all the problems that pex can address, and I'm slowly starting to adopt it in my own deployments. -glyph
On Wed, Feb 22, 2017 at 7:08 AM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
On 22/02/17 11:12, Glyph Lefkowitz wrote:
I'm tempted to launch into a diatribe about namespacing, containers, and
application isolation generally, but before I do - why is it that you /want/ to use the system Python environment? Had you just not considered the option of virtual environments?
Awesome though it is, virtualenv can be very tedious if you need to install hundreds of megabytes of compiler and -devel packages. System packages are attractive precisely because you can avoid this.
I guess this is a big part of the motivation for Twisted itself to shed its C extensions - to avoid 100% of these extra dependencies. And presumably also a big part of the motivation behind the development of support for distributable binary wheels for Linux (which seems to have largely succeeded at this point, though their use is not as widespread as one would hope).
I've had to do all sorts of tedious things with containers where I spin up a temporary container to build a bunch of .whl files, then actually install them in the final container - all to avoid bloating the container image with the build tools.
And I've been meaning to really dig in to the Nix toolchain for image creation. That seems like another thing that should be more widely used (but it's hard to beat the "batteries included" tools Docker has, even if those tools are awful).
It's a real shame that binary wheels on Linux/PyPI aren't a thing. I understand the reasons why, but TBH it substantially reduces the value of the Python ecosystem for me.
Ah! They are now, maybe you haven't seen them? Check out https://github.com/pypa/manylinux and https://github.com/pypa/python-manylinux-demo Go is looking like a more and more attractive competitor to Python,
frankly, having sidestepped this problem entirely.
A lot of people seem to be thinking that way. *Personally*, with my experience with Go (about five months solid at previous employer), I wouldn't go anywhere near that stack. There are plenty of *other *things with appealing features that Python lacks which would seem to make a better move. :) But check out the story for binary wheels. The state of things in Python may not be *quite* as bad as you think. Jean-Paul
On 22/02/17 13:06, Jean-Paul Calderone wrote:
Ah! They are now, maybe you haven't seen them?
Check out https://github.com/pypa/manylinux and https://github.com/pypa/python-manylinux-demo
I had not seen this. Thanks, this looks like a big leap forward.
A lot of people seem to be thinking that way. /Personally/, with my experience with Go (about five months solid at previous employer), I wouldn't go anywhere near that stack. There are plenty of /other /things with appealing features that Python lacks which would seem to make a better move. :)
Very possibly! Go as a language leaves me cold, and there are downsides to the one-big-binary approach. But it's undeniably easy to deploy in a container, and the concurrency is very attractive.
But check out the story for binary wheels. The state of things in Python may not be /quite/ as bad as you think.
Indeed, it does look like a promising improvement.
On Feb 22, 2017, at 9:17 AM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
On 22/02/17 13:06, Jean-Paul Calderone wrote:
Ah! They are now, maybe you haven't seen them?
Check out https://github.com/pypa/manylinux and https://github.com/pypa/python-manylinux-demo
I had not seen this. Thanks, this looks like a big leap forward.
A lot of people seem to be thinking that way. /Personally/, with my experience with Go (about five months solid at previous employer), I wouldn't go anywhere near that stack. There are plenty of /other /things with appealing features that Python lacks which would seem to make a better move. :)
Very possibly!
Go as a language leaves me cold, and there are downsides to the one-big-binary approach. But it's undeniably easy to deploy in a container, and the concurrency is very attractive.
Python is easier to deploy in a container than you might think. Check out this three-line Dockerfile from Moshe Zadka (which, by the way, deploys Twisted...): https://moshez.wordpress.com/2016/11/19/belt-suspenders-why-put-a-pex-file-i... <https://moshez.wordpress.com/2016/11/19/belt-suspenders-why-put-a-pex-file-inside-a-docker-container/> -glyph
But check out the story for binary wheels. The state of things in Python may not be /quite/ as bad as you think.
Indeed, it does look like a promising improvement.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 02/22/2017 07:08 AM, Phil Mayers wrote:
On 22/02/17 11:12, Glyph Lefkowitz wrote:
I'm tempted to launch into a diatribe about namespacing, containers, and application isolation generally, but before I do - why is it that you /want/ to use the system Python environment? Had you just not considered the option of virtual environments?
Awesome though it is, virtualenv can be very tedious if you need to install hundreds of megabytes of compiler and -devel packages. System packages are attractive precisely because you can avoid this.
Have you considered the 'conda' package manager? https://conda.io/docs/index.html If not, and for the benefit of others (I'm amazed it's not mentioned on this list) here is my conda rant: TL;DR: use conda. IMO, conda is superior to pip/wheels. The Python packaging orthodoxy's story is "oh, yes, conda -- that solves a different problem ..." which seems disingenuous to me. Conda was originally created by Continuum Analytics to solve the Numpy/SciPy packaging problem(s), which it has done with brilliant success. Conda has its own kind of packages, but they are relatively easy to create (I can do it, proving that anyone can ;) both for your own packages and for PyPI packages -- see: https://conda.io/docs/build_tutorials/pkgs.html Conda is a system-level package manager in the sense that it can install any package, including Python itself, and creates its own type of 'virtual environments' with their own Python versions (which it will install). Conda does a great job of keeping non-Python libraries that Python packages depend on isolated from system libraries. If the version of a Python package you want to use depends on a later version of a library that exists on your system, conda will install the required version of the library in the virtual environment, keeping it separate from other versions of that library on your system. The latest version of Twisted is available from the main conda package repository and is well supported. I am developing an application with many dependencies, including Twisted, PyQt5, pythonOCC (a massive python-wrapped C++ 3D CAD library), and many scientific libraries, and conda is the only Python packaging and distribution technology that can support my app -- I would not even begin to attempt it using pip. I've also used conda to manage large Django applications. After I showed conda to my sysadmins at work, they jumped on it and now use it for all Python installations on our servers. They will never have to compile Python again! I do all my development on an Ubuntu 14.04 laptop and use conda for everything Python on my machine. Cheers, Steve
On 02/22/2017 12:23 PM, Phil Mayers wrote:
On 22/02/17 15:00, Steve Waterbury wrote:
Have you considered the 'conda' package manager?
I've never come across it. It looks too big for me to give a quick opinion on, but I'll bear it in mind. Thanks for the pointer.
"Anaconda" is huge; conda is not. You can get conda by installing "Miniconda": https://conda.io/miniconda.html The 64-bit bash Miniconda installer for Linux, for example, is about 28 MB, and includes just Python (either 2 or 3) with conda and its dependencies, and includes pip in case you want to use a package that is not yet available in the conda package channels -- 'pip install' works in a conda environment. (The only reason to install Anaconda, which includes over 100 scientific Python packages, is if you are developing an app that requires a lot of Numpy or SciPy libraries.) Steve
On 22/02/17 17:35, Steve Waterbury wrote:
On 02/22/2017 12:23 PM, Phil Mayers wrote:
On 22/02/17 15:00, Steve Waterbury wrote:
Have you considered the 'conda' package manager?
I've never come across it. It looks too big for me to give a quick opinion on, but I'll bear it in mind. Thanks for the pointer.
"Anaconda" is huge; conda is not. You can get conda
I was referring to scope as opposed to size in bytes, but it's entirely possible I've misunderstood how it works.
On Feb 22, 2017, at 9:35 AM, Steve Waterbury <waterbug@pangalactic.us> wrote:
On 02/22/2017 12:23 PM, Phil Mayers wrote:
On 22/02/17 15:00, Steve Waterbury wrote:
Have you considered the 'conda' package manager?
I've never come across it. It looks too big for me to give a quick opinion on, but I'll bear it in mind. Thanks for the pointer.
"Anaconda" is huge; conda is not. You can get conda by installing "Miniconda":
https://conda.io/miniconda.html
The 64-bit bash Miniconda installer for Linux, for example, is about 28 MB, and includes just Python (either 2 or 3) with conda and its dependencies, and includes pip in case you want to use a package that is not yet available in the conda package channels -- 'pip install' works in a conda environment.
(The only reason to install Anaconda, which includes over 100 scientific Python packages, is if you are developing an app that requires a lot of Numpy or SciPy libraries.)
It is "huge" in the sense that it is a parallel universe to your normal Linux distribution though; you need to install duplicate versions of various C libraries in multiple locations, whereas pip & virtualenv allow you the flexibility of "package everything as a static generic manylinux1 wheel for portability" or "depend on dynamically linked libraries in your operating system" depending on your deployment preferences. -glyph
So my problem is that while my protocol stack spins up very nicely and trivially easy using the twistd daemon, the standard python3-twisted package for 14.04 is waaaay behind and doesn't include twistd (!), and I haven't found a good backport. I'd also like to avoid doing things that make life hard for DevOps, such as pip3 installing and risking a package manager fight.
The general solution to this problem is "don't use the system Python environment to deploy your applications".
This is what virtualenv is for.
Yes, well, the standard practice around here is to wrap up deliverables in a .deb so that the DevOps tools can do automated deploys. I suppose it is possible to wrap up a venv in a .deb, but I’ve never done looked into it. Pointers welcome.
So some options: - Is there a reliable backport of the python3-twisted package for 14.04? Google is failing me on that one. [...] - Perhaps I should create my own backport for Ubuntu 14.04 of the current python3-twisted .deb. (This is probably not the right list to ask, but I'm happy to dereference a pointer.)
Both of these options are bad. Installing or creating a backport will break any system tools that depend on the python3-twisted package. This is better than `sudo pip3 install`, but only a little.
- All I need is the most basic twistd functionality. Perhaps I should spin up my own by snarfing the code out of the twisted source repo. I can package this with my stuff temporarily and get on with life. Clues to the relevant parts of the repo welcome, I haven't poked around much in Twisted sources so need a road map.
If you have the ability to ship parts of the Twisted stack, there's a tool that can automate the "snarfing": pip :-).
This is a short-term problem for me, we are transitioning to 16.04 soon, but the process is driven by other parts of the software stack. So I don't want to over-invest. I'm looking for a reasonable band-aid.
16.04 will also be out of date.
So… If you would like to dive into reasons… we do robots. We use ROS, the Robot Operating System. LTS versions of ROS are tied to LTS versions of Ubuntu. Transitioning from one LTS version of ROS to another is, as they say, “non-trivial”. Being on a stale version of Ubuntu is part of the price for using ROS, because ROS just can’t move that fast. No one likes it. A lot of people (in many ROS-based companies, and the OSRF) are working to improve the situation. That doesn’t help me this week. Now, why are the DevOps people here so conservative? Because doing a remote update on a robot in the field, and having it fail, has two impacts: 1) we stop billing the customer until the robot is running again, and 2) we put someone on an airplane to make the robot run again.
I'm tempted to launch into a diatribe about namespacing, containers, and application isolation generally, but before I do - why is it that you want to use the system Python environment? Had you just not considered the option of virtual environments?
Legitimate questions. Fitting into the existing deployment flow, and ummm… can I wrap a venv in a .deb? ‘Cuz that would be cool. Another poster mentioned Anaconda…. I’m not a huge fan. I am using it extensively as part of the Udacity self-driving car class, and I will admit that it would be nearly impossible for Udacity to deploy the project environment any other way. But I see Anaconda as a very heavy-weight solution for projects requiring a huge infrastructure, and my problem is no where near large enough that the benefits justifies the cost. A simple venv is more in line.
-glyph _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
So my problem is that while my protocol stack spins up very nicely and trivially easy using the twistd daemon, the standard python3-twisted package for 14.04 is waaaay behind and doesn't include twistd (!), and I haven't found a good backport. I'd also like to avoid doing things that make life hard for DevOps, such as pip3 installing and risking a package manager fight.
The general solution to this problem is "don't use the system Python environment to deploy your applications".
This is what virtualenv is for.
Yes, well, the standard practice around here is to wrap up deliverables in a .deb so that the DevOps tools can do automated deploys. I suppose it is possible to wrap up a venv in a .deb, but I’ve never done looked into it. Pointers welcome.
https://github.com/spotify/dh-virtualenv <https://github.com/spotify/dh-virtualenv> and https://hynek.me/articles/python-app-deployment-with-native-packages/ <https://hynek.me/articles/python-app-deployment-with-native-packages/> come to mind. N.B. the article is by me and I’m currently migrating to Docker/Nomad. But it served us very well for many years.
participants (12)
-
Clayton Daley -
Cory Benfield -
Dave Curtis -
Glyph Lefkowitz -
Hynek Schlawack -
James Broadhead -
Jean-Paul Calderone -
L. Daniel Burr -
Phil Mayers -
Steve Waterbury -
steven meiers -
Tobias Oberstein