Support for current versions of Pypy on Heroku and Cloud Foundry platforms?
Hi Folks, Currently Heroku only has Pypy 1.9 on an unsupported, experimental basis. https://devcenter.heroku.com/articles/python-runtimes#supported-python-runti... and https://github.com/heroku/heroku-buildpack-python/issues/139 Other Cloud Foundry-based IaaS offering (IBM BlueMix, etc.) seem to use similar buildpacks. https://github.com/cloudfoundry-community/cf-docs-contrib/wiki/Buildpacks Do any of you know of an Infrastructure as a Service provider that supports current versions of Pypy? Thanks for any pointers that you can provide. Chris Clauss
Hi Chris, Are you looking for an Infrastructure as a Service (something like AWS, or Rackspace Cloud) or a Platform as a Service (Heroku)? Typically IaaS providers just give you a bare linux box, where you can of course install your own PyPy; as you've seen it looks like the PyPy on Heroku is out of date. I think the easiest move is probably to look into writing a custom build pack for Heroku, if that's what you're interested in. Alex PS: Disclaimer, I work at Rackspace. On Tue, Jun 17, 2014 at 6:19 PM, cclauss <cclauss@me.com> wrote:
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084
Hi Folks, The plan is to support Pypy on Heroku, but we need to have full libffi support before we move forward. As it stands, the following formulas produce a broken version of PyPy and Pypy3. Does anyone on this list have the required skills to suggest a working solution? https://github.com/heroku/heroku-buildpack-python/pull/154 and https://github.com/heroku/heroku-buildpack-python/pull/155 Thanks, CCC
Hi Chris, On 16 July 2014 14:29, cclauss <cclauss@me.com> wrote:
The plan is to support Pypy on Heroku, but we need to have full libffi support before we move forward.
What does this mean, exactly? You don't provide libffi on your build system, and so PyPy cannot be built there? A bientôt, Armin.
Hi Armin, The way to think about cloud application platforms like Heroku, Bluemix, and Cloud Foundry is that they provide you with a working Linux box and little else. Therefore you use a "buildpack" (like an install script) to bundle up all your executable environment, libraries, and code so that it can be loaded and run on a remote Linux box. In the case of CPython that means that the buildpack needs to install a working CPython, setuptools, and pip, etc. and then look in your requirements.txt to find which pypi modules pip needs to install and then it launches your webapp (written to django, flask, bottle, etc.). To get Pypy to work in place of CPython, the buildpack would need to install a working Pypy, setuptools, and pip. Libffi is an essential precursor to having Pypy work properly. Some interesting work was done in https://github.com/mfenniak/heroku-buildpack-python-libffi/blob/master/bin/s... to get libffi working in a buildpack but my limited understanding does not allow me to take that further. It would be of interest to get Pypy working on these Platform-as-a-Service environments in place of CPython but it is beyond my limited understanding to actually make it happen. I hope to see your presentation at EuroPython. On 16 Jul 2014, at 14:50, Armin Rigo <arigo@tunes.org> wrote:
Hi, On 17 July 2014 09:09, Maciej Fijalkowski <fijall@gmail.com> wrote:
Note that CPython bundles libffi, so you just bundle libffi with pypy in a buildpack and be done.
Also, I'm still not clear if a "bundle" contains all commands to recompile everything, or if it's just a binary package. Anyways, you can translate PyPy in a way that links to libffi statically instead of dynamically (I think it's the default, but I'm not sure). Then there is no libffi issue any more... I'm also quite unclear about why libffi in particular is the issue. As far as I know, on non-standard platforms you may have troubles compiling it, but on a standard Linux it should just work. If the only missing thing is knowledge about how to declare a "bundle" for Platform-as-service XYZ, then I'm afraid we can't help you on pypy-dev, but I don't see why making a "bundle" for PyPy would be easy but making a "bundle" for libffi not... And, anyway, as Maciej says you can put everything in the same bundle. It's not like the size of libffi (15KB?) matters. A bientôt, Armin.
On Thu, 2014-07-17 at 10:44 +0200, Armin Rigo wrote:
Also, I'm still not clear if a "bundle" contains all commands to recompile everything, or if it's just a binary package.
In particular, the above-mentioned libffi bundle fetches a pre-compiled version of libffi from some random AWS account and the PyPy bundle fetches the pre-compiled binaries the PyPy bitbucket page. The problem is, I guess, that the PyPy binaries are dynamically linked against libffi, but installing libffi bundle will not help the matters, if it's not going to be added to the LD_LIBRARY_PATH for PyPy to see. So, it's either that, or else I would recommend trying this distribution out instead of the official PyPy binaries: https://github.com/squeaky-pl/portable-pypy It seems to include everything that can be reasonably bundled up in a single archive. Anyways, the buildpack moniker is misleading, isn't it? One would naively assume that it has something to do with building, whereas this doesn't seem to be the case. -- Sincerely yours, Yury V. Zaytsev
Hi Yury, On 17 July 2014 11:04, Yury V. Zaytsev <yury@shurup.com> wrote:
Thanks for the explanations! And, just to be complete, may I ask how PyPy finds the other libraries it needs, and how in general project X finds library Y that it depends on? I still fail to see why we're discussing libffi specifically here... A bientôt, Armin.
On Thu, 2014-07-17 at 15:10 +0200, Armin Rigo wrote:
Thanks for the explanations! And, just to be complete, may I ask how PyPy finds the other libraries it needs,
Excellent question for a PyPy developer; to be honest, I actually have very little idea :-) I guess it just adds the -l / -L / -I flags (discovered either by checking a pre-defined set of paths, or by running pkg-config) and hopes that all the right libraries are installed and headers are at the correct place, and blows up otherwise. It seems from a cursory grep that the Makefile writing logic is here: pypy/pypy/rpython/translator/platform Specifically, see posix.py and linux.py.
and how in general project X finds library Y that it depends on?
The build systems usually try pkg-config, then a list of pre-defined locations and flags. Then a dummy executable is compiled, linked and run. If it works, everything is great, if not, it complains and/or disables the dependency. Now, this has to do with building, but as we've established, the buildpacks don't actually build anything, but rather unpack pre-compiled binaries. In this case, it's a dynamic linker thing. In brief, it looks for libraries in RPATH / LD_LIBRARY_PATH and standard locations. Does this make sense to you?
I still fail to see why we're discussing libffi specifically here...
Actually, I'm not sure about that either :-) Maybe the machines provided by Heroku already have all other libraries that the binaries of PyPy that you provide via the buildbot are dynamically linked against, and so libffi is the only problematic one... -- Sincerely yours, Yury V. Zaytsev
It looks like Heroku will soon have native support for PyPy 2.3.1: https://github.com/heroku/heroku-buildpack-python/issues/139 Alex On Thu, Jul 17, 2014 at 7:51 AM, Yury V. Zaytsev <yury@shurup.com> wrote:
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084
Hi Chris, Are you looking for an Infrastructure as a Service (something like AWS, or Rackspace Cloud) or a Platform as a Service (Heroku)? Typically IaaS providers just give you a bare linux box, where you can of course install your own PyPy; as you've seen it looks like the PyPy on Heroku is out of date. I think the easiest move is probably to look into writing a custom build pack for Heroku, if that's what you're interested in. Alex PS: Disclaimer, I work at Rackspace. On Tue, Jun 17, 2014 at 6:19 PM, cclauss <cclauss@me.com> wrote:
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084
Hi Folks, The plan is to support Pypy on Heroku, but we need to have full libffi support before we move forward. As it stands, the following formulas produce a broken version of PyPy and Pypy3. Does anyone on this list have the required skills to suggest a working solution? https://github.com/heroku/heroku-buildpack-python/pull/154 and https://github.com/heroku/heroku-buildpack-python/pull/155 Thanks, CCC
Hi Chris, On 16 July 2014 14:29, cclauss <cclauss@me.com> wrote:
The plan is to support Pypy on Heroku, but we need to have full libffi support before we move forward.
What does this mean, exactly? You don't provide libffi on your build system, and so PyPy cannot be built there? A bientôt, Armin.
Hi Armin, The way to think about cloud application platforms like Heroku, Bluemix, and Cloud Foundry is that they provide you with a working Linux box and little else. Therefore you use a "buildpack" (like an install script) to bundle up all your executable environment, libraries, and code so that it can be loaded and run on a remote Linux box. In the case of CPython that means that the buildpack needs to install a working CPython, setuptools, and pip, etc. and then look in your requirements.txt to find which pypi modules pip needs to install and then it launches your webapp (written to django, flask, bottle, etc.). To get Pypy to work in place of CPython, the buildpack would need to install a working Pypy, setuptools, and pip. Libffi is an essential precursor to having Pypy work properly. Some interesting work was done in https://github.com/mfenniak/heroku-buildpack-python-libffi/blob/master/bin/s... to get libffi working in a buildpack but my limited understanding does not allow me to take that further. It would be of interest to get Pypy working on these Platform-as-a-Service environments in place of CPython but it is beyond my limited understanding to actually make it happen. I hope to see your presentation at EuroPython. On 16 Jul 2014, at 14:50, Armin Rigo <arigo@tunes.org> wrote:
Hi, On 17 July 2014 09:09, Maciej Fijalkowski <fijall@gmail.com> wrote:
Note that CPython bundles libffi, so you just bundle libffi with pypy in a buildpack and be done.
Also, I'm still not clear if a "bundle" contains all commands to recompile everything, or if it's just a binary package. Anyways, you can translate PyPy in a way that links to libffi statically instead of dynamically (I think it's the default, but I'm not sure). Then there is no libffi issue any more... I'm also quite unclear about why libffi in particular is the issue. As far as I know, on non-standard platforms you may have troubles compiling it, but on a standard Linux it should just work. If the only missing thing is knowledge about how to declare a "bundle" for Platform-as-service XYZ, then I'm afraid we can't help you on pypy-dev, but I don't see why making a "bundle" for PyPy would be easy but making a "bundle" for libffi not... And, anyway, as Maciej says you can put everything in the same bundle. It's not like the size of libffi (15KB?) matters. A bientôt, Armin.
On Thu, 2014-07-17 at 10:44 +0200, Armin Rigo wrote:
Also, I'm still not clear if a "bundle" contains all commands to recompile everything, or if it's just a binary package.
In particular, the above-mentioned libffi bundle fetches a pre-compiled version of libffi from some random AWS account and the PyPy bundle fetches the pre-compiled binaries the PyPy bitbucket page. The problem is, I guess, that the PyPy binaries are dynamically linked against libffi, but installing libffi bundle will not help the matters, if it's not going to be added to the LD_LIBRARY_PATH for PyPy to see. So, it's either that, or else I would recommend trying this distribution out instead of the official PyPy binaries: https://github.com/squeaky-pl/portable-pypy It seems to include everything that can be reasonably bundled up in a single archive. Anyways, the buildpack moniker is misleading, isn't it? One would naively assume that it has something to do with building, whereas this doesn't seem to be the case. -- Sincerely yours, Yury V. Zaytsev
Hi Yury, On 17 July 2014 11:04, Yury V. Zaytsev <yury@shurup.com> wrote:
Thanks for the explanations! And, just to be complete, may I ask how PyPy finds the other libraries it needs, and how in general project X finds library Y that it depends on? I still fail to see why we're discussing libffi specifically here... A bientôt, Armin.
On Thu, 2014-07-17 at 15:10 +0200, Armin Rigo wrote:
Thanks for the explanations! And, just to be complete, may I ask how PyPy finds the other libraries it needs,
Excellent question for a PyPy developer; to be honest, I actually have very little idea :-) I guess it just adds the -l / -L / -I flags (discovered either by checking a pre-defined set of paths, or by running pkg-config) and hopes that all the right libraries are installed and headers are at the correct place, and blows up otherwise. It seems from a cursory grep that the Makefile writing logic is here: pypy/pypy/rpython/translator/platform Specifically, see posix.py and linux.py.
and how in general project X finds library Y that it depends on?
The build systems usually try pkg-config, then a list of pre-defined locations and flags. Then a dummy executable is compiled, linked and run. If it works, everything is great, if not, it complains and/or disables the dependency. Now, this has to do with building, but as we've established, the buildpacks don't actually build anything, but rather unpack pre-compiled binaries. In this case, it's a dynamic linker thing. In brief, it looks for libraries in RPATH / LD_LIBRARY_PATH and standard locations. Does this make sense to you?
I still fail to see why we're discussing libffi specifically here...
Actually, I'm not sure about that either :-) Maybe the machines provided by Heroku already have all other libraries that the binaries of PyPy that you provide via the buildbot are dynamically linked against, and so libffi is the only problematic one... -- Sincerely yours, Yury V. Zaytsev
It looks like Heroku will soon have native support for PyPy 2.3.1: https://github.com/heroku/heroku-buildpack-python/issues/139 Alex On Thu, Jul 17, 2014 at 7:51 AM, Yury V. Zaytsev <yury@shurup.com> wrote:
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084
participants (5)
-
Alex Gaynor -
Armin Rigo -
cclauss -
Maciej Fijalkowski -
Yury V. Zaytsev