Docker builds, ~/.pip/cache, and O(1) or O(n) bandwidth
What is the best way to build docker images without constantly re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n) for every build) (AFAIK, nobody has any issue with the amount of bandwidth PyPI uses) Thus far, Docker doesn't want to support a build-time -v option (that could be used to bind-mount .pip/cache in at build time): "build time only -v option" https://github.com/moby/moby/issues/14080 Buildah *does* support a build-time -v option (and can also do rootless builds without a docker socket)
You can use a local PyPI mirror, e.g. devpi, and point your docker builds at that, basically tricking docker by going through the (local) network stack instead of the filesystem. On Tue, May 7, 2019 at 8:12 AM Wes Turner <wes.turner@gmail.com> wrote:
What is the best way to build docker images without constantly re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n) for every build)
(AFAIK, nobody has any issue with the amount of bandwidth PyPI uses)
Thus far, Docker doesn't want to support a build-time -v option (that could be used to bind-mount .pip/cache in at build time):
"build time only -v option" https://github.com/moby/moby/issues/14080
Buildah *does* support a build-time -v option (and can also do rootless builds without a docker socket) -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/IL6MG...
Have you tried using buildkit and the RUN --mount option? I've done extra stuff here (downloading rpms in a second image first) but I think you could just use the cache option. It would also be easy to use a second image and COPY in old docker. https://github.com/dholth/vagrant-docker/blob/master/Dockerfile#L86 On Tue, May 7, 2019, 11:59 Alex Becker <alcubecker@gmail.com> wrote:
You can use a local PyPI mirror, e.g. devpi, and point your docker builds at that, basically tricking docker by going through the (local) network stack instead of the filesystem.
On Tue, May 7, 2019 at 8:12 AM Wes Turner <wes.turner@gmail.com> wrote:
What is the best way to build docker images without constantly re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n) for every build)
(AFAIK, nobody has any issue with the amount of bandwidth PyPI uses)
Thus far, Docker doesn't want to support a build-time -v option (that could be used to bind-mount .pip/cache in at build time):
"build time only -v option" https://github.com/moby/moby/issues/14080
Buildah *does* support a build-time -v option (and can also do rootless builds without a docker socket) -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/IL6MG...
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/2MNMG...
On Tuesday, May 7, 2019, Daniel Holth <dholth@gmail.com> wrote:
Have you tried using buildkit and the RUN --mount option? I've done extra stuff here (downloading rpms in a second image first) but I think you could just use the cache option.
It would also be easy to use a second image and COPY in old docker.
https://github.com/dholth/vagrant-docker/blob/master/Dockerfile#L86
Thanks. Is this a multi-stage Dockerfile (with multiple FROM instructions)? Is there a guide for a PyPA recommended way to create network-efficient reproducible (manylinux2010) wheels?
On Tue, May 7, 2019, 11:59 Alex Becker <alcubecker@gmail.com> wrote:
You can use a local PyPI mirror, e.g. devpi, and point your docker builds at that, basically tricking docker by going through the (local) network stack instead of the filesystem.
On Tue, May 7, 2019 at 8:12 AM Wes Turner <wes.turner@gmail.com> wrote:
What is the best way to build docker images without constantly re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n) for every build)
(AFAIK, nobody has any issue with the amount of bandwidth PyPI uses)
Thus far, Docker doesn't want to support a build-time -v option (that could be used to bind-mount .pip/cache in at build time):
"build time only -v option" https://github.com/moby/moby/issues/14080
Buildah *does* support a build-time -v option (and can also do rootless builds without a docker socket) -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@ python.org/message/IL6MGTSAA4X5UZDERCAAAFGGX5DGRPF3/
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@ python.org/message/2MNMGYBNFOOVGXA2JQ2PLDTZITZXANJU/
That one is multistage, great feature, but I think cache mounts would work fine with any number of stages. The mount option is a pretty new feature. On Tue, May 7, 2019, 12:24 Wes Turner <wes.turner@gmail.com> wrote:
On Tuesday, May 7, 2019, Daniel Holth <dholth@gmail.com> wrote:
Have you tried using buildkit and the RUN --mount option? I've done extra stuff here (downloading rpms in a second image first) but I think you could just use the cache option.
It would also be easy to use a second image and COPY in old docker.
https://github.com/dholth/vagrant-docker/blob/master/Dockerfile#L86
Thanks. Is this a multi-stage Dockerfile (with multiple FROM instructions)?
Is there a guide for a PyPA recommended way to create network-efficient reproducible (manylinux2010) wheels?
On Tue, May 7, 2019, 11:59 Alex Becker <alcubecker@gmail.com> wrote:
You can use a local PyPI mirror, e.g. devpi, and point your docker builds at that, basically tricking docker by going through the (local) network stack instead of the filesystem.
On Tue, May 7, 2019 at 8:12 AM Wes Turner <wes.turner@gmail.com> wrote:
What is the best way to build docker images without constantly re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n) for every build)
(AFAIK, nobody has any issue with the amount of bandwidth PyPI uses)
Thus far, Docker doesn't want to support a build-time -v option (that could be used to bind-mount .pip/cache in at build time):
"build time only -v option" https://github.com/moby/moby/issues/14080
Buildah *does* support a build-time -v option (and can also do rootless builds without a docker socket) -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/IL6MG...
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/2MNMG...
Also you must set an environment variable for docker build to be able to use --mount On Tue, May 7, 2019, 12:28 Daniel Holth <dholth@gmail.com> wrote:
That one is multistage, great feature, but I think cache mounts would work fine with any number of stages. The mount option is a pretty new feature.
On Tue, May 7, 2019, 12:24 Wes Turner <wes.turner@gmail.com> wrote:
On Tuesday, May 7, 2019, Daniel Holth <dholth@gmail.com> wrote:
Have you tried using buildkit and the RUN --mount option? I've done extra stuff here (downloading rpms in a second image first) but I think you could just use the cache option.
It would also be easy to use a second image and COPY in old docker.
https://github.com/dholth/vagrant-docker/blob/master/Dockerfile#L86
Thanks. Is this a multi-stage Dockerfile (with multiple FROM instructions)?
Is there a guide for a PyPA recommended way to create network-efficient reproducible (manylinux2010) wheels?
On Tue, May 7, 2019, 11:59 Alex Becker <alcubecker@gmail.com> wrote:
You can use a local PyPI mirror, e.g. devpi, and point your docker builds at that, basically tricking docker by going through the (local) network stack instead of the filesystem.
On Tue, May 7, 2019 at 8:12 AM Wes Turner <wes.turner@gmail.com> wrote:
What is the best way to build docker images without constantly re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n) for every build)
(AFAIK, nobody has any issue with the amount of bandwidth PyPI uses)
Thus far, Docker doesn't want to support a build-time -v option (that could be used to bind-mount .pip/cache in at build time):
"build time only -v option" https://github.com/moby/moby/issues/14080
Buildah *does* support a build-time -v option (and can also do rootless builds without a docker socket) -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/IL6MG...
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/2MNMG...
RUN --mount needs a magic comment to opt into experimental syntax: # syntax=docker/dockerfile:experimental https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experi... but AFAIK, was designed for this purpose, among others. Thanks, On Tue, May 7, 2019, 11:33 AM Daniel Holth <dholth@gmail.com> wrote:
Also you must set an environment variable for docker build to be able to use --mount
On Tue, May 7, 2019, 12:28 Daniel Holth <dholth@gmail.com> wrote:
That one is multistage, great feature, but I think cache mounts would work fine with any number of stages. The mount option is a pretty new feature.
On Tue, May 7, 2019, 12:24 Wes Turner <wes.turner@gmail.com> wrote:
On Tuesday, May 7, 2019, Daniel Holth <dholth@gmail.com> wrote:
Have you tried using buildkit and the RUN --mount option? I've done extra stuff here (downloading rpms in a second image first) but I think you could just use the cache option.
It would also be easy to use a second image and COPY in old docker.
https://github.com/dholth/vagrant-docker/blob/master/Dockerfile#L86
Thanks. Is this a multi-stage Dockerfile (with multiple FROM instructions)?
Is there a guide for a PyPA recommended way to create network-efficient reproducible (manylinux2010) wheels?
On Tue, May 7, 2019, 11:59 Alex Becker <alcubecker@gmail.com> wrote:
You can use a local PyPI mirror, e.g. devpi, and point your docker builds at that, basically tricking docker by going through the (local) network stack instead of the filesystem.
On Tue, May 7, 2019 at 8:12 AM Wes Turner <wes.turner@gmail.com> wrote:
What is the best way to build docker images without constantly re-downloading packages from PyPI (to use ~O(1) bandwidth instead of O(n) for every build)
(AFAIK, nobody has any issue with the amount of bandwidth PyPI uses)
Thus far, Docker doesn't want to support a build-time -v option (that could be used to bind-mount .pip/cache in at build time):
"build time only -v option" https://github.com/moby/moby/issues/14080
Buildah *does* support a build-time -v option (and can also do rootless builds without a docker socket) -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/IL6MG...
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/2MNMG...
--
Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/JXVF2...
participants (4)
-
Alex Becker -
C Anthony Risinger -
Daniel Holth -
Wes Turner