Configure BuildKit

If you create a docker-container or kubernetes builder with Buildx, you can apply a custom BuildKit configuration by passing the --config flag to the docker buildx create command.

Registry mirror

You can define a registry mirror to use for your builds. Doing so redirects BuildKit to pull images from a different hostname. The following steps exemplify defining a mirror for docker.io (Docker Hub) to mirror.gcr.io .

1. Create a TOML at /etc/buildkitd.toml with the following content:

   debug = true
   [registry."docker.io"]
     mirrors = ["mirror.gcr.io"]
   

   Note

   debug = true turns on debug requests in the BuildKit daemon, which logs a message that shows when a mirror is being used.

2. Create a docker-container builder that uses this BuildKit configuration:

   $ docker buildx create --use --bootstrap \
     --name mybuilder \
     --driver docker-container \
     --config /etc/buildkitd.toml
   

3. Build an image:

   docker buildx build --load . -f - <<EOF
   FROM alpine
   RUN echo "hello world"
   EOF
   

The BuildKit logs for this builder now shows that it uses the GCR mirror. You can tell by the fact that the response messages include the x-goog-* HTTP headers.

$ docker logs buildx_buildkit_mybuilder0

...
time="2022-02-06T17:47:48Z" level=debug msg="do request" request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.8+unknown request.method=GET spanID=9460e5b6e64cec91 traceID=b162d3040ddf86d6614e79c66a01a577
time="2022-02-06T17:47:48Z" level=debug msg="fetch response received" response.header.accept-ranges=bytes response.header.age=1356 response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.cache-control="public, max-age=3600" response.header.content-length=1469 response.header.content-type=application/octet-stream response.header.date="Sun, 06 Feb 2022 17:25:17 GMT" response.header.etag="\"774380abda8f4eae9a149e5d5d3efc83\"" response.header.expires="Sun, 06 Feb 2022 18:25:17 GMT" response.header.last-modified="Wed, 24 Nov 2021 21:07:57 GMT" response.header.server=UploadServer response.header.x-goog-generation=1637788077652182 response.header.x-goog-hash="crc32c=V3DSrg==" response.header.x-goog-hash.1="md5=d0OAq9qPTq6aFJ5dXT78gw==" response.header.x-goog-metageneration=1 response.header.x-goog-storage-class=STANDARD response.header.x-goog-stored-content-encoding=identity response.header.x-goog-stored-content-length=1469 response.header.x-guploader-uploadid=ADPycduqQipVAXc3tzXmTzKQ2gTT6CV736B2J628smtD1iDytEyiYCgvvdD8zz9BT1J1sASUq9pW_ctUyC4B-v2jvhIxnZTlKg response.status="200 OK" spanID=9460e5b6e64cec91 traceID=b162d3040ddf86d6614e79c66a01a577
time="2022-02-06T17:47:48Z" level=debug msg="fetch response received" response.header.accept-ranges=bytes response.header.age=760 response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.cache-control="public, max-age=3600" response.header.content-length=1471 response.header.content-type=application/octet-stream response.header.date="Sun, 06 Feb 2022 17:35:13 GMT" response.header.etag="\"35d688bd15327daafcdb4d4395e616a8\"" response.header.expires="Sun, 06 Feb 2022 18:35:13 GMT" response.header.last-modified="Wed, 24 Nov 2021 21:07:12 GMT" response.header.server=UploadServer response.header.x-goog-generation=1637788032100793 response.header.x-goog-hash="crc32c=aWgRjA==" response.header.x-goog-hash.1="md5=NdaIvRUyfar8201DleYWqA==" response.header.x-goog-metageneration=1 response.header.x-goog-storage-class=STANDARD response.header.x-goog-stored-content-encoding=identity response.header.x-goog-stored-content-length=1471 response.header.x-guploader-uploadid=ADPycdtR-gJYwC7yHquIkJWFFG8FovDySvtmRnZBqlO3yVDanBXh_VqKYt400yhuf0XbQ3ZMB9IZV2vlcyHezn_Pu3a1SMMtiw response.status="200 OK" spanID=9460e5b6e64cec91 traceID=b162d3040ddf86d6614e79c66a01a577
time="2022-02-06T17:47:48Z" level=debug msg=fetch spanID=9460e5b6e64cec91 traceID=b162d3040ddf86d6614e79c66a01a577
time="2022-02-06T17:47:48Z" level=debug msg=fetch spanID=9460e5b6e64cec91 traceID=b162d3040ddf86d6614e79c66a01a577
time="2022-02-06T17:47:48Z" level=debug msg=fetch spanID=9460e5b6e64cec91 traceID=b162d3040ddf86d6614e79c66a01a577
time="2022-02-06T17:47:48Z" level=debug msg=fetch spanID=9460e5b6e64cec91 traceID=b162d3040ddf86d6614e79c66a01a577
time="2022-02-06T17:47:48Z" level=debug msg="do request" request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.8+unknown request.method=GET spanID=9460e5b6e64cec91 traceID=b162d3040ddf86d6614e79c66a01a577
time="2022-02-06T17:47:48Z" level=debug msg="fetch response received" response.header.accept-ranges=bytes response.header.age=1356 response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"" response.header.cache-control="public, max-age=3600" response.header.content-length=2818413 response.header.content-type=application/octet-stream response.header.date="Sun, 06 Feb 2022 17:25:17 GMT" response.header.etag="\"1d55e7be5a77c4a908ad11bc33ebea1c\"" response.header.expires="Sun, 06 Feb 2022 18:25:17 GMT" response.header.last-modified="Wed, 24 Nov 2021 21:07:06 GMT" response.header.server=UploadServer response.header.x-goog-generation=1637788026431708 response.header.x-goog-hash="crc32c=ZojF+g==" response.header.x-goog-hash.1="md5=HVXnvlp3xKkIrRG8M+vqHA==" response.header.x-goog-metageneration=1 response.header.x-goog-storage-class=STANDARD response.header.x-goog-stored-content-encoding=identity response.header.x-goog-stored-content-length=2818413 response.header.x-guploader-uploadid=ADPycdsebqxiTBJqZ0bv9zBigjFxgQydD2ESZSkKchpE0ILlN9Ibko3C5r4fJTJ4UR9ddp-UBd-2v_4eRpZ8Yo2llW_j4k8WhQ response.status="200 OK" spanID=9460e5b6e64cec91 traceID=b162d3040ddf86d6614e79c66a01a577
...

Setting registry certificates

If you specify registry certificates in the BuildKit configuration, the daemon copies the files into the container under /etc/buildkit/certs . The following steps show adding a self-signed registry certificate to the BuildKit configuration.

1. Add the following configuration to /etc/buildkitd.toml :

   # /etc/buildkitd.toml
   debug = true
   [registry."myregistry.com"]
     ca=["/etc/certs/myregistry.pem"]
     [[registry."myregistry.com".keypair]]
       key="/etc/certs/myregistry_key.pem"
       cert="/etc/certs/myregistry_cert.pem"
   

   This tells the builder to push images to the myregistry.com registry using the certificates in the specified location ( /etc/certs ).

2. Create a docker-container builder that uses this configuration:

   $ docker buildx create --use --bootstrap \
     --name mybuilder \
     --driver docker-container \
     --config /etc/buildkitd.toml
   

3. Inspect the builder’s configuration file ( /etc/buildkit/buildkitd.toml ), it shows that the certificate configuration is now configured in the builder.

   $ docker exec -it buildx_buildkit_mybuilder0 cat /etc/buildkit/buildkitd.toml
   

   debug = true
   
   [registry]
   
     [registry."myregistry.com"]
       ca = ["/etc/buildkit/certs/myregistry.com/myregistry.pem"]
   
       [[registry."myregistry.com".keypair]]
         cert = "/etc/buildkit/certs/myregistry.com/myregistry_cert.pem"
         key = "/etc/buildkit/certs/myregistry.com/myregistry_key.pem"
   

4. Verify that the certificates are inside the container:

   $ docker exec -it buildx_buildkit_mybuilder0 ls /etc/buildkit/certs/myregistry.com/
   myregistry.pem    myregistry_cert.pem   myregistry_key.pem
   

Now you can push to the registry using this builder, and it will authenticate using the certificates:

$ docker buildx build --push --tag myregistry.com/myimage:latest .

CNI networking

CNI networking for builders can be useful for dealing with network port contention during concurrent builds. CNI is not yet available in the default BuildKit image. But you can create your own image that includes CNI support.

The following Dockerfile example shows a custom BuildKit image with CNI support. It uses the CNI config for integration tests in BuildKit as an example. Feel free to include your own CNI configuration.

# syntax=docker/dockerfile:1

ARG BUILDKIT_VERSION=v{{ site.buildkit_version }}
ARG CNI_VERSION=v1.0.1

FROM --platform=$BUILDPLATFORM alpine AS cni-plugins
RUN apk add --no-cache curl
ARG CNI_VERSION
ARG TARGETOS
ARG TARGETARCH
WORKDIR /opt/cni/bin
RUN curl -Ls https://github.com/containernetworking/plugins/releases/download/$CNI_VERSION/cni-plugins-$TARGETOS-$TARGETARCH-$CNI_VERSION.tgz | tar xzv

FROM moby/buildkit:${BUILDKIT_VERSION}
ARG BUILDKIT_VERSION
RUN apk add --no-cache iptables
COPY --from=cni-plugins /opt/cni/bin /opt/cni/bin
ADD https://raw.githubusercontent.com/moby/buildkit/${BUILDKIT_VERSION}/hack/fixtures/cni.json /etc/buildkit/cni.json

Now you can build this image, and create a builder instance from it using the --driver-opt image option:

$ docker buildx build --tag buildkit-cni:local --load .
$ docker buildx create --use --bootstrap \
  --name mybuilder \
  --driver docker-container \
  --driver-opt "image=buildkit-cni:local" \
  --buildkitd-flags "--oci-worker-net=cni"

Resource limiting

Max parallelism

You can limit the parallelism of the BuildKit solver, which is particularly useful for low-powered machines, using a BuildKit configuration while creating a builder with the --config flags.

# /etc/buildkitd.toml
[worker.oci]
  max-parallelism = 4

Now you can create a docker-container builder that will use this BuildKit configuration to limit parallelism.

$ docker buildx create --use \
  --name mybuilder \
  --driver docker-container \
  --config /etc/buildkitd.toml

TCP connection limit

TCP connections are limited to 4 simultaneous connections per registry for pulling and pushing images, plus one additional connection dedicated to metadata requests. This connection limit prevents your build from getting stuck while pulling images. The dedicated metadata connection helps reduce the overall build time.

More information: moby/buildkit#2259

Custom Dockerfile syntax

Dockerfile frontend

BuildKit supports loading frontends dynamically from container images. To use an external Dockerfile frontend, the first line of your Dockerfile needs to set the syntax directive pointing to the specific image you want to use:

# syntax=[remote image reference]

For example:

# syntax=docker/dockerfile:1
# syntax=docker.io/docker/dockerfile:1
# syntax=example.com/user/repo:tag@sha256:abcdef...

This defines the location of the Dockerfile syntax that is used to build the Dockerfile. The BuildKit backend allows seamlessly using external implementations that are distributed as Docker images and execute inside a container sandbox environment.

Custom Dockerfile implementations allow you to:

• Automatically get bugfixes without updating the Docker daemon
• Make sure all users are using the same implementation to build your Dockerfile
• Use the latest features without updating the Docker daemon
• Try out new features or third-party features before they are integrated in the Docker daemon
• Use alternative build definitions, or create your own

Note

BuildKit also ships with a built-in Dockerfile frontend, but it’s recommended to use an external image to make sure that all users use the same version on the builder and to pick up bugfixes automatically without waiting for a new version of BuildKit or Docker Engine.

Official releases

Docker distributes official versions of the images that can be used for building Dockerfiles under docker/dockerfile repository on Docker Hub. There are two channels where new images are released: stable and labs .

Stable channel

The stable channel follows semantic versioning. For example:

• docker/dockerfile:1 - kept updated with the latest 1.x.x minor and patch release.
• docker/dockerfile:1.2 - kept updated with the latest 1.2.x patch release, and stops receiving updates once version 1.3.0 is released.
• docker/dockerfile:1.2.1 - immutable: never updated.

We recommend using docker/dockerfile:1 , which always points to the latest stable release of the version 1 syntax, and receives both “minor” and “patch” updates for the version 1 release cycle. BuildKit automatically checks for updates of the syntax when performing a build, making sure you are using the most current version.

If a specific version is used, such as 1.2 or 1.2.1 , the Dockerfile needs to be updated manually to continue receiving bugfixes and new features. Old versions of the Dockerfile remain compatible with the new versions of the builder.

Labs channel

The labs channel provides early access to Dockerfile features that are not yet available in the stable channel. labs images are released at the same time as stable releases, and follow the same version pattern, but use the -labs suffix, for example:

• docker/dockerfile:labs - latest release on labs channel.
• docker/dockerfile:1-labs - same as dockerfile:1 , with experimental features enabled.
• docker/dockerfile:1.2-labs - same as dockerfile:1.2 , with experimental features enabled.
• docker/dockerfile:1.2.1-labs - immutable: never updated. Same as dockerfile:1.2.1 , with experimental features enabled.

Choose a channel that best fits your needs. If you want to benefit from new features, use the labs channel. Images in the labs channel contain all the features in the stable channel, plus early access features. Stable features in the labs channel follow semantic versioning, but early access features don’t, and newer releases may not be backwards compatible. Pin the version to avoid having to deal with breaking changes.

Other resources

For documentation on “labs” features, master builds, and nightly feature releases, refer to the description in the BuildKit source repository on GitHub. For a full list of available images, visit the docker/dockerfile repository on Docker Hub, and the docker/dockerfile-upstream repository on Docker Hub for development builds.

BuildKit

Overview

BuildKit is an improved backend to replace the legacy builder. It comes with new and much improved functionality for improving your builds’ performance and the reusability of your Dockerfiles. It also introduces support for handling more complex scenarios:

• Detect and skip executing unused build stages
• Parallelize building independent build stages
• Incrementally transfer only the changed files in your build context between builds
• Detect and skip transferring unused files in your build context
• Use Dockerfile frontend implementations with many new features
• Avoid side effects with rest of the API (intermediate images and containers)
• Prioritize your build cache for automatic pruning

Apart from many new features, the main areas BuildKit improves on the current experience are performance, storage management, and extensibility. From the performance side, a significant update is a new fully concurrent build graph solver. It can run build steps in parallel when possible and optimize out commands that don’t have an impact on the final result. We have also optimized the access to the local source files. By tracking only the updates made to these files between repeated build invocations, there is no need to wait for local files to be read or uploaded before the work can begin.

LLB

At the core of BuildKit is a Low-Level Build (LLB) definition format. LLB is an intermediate binary format that allows developers to extend BuildKit. LLB defines a content-addressable dependency graph that can be used to put together very complex build definitions. It also supports features not exposed in Dockerfiles, like direct data mounting and nested invocation.

Everything about execution and caching of your builds is defined in LLB. The caching model is entirely rewritten compared to the legacy builder. Rather than using heuristics to compare images, LLB directly tracks the checksums of build graphs and content mounted to specific operations. This makes it much faster, more precise, and portable. The build cache can even be exported to a registry, where it can be pulled on-demand by subsequent invocations on any host.

LLB can be generated directly using a golang client package that allows defining the relationships between your build operations using Go language primitives. This gives you full power to run anything you can imagine, but will probably not be how most people will define their builds. Instead, most users would use a frontend component, or LLB nested invocation, to run a prepared set of build steps.

Frontend

A frontend is a component that takes a human-readable build format and converts it to LLB so BuildKit can execute it. Frontends can be distributed as images, and the user can target a specific version of a frontend that is guaranteed to work for the features used by their definition.

For example, to build a Dockerfile with BuildKit, you would use an external Dockerfile frontend.

Getting started

BuildKit is enabled by default for all users on Docker Desktop. If you have installed Docker Desktop, you don’t have to manually enable BuildKit. If you are running Docker on Linux, you can enable BuildKit either by using an environment variable or by making BuildKit the default setting.

To set the BuildKit environment variable when running the docker build command, run:

$ DOCKER_BUILDKIT=1 docker build .

Note

Buildx always enables BuildKit.

To enable docker BuildKit by default, set daemon configuration in /etc/docker/daemon.json feature to true and restart the daemon. If the daemon.json file doesn’t exist, create new file called daemon.json and then add the following to the file.

{
  "features": {
    "buildkit" : true
  }
}

And restart the Docker daemon.

Warning

BuildKit only supports building Linux containers. Windows support is tracked in moby/buildkit#616

BuildKit TOML configuration

The TOML file used to configure the buildkitd daemon settings has a short list of global settings followed by a series of sections for specific areas of daemon configuration.

The file path is /etc/buildkit/buildkitd.toml for rootful mode, ~/.config/buildkit/buildkitd.toml for rootless mode.

The following is a complete buildkitd.toml configuration example, please note some configuration is only good for edge cases, please take care of it carefully.

debug = true
# root is where all buildkit state is stored.
root = "/var/lib/buildkit"
# insecure-entitlements allows insecure entitlements, disabled by default.
insecure-entitlements = [ "network.host", "security.insecure" ]

[grpc]
  address = [ "tcp://0.0.0.0:1234" ]
  # debugAddress is address for attaching go profiles and debuggers.
  debugAddress = "0.0.0.0:6060"
  uid = 0
  gid = 0
  [grpc.tls]
    cert = "/etc/buildkit/tls.crt"
    key = "/etc/buildkit/tls.key"
    ca = "/etc/buildkit/tlsca.crt"

# config for build history API that stores information about completed build commands
[history]
  # maxAge is the maximum age of history entries to keep, in seconds.
  maxAge = 172800
  # maxEntries is the maximum number of history entries to keep.
  maxEntries = 50

[worker.oci]
  enabled = true
  # platforms is manually configure platforms, detected automatically if unset.
  platforms = [ "linux/amd64", "linux/arm64" ]
  snapshotter = "auto" # overlayfs or native, default value is "auto".
  rootless = false # see docs/rootless.md for the details on rootless mode.
  # Whether run subprocesses in main pid namespace or not, this is useful for
  # running rootless buildkit inside a container.
  noProcessSandbox = false
  gc = true
  gckeepstorage = 9000
  # alternate OCI worker binary name(example 'crun'), by default either 
  # buildkit-runc or runc binary is used
  binary = ""
  # name of the apparmor profile that should be used to constrain build containers.
  # the profile should already be loaded (by a higher level system) before creating a worker.
  apparmor-profile = ""
  # limit the number of parallel build steps that can run at the same time
  max-parallelism = 4
  # maintain a pool of reusable CNI network namespaces to amortize the overhead
  # of allocating and releasing the namespaces
  cniPoolSize = 16

  [worker.oci.labels]
    "foo" = "bar"

  [[worker.oci.gcpolicy]]
    keepBytes = 512000000
    keepDuration = 172800
    filters = [ "type==source.local", "type==exec.cachemount", "type==source.git.checkout"]
  [[worker.oci.gcpolicy]]
    all = true
    keepBytes = 1024000000

[worker.containerd]
  address = "/run/containerd/containerd.sock"
  enabled = true
  platforms = [ "linux/amd64", "linux/arm64" ]
  namespace = "buildkit"
  gc = true
  # gckeepstorage sets storage limit for default gc profile, in MB.
  gckeepstorage = 9000
  # maintain a pool of reusable CNI network namespaces to amortize the overhead
  # of allocating and releasing the namespaces
  cniPoolSize = 16

  [worker.containerd.labels]
    "foo" = "bar"

  [[worker.containerd.gcpolicy]]
    keepBytes = 512000000
    keepDuration = 172800 # in seconds
    filters = [ "type==source.local", "type==exec.cachemount", "type==source.git.checkout"]
  [[worker.containerd.gcpolicy]]
    all = true
    keepBytes = 1024000000

# registry configures a new Docker register used for cache import or output.
[registry."docker.io"]
  # mirror configuration to handle path in case a mirror registry requires a /project path rather than just a host:port
  mirrors = ["yourmirror.local:5000", "core.harbor.domain/proxy.docker.io"]
  http = true
  insecure = true
  ca=["/etc/config/myca.pem"]
  [[registry."docker.io".keypair]]
    key="/etc/config/key.pem"
    cert="/etc/config/cert.pem"

# optionally mirror configuration can be done by defining it as a registry.
[registry."yourmirror.local:5000"]
  http = true

Azure Blob Storage cache

Warning

This cache backend is unreleased. You can use it today, by using the moby/buildkit:master image in your Buildx driver.

The azblob cache store uploads your resulting build cache to Azure’s blob storage service.

Note

This cache storage backend requires using a different driver than the default docker driver - see more information on selecting a driver here. To create a new driver (which can act as a simple drop-in replacement):

$ docker buildx create --use --driver=docker-container

Synopsis

$ docker buildx build --push -t <registry>/<image> \
  --cache-to type=azblob,name=<cache-image>[,parameters...] \
  --cache-from type=azblob,name=<cache-image>[,parameters...] .

The following table describes the available CSV parameters that you can pass to --cache-to and --cache-from .

Authentication

The secret_access_key , if left unspecified, is read from environment variables on the BuildKit server following the scheme for the Azure Go SDK. The environment variables are read from the server, not the Buildx client.

Further reading

For an introduction to caching see Optimizing builds with cache.

For more information on the azblob cache backend, see the BuildKit README.