Docker — Community Edition (CE)

Containers

Published

January 31, 2025

Modified

March 5, 2025

Docker Community Edition (Docker CE) — Free and open-source version of Docker

…uses the DockerHub¹ container registry by default (rate limits apply)
Commercial: Docker Enterprise Edition (Docker EE)
- …official technical support or guaranteed response times for issues
- …role-based access control (RBAC), image signing, and vulnerability scanning
- …advanced orchestration features …multi-host networking …enterprise authentication …automatic scaling of containers based on load or demand
- …advanced management and monitoring tools for container performance
- …advanced integration with CI/CD tools

Installation

Install Docker from the official package repository²:

# Enterprise Linux 9
sudo dnf remove -y podman-docker
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf install -y git docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

# Fedora >=41
sudo dnf remove -y podman-docker
sudo dnf config-manager addrepo --from-repofile=https://download.docker.com/linux/fedora/docker-ce.repo
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

Configuration

Configure Docker…

…CLI in ~/.docker/config.json …environment variable DOCKER_CONFIG=path/to/dir
…daemon in /etc/docker/daemon.json …or Systemd drop-in

HTTP Proxy

Docker daemon:

mkdir -p /etc/systemd/system/docker.service.d
cat > /etc/systemd/system/docker.service.d/http-proxy.conf <<EOF
[Service]
Environment="HTTPS_PROXY=$HTTPS_PROXY"
EOF
systemctl daemon-reload && systemctl restart docker.service

Docker client:

# building a container
docker build --build-arg HTTPS_PROXY="$HTTPS_PROXY" .

# executing a container
docker run --env HTTPS_PROXY="$HTTPS_PROXY" #…

…persistent…

# proxy configurations for the Docker client 
cat > ~/.docker/config.json <<EOF
{
 "proxies": {
   "default": {
     "httpProxy": "$HTTP_PROXY",
     "httpsProxy": "$HTTPS_PROXY",
     "noProxy": "$NO_PROXY"
   }
 }
<}
EOF

# check functionality by printing the environment variables in a container instance
docker run --rm alpine sh -c 'env | grep -i  _PROXY'

Build

How to minimize container image size?

Use a minimal base image
- Distroless³ does not include a Linux distribution environment
- Alpine⁴ Linux distribution specifically for containers
- Stageˣ⁵ minimal, security first container images
Minify container images with extra tools: slim⁶
Minimize dependencies your application requires (libraries/packages)
Multi-stage builds

Instructions

Dockerfile — Set of instruction that define how to build a container image

Instruction	Description
`FROM`	Specify the container base images
`ENV`	Set environment variables
`ARG`	Define variables
`LABEL`	Image metadata …key-value pairs
`SHELL`	Default shell
`USER`	User to execute subsequent instructions
`WORKDIR`	Set directory to execute subsequent instructions
`RUN`	Executes a command in a new layer
`COPY`	Copies files or directories from the host
`ADD`	Like `COPY` …extract archive …download URLs
`ENTRYPOINT`	Command that runs when the container is started
`CMD`	Default arguments for the `ENTRYPOINT` instruction
`EXPOSE`	List runtime network ports (does not publish ports)
`VOLUME`	Mount external storage
`HEALTHCHECK`	How to test a container to check that it is still working
`STOPSIGNAL`	System call signal that will be sent to the container to stop
`ONBUILD`	Trigger instruction …when used as a base for another build

Build Context

Build context…

…typically the directory where the Dockerfile is located
…includes all files and directories referenced in the Dockerfile
…entire build context send to the Docker daemon (impacts build performance)

# build in the current working directory
docker build -t $app:$app_version .

.dockerignore file — Exclude unnecessary files and directories

…minimizes the number of files in the build context
…avoid large files, temporary files, or development artifacts
…organize the directory structure to streamline the build

Layer Cache

Container images are built in layers …explore with dive⁷:

…each layer represents a set of file-system changes
…once a layer is generated it becomes immutable
…each instruction in a Dockerfile creates a new layer
…each layer is cached …reused when possible to speed up build
Layer dependency…
- …each layer depends on the layers that precede it
- …layer invalidation forces a rebuild of all subsequent layers
- …changes to early layers have a cascading effect
Cache invalidation — Determines when one or more layers need rebuild
- COPY or ADD …lines that copy files into an image
- RUN and other instruction that modify the file-system

Best practices for cache optimization…

Maximize cache hits — Order instruction wisely
- …ordering your Dockerfile instructions strategically
- …place less frequently changed instructions at the top
- …use specific versions of dependencies
Layer squashing — Combine instructions to reduce invalidation
- …reduce the number of layers to minimize image size
- …for example combine multiple commands into a singe RUN instruction
Deleting files doesn’t remove them from earlier layers
- …marked as “not accessible” …still consume space
- …example: package install & clean up require a single RUN instruction

Multi-Stage

Multiple FROM statements — Define multiple stages in your Dockerfile

Why use multi-stage builds?⁸
- Separate the build environment from the runtime environment
- Significantly reduce the size of a final container image
- Improve security by excluding unnecessary build tools and dependencies
Each stage can use a different base…
- …by default stages aren’t named
- AS keyword — Give a stage a name…
  - …allows reference to it in a later stage
  - …the base of a FROM statement can be a previous stage
- COPY statement — Copy files from one stage to another
  - …option --from= used to identify a specific stage
  - …either by name or index (starts with zero)

FROM rockylinux:9 AS build
# …install build dependencies, build software/packages

FROM rockylinux:9 AS base
# copy artifactes from the build stage (packages in this example) 
COPY --from=build /root/rpmbuild/RPMS/**/*.rpm /tmp

FROM base AS #…
#…install & configure application

Build a specific stage with docker build with option --target:

# use tags to distinguish build stages
docker build --target build -t $app:build .

# build final stage ready for production
docker build --target release -t $app:$app_version -t $app:latest .

Usage

Work with Docker container instances:

docker ps               # list running containers
docker stop $container  # stop running container
docker rm $container    # remove container instance

# ephemeral interactive container instance
docker run -it --rm $image:$tag

# overwrite the entrypoint
docker run -it --rm --entrypoint /bin/bash $image -c "echo hello"

Images

Docker commands to work with container images:

docker pull $image           # download container from registry
docker images                # list containers on local system
docker rmi $image            # remove a container from local system
docker inspect $image:$tag   # details a bout an image

# find container instances using an container image
docker container ls --all --filter=ancestor=$image_id --format "{{.ID}}"

Buildx

docker buildx ls                 # list builders
docker buildx inspect $name      # print details on a builder
docker buildx rm $name           # remove a builder
docker buildx use $name          # set builder as default

docker buildx — Use BuildKit⁹ to build container images…

…improved back-end to replace the legacy builder
…default for Docker Engine >=23.0
Features…
- …parallelize multi-stage builds with custom contexts
- …build images for different platforms/architectures

`docker buildx create`

# create a new build instance
docker buildx create $options --name $name #…

Common options for buildx create…

--driver — Configure where the BuildKit back-end runs…
- docker (default) BuildKit library bundled into the Docker daemon
- docker-container dedicated BuildKit container using Docker
- kubernetes creates BuildKit pods in a Kubernetes cluster
- remote connects directly to a manually managed BuildKit daemon
--driver-opt for the docker-container driver¹⁰
- network=host host network mode for the container
- image=moby/buildkit:latest¹¹ sets the image to use for the container

`docker buildx bake`

docker buildx bake          # build all targets
docker buildx bake $target  # build a specific target

Bake file¹² — Build images using a declarative approach

…define your build configuration & build arguments
…in a file typical docker-bake.hcl¹³ or docker-compose.yml
Property types…
- target specifies a build target
- group defines a collections of build targets
- variable sets build arguments
- function defines a custome Bake function

docker-bake.hcl

variable "APP_VERSION" { default = "24.11.1" }

variable "REGISTRY" {
    default = "containers.example.org"
}

target "default" {
  matrix = {
    stage = [
      "slurmctld",
      "slurmdbd",
      "slurmd",
      "slurmrestd",
      "sackd"
    ]
  }
  context    = "."
  dockerfile = "Dockerfile"
  output     = ["type=registry"]
  name       = "${stage}"
  target     = stage
  tags       = [ "${REGISTRY}/slurm/${stage}:${APP_VERSION}" ]
}

Clean Up

docker system df                        # check disk usage of all resources

docker stop $(docker ps -aq)            # stop all running containers
docker rm $(docker ps -aq)              # remove all containers
docker rmi $(docker images -q)          # remove all container images
docker volume rm $(docker volume ls -q) # remove all volumes

docker container prune                  # remove all stopped containers
docker image prune                      # remove dangling images
docker image prune -a                   # remove all images (not just dangling)
docker network prun                     # remove unused networks
docker volume prune                     # remove unused volumes

# removes unused data, including stopped containers
# unused networks, and dangling images
docker system prune --volumes

Dangling images — Images not tagged and not referenced by any container
Option -f — Forcefully remove containers/images even if running/used

Footnotes

DockerHub
https://hub.docker.com ↩︎
Official Docker Packages
https://download.docker.com/linux ↩︎
“Distroless” Container Images, Google
https://github.com/GoogleContainerTools/distroless ↩︎
Alpine, GitHub
https://github.com/alpinelinux/docker-alpine ↩︎
Stageˣ Project
https://stagex.tools/about
https://codeberg.org/stagex/stagex ↩︎
slim roolkit, GitHub
https://github.com/slimtoolkit/slim ↩︎
dive Project, GitHub
https://github.com/wagoodman/dive ↩︎
Multi-Stage Builds, Docker Documentation
https://docs.docker.com/build/building/multi-stage ↩︎
BuildKit, Docker
https://docs.docker.com/build/buildkit
https://github.com/moby/buildkit ↩︎
Docker Container Build Driver
https://docs.docker.com/build/builders/drivers/docker-container ↩︎
BuildKit Container Image, DockerHub
https://hub.docker.com/r/moby/buildkit ↩︎
Bake file reference, Docker Documentation
https://docs.docker.com/build/bake/reference ↩︎
HCL (HashiCorp configuration language), GitHub
https://github.com/hashicorp/hcl ↩︎

--- title: 'Docker — Community Edition (CE)' categories: - Containers date: 2025/01/31 date-modified: 2025/03/05 toc-expand: 3 tbl-colwidths: [25,75] --- Docker Community Edition (Docker CE) — Free and open-source version of Docker - **…uses the DockerHub[^Q9X9l] container registry by default (rate limits apply)** - Commercial: Docker Enterprise Edition (Docker EE) - …official technical support or guaranteed response times for issues - …role-based access control (RBAC), image signing, and vulnerability scanning - …advanced orchestration features …multi-host networking …enterprise authentication …automatic scaling of containers based on load or demand - …advanced management and monitoring tools for container performance - …advanced integration with CI/CD tools [^Q9X9l]: DockerHub <https://hub.docker.com> ## Installation Install Docker from the official package repository[^JET2K]: [^JET2K]: Official Docker Packages <https://download.docker.com/linux> ```bash # Enterprise Linux 9 sudo dnf remove -y podman-docker sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo sudo dnf install -y git docker-ce docker-ce-cli containerd.io docker-compose-plugin sudo systemctl enable --now docker sudo usermod -aG docker $USER # Fedora >=41 sudo dnf remove -y podman-docker sudo dnf config-manager addrepo --from-repofile=https://download.docker.com/linux/fedora/docker-ce.repo sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin sudo systemctl enable --now docker sudo usermod -aG docker $USER ``` # Configuration Configure Docker… - …CLI in `~/.docker/config.json` …environment variable `DOCKER_CONFIG=path/to/dir` - …daemon in `/etc/docker/daemon.json` …or Systemd drop-in ## HTTP Proxy Docker **daemon**: ```bash mkdir -p /etc/systemd/system/docker.service.d cat > /etc/systemd/system/docker.service.d/http-proxy.conf <<EOF [Service] Environment="HTTPS_PROXY=$HTTPS_PROXY" EOF systemctl daemon-reload && systemctl restart docker.service ``` Docker **client**: ```bash # building a container docker build --build-arg HTTPS_PROXY="$HTTPS_PROXY" . # executing a container docker run --env HTTPS_PROXY="$HTTPS_PROXY" #… ``` …persistent… ```bash # proxy configurations for the Docker client cat > ~/.docker/config.json <<EOF { "proxies": { "default": { "httpProxy": "$HTTP_PROXY", "httpsProxy": "$HTTPS_PROXY", "noProxy": "$NO_PROXY" } } <} EOF # check functionality by printing the environment variables in a container instance docker run --rm alpine sh -c 'env | grep -i _PROXY' ``` # Build _How to minimize container image size?_ - Use a minimal base image - Distroless[^sEObv] does not include a Linux distribution environment - Alpine[^abguP] Linux distribution specifically for containers - Stageˣ[^uH4aM] minimal, security first container images - Minify container images with extra tools: `slim`[^DOVdg] - Minimize dependencies your application requires (libraries/packages) - Multi-stage builds [^sEObv]: "Distroless" Container Images, Google <https://github.com/GoogleContainerTools/distroless> [^abguP]: Alpine, GitHub <https://github.com/alpinelinux/docker-alpine> [^DOVdg]: `slim` roolkit, GitHub <https://github.com/slimtoolkit/slim> [^uH4aM]: Stageˣ Project <https://stagex.tools/about> <https://codeberg.org/stagex/stagex> ## Instructions `Dockerfile` — **Set of instruction that define how to build a container image** Instruction | Description -------------|--------------- `FROM` | Specify the container base images `ENV` | Set environment variables `ARG` | Define variables `LABEL` | Image metadata …key-value pairs `SHELL` | Default shell `USER` | User to execute subsequent instructions `WORKDIR` | Set directory to execute subsequent instructions `RUN` | Executes a command in a new layer `COPY` | Copies files or directories from the host `ADD` | Like `COPY` …extract archive …download URLs `ENTRYPOINT` | Command that runs when the container is started `CMD` | Default arguments for the `ENTRYPOINT` instruction `EXPOSE` | List runtime network ports (does not publish ports) `VOLUME` | Mount external storage `HEALTHCHECK`| How to test a container to check that it is still working `STOPSIGNAL` | System call signal that will be sent to the container to stop `ONBUILD` | Trigger instruction …when used as a base for another build ## Build Context Build context… - …typically the directory where the `Dockerfile` is located - …includes all files and directories referenced in the `Dockerfile` - …entire build context send to the Docker daemon (impacts build performance) ```bash # build in the current working directory docker build -t $app:$app_version . ``` **`.dockerignore` file** — Exclude unnecessary files and directories - …minimizes the number of files in the build context - …avoid large files, temporary files, or development artifacts - …organize the directory structure to streamline the build ## Layer Cache Container images are built in layers …explore with `dive`[^6PFA7]: [^6PFA7]: `dive` Project, GitHub <https://github.com/wagoodman/dive> - …each layer represents a set of file-system changes - …once a layer is generated it becomes immutable - …each **instruction in a `Dockerfile` creates a new layer** - …each layer is cached …reused when possible to speed up build - Layer dependency… - …each layer depends on the layers that precede it - …layer invalidation forces a rebuild of all subsequent layers - …changes to early layers have a cascading effect - **Cache invalidation** — Determines when one or more layers need rebuild - `COPY` or `ADD` …lines that copy files into an image - `RUN` and other instruction that modify the file-system Best practices for cache optimization… - **Maximize cache hits** — Order instruction wisely - …ordering your `Dockerfile` instructions strategically - …place less frequently changed instructions at the top - …use specific versions of dependencies - **Layer squashing** — Combine instructions to reduce invalidation - …reduce the number of layers to minimize image size - …for example combine multiple commands into a singe `RUN` instruction - **Deleting files doesn't remove them from earlier layers** - …marked as "not accessible" …still consume space - …example: package install & clean up require a single `RUN` instruction ## Multi-Stage Multiple **`FROM` statements** — Define multiple stages in your `Dockerfile` - _Why use multi-stage builds?_[^ojb6U] - Separate the build environment from the runtime environment - Significantly reduce the size of a final container image - Improve security by excluding unnecessary build tools and dependencies - Each stage can use a different base… - …by default stages aren't named - **`AS` keyword** — Give a stage a name… - …allows reference to it in a later stage - …the base of a `FROM` statement can be a previous stage - **`COPY` statement** — Copy files from one stage to another - …option `--from=` used to identify a specific stage - …either by name or index (starts with zero) ```Dockerfile FROM rockylinux:9 AS build # …install build dependencies, build software/packages FROM rockylinux:9 AS base # copy artifactes from the build stage (packages in this example) COPY --from=build /root/rpmbuild/RPMS/**/*.rpm /tmp FROM base AS #… #…install & configure application ``` Build a specific stage with **`docker build` with option `--target`**: ```bash # use tags to distinguish build stages docker build --target build -t $app:build . # build final stage ready for production docker build --target release -t $app:$app_version -t $app:latest . ``` [^ojb6U]: Multi-Stage Builds, Docker Documentation <https://docs.docker.com/build/building/multi-stage> # Usage Work with Docker container instances: ```bash docker ps # list running containers docker stop $container # stop running container docker rm $container # remove container instance # ephemeral interactive container instance docker run -it --rm $image:$tag # overwrite the entrypoint docker run -it --rm --entrypoint /bin/bash $image -c "echo hello" ``` ## Images Docker commands to work with container images: ```bash docker pull $image # download container from registry docker images # list containers on local system docker rmi $image # remove a container from local system docker inspect $image:$tag # details a bout an image # find container instances using an container image docker container ls --all --filter=ancestor=$image_id --format "{{.ID}}" ``` ## Buildx ```bash docker buildx ls # list builders docker buildx inspect $name # print details on a builder docker buildx rm $name # remove a builder docker buildx use $name # set builder as default ``` **`docker buildx`** — Use BuildKit[^O8QBi] to build container images… - …improved back-end to replace the legacy builder - …default for Docker Engine >=23.0 - Features… - …parallelize multi-stage builds with custom contexts - …build images for different platforms/architectures [^O8QBi]: BuildKit, Docker <https://docs.docker.com/build/buildkit> <https://github.com/moby/buildkit> ### `docker buildx create` ```bash # create a new build instance docker buildx create $options --name $name #… ``` Common options for `buildx create`… - **`--driver`** — Configure where the BuildKit back-end runs… - `docker` (default) BuildKit library bundled into the Docker daemon - `docker-container` dedicated BuildKit container using Docker - `kubernetes` creates BuildKit pods in a Kubernetes cluster - `remote` connects directly to a manually managed BuildKit daemon - **`--driver-opt`** for the `docker-container` driver[^MRK3Z] - `network=host` host network mode for the container - `image=moby/buildkit:latest`[^zhdSl] sets the image to use for the container [^MRK3Z]: Docker Container Build Driver <https://docs.docker.com/build/builders/drivers/docker-container> [^zhdSl]: BuildKit Container Image, DockerHub <https://hub.docker.com/r/moby/buildkit> ### `docker buildx bake` ```bash docker buildx bake # build all targets docker buildx bake $target # build a specific target ``` **Bake file**[^bDifB] — Build images using a declarative approach [^bDifB]: Bake file reference, Docker Documentation <https://docs.docker.com/build/bake/reference> - …define your build configuration & build arguments - …in a file **typical `docker-bake.hcl`**[^66ksp] or `docker-compose.yml` - Property types… - `target` specifies a build target - `group` defines a collections of build targets - `variable` sets build arguments - `function` defines a custome Bake function [^66ksp]: HCL (HashiCorp configuration language), GitHub <https://github.com/hashicorp/hcl> ```{.go filename=docker-bake.hcl} variable "APP_VERSION" { default = "24.11.1" } variable "REGISTRY" { default = "containers.example.org" } target "default" { matrix = { stage = [ "slurmctld", "slurmdbd", "slurmd", "slurmrestd", "sackd" ] } context = "." dockerfile = "Dockerfile" output = ["type=registry"] name = "${stage}" target = stage tags = [ "${REGISTRY}/slurm/${stage}:${APP_VERSION}" ] } ``` ## Clean Up ```bash docker system df # check disk usage of all resources docker stop $(docker ps -aq) # stop all running containers docker rm $(docker ps -aq) # remove all containers docker rmi $(docker images -q) # remove all container images docker volume rm $(docker volume ls -q) # remove all volumes docker container prune # remove all stopped containers docker image prune # remove dangling images docker image prune -a # remove all images (not just dangling) docker network prun # remove unused networks docker volume prune # remove unused volumes # removes unused data, including stopped containers # unused networks, and dangling images docker system prune --volumes ``` - **Dangling images** — Images not tagged and not referenced by any container - **Option `-f`** — Forcefully remove containers/images even if running/used