Bootstrap Containers

Setting up the host with containers that start during boot.

Bootstrap containers are a form of host container that you can use to run critical software before the node connects to an orchestrator. They give you substantial power to configure and modify the system in ways that would otherwise be difficult or impossible.

Lifecycle

When you define a bootstrap container using user-data or the Bottlerocket API, the following sequence occurs on next boot:

  • systemd starts all of the bootstrap containers concurrently,
  • systemd will not move to the next target until all of the bootstrap containers start, run, and exit,
  • any bootstrap container configured with mode=once will change to mode=off once complete,
  • any bootstrap container set to essential=true that exits with an non-zero exit code halts the boot process,
  • unless stopped in the previous step, the boot process continues by loading the orchestrator agent ( kubelet on *-k8s-* variants or the ECS agent on *-ecs-* variants).

Examples

systemdbootstrap containersB1B2B3B4next boot stage

This example shows four bootstrap containers (B1,B2,B3, and B4) starting. The start order of these containers is not sequential and could be in any order. Container B2 runs the longest and the next stage of the boot does not proceed until it completes.

systemdbootstrap containersB1B2essential = trueB3B4boot haltexit code 1

This example is the same as the previous one except container B2 is configured with essential=true. Container B2 exits with a non-zero exit code and consequently the boot halts.

systemdbootstrap containersB1B2B3B4mode=onceUpdate containerB4 mode to offnext boot stage

This example is the same as the first except container B4 is configured with mode=once. After all the bootstrap containers finish, Bottlerocket (via a systemd unit) updates container B4’s mode to off.

Considerations

As a consequence of this lifecycle, you should keep a few things in mind when using bootstrap containers:

  1. Bootstrap containers should be short-running and cleanly exit when complete otherwise the node will hang.
  2. Since systemd starts all the bootstrap containers concurrently, you cannot rely on a deterministic starting order.
  3. Bootstrap containers do not have access to the container orchestrator nor the rest of the cluster.
  4. Bootstrap containers should not modify critical configuration files (e.g. /etc/).

Capabilities and permissions

Bootstrap containers sit somewhat between default host containers and superpowered host container permissions. They have access to the underlying host filesystem at /.bottlerocket/ which contains the root filesystem (/.bottlerocket/rootfs). Additionally, bootstrap containers run with the CAP_SYS_ADMIN capability, allowing for the creation of files, directories, and mounts accessible to the host (however the root filesystem remains immutable). You cannot elevate the permissions of bootstrap containers.

Use cases

Bootstrap containers have a variety of uses.

  1. Configure large setting values. On many platforms there are limits to the size of user-data but you can use a bootstrap container to pass large values into apiclient.
  2. Conditionally halt boot. You can perform checks or evaluation of the system in a bootstrap container with `essential` set to `true`.
  3. Apply settings you do not want to store in plaintext. In cases where you need to pass values that you would not want to store in plain-text on external systems (such as credentials), you can instead use a bootstrap container that sets the values via apiclient.
  4. Configure ephemeral disks. Since bootstrap containers have access to /.bottlerocket/rootfs/mnt and all bootstrap or superpowered host containers share this bind mount, you can configure ephemeral disks attached to your host in a bootstrap container and use it elsewhere.

Also see

See a problem with this page? File an issue. All feedback is appreciated.
You can also directly contribute a change to the source file of this page on GitHub.