FAQ

How do I regain access to a Bottlerocket node after disabling the control and admin containers?

There is a way to regain access to a Bottlerocket node in such a state. See the Regaining Access documentation for further details.

How do I SSH into Bottlerocket?

Bottlerocket’s control container includes the capability for accessing the host through SSM. SSM is the preferred way to access a Bottlerocket node.

If SSM does not work in your use case, there is a way to access the host through SSH by using the admin container.

How do I use kubectl debug node with Bottlerocket?

The debug node option from kubectl can be used to gain access to a Bottlerocket node as follows:

kubectl debug node/<node name> -it --image=ubuntu --profile=sysadmin

This is a useful alternative to SSH or SSM. Use a kubectl version of 1.30 or later to ensure that the sysadmin profile is available. Once the debug pod is started and you have a shell, you can run chroot /host apiclient exec admin bash to access the admin container (if enabled) or chroot /host <command> to run commands from the Bottlerocket OS.

The Pod created is privileged so ensure it is removed after use.

How do I perform fault injection with Bottlerocket?

Fault injection testing on a Bottlerocket cluster is not different from testing on a cluster running another operating system. You can use fault injection tools such as those in Istio or Chaos Mesh.

Alternatively, if you are using EKS, you can use AWS Fault Injection Simulator (FIS). In order to use AWS FIS, you need SSM access to the node, which is available through the control container.

What outbound endpoints need to be permitted to run Bottlerocket on EKS?

You may wish to limit outbound access for your EKS nodes. Bottlerocket needs ECR (even if you use another registry, we have images on ECR), EKS, IMDS, SSM, and possibly STS endpoints.

See Bottlerocket Without Internet Access.

Why is my allocatable memory lower than expected?

You may observe that allocatable memory is lower than on general-purpose Linux distributions. There are 2 possible reasons for this:

• As the control container runs in a second instance of containerd, a small amount of memory is reserved for this process. Typically this is just a few MiB.

• If you increase the maxPods setting on the kubelet, you may see a linear decrease in allocatable memory proportional to the max number of pods. The formula for this reservation is detailed in the Settings Reference. This may be different to the calculation used by other Linux distributions. This setting can be overridden by providing a static value to the settings user data.

How do I clean up old container images from Bottlerocket?

You should not need to manually clean up container images from Bottlerocket, this is handled automatically by the orchestrator agent.

On Kubernetes variants, kubelet manages the container image clean up, the settings settings.kubernetes.image-gc-high-threshold-percent and settings.kubernetes.image-gc-low-threshold-percent allow you to control how this clean up occurs.

On ECS variants, the ECS Agent manages the container image clean up, the settings settings.ecs.image-cleanup-enabled, settings.ecs.image-cleanup-age, settings.ecs.image-cleanup-delete-per-cycle, and settings.ecs.image-cleanup-wait allow you to control how this clean up occurs.

If you want to manually clean up images, log into the admin container and use ctr’s image commands with the address pointed at /run/dockershim.sock. For example, on Kubernetes variants to list all images:

sudo chroot /.bottlerocket/rootfs ctr -n k8s.io image ls

From here you can decide which images you want to manually remove.

How do I manage an on-prem Bottlerocket node similar to how I manage a cloud Bottlerocket node?

If you are using the control container, you can use SSM to manage on-prem Bottlerocket nodes by passing in your own activation information for SSM.

Otherwise, you are in control of what host containers you have running and accessible. Host containers have the apiclient binary available. This allows you to use custom host containers to manage a Bottlerocket node in a similar fashion to the control container.

How do I disable Secure Boot?

If you have a specific reason to disable Secure Boot in Bottlerocket, you will need to make infrastructure-level changes which vary by platform.

• aws-* variants: Re-register the AMI without UEFI metadata.
• vmware-* variants: configure the virtual machine to disable Secure Boot.

How do I scan my cluster for software vulnerabilities?

The ability to scan your hosts for vulnerabilities is included with Bottlerocket by enabling the control container in order to utilize Amazon Inspector. Amazon Inspector works via the SSM agent, which is included in the control container.

I heard Bottlerocket is immutable. What does that really mean?

There are both immutable and mutable areas on a Bottlerocket host.

dm-verity is used for Bottlerocket’s root filesystem, meaning it is read-only. More details about the immutable filesystem are available in the "Immutable Filesystem" section of the Restricted Filesystem documentation.

On the other hand, the non-root filesystem uses SELinux to protect files at a granular level. There are rules and policies which determine the mutability of different areas of the non-root filesystem. With sufficient privilege, a user can modify the SELinux labels of a file or resource. More details about the mutability of the non-root filesystem are available in the "Mutable Filesystem" section of the Restricted Filesystem documentation.

What language does Bottlerocket use for settings?

When defining settings in user data, Bottlerocket uses TOML. Some settings have examples available for reference, such as the container image registry settings.

How do I access logs for bootstrap containers?

Bootstrap container output should be present in the system journal. You can access the system journal from the admin container as follows:

sudo chroot /.bottlerocket/rootfs journalctl

My FSx Lustre file system is not responsive. How can I troubleshoot it?

Bottlerocket has supported FSx for Lustre on all AWS variants since v1.19.3. If you find that your Lustre filesystem is unresponsive, the discussion here may help: Troubleshooting FSx for Lustre on Bottlerocket

How long do variants receive updates?

Each variant has specific dependency constraints which dictate how long a variant can be reasonably patched. See the Security Features document in the Bottlerocket GitHub repo for more information.

Why are my nodes egressing to updates.bottlerocket.aws?

The Bottlerocket Updater API uses TUF metadata served from a public endpoint. The default AWS variants endpoint is updates.bottlerocket.aws.

Why do some of the nodes in my cluster have an update available and others do not?

This is normal. Bottlerocket uses “waves” to stagger deployment of updates. When a node starts for the first time, the boot process generates a random seed (or uses the value from settings.updates.seed). Bottlerocket’s update process uses the seed to determine if a node should update, so in the situation where some of your nodes have an available update and some do not, it just means that the update wave hasn’t reached that seed of some nodes and it has for the others.