kubernetes

Settings related to Kubernetes (settings.kubernetes.*)

Topic list

Setting list for settings.kubernetes

Topics

Node labels & taints

You can use these optional settings to customize the node labels and taints.

Settings

Settings needed for aws-k8s-* variants

Required settings for Kubernetes variants in AWS. These settings are typically specified in user data.

See the EKS Quickstart and the EKS setup guide for more details on setting up Bottlerocket and Kubernetes on Amazon EKS.

Settings

Settings needed for metal-k8s-* variants

Required settings for Kubernetes bare metal variants. These settings are typically specified in user data.

See metal provisioning guide for more information.

Settings

Settings needed for vmware-k8s-* variants

Required settings for Kubernetes VMware variants. These settings are typically specified in user data.

See the VMware setup guide for more information.

Settings

Static Pods & standalone mode

You can also optionally specify static pods for your node. Static pods can be particularly useful when running in standalone mode.

Settings

Full Reference

settings.kubernetes.allowed-unsafe-sysctls

Enables specified list of unsafe sysctls.

[settings.kubernetes]
allowed-unsafe-sysctls = ["net.core.somaxconn", "net.ipv4.ip_local_port_range"]
apiclient apply <<EOF
[settings.kubernetes]
allowed-unsafe-sysctls = ["net.core.somaxconn", "net.ipv4.ip_local_port_range"]
EOF
Also see: 

settings.kubernetes.api-server

The cluster’s Kubernetes API endpoint. This is typically specified in user data.

Also see: 

settings.kubernetes.authentication-mode

The authentication method kubelet should use to connect to the API server, and for incoming requests.

Default: aws for AWS variants, tls for other variants

Also see: 

settings.kubernetes.bootstrap-token

The token to use for TLS bootstrapping. Only used when settings.kubernetes.authentication-mode is set to tls (ignored otherwise).

Also see: 

settings.kubernetes.cloud-provider

The cloud provider for the cluster.

Default: aws for AWS variants, external for other variants

Also see: 

settings.kubernetes.cluster-certificate

The base64-encoded certificate authority of the cluster.

Also see: 

settings.kubernetes.cluster-dns-ip

The IP of the DNS service running in the cluster. On AWS variants, this is derived from the EKS Service IP CIDR or the CIDR block of the primary network interface. This value can be set as a string containing a single IP address, or as a list containing multiple IP addresses.

# Valid, single IP
[settings.kubernetes]
cluster-dns-ip = "10.0.0.1"
# Also valid, multiple nameserver IPs
[settings.kubernetes]
cluster-dns-ip = ["10.0.0.1", "10.0.0.2"]
Valid, single IP
apiclient set settings.kubernetes.cluster-dns-ip="10.0.0.1"
Also valid, multiple nameserver IPs
apiclient apply <<EOF
[settings.kubernetes]
cluster-dns-ip = ["10.0.0.1", "10.0.0.2"]
EOF
Also see: 

settings.kubernetes.cluster-domain

The DNS domain for the cluster, allowing all Kubernetes-run containers to search this domain before the host’s search domains

Default: cluster.local

Also see: 

settings.kubernetes.cluster-name

The cluster name you chose during setup.

Also see: 

settings.kubernetes.container-log-max-files

The maximum number of container log files that can be present for a container.

Also see: 

settings.kubernetes.container-log-max-size

The maximum size of container log file before it is rotated.

Also see: 

settings.kubernetes.cpu-cfs-quota-enforced

Whether CPU CFS quotas are enforced

Default: true

Also see: 

settings.kubernetes.cpu-manager-policy

Specifies the CPU manager policy. If you want to allow pods with certain resource characteristics to be granted increased CPU affinity and exclusivity on the node, you can set this setting to static. You should reboot if you change this setting after startup - try apiclient reboot

Default: none

Accepted values:
  • static
  • none
Also see: 

settings.kubernetes.cpu-manager-policy-options

Policy options to apply when settings.kubernetes.cpu-manager-policy is set to static. There currently there is only one allowed option, so the default is implict if not the setting is not defined.

Accepted values:
  • full-pcpus-only
# When `settings.kubernetes.cpu-manager-policy` is set to `static`
[settings.kubernetes]
cpu-manager-policy-options = ["full-pcpus-only"]
When `settings.kubernetes.cpu-manager-policy` is set to `static`
apiclient apply <<EOF
[settings.kubernetes]
cpu-manager-policy-options = ["full-pcpus-only"]
EOF
Also see: 

settings.kubernetes.cpu-manager-reconcile-period

Specifies the CPU manager reconcile period, which controls how often updated CPU assignments are written to cgroupfs. The value is a duration like 30s for 30 seconds or 1h5m for 1 hour and 5 minutes.

Also see: 

settings.kubernetes.credential-providers

Contains a collection of Kubelet image credential provider settings. Each key under this setting is the name of the plugin to configure. See Example 1 below. The ecr-credential-provider plugin can also be used for AWS IAM Roles Anywhere support. IAM Roles Anywhere is configured using the settings.aws.config setting. The content of that setting needs to configure the credential_process using the aws_signing_helper using your IAM Roles Anywhere settings, see Example 2 below.

# Example 1: user data for configuring the `ecr-credential-provider` credential provider plug-in
[settings.kubernetes.credential-providers.ecr-credential-provider]
enabled = true
# (optional - defaults to "12h")
cache-duration = "30m"
image-patterns = [
  # One or more URL paths to match an image prefix. Supports globbing of subdomains.
  "*.dkr.ecr.us-east-2.amazonaws.com",
  "*.dkr.ecr.us-west-2.amazonaws.com"
]

[settings.kubernetes.credential-providers.ecr-credential-provider.environment]
# The following are not used with ecr-credential-provider, but are provided for illustration
"KEY" = "abc123xyz"
"GOMAXPROCS" = "2"

# Example 2: `credential_process` using the `aws_signing_helper`
[default]
region = us-west-2
credential_process = aws_signing_helper credential-process --certificate /var/lib/kubelet/pki/kubelet-client-current.pem --private-key /var/lib/kubelet/pki/kubelet-client-current.pem --profile-arn [profile ARN]
   --role-arn [role ARN]
   --trust-anchor-arn [trust anchor ARN]
Also see: 

settings.kubernetes.static-pods.<custom identifier>.enabled

Whether the static pod is enabled.

Also see: 

settings.kubernetes.static-pods.<custom identifier>.manifest

A base64-encoded pod manifest.

Also see: 

settings.kubernetes.event-burst

The maximum size of a burst of event creations.

Also see: 

settings.kubernetes.event-qps

The maximum event creations per second.

Also see: 

settings.kubernetes.eviction-hard

The signals and thresholds that trigger pod eviction. Keys are signals and must be quoted since they contain a dot (.).

[settings.kubernetes.eviction-hard]
"memory.available" = "15%"

apiclient set settings.kubernetes.eviction-hard."memory.available"="15%"
Also see: 

settings.kubernetes.eviction-max-pod-grace-period

Maximum grace period, in seconds, to wait for pod termination before soft eviction.

Default: 0

[settings.kubernetes]
eviction-max-pod-grace-period = 40
apiclient set settings.kubernetes.eviction-max-pod-grace-period=40
Also see: 

settings.kubernetes.eviction-soft

The signals and thresholds that trigger pod eviction with a provided grace period (settings.kubernetes.eviction-soft-grace-period). Keys are signals and must be quoted since they contain a dot (.).

[settings.kubernetes.eviction-soft]
"memory.available" = "12%"

apiclient set settings.kubernetes.eviction-soft."memory.available"="12%"
Also see: 

settings.kubernetes.eviction-soft-grace-period

Delay for each signal to wait for pod termination before eviction. Keys are signals and must be quoted since they contain a dot (.).

[settings.kubernetes.eviction-soft-grace-period]
"memory.available" = "30s"

apiclient set settings.kubernetes.eviction-soft-grace-period."memory.available"="30s"
Also see: 

settings.kubernetes.hostname-override

The node name kubelet uses as identification instead of the hostname or the name determined by the in-tree cloud provider if that’s enabled.

Also see: 

settings.kubernetes.image-gc-high-threshold-percent

The percent of disk usage after which image garbage collection is always run, expressed as an integer from 0-100 inclusive.

If you downgrade from 1.14.0 to an earlier version, the values will be automatically converted to strings.

# After 1.14.0, the value can be represented as a integer or string for backwards compatiblity.
[settings.kubernetes]
image-gc-high-threshold-percent = 85
# Before 1.14.0, the value must be represented as a string.
[settings.kubernetes]
image-gc-high-threshold-percent = "85"
After 1.14.0, the value can be represented as a integer or string for backwards compatiblity.
apiclient set settings.kubernetes.image-gc-high-threshold-percent=85
Before 1.14.0, the value must be represented as a string.
apiclient set settings.kubernetes.image-gc-high-threshold-percent="85"
Also see: 

settings.kubernetes.image-gc-low-threshold-percent

The percent of disk usage before which image garbage collection is never run, expressed as an integer from 0-100 inclusive.

If you downgrade from 1.14.0 to an earlier version, the values will be automatically converted to strings.

# After 1.14.0, the value can be represented as a integer or string for backwards compatiblity.
[settings.kubernetes]
image-gc-low-threshold-percent = 80
# Before 1.14.0, the value must be represented as a string.
[settings.kubernetes]
image-gc-low-threshold-percent = "80"
After 1.14.0, the value can be represented as a integer or string for backwards compatiblity.
apiclient set settings.kubernetes.image-gc-low-threshold-percent=80
Before 1.14.0, the value must be represented as a string.
apiclient set settings.kubernetes.image-gc-low-threshold-percent="80"
Also see: 

settings.kubernetes.kube-api-burst

The burst to allow while talking with kubernetes.

Also see: 

settings.kubernetes.kube-api-qps

The QPS to use while talking with kubernetes apiserver.

Also see: 

settings.kubernetes.kube-reserved

Resources reserved for node components. The following keys are valid:

  • cpu: in millicores from the total number of vCPUs available on the instance.
  • memory: in mebibytes from the max num of pods on the instance. memory_to_reserve = max_num_pods * 11 + 255.
  • ephemeral-storage: defaults to 1Gi.

Also see: 

settings.kubernetes.log-level

The logging verbosity of the kubelet process. Higher numbers enabling more verbose logging.

Default: 2

Also see: 

settings.kubernetes.max-pods

The maximum number of pods that can be scheduled on this node (limited by number of available IPv4 addresses).

Also see: 

settings.kubernetes.memory-manager-policy

The memory management policy to use: None or Static. When using the Static policy you should also set settings.kubernetes.memory-manager-reserved-memory values.

Default: None

Also see: 

settings.kubernetes.memory-manager-reserved-memory

Set the total amount of reserved memory for a node. settings.kubernetes.memory-manager-reserved-memory is set per NUMA node. These settings are used to configure memory manager policy when settings.kubernetes.memory-manager-policy is set to Static.

[settings.kubernetes]
"memory-manager-policy" = "Static"

[settings.kubernetes.memory-manager-reserved-memory.0]
# Reserve a single 1GiB huge page along with 674MiB of memory
"enabled" = true
"memory" = "674Mi"
"hugepages-1Gi" = "1Gi"

[settings.kubernetes.memory-manager-reserved-memory.1]
# Reserve 1,074 2MiB huge pages
"enabled" = true
"hugepages-2Mi" = "2148Mi"
Also see: 

settings.kubernetes.node-ip

The IP address of the node.

settings.kubernetes.node-labels

Labels in the form of key, value pairs added when registering the node in the cluster.

[settings.kubernetes.node-labels]
label1 = foo
label2 = bar

apiclient set settings.kubernetes.node-labels.label1=foo \ 
 	settings.kubernetes.node-labels.label2=bar
Also see: 

settings.kubernetes.node-taints

Taints in the form of key, values and effects entries added when registering the node in the cluster.

[settings.kubernetes.node-taints]
dedicated = ["experimental:PreferNoSchedule", "experimental:NoExecute"]
special = ["true:NoSchedule"]

apiclient set settings.kubernetes.node-taints.dedicated=["experimental:PreferNoSchedule", "experimental:NoExecute"] \ 
 	settings.kubernetes.node-taints.special=["true:NoSchedule"]
Also see: 

settings.kubernetes.pod-infra-container-image

The URI of the ‘pause’ container.

Also see: 

settings.kubernetes.pod-pids-limit

The maximum number of processes per pod.

Also see: 

settings.kubernetes.provider-id

The way an external provider identifies a node.

Also see: 

settings.kubernetes.registry-burst

The maximum size of bursty pulls.

Also see: 

settings.kubernetes.registry-qps

The registry pull QPS.

Also see: 

settings.kubernetes.seccomp-default

Enable RuntimeDefault as the default seccomp profile for all workloads via kubelet-configuration

Default: false

Accepted values:
  • true
  • false

settings.kubernetes.server-certificate

The base64 encoded content of an x509 certificate for the kubelet web server, which is used for retrieving logs and executing commands.

Also see: 

settings.kubernetes.server-key

The base64 encoded content of an x509 private key for the kubelet web server.

Also see: 

settings.kubernetes.server-tls-bootstrap

Enables or disables server certificate bootstrap. When enabled, the kubelet will request a certificate from the certificates.k8s.io API. This requires an approver to approve the certificate signing requests (CSR).

Default: true

Also see: 

settings.kubernetes.shutdown-grace-period

Delay the node should wait for pod termination before shutdown.

Default: 0s

Also see: 

settings.kubernetes.shutdown-grace-period-for-critical-pods

The portion of the shutdown delay that should be dedicated to critical pod shutdown. Default is 0s.

Default: false

Also see: 

settings.kubernetes.standalone-mode

It true, kubelet runs in standalone mode without connecting to an API server.

Default: false

Accepted values:
  • true
  • false
Also see: 

settings.kubernetes.system-reserved

Resources reserved for system components.

# Example user data for setting up system reserved
[settings.kubernetes.system-reserved]
cpu = "10m"
ephemeral-storage = "1Gi"
memory = "100Mi"
Example user data for setting up system reserved
apiclient set settings.kubernetes.system-reserved.cpu="10m" \ 
 	settings.kubernetes.system-reserved.ephemeral-storage="1Gi" \ 
 	settings.kubernetes.system-reserved.memory="100Mi"
Also see: 

settings.kubernetes.topology-manager-policy

Specifies the topology manager policy.

Default: none

Accepted values:
  • none
  • restricted
  • best-effort
  • single-numa-node
Also see: 

settings.kubernetes.topology-manager-scope

Specifies the topology manager scope. If you want to group all containers in a pod to a common set of NUMA nodes, you can set this setting to pod.

Default: container

Accepted values:
  • container
  • pod
Also see: 

Some setting descriptions come from the Kubelet Configuration or Kubelet Options documentation.