December 23, 2024

Kubernetes DaemonSets: Managing System-Level Components Across Every Node

Blog

Kubernetes DaemonSet for system components

In Kubernetes, daemon set Is a special type of controller used to manage Pod deployment on all or specific nodes in a cluster. They are particularly useful for running system-level services or agents that need to run on every node or subset of nodes in the cluster.

one daemon set Ensures that replicas of a specific Pod are running on all nodes (or on a subset of nodes, depending on configuration). It is typically used for system elements or background services that must run on each node, such as logging agents, monitoring agents, network agents, or any service that needs to interact with the node itself.

In this article, we will explore the following concepts: daemon set In Kubernetes, their use cases and how to implement and manage them effectively.

What is a DaemonSet?

one daemon set Controllers in Kubernetes ensure that a specific pod runs on every node or subset of nodes in a cluster. Unlike other controllers deploy or stateful setDaemonSets are typically used to manage stateless or stateful applications, while DaemonSets are typically used for workloads that need to be deployed on every node in the cluster.

The main functions of DaemonSet include:

One Pod per node:DaemonSet ensures that a pod is running on each node in the cluster.
Dynamic scaling: If you add new nodes to the cluster, DaemonSet will automatically create pods on these new nodes.
Node selection: A DaemonSet can be configured to run only on specific nodes using node selectors, affinity, or taints and tolerances.

DaemonSets are particularly important for deploying system elements that must be available on all nodes in the cluster, such as logging, monitoring, or network services.

Why use DaemonSet?

DaemonSet is primarily used for workloads that need to be executed on every node in a Kubernetes cluster. Common use cases include:

Cluster-level logging:DaemonSets can be used to deploy logging agents (for example, fluentd or logstash) on each node in the cluster, allowing logs to be collected and forwarded to a centralized logging system.
Cluster level monitoring: monitoring agent, e.g. Prometheus Node Exporter or data dog agent DaemonSets can be deployed to collect node-level metrics for the entire cluster.
network components:DaemonSets can be used to deploy network agents or VPN agents (for example, calico or eyelash) to each node to control network traffic and security.
System daemon: System-level daemon process, such as firewall proxy, load balanceror network routeroften needs to be deployed on each node in the cluster.
service mesh: In a service mesh scenario, DaemonSets can be used to deploy agents (e.g. messenger or Istio sidecar) manages communication between microservices on each node.

How DaemonSet works

When you create a daemon set In Kubernetes, a controller ensures that pods are scheduled on each node based on defined criteria such as node affinity, selectors, or tolerances. Key workflows include:

Pod scheduling: Kubernetes schedules a pod on each node that matches the node selector and constraints of the DaemonSet.
Auto scaling: If you add new nodes to the cluster, DaemonSet will automatically create pods on these nodes. Conversely, if nodes are deleted, the DaemonSet will ensure that the pods on those nodes are deleted.
Pod delete: When you delete a DaemonSet, Kubernetes terminates pods executing on all nodes associated with the DaemonSet.

DaemonSet is highly dynamic, ensuring that as the cluster expands, relevant services are deployed on newly added nodes without any manual intervention.

How to define DaemonSet in Kubernetes

one daemon set Definition using a YAML profile that specifies the desired state of the Pod, including the container to execute, resource requests, and any node-specific constraints.

Example YAML for DaemonSet (For example, to run a logging agent):

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  labels:
    app: logging
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
        - name: fluentd
          image: fluent/fluentd:v1.12-1
          volumeMounts:
            - mountPath: /fluentd/etc
              name: config-volume
            - mountPath: /var/log
              name: log-volume
      volumes:
        - name: config-volume
          configMap:
            name: fluentd-config
        - name: log-volume
          hostPath:
            path: /var/log
            type: DirectoryOrCreate

In this example:

one fluent Containers are deployed on each node as a DaemonSet.
this volumes section is used to mount the host directory (in this case, /var/log) to collect logs.
this Configure mapping table fluentd-config Provide configuration data to the fluidd container.

DaemonSet node selection

DaemonSet provides multiple methods to control the operating location of pods node selector, Affinity, stainand Tolerance. These mechanisms ensure that Pods only run on the appropriate nodes based on specific conditions.

node selector: You can use node selectors to specify that the DaemonSet pod should only execute on nodes with specific labels.
```
spec:
  template:
    spec:
      nodeSelector:
        disktype: ssd
```
This ensures that the DaemonSet Pod only runs on nodes marked as disktype=ssd.

node affinity:Node affinity allows for more complex scheduling rules based on node labels.

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: disktype
                    operator: In
                    values:
                      - ssd

contamination and tolerance:Taints and Tolerations are useful for ensuring that DaemonSet pods only run on “tainted” nodes with certain conditions (e.g., only run on dedicated nodes).
```
spec:
  template:
    spec:
      tolerations:
        - key: "dedicated"
          operator: "Equal"
          value: "logging"
          effect: "NoSchedule"
```
This ensures that the DaemonSet Pod is only executed on nodes with the following properties: dedicated=logging stain.

DaemonSet rolling update

DaemonSet supports rolling updates, which allows you to update pods managed by the DaemonSet one at a time without disrupting the entire system. This is particularly useful when updating system components or services.

Rolling updates can be controlled through the following parameters:

Maximum unavailable: Limit the number of pods that are unavailable during updates.
maximum surge: Allow new Pods to be established during the update process before the old Pods are terminated.

Scroll update setting example:

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1

In this case, up to 1 pod can be unavailable during the update, and 1 additional pod can be established before the old pod is terminated.

Common use cases for DaemonSet

Log collection:Run the log collection agent (for example, fluent, Log storage) collects logs on each node and forwards them to a central location.
monitoring agent: Deploy monitoring agent, for example Prometheus Node Exporter or data dog agent Metrics are collected on each node.
Internet proxy: Run a network proxy, for example messenger or Istio sidecar proxymanages communication between services on each node.
security agent: Deploy a security agent, e.g. Antivirus software, intrusion detection systemor firewall proxy on every node.
System daemon: Run background service, for example backup agent, Configuration management (For example, ansipur), or Service mesh components on all nodes.

Best practices for using DaemonSet

for system-level daemons: DaemonSet is best suited for system-level components (such as logging, monitoring, network proxies) that need to execute on every node in the cluster.
Resource requests and limits: Ensure that each pod in the DaemonSet has resource requests and limits defined to prevent cross-node resource contention.
Consider node affinity: If your cluster has different types of nodes (for example, GPU nodes, dedicated storage nodes), use node selectors or affinity rules to ensure that DaemonSet pods only run on the appropriate nodes.
rolling update: Use rolling updates for DaemonSets to minimize disruption when updating system components.
contamination and tolerance: Use taints and tolerations to ensure that DaemonSets only run on nodes dedicated to these workloads.

in conclusion

daemon set Services in Kubernetes are critical for managing system-level services that need to be executed on every node in a cluster or on a specific subset of nodes. They ensure that every node gets a copy of the Pod, providing benefits such as scaling, dynamic node addition, and automatic management of system daemons such as logging, monitoring, and network proxies.

By effectively using DaemonSets, you can ensure that your Kubernetes cluster is equipped with key components for operational monitoring, security, and logging, while also ensuring that your infrastructure remains scalable and highly available.

2024-12-23 14:48:05

Components DaemonSets Kubernetes Managing node SystemLevel video dawnloader free video dawnloader free online VideoDDD