# Introduction

I started using Kubernetes professionally back in January 2019. I was solely responsible for making sure Kubernetes ran smoothly in a production environment. Over the years, I’ve learned a lot. For those just starting their Kubernetes journey I wanted to share some tips for managing a Kubernetes cluster effectively.

# Start Small and Simple

Why: Overcomplicating your Kubernetes setup too early introduces unnecessary complexity and can cause cascading failures.

Tip: Focus on the basics first. Use Kubernetes’ core features like Deployments, Services, and Ingress before jumping into advanced tools like Helm, Istio or Operators. Once your infrastructure is stable and your team is familiar with Kubernetes management, then consider adding these tools as needed. Try to go as long as possible without adding additional complexity.

# Be Cautious with the aws-auth Config

Why: This is specific to AWS and EKS: Mismanaging the aws-auth config can lock you or your team out of your Kubernetes cluster, especially if manual changes lead to incorrect IAM role mappings.

Tip: Automate aws-auth management using infrastructure-as-code tools like ArgoCD. I know this is immediately counter to Start Small and Simple, but a broken cluster is not fun to deal with. Automating this process reduces the risk of manual errors and provides a version-controlled way to manage access. The user/role that creates the EKS cluster will always be an admin, so try to use a shared role whenever possible. This way, you’ll always have access to the cluster, even if aws-auth is misconfigured.

# Match Your Resource Requests and Limits

Why: Setting different resource requests and limits (especially for memory) can lead to unexpected pod terminations. Some containers may not perform garbage collection correctly if they see more memory than requested.

Tip: Keep your memory requests and limits the same to ensure garbage collection runs properly within the container. This will prevent Kubernetes from killing your pods unexpectedly due to memory pressure.

# Use PodDisruptionBudgets (PDBs) for Safe Rollouts and Drains

Why: PDBs ensure that a certain number or percentage of replicas are available during voluntary disruptions (e.g., rolling updates, node drains).

Tip: Set PDBs to prevent evictions of critical pods during cluster upgrades or scaling activities. Without it, Kubernetes could kill too many pods, impacting application availability and stability.

# Leverage Topology Spread Constraints

Why: This ensures that pods are spread across different failure domains, such as nodes and zones, to improve fault tolerance and availability.

Tip: Use topology spread constraints to spread pods across nodes and availability zones. Here’s an example of spreading pods across both nodes and AWS availability zones:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: "kubernetes.io/hostname"  # Spread across nodes
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-app
  - maxSkew: 1
    topologyKey: "topology.kubernetes.io/zone"  # Spread across AWS availability zones
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-app

This ensures that no single node or availability zone becomes overloaded, improving resilience and availability in case of node or zone failures.

# Use Liveness, Readiness, and Startup Probes Properly

Why: Misconfigured probes can cause cascading failures and application downtime.

Tip: Kubernetes provides three types of probes—liveness, readiness, and startup—each serving a different purpose.

• Liveness Probe: Ensures that your application is still running. If it fails, Kubernetes will restart the pod.
• Readiness Probe: Ensures that your application is ready to receive traffic. Kubernetes will only send traffic to pods that pass the readiness check.
• Startup Probe: Used when your application has a long startup time. It prevents Kubernetes from killing the container prematurely during initialization.

Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

startupProbe:
  httpGet:
    path: /start
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10

Use startup probes for slow-starting applications, which can prevent unnecessary restarts while the app is still initializing. Readiness probes should be used to ensure that traffic is only routed to fully functional pods, while liveness probes can be more aggressive but should be tuned carefully to avoid killing slow but healthy pods.

# Limit NodePort Usage

Why: NodePort services expose specific ports on all nodes in the cluster, which can lead to security risks and inefficient use of resources.

Tip: Use LoadBalancer services or Ingress for external traffic handling. Limit NodePort usage to development environments or internal traffic routing.

# Limit Secret Exposure with RBAC and Cloud Secret Managers

Why: Secrets in Kubernetes are only base64-encoded, which is not secure. Exposing secrets to unauthorized users or applications can lead to security breaches.

Tip: Instead of storing sensitive secrets directly in Kubernetes, use tools like HashiCorp Vault, AWS Secrets Manager, or another cloud-based secret manager. These tools provide robust encryption and secret management capabilities.

Additionally, implement Role-Based Access Control (RBAC) in Kubernetes to limit who can access secrets, and always enable secret encryption at rest for an extra layer of security.

# Leverage Horizontal Pod Autoscaling with Custom Metrics

Why: Default autoscaling is based on CPU or memory utilization, which may not be the best indicator of your app’s performance. Especially for workloads based on queues like RabbitMQ.

Tip: Use custom metrics for Horizontal Pod Autoscaling (HPA) to scale based on more relevant indicators, such as queue length, request latency, or other custom metrics. Datadog provides an easy way to define and use custom metrics.

Example: Datadog has great documentation on how to achieve this

Using custom metrics gives you more control and ensures your autoscaling logic is aligned with business-relevant performance indicators, rather than just CPU or memory utilization.