codezhong

Kubernetes容器编排生产环境最佳实践

2026/04/28 SRE DevOps Kubernetes 共 8089 字，约 24 分钟

Kubernetes容器编排生产环境最佳实践

情境(Situation)

Kubernetes已成为容器编排的事实标准，是构建云原生应用的核心平台。作为DevOps/SRE工程师，掌握Kubernetes的设计、部署和运维是必备技能。

冲突(Conflict)

许多团队在Kubernetes实践中面临以下挑战：

  • 集群稳定性问题：节点故障导致服务中断
  • 资源管理困难：Pod资源争用影响性能
  • 网络配置复杂：Service、Ingress、NetworkPolicy配置不当
  • 存储管理挑战：持久化存储与状态应用部署
  • 大规模集群运维：管理数百个节点和数千个Pod

问题(Question)

如何设计和管理一个稳定、高效、可扩展的Kubernetes集群？

答案(Answer)

本文将基于真实生产案例，提供一套完整的Kubernetes容器编排最佳实践指南。

一、Kubernetes集群架构设计

1.1 集群架构概览

flowchart TD
    subgraph 控制平面["Control Plane"]
        API["API Server"]
        ETCD["etcd"]
        SCH["Scheduler"]
        CTL["Controller Manager"]
    end
    
    subgraph Worker节点["Worker Nodes"]
        subgraph Node1["Node 1"]
            KUBELET1["Kubelet"]
            KUBE_PROXY1["Kube-proxy"]
            PODS1["Pods"]
        end
        
        subgraph Node2["Node 2"]
            KUBELET2["Kubelet"]
            KUBE_PROXY2["Kube-proxy"]
            PODS2["Pods"]
        end
        
        subgraph Node3["Node 3"]
            KUBELET3["Kubelet"]
            KUBE_PROXY3["Kube-proxy"]
            PODS3["Pods"]
        end
    end
    
    subgraph 外部访问["External Access"]
        LB["Load Balancer"] --> API
        ING["Ingress Controller"] --> PODS1
        ING --> PODS2
        ING --> PODS3
    end
    
    API <--> ETCD
    API <--> SCH
    API <--> CTL
    SCH --> KUBELET1
    SCH --> KUBELET2
    SCH --> KUBELET3
    
    style 控制平面 fill:#e3f2fd
    style Worker节点 fill:#fff3e0
    style 外部访问 fill:#c8e6c9

1.2 控制平面高可用配置

# 控制平面节点配置示例
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.28.0
controlPlaneEndpoint: "k8s-api.example.com:6443"
etcd:
  local:
    dataDir: /var/lib/etcd
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://${HOST_IP}:2379"
      advertise-client-urls: "https://${HOST_IP}:2379"
      listen-peer-urls: "https://${HOST_IP}:2380"
      initial-advertise-peer-urls: "https://${HOST_IP}:2380"
      peer-cert-file: /etc/kubernetes/pki/etcd/peer.crt
      peer-key-file: /etc/kubernetes/pki/etcd/peer.key
      peer-client-cert-auth: "true"
      client-cert-auth: "true"
      trusted-ca-file: /etc/kubernetes/pki/etcd/ca.crt
      cert-file: /etc/kubernetes/pki/etcd/server.crt
      key-file: /etc/kubernetes/pki/etcd/server.key
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"

二、Pod与容器设计最佳实践

2.1 Pod设计模式

模式适用场景示例
Sidecar日志收集、监控代理Fluentd、Envoy
Ambassador网络代理、服务发现MySQL代理
Adapter指标转换、格式适配Prometheus Exporter
Init Container初始化任务、依赖准备数据库迁移、配置下载
# Sidecar模式示例
apiVersion: v1
kind: Pod
metadata:
  name: app-with-sidecar
spec:
  containers:
  - name: main-app
    image: myapp:latest
    ports:
    - containerPort: 8080
    volumeMounts:
    - name: logs
      mountPath: /var/log/myapp
  - name: fluentd-sidecar
    image: fluent/fluentd:latest
    volumeMounts:
    - name: logs
      mountPath: /var/log/myapp
    env:
    - name: FLUENTD_CONF
      value: "fluentd.conf"
  volumes:
  - name: logs
    emptyDir: {}

2.2 资源管理配置

# Pod资源请求和限制
apiVersion: v1
kind: Pod
metadata:
  name: resource-demo
spec:
  containers:
  - name: main
    image: myapp:latest
    resources:
      requests:
        memory: "256Mi"
        cpu: "250m"
      limits:
        memory: "512Mi"
        cpu: "500m"
    env:
    - name: JAVA_OPTS
      value: "-Xms256m -Xmx512m"

2.3 健康检查配置

# Liveness和Readiness探针
apiVersion: v1
kind: Pod
metadata:
  name: health-check-demo
spec:
  containers:
  - name: main
    image: myapp:latest
    ports:
    - containerPort: 8080
    livenessProbe:
      httpGet:
        path: /health/live
        port: 8080
      initialDelaySeconds: 30
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /health/ready
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 3
      failureThreshold: 2
    startupProbe:
      httpGet:
        path: /health/startup
        port: 8080
      failureThreshold: 30
      periodSeconds: 10

三、Deployment与ReplicaSet最佳实践

3.1 Deployment配置

# Deployment配置示例
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deployment
  labels:
    app: myapp
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
        version: v1.0.0
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - myapp
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: myapp
        image: registry.example.com/myapp:v1.0.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        envFrom:
        - configMapRef:
            name: myapp-config
        - secretRef:
            name: myapp-secrets
        volumeMounts:
        - name: data
          mountPath: /data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: myapp-pvc

3.2 部署策略对比

策略适用场景特点风险
RollingUpdate常规更新渐进式更新，零停机可能影响部分用户
Recreate状态应用更新先销毁再创建服务中断
Blue/Green关键业务发布完整切换，快速回滚资源需求翻倍
Canary新功能验证小流量验证需要流量控制

四、Service与网络配置

4.1 Service类型对比

类型适用场景特点
ClusterIP集群内部通信默认类型，仅集群内访问
NodePort外部简单访问每个节点暴露端口
LoadBalancer云平台负载均衡自动创建LB
ExternalName外部服务访问别名方式
# Service配置示例
apiVersion: v1
kind: Service
metadata:
  name: myapp-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
  type: LoadBalancer
  selector:
    app: myapp
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800

4.2 Ingress配置

# Ingress配置示例
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
spec:
  tls:
  - hosts:
    - api.example.com
    secretName: api-tls
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /api/v1/
        pathType: Prefix
        backend:
          service:
            name: myapp-service
            port:
              number: 80

4.3 NetworkPolicy配置

# NetworkPolicy配置示例
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: myapp-network-policy
spec:
  podSelector:
    matchLabels:
      app: myapp
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend
    - ipBlock:
        cidr: 10.0.0.0/24
        except:
        - 10.0.0.5/32
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: database
    ports:
    - protocol: TCP
      port: 3306

五、存储与持久化

5.1 PV/PVC配置

# PersistentVolume配置
apiVersion: v1
kind: PersistentVolume
metadata:
  name: myapp-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  awsElasticBlockStore:
    volumeID: vol-1234567890abcdef0
    fsType: ext4

# PersistentVolumeClaim配置
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myapp-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: gp2

5.2 StorageClass配置

# StorageClass配置
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  fsType: ext4
  encrypted: "true"
reclaimPolicy: Retain
allowVolumeExpansion: true

六、大规模集群运维

6.1 集群监控配置

# Prometheus ServiceMonitor配置
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubernetes-services
  labels:
    release: prometheus
spec:
  endpoints:
  - port: http-metrics
    interval: 30s
  selector:
    matchLabels:
      k8s-app: kube-state-metrics

6.2 集群自动扩缩容

# Horizontal Pod Autoscaler配置
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp-deployment
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

6.3 节点管理

# 节点管理命令
kubectl cordon node-01                    # 标记节点不可调度
kubectl drain node-01 --ignore-daemonsets # 驱逐节点上的Pod
kubectl uncordon node-01                  # 取消节点不可调度标记

# 节点标签管理
kubectl label nodes node-01 zone=east
kubectl label nodes node-01 tier=production

# 节点污点管理
kubectl taint nodes node-01 dedicated=production:NoSchedule
kubectl taint nodes node-01 dedicated=production:PreferNoSchedule

七、安全最佳实践

7.1 RBAC配置

# Role配置
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: myapp
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

# RoleBinding配置
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: myapp
subjects:
- kind: User
  name: jane
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

7.2 Pod安全标准

策略限制级别适用场景
Restricted最严格生产环境
Baseline中等测试环境
Privileged无限制开发环境
# Pod安全准入配置
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
  configuration:
    defaults:
      enforce: "restricted"
      enforce-version: "latest"
      audit: "restricted"
      audit-version: "latest"
      warn: "restricted"
      warn-version: "latest"

八、最佳实践总结

8.1 Kubernetes设计原则

原则说明实践建议
高可用消除单点故障多Master节点、Pod反亲和性
弹性伸缩根据负载自动调整HPA + VPA
网络隔离控制Pod间通信NetworkPolicy
资源管理合理分配资源Requests/Limits配置
安全优先最小权限原则RBAC + Pod安全标准
可观测性全面监控Prometheus + Grafana

8.2 常见问题与解决方案

问题症状解决方案
Pod调度失败Pending状态检查资源不足、节点污点、亲和性配置
服务不可访问Connection refused检查Service、Pod状态、网络策略
Pod频繁重启CrashLoopBackOff检查日志、资源限制、健康检查
存储挂载失败FailedMount检查PV/PVC状态、存储类配置
网络延迟高响应慢检查网络插件、Service类型

总结

Kubernetes是一个复杂但强大的容器编排平台。通过合理的架构设计、资源配置、网络管理和安全策略，可以构建一个稳定、高效、可扩展的生产环境集群。

延伸阅读：更多Kubernetes相关面试题，请参考 SRE面试题解析：基于JD与简历匹配分析。

参考资料

文档信息

Search

    Table of Contents