Ingress使用及生产案例
ingress
一、ingress架构
Ingress Controller nginx，k8s官方维护的
生产级架构图
1. 外部接入层（Internet/DMZ）
- 功能：接收来自互联网的HTTP/HTTPS请求，提供安全隔离（DMZ区）。
- 关键组件：
- 企业级负载均衡设备：如F5、LVS、HAProxy等[1][2]，用于SSL终止和请求分发。
- 端口暴露：开放80（HTTP）和443（HTTPS）端口，作为入口端口。
2. 负载均衡层（高可用设计）
- 高可用机制：
- 多实例负载均衡：部署多个Nginx或HAProxy实例，通过健康检查实现流量动态分配。
- 流量分发逻辑：
- 将请求均匀分发至后端的
Ingress Controller实例（Gateway Node）。
- 将请求均匀分发至后端的
3. Ingress控制层（Kubernetes节点）
- 部署方式：
- HostNetwork模式：
Ingress Controller Pod直接绑定宿主机网络，跳过Kubernetes Service（如NodePort），降低性能损耗。 - 多节点部署：通过
Deployment部署多个Controller实例，结合节点亲和性（nodeSelector）和Pod反亲和性（PodAntiAffinity）确保跨节点分布。
- HostNetwork模式：
- 动态配置更新：
- 监听K8s资源：实时监听
Ingress、Service、Endpoint等资源变化，自动生成Nginx配置并触发热更新（Reload）。 - 热更新优化：采用OpenResty减少Reload性能影响，避免长尾请求中断。
- 监听K8s资源：实时监听
4. 后端服务层（Real Server）
服务路由：
- 根据Ingress规则（域名/路径）将请求路由至对应的后端
Service（如Web应用、API服务）。
- 根据Ingress规则（域名/路径）将请求路由至对应的后端
服务发现：
- 通过
Kubernetes Endpoints动态获取Pod IP，支持扩缩容和故障转移。
- 通过
部署yaml：
apiVersion: v1
kind: Namespace
metadata:
labels:
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
name: ingress-nginx
---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx
namespace: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx-admission
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx
namespace: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- coordination.k8s.io
resourceNames:
- ingress-nginx-leader
resources:
- leases
verbs:
- get
- update
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- create
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx-admission
namespace: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
- namespaces
verbs:
- list
- watch
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx-admission
rules:
- apiGroups:
- admissionregistration.k8s.io
resources:
- validatingwebhookconfigurations
verbs:
- get
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx-admission
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx-admission
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
apiVersion: v1
data:
allow-snippet-annotations: "true"
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx-controller
namespace: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- appProtocol: http
name: http
port: 80
protocol: TCP
targetPort: http
- appProtocol: https
name: https
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: NodePort
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx-controller-admission
namespace: ingress-nginx
spec:
ports:
- appProtocol: https
name: https-webhook
port: 443
targetPort: webhook
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: ClusterIP
---
apiVersion: apps/v1
#kind: Deployment
kind: DaemonSet
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
minReadySeconds: 0
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
spec:
hostNetwork: true
containers:
- args:
- /nginx-ingress-controller
- --election-id=ingress-nginx-leader
- --controller-class=k8s.io/ingress-nginx
- --ingress-class=nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LD_PRELOAD
value: /usr/local/lib/libmimalloc.so
image: registry.cn-beijing.aliyuncs.com/dotbalo/ingress-nginx-controller:v1.7.1
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: controller
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
- containerPort: 8443
name: webhook
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
cpu: 100m
memory: 90Mi
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- NET_BIND_SERVICE
drop:
- ALL
runAsUser: 101
volumeMounts:
- mountPath: /usr/local/certificates/
name: webhook-cert
readOnly: true
// 更改dns解析策略
#dnsPolicy: ClusterFirst
dnsPolicy: ClusterFirstWithHostNet
nodeSelector:
kubernetes.io/os: linux
ingress: "true"
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
---
apiVersion: batch/v1
kind: Job
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx-admission-create
namespace: ingress-nginx
spec:
template:
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx-admission-create
spec:
containers:
- args:
- create
- --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
- --namespace=$(POD_NAMESPACE)
- --secret-name=ingress-nginx-admission
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: registry.cn-beijing.aliyuncs.com/dotbalo/kube-webhook-certgen:v20230312
imagePullPolicy: IfNotPresent
name: create
securityContext:
allowPrivilegeEscalation: false
nodeSelector:
kubernetes.io/os: linux
restartPolicy: OnFailure
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccountName: ingress-nginx-admission
---
apiVersion: batch/v1
kind: Job
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx-admission-patch
namespace: ingress-nginx
spec:
template:
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx-admission-patch
spec:
containers:
- args:
- patch
- --webhook-name=ingress-nginx-admission
- --namespace=$(POD_NAMESPACE)
- --patch-mutating=false
- --secret-name=ingress-nginx-admission
- --patch-failure-policy=Fail
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: registry.cn-beijing.aliyuncs.com/dotbalo/kube-webhook-certgen:v20230312
imagePullPolicy: IfNotPresent
name: patch
securityContext:
allowPrivilegeEscalation: false
nodeSelector:
kubernetes.io/os: linux
restartPolicy: OnFailure
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccountName: ingress-nginx-admission
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: nginx
spec:
controller: k8s.io/ingress-nginx
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.7.1
name: ingress-nginx-admission
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
service:
name: ingress-nginx-controller-admission
namespace: ingress-nginx
path: /networking/v1/ingresses
failurePolicy: Fail
matchPolicy: Equivalent
name: validate.nginx.ingress.kubernetes.io
rules:
- apiGroups:
- networking.k8s.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- ingresses
sideEffects: None
root@VM-26-130-ubuntu:/usr/local/src/k8s# kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create--1-5c74h 0/1 Completed 0 13m
ingress-nginx-admission-patch--1-gs92r 0/1 Completed 0 13m
ingress-nginx-controller-4frtd 1/1 Running 0 13m
ingress-nginx-controller-sgcg2 1/1 Running 0 13m
ingress-nginx-controller-tpndn 1/1 Running 0 13m二、ingress使用
2.1 资源定义
kind: Ingress
metadata:
name: nginx-ingress
spec:
ingressClassName: nginx # 指定该 Ingress 被哪个 Controller 解析
rules: # 定义路由匹配规则，可以配置多个
- host: nginx.test.com # 定义域名
http: #
paths: # 详细的路由配置，可以配置多个
- backend: # 指定该路由的后端
service:
name: nginx
port:
number: 80
path: / # 指定 PATH
pathType: ImplementationSpecific # 指定匹配规则pathType：路径的匹配方式，目前有 ImplementationSpecific、Exact 和 Prefix 方式
Exact：精确匹配，比如配置的 path 为/bar，那么/bar/将不能被路由；Prefix：前缀匹配，基于以 / 分隔的 URL 路径。比如 path 为/abc，可以匹配到/abc/bbb等，比较常用的配置；ImplementationSpecific：这种类型的路由匹配根据 Ingress Controller 来实现，可以当做一个单独的类型，也可以当做 Prefix 和 Exact。ImplementationSpecific 是 1.18 版本引入 Prefix 和 Exact 的默认配置。
2.2 Ingress入门
Ingress Nginx 配置文档： https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/
2.2.1 使用域名发布K8S服务
创建一个web服务：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl create deploy nginx --image=registry.cn-beijing.aliyuncs.com/dotbalo/nginx:1.15.12
暴露服务
kubectl expose deploy nginx --port 80创建ingress
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# vim web-ingress.yaml
apiVersion: networking.k8s.io/v1 # k8s >= 1.22 必须 v1
kind: Ingress
metadata:
name: nginx-ingress
spec:
ingressClassName: nginx # 指定该 Ingress 被哪个 Controller 解析
rules: # 定义路由匹配规则，可以配置多个
- host: dujie.test.com # 定义域名
http: #
paths: # 详细的路由配置，可以配置多个
- backend: # 指定该路由的后端
service:
name: nginx
port:
number: 80
path: / # 指定 PATH
pathType: ImplementationSpecific # 指定匹配规则2.2.2 Ingress特例：不配置域名发布服务
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# cat ingress-no-host.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-ingress-no-host
spec:
ingressClassName: nginx
rules:
- http:
paths:
- backend:
service:
name: nginx
port:
number: 80
path: /no-host
pathType: ImplementationSpecific2.3 Ingress实战
环境准备
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl create ns study-ingress
namespace/study-ingress created
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress#
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl create deployment nginx --image=registry.cn-beijing.aliyuncs.com/dotbalo/nginx:1.15.12 -n study-ingress
deployment.apps/nginx created
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl expose deployment nginx --port 80 -n study-ingress
service/nginx exposed2.3.1 使用HTTPS发布服务
生产环境对外的服务，一般需要配置https协议，使用Ingress可以非常方便添加https证书。
测试环境，手动生成证书
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=dujie.test.com"
Can't load /root/.rnd into RNG
140457963239232:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/root/.rnd
Generating a RSA private key
......................+++++
..............................+++++
writing new private key to 'tls.key'
-----
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl create secret tls ca-secret --cert=tls.crt --key=tls.key
secret/ca-secret created配置Ingress添加TLS配置：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# cat tls-ingress.yaml
apiVersion: networking.k8s.io/v1 # k8s >= 1.22 必须 v1
kind: Ingress
metadata:
name: nginx-ingress
namespace: study-ingress
spec:
ingressClassName: nginx # 指定该 Ingress 被哪个 Controller 解析
tls:
- hosts:
- dujie.test.com
secretName: ca-secret
rules: # 定义路由匹配规则，可以配置多个
- host: dujie.test.com # 定义域名
http: #
paths: # 详细的路由配置，可以配置多个
- backend: # 指定该路由的后端
service:
name: nginx
port:
number: 80
path: / # 指定 PATH
pathType: ImplementationSpecific # 指定匹配规则更新ingress之后访问测试可以看到域名已经被重定向到https
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# curl http://dujie.test.com -I
HTTP/1.1 308 Permanent Redirect
Date: Tue, 25 Mar 2025 07:14:44 GMT
Content-Type: text/html
Content-Length: 164
Connection: keep-alive
Location: https://dujie.test.com2.3.2 域名添加用户名密码认证
有些开源工具本身不提供密码认证，如果暴露出去会有很大风险，对于这类工具可以使用Nginx 的 basic-auth 设置密码访问，具体方法如下，由于需要使用 htpasswd 工具，所以需要安装httpd：
ubuntu是apache2，centos是httpd
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# apt install apache2使用 htpasswd 创建 dujie 用户的密码：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# htpasswd -c auth dujie
New password:
Re-type new password:
Adding password for user dujie
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress#
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# cat auth
dujie:$apr1$3MhHcDfm$CRmPVqklX2qhNg8FtV9pR/基于之前创建的密码文件创建secret
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl create secret generic basic-auth --from-file=auth -n study-ingress
secret/basic-auth created创建包含密码认证的Ingress
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# cat auth-ingress.yaml
apiVersion: networking.k8s.io/v1 # k8s >= 1.22 必须 v1
kind: Ingress
metadata:
name: nginx-ingress
namespace: study-ingress
annotations:
nginx.ingress.kubernetes.io/auth-realm: Please Input Your Username and Password
nginx.ingress.kubernetes.io/auth-secret: basic-auth
nginx.ingress.kubernetes.io/auth-type: basic
spec:
ingressClassName: nginx # 指定该 Ingress 被哪个 Controller 解析
tls:
- hosts:
- dujie.test.com
secretName: ca-secret
rules: # 定义路由匹配规则，可以配置多个
- host: dujie.test.com # 定义域名
http: #
paths: # 详细的路由配置，可以配置多个
- backend: # 指定该路由的后端
service:
name: nginx
port:
number: 80
path: / # 指定 PATH
pathType: ImplementationSpecific # 指定匹配规则- nginx.ingress.kubernetes.io/auth-type：认证类型，可以是 basic 和 digest
- nginx.ingress.kubernetes.io/auth-secret：密码文件的 Secret 名称
- nginx.ingress.kubernetes.io/auth-realm：需要密码认证的消息提醒
创建该 Ingress，并访问测试：
2.3.3 开启会话保持
和 Nginx 一样，Ingress Nginx 也支持基于 cookie 的会话保持。
首先扩容 nginx 服务至多个副本：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl scale deployment -n study-ingress nginx --replicas=3
deployment.apps/nginx scaled
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl get pods -n study-ingress
NAME READY STATUS RESTARTS AGE
nginx-785999d887-77kls 1/1 Running 0 43m
nginx-785999d887-8sld8 0/1 ContainerCreating 0 2s
nginx-785999d887-gg6wz 0/1 ContainerCreating 0 2s未开启会话保持，同一个主机访问可以看到流量会在三个副本中都有。
通过以下配置，即可看到流量只会进入到一个 Pod：
apiVersion: networking.k8s.io/v1 # k8s >= 1.22 必须 v1
kind: Ingress
metadata:
name: nginx-ingress
namespace: study-ingress
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 16m
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "route"
# 过期时间，expires 兼容旧版浏览器
nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
# 后端负载扩容后，是否需要重新分配流量，balanced: 重新分配 persistent: 保持粘性
nginx.ingress.kubernetes.io/affinity-mode: persistent
spec:
ingressClassName: nginx # 指定该 Ingress 被哪个 Controller 解析
tls:
- hosts:
- dujie.test.com
secretName: ca-secret
rules: # 定义路由匹配规则，可以配置多个
- host: dujie.test.com # 定义域名
http: #
paths: # 详细的路由配置，可以配置多个
- backend: # 指定该路由的后端
service:
name: nginx
port:
number: 80
path: / # 指定 PATH
pathType: ImplementationSpecific # 指定匹配规则同时浏览器的 request headers 会添加一个 Cookie 的属性：
流量重新分配，更改 nginx.ingress.kubernetes.io/affinity-mode: balanced 即可：
再次扩容后，就会发现流量会重新分配到其他节点。
2.3.4 配置流式返回SSE（代理大模型服务）
如果后端需要持续的输出数据，或者需要长连接，此时需要更改请求头升级链接为长连接
apiVersion: networking.k8s.io/v1 # k8s >= 1.22 必须 v1
kind: Ingress
metadata:
name: nginx-ingress
namespace: study-ingress
annotations:
nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
nginx.ingress.kubernetes.io/proxy-buffering: "off"
# snippet 通常用于配置一些不支持或者复杂的参数，比如配置请求头，或者逻辑控制
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header Updrade $http_updrade;
proxy_set_header Connection 'upgrade';
spec:
ingressClassName: nginx # 指定该 Ingress 被哪个 Controller 解析
tls:
- hosts:
- dujie.test.com
secretName: ca-secret
rules: # 定义路由匹配规则，可以配置多个
- host: dujie.test.com # 定义域名
http: #
paths: # 详细的路由配置，可以配置多个
- backend: # 指定该路由的后端
service:
name: nginx
port:
number: 80
path: / # 指定 PATH
pathType: ImplementationSpecific # 指定匹配规则登录到 ingress-nginx 的 Pod 查看配置：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create--1-5c74h 0/1 Completed 0 110m
ingress-nginx-admission-patch--1-gs92r 0/1 Completed 0 110m
ingress-nginx-controller-4frtd 1/1 Running 0 110m
ingress-nginx-controller-sgcg2 1/1 Running 0 110m
ingress-nginx-controller-tpndn 1/1 Running 0 110m
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl exec -it -n ingress-nginx ingress-nginx-controller-4frtd bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
VM-26-136-ubuntu:/etc/nginx$ cat nginx.conf |grep Upgrade
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Upgrade $http_upgrade;
VM-26-136-ubuntu:/etc/nginx$ cat nginx.conf |grep connection_upgrade
map $http_upgrade $connection_upgrade {
proxy_set_header Connection $connection_upgrade;
proxy_set_header Connection $connection_upgrade;2.3.5 域名重定向Redirect
在使用 Nginx 作为代理服务器时，Redirect 可用于域名的重定向，比如访问 old.com 被重定向到 new.com。Ingress 也可以实现 Redirect 功能，接下来用 nginx.redirect.com 作为旧域名，baidu.com 作为新域名进行演示：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# cat redirect-ingress.yaml
apiVersion: networking.k8s.io/v1 # k8s >= 1.22 必须 v1
kind: Ingress
metadata:
name: nginx-ingress
namespace: study-ingress
annotations:
nginx.ingress.kubernetes.io/permanent-redirect: https://www.baidu.com
nginx.ingress.kubernetes.io/permanent-redirect-code: "308"
spec:
ingressClassName: nginx # 指定该 Ingress 被哪个 Controller 解析
tls:
- hosts:
- dujie.test.com
secretName: ca-secret
rules: # 定义路由匹配规则，可以配置多个
- host: dujie.test.com # 定义域名
http: #
paths: # 详细的路由配置，可以配置多个
- backend: # 指定该路由的后端
service:
name: nginx
port:
number: 80
path: / # 指定 PATH
pathType: ImplementationSpecific # 指定匹配规则使用 curl 访问域名 nginx.redirect.com，可以看到 308（使用浏览器会自动跳转到百度的首页）：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# curl dujie.test.com -I
HTTP/1.1 308 Permanent Redirect
Date: Tue, 25 Mar 2025 08:15:28 GMT
Content-Type: text/html
Content-Length: 164
Connection: keep-alive
Location: https://www.baidu.com2.3.6 访问地址重写Rewrite
创建一个应用模拟后端服务
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl create deploy backend-api --image=registry.cn-beijing.aliyuncs.com/dotbalo/nginx:backend-api -n study-ingress
deployment.apps/backend-api created创建service暴露应用
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl expose deploy -n study-ingress backend-api --port 80
service/backend-api exposed查看service地址，并且通过/api-a访问测试
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl get svc -n study-ingress
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
backend-api ClusterIP 10.16.252.90 <none> 80/TCP 7s
nginx ClusterIP 10.16.238.11 <none> 80/TCP 69m
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# curl 10.16.252.90
<h1> backend for ingress rewrite </h1>
<h2> Path: /api-a </h2>
<a href="http://gaoxin.kubeasy.com"> Kubeasy </a>
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress#
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# curl 10.16.252.90/api-a
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.15.12</center>
</body>
</html>配置 Ingress，假设需要配置一个/api-a 代理到该服务：
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: backend-api
namespace: study-ingress
spec:
ingressClassName: nginx
rules:
- host: nginx.rewrite.com
http:
paths:
- backend:
service:
name: backend-api
port:
number: 80
path: /api-a
pathType: ImplementationSpecific测试通过浏览器访问，会报 404：
此时需要通过 Ingress Nginx 的 Rewrite 功能，将/api-a 重写为“/”，配置示例如下：
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: backend-api
namespace: study-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
ingressClassName: nginx
rules:
- host: nginx.rewrite.com
http:
paths:
- backend:
service:
name: backend-api
port:
number: 80
path: /api-a(/|$)(.*)
pathType: ImplementationSpecific再次访问 nginx.test.com/api-a 即可访问到后端服务：
2.3.7 访问速率限制
有时候可能需要限制速率以降低后端压力，或者限制单个 IP 每秒的访问速率防止攻击。此时可以使用 Nginx 的 rate limit 进行配置。
首先没有加速率限制，使用 ab 进行访问，Failed 为 0：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# ab -c 10 -n 100 http://dujie.test.com/ |grep requests
Complete requests: 100
Failed requests: 0
Time per request: 0.061 [ms] (mean, across all concurrent requests)
Percentage of the requests served within a certain time (ms)添加速率限制，限制只能有一个连接，只需要添加nginx.ingress.kubernetes.io/limitconnections为 1 即可：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# cat web-ingress.yaml
apiVersion: networking.k8s.io/v1 # k8s >= 1.22 必须 v1
kind: Ingress
metadata:
name: nginx-ingress
namespace: study-ingress
annotations:
nginx.ingress.kubernetes.io/limit-connections: "1"
spec:
ingressClassName: nginx # 指定该 Ingress 被哪个 Controller 解析
rules: # 定义路由匹配规则，可以配置多个
- host: dujie.test.com # 定义域名
http: #
paths: # 详细的路由配置，可以配置多个
- backend: # 指定该路由的后端
service:
name: nginx
port:
number: 80
path: / # 指定 PATH
pathType: ImplementationSpecific # 指定匹配规则再次使用 ab 测试，
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# ab -c 10 -n 100 http://dujie.test.com/ |grep requests
Complete requests: 100
Failed requests: 13
Time per request: 0.065 [ms] (mean, across all concurrent requests)
Percentage of the requests served within a certain time (ms)还有很多其它方面的限制，常用的配置如下：
#限制每秒的连接，单个 IP：
nginx.ingress.kubernetes.io/limit-rps
#限制每分钟的连接，单个 IP：
nginx.ingress.kubernetes.io/limit-rpm
#限制客户端每秒传输的字节数，单位为 K，需要开启 proxy-buffering：
nginx.ingress.kubernetes.io/limit-rate
# 速率限制白名单
nginx.ingress.kubernetes.io/limit-whitelist2.3.8 IngressNginx 黑/白名单
####### 2.3.8.1 配置黑名单
使用 nginx.ingress.kubernetes.io/denylist-source-range 添加黑名单，支持 IP、网段，多个黑名单逗号分隔：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# cat ingress-with-auth.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/denylist-source-range: 10.122.26.134
#10.0.0.0/24,172.10.0.1
name: ingress-with-auth
namespace: study-ingress
spec:
ingressClassName: nginx
rules:
- host: auth.test.com
http:
paths:
- backend:
service:
name: nginx
port:
number: 80
path: /
pathType: ImplementationSpecific访问 auth.test.com 会被拒绝：
root@VM-26-134-ubuntu:~# curl -H "Host:auth.test.com" 10.122.26.134
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>访问其他域名正常：
root@VM-26-134-ubuntu:~# curl -I dujie.test.com
HTTP/1.1 200 OK
Date: Tue, 25 Mar 2025 08:57:48 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Last-Modified: Tue, 16 Apr 2019 13:08:19 GMT
ETag: "5cb5d3c3-264"
Accept-Ranges: bytes
####### 2.3.8.2 配置白名单
白名单表示只允许某个 IP 可以访问，直接在 yaml 文件中配置即可，比如只允许192.168.181.141 访问，只需要添加一个 nginx.ingress.kubernetes.io/whitelist-source-range 注释即可：
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range: 10.122.26.134
name: ingress-with-auth
namespace: study-ingress
spec:
ingressClassName: nginx
rules:
- host: auth.test.com
http:
paths:
- backend:
service:
name: nginx
port:
number: 80
path: /
pathType: ImplementationSpecific####### 2.3.8.2 全局黑白名单
Ingress-nginx 支持全局的黑白名单（白名单慎用），只需要在 ingress nginx 的配置文件中添加即可，添加后无需重启 Controller，加一个全局黑名单：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl edit cm -n ingress-nginx ingress-nginx-controller
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
allow-snippet-annotations: "true"
denylist-source-range: 10.122.26.134
配置全局黑名单后，访问任何域名都会被拒绝：
# curl -H "Host:nginx.test.com" https://10.122.26.134 -k
<html>
<head><title>403 Forbidden</title></head>
# curl -H "Host:auth.test.com" https://10.122.26.134 -k
<html>
<head><title>403 Forbidden</title></head>添加一个全局白名单：
# kubectl edit cm ingress-nginx-controller -n ingress-nginx
data:
allow-snippet-annotations: "true"
whitelist-source-range: 10.122.26.134添加后除了 192.168.181.134，任何 IP 都不能访问该 Ingress Controller 下的域名：
# curl -H "Host:auth.test.com" https://10.122.26.134 -k
# 10.122.26.134 正常
<head><title>401 Authorization Required</title></head>
# 其他节点均拒绝
# curl -H "Host:auth.test.com" https://10.122.26.134 -k
<html>
<head><title>403 Forbidden</title></head>2.3.9 自定义错误页
官方文档：https://kubernetes.github.io/ingress-nginx/examples/customization/custom-errors/
每个项目在对外发布时，难免不了会有一些未知的错误，比如 404/502/503 等，为了给客户更加友好的提示，可以使用 default backend 自定义错误页。
首先需要安装 default backend 服务：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# cat custom-default-backend.yaml
---
apiVersion: v1
kind: Service
metadata:
name: nginx-errors
labels:
app.kubernetes.io/name: nginx-errors
app.kubernetes.io/part-of: ingress-nginx
spec:
selector:
app.kubernetes.io/name: nginx-errors
app.kubernetes.io/part-of: ingress-nginx
ports:
- port: 80
targetPort: 8080
name: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-errors
labels:
app.kubernetes.io/name: nginx-errors
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: nginx-errors
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: nginx-errors
app.kubernetes.io/part-of: ingress-nginx
spec:
containers:
- name: nginx-error-server
image: registry.cn-beijing.aliyuncs.com/dotbalo/custom-error-pages:v1.0.1
ports:
- containerPort: 8080
# Setting the environment variable DEBUG we can see the headers sent
# by the ingress controller to the backend in the client response.
# env:
# - name: DEBUG
# value: "true"
# Mounting custom error page from configMap
# volumeMounts:
# - name: custom_error_pages
# mountPath: /www
# Mounting custom error page from configMap
# volumes:
# - name: custom_error_pages
# configMap:
# name: custom_error_pages
# items:
# - key: "404"
# path: "404.html"
# - key: "503"
# path: "503.html"
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl apply -f custom-default-backend.yaml -n ingress-nginx
service/nginx-errors created
deployment.apps/nginx-errors created
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create--1-5c74h 0/1 Completed 0 23h
ingress-nginx-admission-patch--1-gs92r 0/1 Completed 0 23h
ingress-nginx-controller-4frtd 1/1 Running 0 23h
ingress-nginx-controller-sgcg2 1/1 Running 0 23h
ingress-nginx-controller-tpndn 1/1 Running 0 23h
nginx-errors-6bf9bfd77f-54mpc 1/1 Running 0 9s更改ingress-nginx的启动参数，支持自定义错误页
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl edit ds -n ingress-nginx
daemonset.apps/ingress-nginx-controller edited
配置 ConfigMap，定义哪些错误码被重定向到自定义错误页：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl edit cm -n ingress-nginx ingress-nginx-controller
apiVersion: v1
data:
allow-snippet-annotations: "true"
custom-http-errors: 404,502,503
denylist-source-range: 10.122.26.134更新完成以后访问一个不存在的页面，比如之前定义的 nginx.test.com。访问一个不存在的页面 123，就会返回 Error Server 中的页面：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# curl dujie.test.com/asdasd
<span>The page you're looking for could not be found.</span>root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# 2.3.10 匹配请求头区分不同客户端
首先部署移动端应用：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl create deploy phone --image=registry.cn-beijing.aliyuncs.com/dotbalo/nginx:phone -n study-ingress
deployment.apps/phone created
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl expose deploy phone --port 80 -n study-ingress
service/phone exposed部署电脑端应用：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl create deploy laptop --image=registry.cn-beijing.aliyuncs.com/dotbalo/nginx:laptop -n study-ingress
deployment.apps/laptop created
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl expose deploy laptop --port 80 -n study-ingress
service/laptop exposed
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl get pods -n study-ingress -l app=laptop
NAME READY STATUS RESTARTS AGE
laptop-664b565969-n4mpl 1/1 Running 0 18s之后创建电脑端的 Ingress，注意 Ingress annotations 的 nginx.ingress.kubernetes.io/server-snippet 配置。Snippet 配置是专门用于一些复杂的 Nginx 配置，和 Nginx 配置通用。匹配移动端实例如下：
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/server-snippet: |
set $agentflag 0;
if ($http_user_agent ~* "(Android|iPhone|Windows Phone|UC|Kindle)" ){
set $agentflag 1;
}
if ( $agentflag = 1 ) {
return 301 http://phone.test.com;
}
name: laptop
namespace: study-ingress
spec:
ingressClassName: nginx
rules:
- host: test.com
http:
paths:
- backend:
service:
name: laptop
port:
number: 80
path: /
pathType: ImplementationSpecific首先通过浏览器访问 test.com，可以看到页面是 Laptop：
接下来使用浏览器的开发者工具将终端类型改为 iPhone，或者直接用 iPhone 手机访问（线上业务一般配置的都有 DNS，可以直接解析域名，测试环境可能需要自己单独配置），如下图所示：
2.3.11 灰度/金丝雀/蓝绿发布
####### 2.3.11.1 创建v1版本
首先创建模拟 Production（生产）环境的 Namespace 和服务：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl create ns production
namespace/production created
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress#
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl create deploy canary-v1 --image=registry.cn-beijing.aliyuncs.com/dotbalo/canary:v1 -n production
deployment.apps/canary-v1 created
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl expose deploy canary-v1 -n production --port 8080
service/canary-v1 exposed
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress#
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl get pods -n production
NAME READY STATUS RESTARTS AGE
canary-v1-64b9cdfd5d-zjtgh 1/1 Running 0 22s创建V1的ingress
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# cat canary-v1-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: canary-v1
namespace: production
spec:
ingressClassName: nginx
rules:
- host: canary.com
http:
paths:
- backend:
service:
name: canary-v1
port:
number: 8080
path: /
pathType: Prefix使用浏览器访问该服务，可以看到 Canary v1 的页面：
####### 2.3.11.2 创建v2版本
下面创建v2版本，充当灰度环境
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl create ns canary
namespace/canary created
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl create deploy canary-v2 --image=registry.cn-beijing.aliyuncs.com/dotbalo/canary:v2 -n canary
deployment.apps/canary-v2 created
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress#
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl expose deploy canary-v2 --port 8080 -n canary
service/canary-v2 exposed待程序启动完成后，通过 Service 访问该服务，会返回 Canary v2：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl get svc -n canary
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
canary-v2 ClusterIP 10.16.245.1 <none> 8080/TCP 23s
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# curl 10.16.245.1:8080
<h1>Canary v2</h1>接下来通过 Ingress 控制流量，实现逐步把流量切入到新版本。
####### 2.3.11.3 Canary版本切入部分流量
创建 v2 版本的 Ingress 时，需要添加两个注释，一个是 nginx.ingress.kubernetes.io/canary，表明是灰度环境，nginx.ingress.kubernetes.io/canary-weight 表明切多少流量到该环境，本示例为10%：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# cat canary-v2-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "10"
name: canary-v2
namespace: canary
spec:
ingressClassName: nginx
rules:
- host: canary.com
http:
paths:
- backend:
service:
name: canary-v2
port:
number: 8080
path: /
pathType: ImplementationSpecific此时通过 nginx.ingress.kubernetes.io/canary-weight: “10”设置的权重是 10，即 v1:v2 为 9:1。
再次访问可以看到页面会来回显示 v1 和 v2：
####### 2.3.11.4 测试灰度发布
接下来使用 Ruby 脚本进行测试，此脚本会输出 v1 和 v2 的访问次数比值：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# cat test-canary.rb
counts = Hash.new(0)
100.times do
output = `curl -s canary.com | grep 'Canary' | awk '{print $2}' | awk -F"<" '{print $1}'`
counts[output.strip.split.last] += 1
end
puts counts
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# apt install ruby
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# ruby test-canary.rb
{"v1"=>87, "v2"=>13}
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress#
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# ruby test-canary.rb
{"v1"=>92, "v2"=>8}2.3.12 Ingress常见报错排查
2.3.12.1 404（Not Found）报错
404 表示访问的路由不存在。通常问题如下：
- ingress 路径配置的不正确
- ingress的配置未被Controller解析
- 未使用正确的域名和路径访问
- 代理的服务没有该路径
场景分析：有一个项目包含一个前端，两个后端，域名为project.test.com。
其中路径 / 指向前端
路径/userapi 指向user服务
路径paymentapi指向payment服务
部署服务：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl create deploy user --image=registry.cn-beijing.aliyuncs.com/dotbalo/ingress-err:v1 -n study-ingress
deployment.apps/user created
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl create deploy payment --image=registry.cn-beijing.aliyuncs.com/dotbalo/ingress-err:v1 -n study-ingress
deployment.apps/payment created
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress#
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl get pods -n study-ingress
NAME READY STATUS RESTARTS AGE
payment-6db9544664-mw58v 1/1 Running 0 4s
user-6c76495bf9-8qw9d 1/1 Running 0 15s配置service
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl expose deployment -n study-ingress user --port 8080
service/user exposed
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl expose deployment -n study-ingress payment --port 8080
service/payment exposed配置ingress
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# cat user-payment-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: user-payment
namespace: study-ingress
annotations:
spec:
ingressClassName: nginx
rules:
- host: project.test.com
http:
paths:
- backend:
service:
name: user
port:
number: 8080
path: /user
pathType: Prefix
- backend:
service:
name: payment
port:
number: 8080
path: /payment
pathType: Prefix测试请求：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# curl project.test.com/payment
{"msg":"支付服务","商品名称":"未知商品"}
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# curl -X POST project.test.com/user
{"msg":"登录成功","用户名":"匿名用户"}此时访问是没有任何问题的，但是有可能程序提供的接口/api 或者/user，但是要求的接口并不是/api 和/user，比如要求的是 userapi 和 paymentapi，此时直接配置会报 404。
2.3.12.2 404（Request Entity Too Large）报错
有时候需要上传一些大文件给程序，但是 nginx 默认允许的最大文件大小只有 8M，不足以满足生产最大上传需求，此时可以通过 nginx.ingress.kubernetes.io/proxy-body-size 参数进行更改（也可以在 ConfigMap 中全局添加）：
apiVersion: networking.k8s.io/v1 # k8s >= 1.22 必须 v1
kind: Ingress
metadata:
name: nginx-ingress
namespace: study-ingress
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 32m
spec:
ingressClassName: nginx # 指定该 Ingress 被哪个 Controller 解析
rules: # 定义路由匹配规则，可以配置多个
- host: dujie.test.com # 定义域名
http: #
paths: # 详细的路由配置，可以配置多个
- backend: # 指定该路由的后端
service:
name: nginx
port:
number: 80
path: / # 指定 PATH
pathType: ImplementationSpecific # 指定匹配规则2.3.12.3 503（Service Unavaiable）报错
503 一般是代理的服务不可用导致的，通常问题如下：
1. Ingress 代理配置错误，比如 Service 名字或端口写错
2. Ingress 代理的 Service 不存在
3. Ingress 代理的 Service 后端 Pod 不正常
2.3.12.4 504（Gateway Timeout）报错
504 一般是代理的服务处理请求的时间过长，导致 Nginx 等待超时，此时需要确认服务的处理时长，或者查看服务是否有问题。
部署后端程序模拟报错：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl create deploy ingress-err --image=registry.cn-beijing.aliyuncs.com/dotbalo/ingress-err:v1 -n study-ingress
deployment.apps/ingress-err created
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress#
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl expose deploy ingress-err --port 8080 -n study-ingress
service/ingress-err exposed
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# kubectl get svc -n study-ingress
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-err ClusterIP 10.16.231.176 <none> 8080/TCP 6s请求接口，返回时间比较长：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# curl 10.16.231.176:8080/longtime
{"msg":"任务处理成功","处理时间":45}
配置ingress
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# cat longtime-ingress.yaml
apiVersion: networking.k8s.io/v1 # k8s >= 1.22 必须 v1
kind: Ingress
metadata:
name: longtime
namespace: study-ingress
annotations:
spec:
ingressClassName: nginx # 指定该 Ingress 被哪个 Controller 解析
rules: # 定义路由匹配规则，可以配置多个
- host: longtime.test.com # 定义域名
http: #
paths: # 详细的路由配置，可以配置多个
- backend: # 指定该路由的后端
service:
name: ingress-err
port:
number: 8080
path: / # 指定 PATH
pathType: ImplementationSpecific # 指定匹配规则测试访问
# curl http://longtime.test.com/longtime
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>配置合适的超时时间：
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# cat longtime-ingress.yaml
apiVersion: networking.k8s.io/v1 # k8s >= 1.22 必须 v1
kind: Ingress
metadata:
name: longtime
namespace: study-ingress
annotations:
nginx.ingress.kubernetes.io/proxy-connect-timeout: "120"
nginx.ingress.kubernetes.io/proxy-send-timeout: "120"
nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
spec:
ingressClassName: nginx # 指定该 Ingress 被哪个 Controller 解析
rules: # 定义路由匹配规则，可以配置多个
- host: longtime.test.com # 定义域名
http: #
paths: # 详细的路由配置，可以配置多个
- backend: # 指定该路由的后端
service:
name: ingress-err
port:
number: 8080
path: / # 指定 PATH
pathType: ImplementationSpecific # 指定匹配规则2.3.12.5 CORS跨域报错
root@VM-26-130-ubuntu:/usr/local/src/k8s/ingress# cat cors-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: longtime
namespace: study-ingress
annotations:
nginx.ingress.kubernetes.io/enable-cors: "true"
# 允许跨域的请求方法
nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS, DELETE"
# 允许携带的请求头
nginx.ingress.kubernetes.io/cors-allow-headers: "X-Forwarded-For,X-app123-XPTO"
# 允许跨域的域名
nginx.ingress.kubernetes.io/cors-allow-origin: "*"
spec:
ingressClassName: nginx
rules:
- host: longtime.test.com
http:
paths:
- backend:
service:
name: ingress-err
port:
number: 8080
path: /
pathType: Prefix本博客所有文章除特别声明外，均采用 CC BY-NC-SA 4.0 许可协议。转载请注明来自 J.のblog！
微信
支付宝
相关推荐
评论
