最近看到一篇公众号讲了nginx-ingress-controller的应用。下面有人评论如何做日志持久化，刚好工作上遇到该问题，整理一个方案，仅供参考。
nginx-ingress-controller的日志
nginx-ingress-controller的日志包括三个部分：
- controller日志： 输出到stdout，通过启动参数中的–log_dir可已配置输出到文件，重定向到文件后会自动轮转，但不会自动清理
- accesslog：输出到stdout，通过nginx-configuration中的字段可以配置输出到哪个文件。输出到文件后不会自动轮转或清理
- errorlog：输出到stderr，配置方式与accesslog类似。
给controller日志落盘
- 给nginx-ingress-controller挂一个hostpath： /data/log/nginx_ingress_controller/ 映射到容器里的/var/log/nginx_ingress_controller/ ，
- 给nginx-ingress-controller配置log-dir和logtostderr参数，将日志重定向到/var/log/nginx_ingress_controller/中。
controller的日志需要做定时清理。由于controller的日志是通过klog(k8s.io/klog)输出的，会进行日志滚动，所以我们通过脚本定时清理一定时间之前的日志文件即可。
给nginx日志落盘
修改configmap： nginx-configuration。配置accesslog和errorlog的输出路径，替换默认的stdout和stderr。输出路径我们可以与controller一致，便于查找。
accesslog和errorlog都只有一个日志文件，我们可以使用logrotate进行日志轮转，将输出到宿主机上的日志进行轮转和清理。配置如：
|
1
2
3
4
5
6
7
8
9
10
|
$ cat /etc/logrotate.d/nginx.log/data/log/nginx_ingress_controller/access.log { su root list rotate 7 daily maxsize 50M copytruncate missingok create 0644 www-data root} |
官方提供的模板中，nginx-ingress-controller默认都是以33这个用户登录启动容器的，因此挂载hostpath路径时存在权限问题。我们需要手动在机器上执行chown -R 33:33 /data/log/nginx_ingress_controller.
自动化ops
nginx日志落盘中，第2、3两点均需要人工运维，有什么解决办法吗？
问题的关键是：有什么办法可以在nginx-ingress-controller容器启动之前加一个hook，将宿主机的指定目录执行chown呢？
可以用initContainer。initcontainer必须在containers中的容器运行前运行完毕并成功退出。利用这一k8s特性，我们开发一个docker image，里面只执行如下脚本：
|
1
2
3
4
5
|
#!/bin/bashlogdir=$LOG_DIRuserID=$USER_IDecho "try to set dir: $logdir 's group as $userID"chown -R $userID:$userID $logdir |
脚本读取一些环境变量， 确认需要修改哪个目录，改成怎样的user group。
将脚本打包成dockerimage， 放在nginx-ingress-controller的deploy yaml中，作为initcontainers。 注意要对该initcontainer配置环境变量和volumeMount.
再说第二点，我们注意到nginx-ingress-controller的基础镜像中就自带了logrotate，那么问题就简单了，我们将写好的logrotate配置文件以configmap的形式挂载到容器中就可以了。
一个deploy yaml如下：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
|
---apiVersion: v1kind: Servicemetadata: name: ingress-nginx namespace: kube-systemspec: type: ClusterIP ports: - name: http port: 80 targetPort: 80 protocol: TCP - name: https port: 443 targetPort: 443 protocol: TCP selector: app: ingress-nginx---apiVersion: v1kind: Servicemetadata: name: default-http-backend namespace: kube-system labels: app: default-http-backendspec: ports: - port: 80 targetPort: 8080 selector: app: default-http-backend---apiVersion: extensions/v1beta1kind: Ingressmetadata: name: default namespace: kube-systemspec: backend: serviceName: default-http-backend servicePort: 80---kind: ConfigMapapiVersion: v1metadata: name: nginx-configuration namespace: kube-system labels: app: ingress-nginxdata: use-forwarded-headers: "true" # 此处配置nginx日志的重定向目标 access-log-path: /var/log/nginx_ingress_controller/access.log error-log-path: /var/log/nginx_ingress_controller/error.log---# 创建一个configmap，配置nginx日志的轮转策略，对应的是nginx日志在容器内的日志文件apiVersion: v1data: nginx.log: | {{ user_nginx_log.host_path }}/access.log { rotate {{ user_nginx_log.rotate_count }} daily maxsize {{ user_nginx_log.rotate_size }} minsize 10M copytruncate missingok create 0644 root root } {{ user_nginx_log.host_path }}/error.log { rotate {{ user_nginx_log.rotate_count }} daily maxsize {{ user_nginx_log.rotate_size }} minsize 10M copytruncate missingok create 0644 root root }kind: ConfigMapmetadata: name: nginx-ingress-logrotate namespace: kube-system---kind: ConfigMapapiVersion: v1metadata: name: tcp-services namespace: kube-system---kind: ConfigMapapiVersion: v1metadata: name: udp-services namespace: kube-system---apiVersion: v1kind: ServiceAccountmetadata: name: nginx-ingress-serviceaccount namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRolemetadata: name: nginx-ingress-clusterrolerules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - "extensions" resources: - ingresses verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - "extensions" resources: - ingresses/status verbs: - update---apiVersion: rbac.authorization.k8s.io/v1beta1kind: Rolemetadata: name: nginx-ingress-role namespace: kube-systemrules: - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - apiGroups: - "" resources: - configmaps resourceNames: # Defaults to "<election-id>-<ingress-class>" # Here: "<ingress-controller-leader>-<nginx>" # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller. - "ingress-controller-leader-nginx" verbs: - get - update - apiGroups: - "" resources: - configmaps verbs: - create - apiGroups: - "" resources: - endpoints verbs: - get---apiVersion: rbac.authorization.k8s.io/v1beta1kind: RoleBindingmetadata: name: nginx-ingress-role-nisa-binding namespace: kube-systemroleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: nginx-ingress-rolesubjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: kube-system---apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRoleBindingmetadata: name: nginx-ingress-clusterrole-nisa-bindingroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: nginx-ingress-clusterrolesubjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: kube-system---apiVersion: apps/v1kind: DaemonSetmetadata: name: ingress-nginx namespace: kube-systemspec: selector: matchLabels: app: ingress-nginx template: metadata: labels: app: ingress-nginx annotations: prometheus.io/port: '10254' prometheus.io/scrape: 'true' spec: serviceAccountName: nginx-ingress-serviceaccount tolerations: - key: dedicated value: ingress-nginx effect: NoSchedule affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "system/ingress" operator: In values: - "true" dnsPolicy: ClusterFirstWithHostNet hostNetwork: true # 配置initcontainer，确保在nginx-ingress-controller容器启动前将日志目录的权限配置好 initContainers: - name: adddirperm image: "{{ image_registry.addr }}/{{ image.adddirperm }}" env: - name: LOG_DIR value: /var/log/nginx_ingress_controller - name: USER_ID value: "33" volumeMounts: - name: logdir mountPath: /var/log/nginx_ingress_controller containers: - name: nginx-ingress-controller image: "{{ image_registry.addr }}/{{ image.ingress }}" imagePullPolicy: IfNotPresent args: - /nginx-ingress-controller - --default-backend-service=$(POD_NAMESPACE)/default-http-backend - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --publish-service=$(POD_NAMESPACE)/ingress-nginx - --annotations-prefix=nginx.ingress.kubernetes.io # 设置controller日志的输出路径和方式 - --log_dir=/var/log/nginx_ingress_controller - --logtostderr=false securityContext: capabilities: drop: - ALL add: - NET_BIND_SERVICE # www-data -> 33 runAsUser: 33 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 resources: requests: cpu: 100m memory: 256Mi livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 volumeMounts: # 配置挂载容器中控制器组件和nginx的日志输出路径 - name: logdir mountPath: /var/log/nginx_ingress_controller # 配置nginx日志的logrotate配置挂载路径 - name: logrotateconf mountPath: /etc/logrotate.d/nginx.log subPath: nginx.log volumes: # 控制器组件和nginx的日志输出路径为宿主机的hostpath - name: logdir hostPath: path: {{ user_nginx_log.host_path }} type: "" # nginx日志的轮转配置文件来自于configmap - name: logrotateconf configMap: name: nginx-ingress-logrotate items: - key: nginx.log path: nginx.log---apiVersion: apps/v1kind: DaemonSetmetadata: name: default-http-backend namespace: kube-system labels: app: default-http-backendspec: selector: matchLabels: app: default-http-backend template: metadata: labels: app: default-http-backend spec: terminationGracePeriodSeconds: 60 tolerations: - key: dedicated value: ingress-nginx effect: NoSchedule affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "system/ingress" operator: In values: - "true" containers: - name: default-http-backend # Any image is permissible as long as: # 1. It serves a 404 page at / # 2. It serves 200 on a /healthz endpoint image: "{{ image_registry.addr }}/{{ image.http_backend }}" imagePullPolicy: IfNotPresent livenessProbe: httpGet: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 ports: - containerPort: 8080 resources: limits: cpu: 10m memory: 20Mi requests: cpu: 10m memory: 20Mi--- |
最后，有的人建议将initcontainer去掉，改为基于原有的nginx-ingress-controller镜像加一层layer，将配置路径权限的脚本放在该层执行。 个人认为这种方法既不美观，也不方便。唯一的好处仅在于deploy yaml仍然简洁（但少不了volumeMount之类的配置）。不过还是看个人使用感受吧~
到此这篇关于nginx-ingress-controller日志持久化方案的解决的文章就介绍到这了,更多相关nginx ingress controller日志持久化内容请搜索服务器之家以前的文章或继续浏览下面的相关文章希望大家以后多多支持服务器之家！
原文链接：https://segmentfault.com/a/1190000022200374
