服务器之家:专注于服务器技术及软件下载分享
分类导航

服务器资讯|IT/互联网|云计算|区块链|软件资讯|操作系统|手机数码|百科知识|免费资源|头条新闻|

服务器之家 - 新闻资讯 - 云计算 - 如何快速查看 Kubernetes Pod 崩溃前的日志

如何快速查看 Kubernetes Pod 崩溃前的日志

2023-07-08 12:00未知服务器之家 云计算

当pod处于crash状态的时候,容器不断重启,此时用 kubelet logs 可能出现一直捕捉不到日志。解决方法: kubectl previous 参数作用: If true, print the logs for the previous instance of the container in a pod if it exists. 单容器pod: kubectllogspod-name--previo

当pod处于crash状态的时候,容器不断重启,此时用 kubelet logs可能出现一直捕捉不到日志。解决方法:

kubectl previous参数作用: If true, print the logs for the previous instance of the container in a pod if it exists.

  • 单容器pod:

kubectllogspod-name--previous

  • 多容器pod:

kubectllogspod-name--previous-ccontainer-name

比如:


NAMEREADYSTATUSRESTARTSAGE

nginx-7d8b49557c-c2lx92/2Running5

kubectllogsnginx-7d8b49557c-c2lx9--previous
Error:xxxxxxxxxxx

kubelet会保持pod的前几个失败的容器,这个是查看的前提条件。kubelet实现previous的原理:将pod的日志存放在 /var/log/pods/podname,并且是链接文件,链接到docker的容器的日志文件,同时kubelet还会保留上一个容器,同时有一个链接文件链接到pod上一个崩溃的容器的日志文件,使用previous就是查看的这个文件

比如查看一个pod:


ubuntu@~$kubeletgetpod

NAMEREADYSTATUSRESTARTSAGE
busybox1/1Running239499d
nginx-deployment-6wlhd1/1Running079d
redis1/1Running049d

到pod所在node查看kubelet放的两个日志文件:


ls/var/log/pods/default_busybox_f72ab71a-5b3b-4ecf-940d-28a5c3b30683/busybox

2393.log2394.log

数字的含义: 2393证明是第 2393 次重启后的日志, 2394代表是第2394次重启后的日志。

实际这两个日志文件是链接文件,指向了docker的日志文件:



/busybox#

stat2393.log

File:2393.log->/data/kubernetes/docker/containers/68a5b32c9fdb1ad011b32e6252f9cdb759f69d7850e6b7b8591cb4c2bf00bcca/68a5b32c9fdb1ad011b32e6252f9cdb759f69d7850e6b7b8591cb4c2bf00bcca-json.log
Size:173Blocks:8IOBlock:4096symboliclink
Device:fc02h/64514dInode:529958Links:1
Access:(0777/lrwxrwxrwx)Uid:(0/root)Gid:(0/root)
Access:2023-01-3113:32:03.751514283+0800
Modify:2023-01-3113:32:03.039526838+0800
Change:2023-01-3113:32:03.039526838+0800
Birth:-

/busybox#
stat2394.log
File:2394.log->/data/kubernetes/docker/containers/2ed9ebf0585215602874b076783e12191dbb010116038b8eb4646273ebfe195c/2ed9ebf0585215602874b076783e12191dbb010116038b8eb4646273ebfe195c-json.log
Size:173Blocks:8IOBlock:4096symboliclink
Device:fc02h/64514dInode:529955Links:1
Access:(0777/lrwxrwxrwx)Uid:(0/root)Gid:(0/root)
Access:2023-01-3114:32:03.991106950+0800
Modify:2023-01-3114:32:03.183119308+0800
Change:2023-01-3114:32:03.183119308+0800
Birth:-

看到分别指向了这两个容器的日志文件,一个是当前pod里在跑的容器,一个是pod上次跑的容器,现在已经退出了


dockerps-a

CONTAINERIDIMAGECOMMANDCREATEDSTATUSPORTSNAMES
2ed9ebf05852ff4a8eb070e1"sleep3600"24minutesagoUp24minutesk8s_busybox_busybox_default_f72ab71a-5b3b-4ecf-940d-28a5c3b30683_2394
68a5b32c9fdbff4a8eb070e1"sleep3600"AboutanhouragoExited(0)24minutesagok8s_busybox_busybox_default_f72ab71a-5b3b-4ecf-940d-28a5c3b30683_2393

使用logs的时候读的是当前容器那个文件,使用 –previous的时候,读的是上次退出的容器的日志文件,由于kubelet为pod保留了上次退出的容器。

我们手动编辑这两个文件的内容,看kubelet是否读的是这两个文件



/busybox#
cat2393.log

{"log":"lastcrashlogs\n","stream":"stderr","time":"2022-11-05T08:11:27.31523845Z"}

/busybox#
cat2394.log
{"log":"nowpodlog\n","stream":"stderr","time":"2022-11-05T08:11:27.31523845Z"}

ubuntu@10-234-32-51:~$klogsbusybox--previous
lastcrashlogs
ubuntu@10-234-32-51:~$klogsbusybox
nowpodlog

由于是链接文件,那么可能实际是从别的地方读的,或者说直接读容器目录下的,由于链接文件我们改了后容器目录下的日志文件也跟着改了,我们直接创建两个文件来做验证:


ubuntu@10-234-32-51:~$kgetpod

NAMEREADYSTATUSRESTARTSAGE
busybox1/1Running239599d
nginx-deployment-6wlhd1/1Running079d
redis1/1Running049d

/busybox#
ls
2394.log2395.log

/busybox#
rm2394.log2395.log

我们删除,然后自己创建,这时是regular file,而不是链接文件了:
/busybox# ls
2394.log2395.log

/busybox#
stat2394.log
File:2394.log
Size:100Blocks:8IOBlock:4096regularfile
Device:fc02h/64514dInode:529965Links:1
Access:(0640/-rw-r-----)Uid:(0/root)Gid:(0/root)
Access:2023-01-3115:42:11.307170422+0800
Modify:2023-01-3115:42:07.711225229+0800
Change:2023-01-3115:42:07.711225229+0800
Birth:-

/busybox#
stat2395.log
File:2395.log
Size:86Blocks:8IOBlock:4096regularfile
Device:fc02h/64514dInode:529967Links:1
Access:(0640/-rw-r-----)Uid:(0/root)Gid:(0/root)
Access:2023-01-3115:41:17.539989934+0800
Modify:2023-01-3115:41:14.348038586+0800
Change:2023-01-3115:41:14.352038525+0800
Birth:-

/busybox#
cat2394.log
{"log":"previouslogscreatebymyself\n","stream":"stderr","time":"2022-11-05T08:11:27.31523845Z"}
/busybox# cat2395.log
{"log":"createbymyself\n","stream":"stderr","time":"2022-11-05T08:11:27.31523845Z"}

ubuntu@10-234-32-51:~$klogsbusybox
createbymyself
ubuntu@10-234-32-51:~$klogsbusybox--previous
previouslogscreatebymyself

得出结论:kubelet读的是 /var/log/pods/下的日志文件, –previous读的也是 /var/log/pods/下的日志文件,且专门有个链接文件来指向上一个退出容器的日志文件,以此来获取容器崩溃前的日志。

链接: https://blog.csdn.net/qq_43684922/article/details/128881716


延伸 · 阅读

精彩推荐