zhangguanzhang's Blog

一次 cni-plugins 导致集群 dns 无法解析的排错

kubernetes cni-plugins
字数统计: 1.4k阅读时长: 7 min
2021/08/24

问题

环境是 1.15.5 的 x86_64 的 k8s 。命令输出被我查看日志给冲掉了，大致描述下。
中间件 kafka 无法连上 zookeeper ，看了下日志报错域名无法解析。看了下 coredns 都挂了：

1
2
3
4
5
$ kubectl -n kube-system get po -o wide -l k8s-app=kube-dns
NAME                       READY   STATUS             RESTARTS   AGE     IP           NODE         NOMINATED NODE   READINESS GATES
coredns-5757945748-l2d2g   0/1     CrashLoopBackOff   254        3d11h   172.27.0.2   10.25.1.55   <none>           <none>
coredns-5757945748-w5pfx   0/1     CrashLoopBackOff   254        3d11h   172.27.0.5   10.25.1.55   <none>           <none>
coredns-5757945748-wfndd   0/1     CrashLoopBackOff   254        3d11h   172.27.0.3   10.25.1.55   <none>           <none>

查看了日志是报错无法连 kubernetes svc https://172.26.0.1:443/xxxx ，报错 No route to host

排错

基本排查

去节点 10.25.1.55 上 docker ps -a | grep coredns 找 pause 的容器 id ，docker inspect xxxxx | grep -m1 -i pid 取进程 pid。然后 nsenter 进去

1
2
3
4
5
6
7
8
9
10
$ nsenter --net --target 14659 
$ ip a s 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 42:7d:b0:83:a9:aa brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.27.0.5/16 scope global eth0
       valid_lft forever preferred_lft forever

curl 了下 svc ip https://172.26.0.1 发现报错 No route to host。然后直接用 ep 也就是 kube-apiserver 的真实 ip和端口访问下，发现不通

1
curk -kvL https://10.25.1.51:6443

然后 ping 下宿主机发现也不通。看了下转发都开了。然后也看了下也没安全软件。

桥接

然后看了下桥接发现了问题所在，先写下正常环境桥接信息。我们 flannel，不是二进制，容器都是在 cni-plugins 下桥接在 cni0 上的。下面找个正常环境的 coredns 做下信息展示：

1
2
3
4
5
6
7
8
9
10
11
12
13
# 取容器 pid
$ docker inspect d30 | grep -m1 -i pid
            "Pid": 9079,

$ nsenter --net --target 9079 ip a s 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if210: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 06:3e:42:00:91:33 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.27.0.74/24 brd 172.27.0.255 scope global eth0
       valid_lft forever preferred_lft forever

注意看 if 后面的数字，宿主机上查看下是哪个 veth

1
2
$ ip link | grep -E '^210'
210: vetha1ca1d55@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default

使用 brctl 看下 cni0 下是有这个的。

1
2
brctl show cni0 | grep vetha1ca1d55
							vetha1ca1d55

机器异常的桥接信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

$ nsenter --net --target 14659 ip a s 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 42:7d:b0:83:a9:aa brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.27.0.5/16 scope global eth0
       valid_lft forever preferred_lft forever

$ ip -o link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1\    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc pfifo_fast state UP mode DEFAULT qlen 1000\    link/ether fa:16:3e:35:09:13 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000\    link/ether fa:16:3e:e2:4d:8d brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT \    link/ether 02:42:3c:a2:d7:3b brd ff:ff:ff:ff:ff:ff
6: vethda399ca1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT \    link/ether a2:95:74:65:29:6d brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT \    link/ether 0a:e5:a5:9a:66:8b brd ff:ff:ff:ff:ff:ff
8: veth96d8f326@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT \    link/ether 92:a5:9c:7c:dd:cb brd ff:ff:ff:ff:ff:ff link-netnsid 1
9: vethdf8eb371@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT \    link/ether 4a:fb:37:c3:f2:7e brd ff:ff:ff:ff:ff:ff link-netnsid 2
10: veth44ce32f4@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT \    link/ether 6e:96:73:2b:4e:52 brd ff:ff:ff:ff:ff:ff link-netnsid 3
11: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT qlen 1000\    link/ether 2a:f9:74:b7:11:b8 brd ff:ff:ff:ff:ff:ff
12: veth0f91b3a3@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT \    link/ether 0a:36:90:dc:0a:97 brd ff:ff:ff:ff:ff:ff link-netnsid 4

查看了下 cni0 压根对不上 veth44ce32f4:

1
2
3
4
5
6
7
8
9
10
11
$ ip a s cni0
11: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP qlen 1000
    link/ether 2a:f9:74:b7:11:b8 brd ff:ff:ff:ff:ff:ff
    inet 172.27.1.1/24 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::28f9:74ff:feb7:11b8/64 scope link 
       valid_lft forever preferred_lft forever
$ brctl show
bridge name     bridge id               STP enabled     interfaces
cni0            8000.2af974b711b8       no              veth0f91b3a3
docker0         8000.02423ca2d73b       no

应该是桥接错了，重新创建下发现好了：

1
2
3
4
5
6
7
8
9
$ kubectl -n kube-system delete pod -l k8s-app=kube-dns
pod "coredns-5757945748-l2d2g" deleted
pod "coredns-5757945748-wfndd" deleted
pod "coredns-5757945748-wjrdf" deleted
$ kubectl -n kube-system get po -o wide -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE         NOMINATED NODE   READINESS GATES
coredns-5757945748-4smll   1/1     Running   0          29s   172.27.1.3   10.25.1.55   <none>           <none>
coredns-5757945748-d9wqk   1/1     Running   0          29s   172.27.3.6   10.25.1.54   <none>           <none>
coredns-5757945748-dtvfl   1/1     Running   0          29s   172.27.2.6   10.25.1.56   <none>           <none>

总结

现场拿了下 cni-plugins 的校验值去查了下，我们使用的是 v0.7.0 版本。有必要升级下了。

原文作者：Zhangguanzhang

原文链接：http://zhangguanzhang.github.io/2021/08/24/cni-plugins-bridge-err/

发表日期：August 24th 2021, 1:08:06 pm

更新日期：August 24th 2021, 1:08:06 pm

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
  1. 1. 问题
  2. 2. 排错
    1. 2.1. 基本排查
    2. 2.2. 桥接
      1. 2.2.1. 机器异常的桥接信息
  3. 3. 总结
Total : 186
2024
2023
2022
2021
2020
2019
2018
2017
2016
linux 微信 diy 8266 总结 ipmi CLP k8s Mismatch Altium Designer ansible consul docker lnmp 安卓 Linux kernel prometheus exporter boot grub cad tcp golang centos6 chromedp chrony kubernetes cni-plugins mips64le kubeconfig swap dlv adguardhome dns Dockerfile e2fsck fsck keepavlied ecs xmrig etcd fio flannel ftp go mod go gobot esp8266 hello world host-gw mqtt 原创 jira confluence kube-controller-manager java springboot vip keepalived ConfigMap PodAffinity kylin arm64 Kylin audit fs nginx microsoft graph onedrive nodeport usb-net overlay kernal preseed Prometheus openstack cloud-init python containerd keepalive shell bash skopeo ssh kubelet systemd dockerhub gcr.io api ubuntu18 initrd tftp uos x86 openwrt zabbix zombie 正则 awk EmuELEC ingress N1 cobra hang cifs http lvs ipvsadm kube-proxy 63s timeout lamp nfc pm3 coredns openshift ocp squashfs timezone rpm rpmbuild segfault suse wireguard raid kubeadm operator exsi redhat8.4 vxlan veth panic supervisor pyinstaller ttf woff hostname qemu binfmt_misc loongarch64 externalIP cgo nfs ipset proxmox non-root containers vmware hung apparmor serviceAccountToken gpu nvidia cri-dockerd pprof trivy redis /etc/resolv.conf hostPort hotplug android headscale derper iptables
日志 心得 闲搞 ipmi kubernetes ansible kubelet Linux 心得 prometheus boot Auto cad boot golang kubeconfig grub linux centos6 容器 docker docker 容器 openstack boot vsftp go version panic gobot 闲聊 iptables admission kube-controller-manager keepalived k8s Prometheus Job k8s lua microsoft openwrt preseed Prometheus proxmox 总结 CI Kylin ubuntu18 pxe uos graph 运维 zombie 技巧 travis docker onedrive kickstart consul fstab http vxlan coredns node-cache golang kmem suse wireguard raid kubeadm LVM kickstart prometheus raid