zhangguanzhang's Blog

一次 cni-plugins 导致集群 dns 无法解析的排错

字数统计: 1.4k阅读时长: 7 min
2021/08/24

问题

环境是 1.15.5 的 x86_64 的 k8s 。命令输出被我查看日志给冲掉了,大致描述下。
中间件 kafka 无法连上 zookeeper ,看了下日志报错域名无法解析。看了下 coredns 都挂了:

1
2
3
4
5
$ kubectl -n kube-system get po -o wide -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-5757945748-l2d2g 0/1 CrashLoopBackOff 254 3d11h 172.27.0.2 10.25.1.55 <none> <none>
coredns-5757945748-w5pfx 0/1 CrashLoopBackOff 254 3d11h 172.27.0.5 10.25.1.55 <none> <none>
coredns-5757945748-wfndd 0/1 CrashLoopBackOff 254 3d11h 172.27.0.3 10.25.1.55 <none> <none>

查看了日志是报错无法连 kubernetes svc https://172.26.0.1:443/xxxx ,报错 No route to host

排错

基本排查

去节点 10.25.1.55docker ps -a | grep coredns 找 pause 的容器 id ,docker inspect xxxxx | grep -m1 -i pid 取进程 pid。然后 nsenter 进去

1
2
3
4
5
6
7
8
9
10
$ nsenter --net --target 14659 
$ ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 42:7d:b0:83:a9:aa brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.27.0.5/16 scope global eth0
valid_lft forever preferred_lft forever

curl 了下 svc ip https://172.26.0.1 发现报错 No route to host。然后直接用 ep 也就是 kube-apiserver 的真实 ip和端口访问下,发现不通

1
curk -kvL https://10.25.1.51:6443

然后 ping 下宿主机发现也不通。看了下转发都开了。然后也看了下也没安全软件。

桥接

然后看了下桥接发现了问题所在,先写下正常环境桥接信息。我们 flannel,不是二进制,容器都是在 cni-plugins 下桥接在 cni0 上的。下面找个正常环境的 coredns 做下信息展示:

1
2
3
4
5
6
7
8
9
10
11
12
13
# 取容器 pid
$ docker inspect d30 | grep -m1 -i pid
"Pid": 9079,

$ nsenter --net --target 9079 ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: eth0@if210: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 06:3e:42:00:91:33 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.27.0.74/24 brd 172.27.0.255 scope global eth0
valid_lft forever preferred_lft forever

注意看 if 后面的数字,宿主机上查看下是哪个 veth

1
2
$ ip link | grep -E '^210'
210: vetha1ca1d55@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default

使用 brctl 看下 cni0 下是有这个的。

1
2
brctl show cni0 | grep vetha1ca1d55
vetha1ca1d55

机器异常的桥接信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

$ nsenter --net --target 14659 ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 42:7d:b0:83:a9:aa brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.27.0.5/16 scope global eth0
valid_lft forever preferred_lft forever

$ ip -o link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc pfifo_fast state UP mode DEFAULT qlen 1000\ link/ether fa:16:3e:35:09:13 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000\ link/ether fa:16:3e:e2:4d:8d brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT \ link/ether 02:42:3c:a2:d7:3b brd ff:ff:ff:ff:ff:ff
6: vethda399ca1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT \ link/ether a2:95:74:65:29:6d brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT \ link/ether 0a:e5:a5:9a:66:8b brd ff:ff:ff:ff:ff:ff
8: veth96d8f326@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT \ link/ether 92:a5:9c:7c:dd:cb brd ff:ff:ff:ff:ff:ff link-netnsid 1
9: vethdf8eb371@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT \ link/ether 4a:fb:37:c3:f2:7e brd ff:ff:ff:ff:ff:ff link-netnsid 2
10: veth44ce32f4@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT \ link/ether 6e:96:73:2b:4e:52 brd ff:ff:ff:ff:ff:ff link-netnsid 3
11: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT qlen 1000\ link/ether 2a:f9:74:b7:11:b8 brd ff:ff:ff:ff:ff:ff
12: veth0f91b3a3@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT \ link/ether 0a:36:90:dc:0a:97 brd ff:ff:ff:ff:ff:ff link-netnsid 4

查看了下 cni0 压根对不上 veth44ce32f4:

1
2
3
4
5
6
7
8
9
10
11
$ ip a s cni0
11: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP qlen 1000
link/ether 2a:f9:74:b7:11:b8 brd ff:ff:ff:ff:ff:ff
inet 172.27.1.1/24 scope global cni0
valid_lft forever preferred_lft forever
inet6 fe80::28f9:74ff:feb7:11b8/64 scope link
valid_lft forever preferred_lft forever
$ brctl show
bridge name bridge id STP enabled interfaces
cni0 8000.2af974b711b8 no veth0f91b3a3
docker0 8000.02423ca2d73b no

应该是桥接错了,重新创建下发现好了:

1
2
3
4
5
6
7
8
9
$ kubectl -n kube-system delete pod -l k8s-app=kube-dns
pod "coredns-5757945748-l2d2g" deleted
pod "coredns-5757945748-wfndd" deleted
pod "coredns-5757945748-wjrdf" deleted
$ kubectl -n kube-system get po -o wide -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-5757945748-4smll 1/1 Running 0 29s 172.27.1.3 10.25.1.55 <none> <none>
coredns-5757945748-d9wqk 1/1 Running 0 29s 172.27.3.6 10.25.1.54 <none> <none>
coredns-5757945748-dtvfl 1/1 Running 0 29s 172.27.2.6 10.25.1.56 <none> <none>

总结

现场拿了下 cni-plugins 的校验值去查了下,我们使用的是 v0.7.0 版本。有必要升级下了。

CATALOG
  1. 1. 问题
  2. 2. 排错
    1. 2.1. 基本排查
    2. 2.2. 桥接
      1. 2.2.1. 机器异常的桥接信息
  3. 3. 总结