zhangguanzhang's Blog

flannel下集群有个节点网络不通的一次排查

kubernetes flannel
字数统计: 1.7k阅读时长: 9 min
2021/08/25

故障

问题和版本没关系，客户的 node 信息啥的后面排错里有。有个节点通信有问题，其余节点都没问题。

排查

惯例信息

先看下 flannel 的 vxlan 的 vtep 信息，客户是双网卡的，但是默认路由是这个网卡，不用管另外的网卡了。下面信息看了下 VtepMAC 和 public-ip 都正常。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
$ kubectl get node -o yaml | grep -B4 public
    annotations:
      flannel.alpha.coreos.com/backend-data: '{"VtepMAC":"76:21:69:41:de:fe"}'
      flannel.alpha.coreos.com/backend-type: vxlan
      flannel.alpha.coreos.com/kube-subnet-manager: "true"
      flannel.alpha.coreos.com/public-ip: 10.25.1.51
--
    annotations:
      flannel.alpha.coreos.com/backend-data: '{"VtepMAC":"b6:61:5c:8d:d9:eb"}'
      flannel.alpha.coreos.com/backend-type: vxlan
      flannel.alpha.coreos.com/kube-subnet-manager: "true"
      flannel.alpha.coreos.com/public-ip: 10.25.1.52
--
    annotations:
      flannel.alpha.coreos.com/backend-data: '{"VtepMAC":"1e:8c:3e:12:fc:0f"}'
      flannel.alpha.coreos.com/backend-type: vxlan
      flannel.alpha.coreos.com/kube-subnet-manager: "true"
      flannel.alpha.coreos.com/public-ip: 10.25.1.53
--
    annotations:
      flannel.alpha.coreos.com/backend-data: '{"VtepMAC":"ba:fe:64:36:6e:a1"}'
      flannel.alpha.coreos.com/backend-type: vxlan
      flannel.alpha.coreos.com/kube-subnet-manager: "true"
      flannel.alpha.coreos.com/public-ip: 10.25.1.54
--
    annotations:
      flannel.alpha.coreos.com/backend-data: '{"VtepMAC":"8e:c1:4d:18:e5:d6"}'
      flannel.alpha.coreos.com/backend-type: vxlan
      flannel.alpha.coreos.com/kube-subnet-manager: "true"
      flannel.alpha.coreos.com/public-ip: 10.25.1.55
--
    annotations:
      flannel.alpha.coreos.com/backend-data: '{"VtepMAC":"fe:95:e6:bf:a0:62"}'
      flannel.alpha.coreos.com/backend-type: vxlan
      flannel.alpha.coreos.com/kube-subnet-manager: "true"
      flannel.alpha.coreos.com/public-ip: 10.25.1.56

coredns 的 pod ip 和 node 分布情况

1
2
3
4
5
$ kubectl -n kube-system get pod -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP           NODE         NOMINATED NODE   READINESS GATES
coredns-5757945748-cr67w              1/1     Running   0          19h   172.27.2.7   10.25.1.56   <none>           <none>
coredns-5757945748-krwfd              1/1     Running   0          19h   172.27.1.4   10.25.1.55   <none>           <none>
coredns-5757945748-zf4zm              1/1     Running   0          19h   172.27.3.7   10.25.1.54   <none>           <none>

排查

curl 下 coredns 的 metrics 接口试试，只有 10.25.1.51 和其他节点无法通信。会导致下面的 curl 卡住。

1
curl  172.27.1.4:9153

目标机器 10.25.1.55 上通过 flannel.1 接口抓我们的 curl 包:

1
2
3
4
5
6
7
8
9
10
$ tcpdump -nn -i flannel.1 host 172.27.1.4 and port 9153 -vv
tcpdump: listening on flannel.1, link-type EN10MB (Ethernet), capture size 262144 bytes
10:07:46.203094 IP (tos 0x0, ttl 64, id 56025, offset 0, flags [DF], proto TCP (6), length 60)
    172.27.0.0.57888 > 172.27.1.4.9153: Flags [S], cksum 0x6804 (correct), seq 879302783, win 28200, options [mss 1410,sackOK,TS val 56279718 ecr 0,nop,wscale 7], length 0
10:07:46.203173 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.27.1.4.9153 > 172.27.0.0.57888: Flags [S.], cksum 0x5969 (incorrect -> 0x163b), seq 4197245653, ack 879302784, win 27960, options [mss 1410,sackOK,TS val 431774697 ecr 56279718,nop,wscale 7], length 0
10:07:47.204797 IP (tos 0x0, ttl 64, id 56026, offset 0, flags [DF], proto TCP (6), length 60)
    172.27.0.0.57888 > 172.27.1.4.9153: Flags [S], cksum 0x641a (correct), seq 879302783, win 28200, options [mss 1410,sackOK,TS val 56280720 ecr 0,nop,wscale 7], length 0
10:07:47.204880 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    172.27.1.4.9153 > 172.27.0.0.57888: Flags [S.], cksum 0x5969 (incorrect -> 0x1251), seq 4197245653, ack 879302784, win 27960, options [mss 1410,sackOK,TS val 431775699 ecr 56279718,nop,wscale 7], length 0

看着是回复了报文 172.27.1.4.9153 > 172.27.0.0.57888，在我们 curl 的机器 10.25.1.51上 lsof -nPi :57888 看到的确实是卡住的 curl 命令 pid 。10.25.1.51 上也同时抓包看下

1
2
3
4
5
6
7
8
$ tcpdump -nn -i flannel.1 host 172.27.1.4 and port 9153 -vv
tcpdump: listening on flannel.1, link-type EN10MB (Ethernet), capture size 262144 bytes
10:08:57.241129 IP (tos 0x0, ttl 64, id 34444, offset 0, flags [DF], proto TCP (6), length 60)
    172.27.0.0.57966 > 172.27.1.4.9153: Flags [S], cksum 0x5969 (incorrect -> 0x2fb2), seq 276913734, win 28200, options [mss 1410,sackOK,TS val 56350922 ecr 0,nop,wscale 7], length 0
10:08:58.242423 IP (tos 0x0, ttl 64, id 34445, offset 0, flags [DF], proto TCP (6), length 60)
    172.27.0.0.57966 > 172.27.1.4.9153: Flags [S], cksum 0x5969 (incorrect -> 0x2bc8), seq 276913734, win 28200, options [mss 1410,sackOK,TS val 56351924 ecr 0,nop,wscale 7], length 0
10:09:00.246423 IP (tos 0x0, ttl 64, id 34446, offset 0, flags [DF], proto TCP (6), length 60)
    172.27.0.0.57966 > 172.27.1.4.9153: Flags [S], cksum 0x5969 (incorrect -> 0x23f4), seq 276913734, win 28200, options [mss 1410,sackOK,TS val 56353928 ecr 0,nop,wscale 7], length 0

没收到包，从 eth1 抓下 flannel 的 8475 端口(配置里我们改了 flannel 的端口)试试:

目标机器 10.25.1.55 上抓包

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ tcpdump -nn -i eth1 host 10.25.1.51 and port 8475 -vvv
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
10:09:40.966705 IP (tos 0x0, ttl 64, id 50110, offset 0, flags [none], proto UDP (17), length 110)
    10.25.1.51.42770 > 10.25.1.55.8475: [no cksum] UDP, length 82
10:09:40.966869 IP (tos 0x0, ttl 64, id 46192, offset 0, flags [none], proto UDP (17), length 110)
    10.25.1.55.48472 > 10.25.1.51.8475: [no cksum] UDP, length 82
10:09:41.968322 IP (tos 0x0, ttl 64, id 50327, offset 0, flags [none], proto UDP (17), length 110)
    10.25.1.51.42770 > 10.25.1.55.8475: [no cksum] UDP, length 82
10:09:41.968440 IP (tos 0x0, ttl 64, id 46957, offset 0, flags [none], proto UDP (17), length 110)
    10.25.1.55.48472 > 10.25.1.51.8475: [no cksum] UDP, length 82
10:09:43.099646 IP (tos 0x0, ttl 64, id 47316, offset 0, flags [none], proto UDP (17), length 110)
    10.25.1.55.48472 > 10.25.1.51.8475: [no cksum] UDP, length 82
10:09:43.972322 IP (tos 0x0, ttl 64, id 51119, offset 0, flags [none], proto UDP (17), length 110)
    10.25.1.51.42770 > 10.25.1.55.8475: [no cksum] UDP, length 82
10:09:43.972454 IP (tos 0x0, ttl 64, id 47934, offset 0, flags [none], proto UDP (17), length 110)
    10.25.1.55.48472 > 10.25.1.51.8475: [no cksum] UDP, length 82
^C

目标机器 10.25.1.51 上抓包:

1
2
3
4
5
6
7
8
$ tcpdump -nn -i eth1 host 10.25.1.55 and port 8475 -vvv
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
10:10:21.702308 IP (tos 0x0, ttl 64, id 6079, offset 0, flags [none], proto UDP (17), length 110)
    10.25.1.51.59558 > 10.25.1.55.8475: [no cksum] UDP, length 82
10:10:22.702441 IP (tos 0x0, ttl 64, id 6117, offset 0, flags [none], proto UDP (17), length 110)
    10.25.1.51.59558 > 10.25.1.55.8475: [no cksum] UDP, length 82
10:10:24.706444 IP (tos 0x0, ttl 64, id 7699, offset 0, flags [none], proto UDP (17), length 110)
    10.25.1.51.59558 > 10.25.1.55.8475: [no cksum] UDP, length 82

完全没报文过来，看了下 flannel 的接口流量压根就没收到任何包:

1
2
3
4
5
6
7
8
9
$ ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.27.0.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::7421:69ff:fe41:defe  prefixlen 64  scopeid 0x20<link>
        ether 76:21:69:41:de:fe  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 28900  bytes 2113052 (2.0 MiB)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0

说明报文从 10.25.1.55 发出后没到 51 上，让客户开通 udp 8475 10.25.1.0/24 整个段的东西向安全组后就正常了。

1
2
3
4
$ curl  172.27.1.4:9153
^C
$ curl  172.27.1.4:9153
404 page not found

原文作者：Zhangguanzhang

原文链接：http://zhangguanzhang.github.io/2021/08/25/flannel-a-host-net-tmout/

发表日期：August 25th 2021, 3:08:06 pm

更新日期：August 25th 2021, 3:08:06 pm

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
  1. 1. 故障
  2. 2. 排查
    1. 2.1. 惯例信息
    2. 2.2. 排查
Total : 186
2024
2023
2022
2021
2020
2019
2018
2017
2016
linux 微信 diy 8266 总结 ipmi CLP k8s Mismatch Altium Designer ansible consul docker lnmp 安卓 Linux kernel prometheus exporter boot grub cad tcp golang centos6 chromedp chrony kubernetes cni-plugins mips64le kubeconfig swap dlv adguardhome dns Dockerfile e2fsck fsck keepavlied ecs xmrig etcd fio flannel ftp go mod go gobot esp8266 hello world host-gw mqtt 原创 jira confluence kube-controller-manager java springboot vip keepalived ConfigMap PodAffinity kylin arm64 Kylin audit fs nginx microsoft graph onedrive nodeport usb-net overlay kernal preseed Prometheus openstack cloud-init python containerd keepalive shell bash skopeo ssh kubelet systemd dockerhub gcr.io api ubuntu18 initrd tftp uos x86 openwrt zabbix zombie 正则 awk EmuELEC ingress N1 cobra hang cifs http lvs ipvsadm kube-proxy 63s timeout lamp nfc pm3 coredns openshift ocp squashfs timezone rpm rpmbuild segfault suse wireguard raid kubeadm operator exsi redhat8.4 vxlan veth panic supervisor pyinstaller ttf woff hostname qemu binfmt_misc loongarch64 externalIP cgo nfs ipset proxmox non-root containers vmware hung apparmor serviceAccountToken gpu nvidia cri-dockerd pprof trivy redis /etc/resolv.conf hostPort hotplug android headscale derper iptables
日志 心得 闲搞 ipmi kubernetes ansible kubelet Linux 心得 prometheus boot Auto cad boot golang kubeconfig grub linux centos6 容器 docker docker 容器 openstack boot vsftp go version panic gobot 闲聊 iptables admission kube-controller-manager keepalived k8s Prometheus Job k8s lua microsoft openwrt preseed Prometheus proxmox 总结 CI Kylin ubuntu18 pxe uos graph 运维 zombie 技巧 travis docker onedrive kickstart consul fstab http vxlan coredns node-cache golang kmem suse wireguard raid kubeadm LVM kickstart prometheus raid