内部产品是 toB 和 toG，针对 toG 的完全内网，测试会在搭建的 K8S 集群上所有节点配置个假的 DNS，这样黑盒下测功能，避免业务访问公网而造成功能问题。但是有些后续新业务是依赖公网的，测试测完没公网的部分后就会配置真实 dns 后测这部分功能，之前就遇到过好几次配置节点 DNS 后，个别 Pod 内 DNS 内容不是 k8s 的，而是下面这样类似变成宿主机：
1 2 3 4 5 6 7 8
# Generated by Docker Engine. # This file can be edited; Docker Engine will not make further changes once it # has been modified.
nameserver 10.xx.xx.xxx
# Based on host file: '/run/systemd/resolve/resolv.conf' (legacy) # Overrides: []
我们是使用的 cri-dockerd + k8s 组合。
过程
之前反馈了几次，但是一直没稳定复现手段，就先去看了下大概这块 docker 源码，然后给测试说，下次开发环境遇到了别删除 Pod 和对应容器，直接喊我，昨天下午反馈找到稳定复现步骤了，我们有个 dashboard，测试环境上开发在上面重启了他的 Pod 下的容器，该 Pod 的 spec.containers 只有一个，勾选的是类似 docker ps -a 的那样，/pause 容器和他的容器都勾选点重启的。上面点了下确实发生了。
$ kubectl get pod -o wide -A | grep ai-aixxxxapi default ai-aixxxxapi-6698f696d8-5w8kw 1/1 Running 1 (27m ago) 35m 10.187.x.34 10.1x.5x.251 <none> <none> $ ip r g 1 1.0.0.0 via 10.1x.5x.1 dev eth0 src 10.1x.5x.251 uid 0 cache $ docker ps -a | grep ai-aixxxxapi-6698f696d8-5w8kw 8b4ae64cecc0 reg.xxx.lan:5000/xxx/ai-aixxxxapi "/usr/local/bin/star…" 27 minutes ago Up 27 minutes k8s_ai-aixxxxapi_ai-aixxxxapi-6698f696d8-5w8kw_default_79fc032c-e1d3-4603-b21f-e5400ac6e3b6_1 e8eb18aaca94 reg.xxx.lan:5000/xxx/pause:3.9 "/pause" 27 minutes ago Up 27 minutes k8s_POD_ai-aixxxxapi-6698f696d8-5w8kw_default_79fc032c-e1d3-4603-b21f-e5400ac6e3b6_1 96ea991f9bb3 reg.xxx.lan:5000/xxx/ai-aixxxxapi "/usr/local/bin/star…" 34 minutes ago Exited (2) 27 minutes ago k8s_ai-aixxxxapi_ai-aixxxxapi-6698f696d8-5w8kw_default_79fc032c-e1d3-4603-b21f-e5400ac6e3b6_0 e4c64897be98 reg.xxx.lan:5000/xxx/pause:3.9 "/pause" 35 minutes ago Exited (0) 27 minutes ago k8s_POD_ai-aixxxxapi-6698f696d8-5w8kw_default_79fc032c-e1d3-4603-b21f-e5400ac6e3b6_0 $ docker inspect 96ea991f9bb3 | grep Resolv "ResolvConfPath": "/data/kube/docker/containers/e4c64897be9891d88b999e81bfd55bb0cc1c21d626708749691d43158062f2bb/resolv.conf", $ cat /data/kube/docker/containers/e4c64897be9891d88b999e81bfd55bb0cc1c21d626708749691d43158062f2bb/resolv.conf # Generated by Docker Engine. # This file can be edited; Docker Engine will not make further changes once it # has been modified.
$ docker ps -a | grep testpod 73b346e11ef4 m.daocloud.io/docker.io/library/nginx "/docker-entrypoint.…" 3 minutes ago Up 3 minutes k8s_vulnerable-container_testpod_default_f8215913-32b2-4e18-8536-69e5ecce7c84_0 ad255b51f1b3 reg.xxx.lan:5000/xxx/pause:3.9 "/pause" 3 minutes ago Up 3 minutes k8s_POD_testpod_default_f8215913-32b2-4e18-8536-69e5ecce7c84_0 $ docker inspect 73b346e11ef4 | grep ResolvConfPath "ResolvConfPath": "/data/kube/docker/containers/ad255b51f1b396fdea0d2579b373ac5c497fbd707031fa53936e805ac1b30cc9/resolv.conf", $ cat /data/kube/docker/containers/ad255b51f1b396fdea0d2579b373ac5c497fbd707031fa53936e805ac1b30cc9/resolv.conf nameserver 10.186.0.2 search default.svc.cluster1.local. svc.cluster1.local. cluster1.local. options ndots:5 $ docker restart 73b346e11ef4 ad255b51f1b3 73b346e11ef4 ad255b51f1b3 $ cat /data/kube/docker/containers/ad255b51f1b396fdea0d2579b373ac5c497fbd707031fa53936e805ac1b30cc9/resolv.conf # Generated by Docker Engine. # This file can be edited; Docker Engine will not make further changes once it # has been modified.
nameserver 10.x3.41.103 # Based on host file: '/run/systemd/resolve/resolv.conf' (legacy) # Overrides: []
$ docker ps -a | grep testpod ef9bb3c69dfa reg.xxx.lan:5000/xxx/nginx "/docker-entrypoint.…" 2 minutes ago Up 2 minutes k8s_vulnerable-container_testpod_default_ebb8ace6-84ab-4a28-814c-109c41827908_1 e6868ab3d8ef reg.xxx.lan:5000/xxx/pause:3.9 "/pause" 2 minutes ago Up 2 minutes k8s_POD_testpod_default_ebb8ace6-84ab-4a28-814c-109c41827908_1 $ docker inspect e6868ab3d8ef | grep ResolvConfPath "ResolvConfPath": "/data/kube/docker/containers/e6868ab3d8ef8fa1238a82a15faa88b1d13967a71a1e16c99618663610d21286/resolv.conf", $ cat /data/kube/docker/containers/e6868ab3d8ef8fa1238a82a15faa88b1d13967a71a1e16c99618663610d21286/resolv.conf nameserver 10.186.0.2 search default.svc.cluster2.local. svc.cluster2.local. cluster2.local. options ndots:5 $ docker restart e6868ab3d8ef e6868ab3d8ef $ cat /data/kube/docker/containers/e6868ab3d8ef8fa1238a82a15faa88b1d13967a71a1e16c99618663610d21286/resolv.conf # Generated by Docker Engine. # This file can be edited; Docker Engine will not make further changes once it # has been modified.
nameserver 223.5.5.5 # Based on host file: '/etc/resolv.conf' (legacy) # Overrides: []
相关源码
既然确定是 docker 逻辑造成，那就需要看下这块源码逻辑了，可以看到有生成注释的，代码里搜关键字 Generated by Docker Engine. 搜到：
// https://github.com/moby/moby/blob/v26.1.4/libnetwork/sandbox_dns_unix.go#L248-L278 // loadResolvConf reads the resolv.conf file at path, and merges in overrides for // nameservers, options, and search domains. func(sb *Sandbox) loadResolvConf(path string) (*resolvconf.ResolvConf, error) { rc, err := resolvconf.Load(path) if err != nil && !errors.Is(err, fs.ErrNotExist) { returnnil, err } // Proceed with rc, which might be zero-valued if path does not exist.
rc.SetHeader(`# Generated by Docker Engine. # This file can be edited; Docker Engine will not make further changes once it # has been modified.`) iflen(sb.config.dnsList) > 0 { var dnsAddrs []netip.Addr for _, ns := range sb.config.dnsList { addr, err := netip.ParseAddr(ns) if err != nil { returnnil, errors.Wrapf(err, "bad nameserver address %s", ns) } dnsAddrs = append(dnsAddrs, addr) } rc.OverrideNameServers(dnsAddrs) } iflen(sb.config.dnsSearchList) > 0 { rc.OverrideSearch(sb.config.dnsSearchList) } iflen(sb.config.dnsOptionsList) > 0 { rc.OverrideOptions(sb.config.dnsOptionsList) } return &rc, nil }
$ go install github.com/go-delve/delve/cmd/dlv@master $ dlv attach 19034 Type 'help' for list of commands. (dlv) b libnetwork/sandbox_dns_unix.go:285 Breakpoint 1 set at 0x24c6c8b for github.com/docker/docker/libnetwork.(*Sandbox).setupDNS() ./libnetwork/sandbox_dns_unix.go:285 (dlv) b libnetwork/sandbox_dns_unix.go:300 Breakpoint 2 set at 0x24c6f52 for github.com/docker/docker/libnetwork.(*Sandbox).updateDNS() ./libnetwork/sandbox_dns_unix.go:300 (dlv) c
(dlv) c > [Breakpoint 1] github.com/docker/docker/libnetwork.(*Sandbox).setupDNS() ./libnetwork/sandbox_dns_unix.go:285 (hits goroutine(2045):1 total:1) (PC: 0x24c6c8b) 280: // For a new sandbox, write an initial version of the container's resolv.conf. It'll 281: // be a copy of the host's file, with overrides for nameservers, options and search 282: // domains applied. 283: func (sb *Sandbox) setupDNS() error { 284: // Make sure the directory exists. => 285: sb.restoreResolvConfPath() 286: dir, _ := filepath.Split(sb.config.resolvConfPath) 287: if err := createBasePath(dir); err != nil { 288: return err 289: } 290: (dlv) p sb Sending output to pager... ("*github.com/docker/docker/libnetwork.Sandbox")(0xc0028d2400) *github.com/docker/docker/libnetwork.Sandbox { id: "1695c2a715872e25bcaa9c0268eeea65e528fa3ec6c275dfbf567344cd2cb30c", containerID: "176c492dc77f3ed8020c1d4e59896311fea359135be39c26c04d28460546cf38", config: github.com/docker/docker/libnetwork.containerConfig {
分析
大致看了下 docker 这块逻辑：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
// https://github.com/moby/moby/blob/v26.1.4/libnetwork/sandbox_dns_unix.go#L280C1-L296C2 // For a new sandbox, write an initial version of the container's resolv.conf. It'll // be a copy of the host's file, with overrides for nameservers, options and search // domains applied. func(sb *Sandbox) setupDNS() error { // Make sure the directory exists. sb.restoreResolvConfPath() dir, _ := filepath.Split(sb.config.resolvConfPath) if err := createBasePath(dir); err != nil { return err }
sb.loadResolvConf 方法就是把 Linux 的 DNS 内容解析成结构体，传入的 sb.config.getOriginResolvConfPath() 是获取宿主机的 dns 文件路径：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
(dlv) n > github.com/docker/docker/libnetwork.(*containerConfig).getOriginResolvConfPath() ./libnetwork/sandbox_dns_unix.go:242 (PC: 0x24c64ab) 237: } 238: } 239: 240: func (c *containerConfig) getOriginResolvConfPath() string { 241: if c.originResolvConfPath != "" { => 242: return c.originResolvConfPath 243: } 244: // Fallback if not specified. 245: return resolvconf.Path() 246: } 247: (dlv) p c.originResolvConfPath "/etc/resolv.conf"
然后继续调试
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
(dlv) list > [Breakpoint 1] github.com/docker/docker/libnetwork.(*Sandbox).setupDNS() ./libnetwork/sandbox_dns_unix.go:291 (hits goroutine(52319):1 total:3) (PC: 0x24c6d7a) > github.com/docker/docker/libnetwork/internal/resolvconf.(*ResolvConf).WriteFile() ./libnetwork/internal/resolvconf/resolvconf.go:377 (PC: 0xfad513) 372: 373: // WriteFile generates content and writes it to path. If hashPath is non-zero, it 374: // also writes a file containing a hash of the content, to enable UserModified() 375: // to determine whether the file has been modified. 376: func (rc *ResolvConf) WriteFile(path, hashPath string, perm os.FileMode) error { => 377: content, err := rc.Generate(true) 378: if err != nil { 379: return err 380: } 381: 382: // Write the resolv.conf file - it's bind-mounted into the container, so can't (dlv) p path "/data/kube/docker/containers/c213ae91753573a17948caf3cfa6421045d11e7e269d57cbc129f784f188d99b/resolv.conf" (dlv) p hashPath "/data/kube/docker/containers/c213ae91753573a17948caf3cfa6421045d11e7e269d57cbc129f784f188d99b/resolv.conf.hash"
// https://github.com/moby/moby/blob/v26.1.4/libnetwork/internal/resolvconf/resolvconf.go#L373-L402 // WriteFile generates content and writes it to path. If hashPath is non-zero, it // also writes a file containing a hash of the content, to enable UserModified() // to determine whether the file has been modified. func(rc *ResolvConf) WriteFile(path, hashPath string, perm os.FileMode) error { content, err := rc.Generate(true) if err != nil { return err }
// Write the resolv.conf file - it's bind-mounted into the container, so can't // move a temp file into place, just have to truncate and write it. if err := os.WriteFile(path, content, perm); err != nil { return errdefs.System(err) }
// Write the hash file. if hashPath != "" { hashFile, err := ioutils.NewAtomicFileWriter(hashPath, perm) if err != nil { return errdefs.System(err) } defer hashFile.Close()
$ dlv attach 19035 Type 'help' for list of commands. (dlv) b libnetwork/sandbox.go:263 Breakpoint 1 set at 0x24be2d8 for github.com/docker/docker/libnetwork.(*Sandbox).Refresh() ./libnetwork/sandbox.go:263 (dlv) b libnetwork/controller.go:944 Breakpoint 2 set at 0x2470d2f for github.com/docker/docker/libnetwork.(*Controller).NewSandbox() ./libnetwork/controller.go:944 (dlv) c > [Breakpoint 2] github.com/docker/docker/libnetwork.(*Controller).NewSandbox() ./libnetwork/controller.go:944 (hits goroutine(58233):1 total:1) (PC: 0x2470d2f) 939: } 940: c.mu.Unlock() 941: } 942: }() 943: => 944: if err := sb.setupResolutionFiles(); err != nil { 945: return nil, err 946: } 947: if err := c.setupOSLSandbox(sb); err != nil { 948: return nil, err 949: }
走的 NewSandbox ，打印下 backtrace：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
(dlv) bt 0 0x0000000002470d2f in github.com/docker/docker/libnetwork.(*Controller).NewSandbox at ./libnetwork/controller.go:944 1 0x0000000002f855e5 in github.com/docker/docker/daemon.(*Daemon).connectToNetwork at ./daemon/container_operations.go:762 2 0x0000000002f82825 in github.com/docker/docker/daemon.(*Daemon).allocateNetwork at ./daemon/container_operations.go:525 3 0x0000000002f87d05 in github.com/docker/docker/daemon.(*Daemon).initializeNetworking at ./daemon/container_operations.go:950 4 0x000000000302d26b in github.com/docker/docker/daemon.(*Daemon).containerStart at ./daemon/start.go:117 5 0x0000000003028736 in github.com/docker/docker/daemon.(*Daemon).containerRestart at ./daemon/restart.go:69 6 0x00000000030281e5 in github.com/docker/docker/daemon.(*Daemon).ContainerRestart at ./daemon/restart.go:24 7 0x00000000028a9c15 in github.com/docker/docker/api/server/router/container.(*containerRouter).postContainersRestart at ./api/server/router/container/container_routes.go:267 8 0x00000000028b594c in github.com/docker/docker/api/server/router/container.(*containerRouter).postContainersRestart-fm at <autogenerated>:1
0 0x000000000247da42 in github.com/docker/docker/libnetwork.(*Endpoint).sbJoin at ./libnetwork/endpoint.go:529 1 0x000000000247cc3a in github.com/docker/docker/libnetwork.(*Endpoint).Join at ./libnetwork/endpoint.go:467 2 0x0000000002f85ad1 in github.com/docker/docker/daemon.(*Daemon).connectToNetwork at ./daemon/container_operations.go:780 3 0x0000000002f82ac5 in github.com/docker/docker/daemon.(*Daemon).allocateNetwork at ./daemon/container_operations.go:530 4 0x0000000002f87fa5 in github.com/docker/docker/daemon.(*Daemon).initializeNetworking at ./daemon/container_operations.go:955 5 0x000000000302d50b in github.com/docker/docker/daemon.(*Daemon).containerStart at ./daemon/start.go:117 6 0x00000000030289d6 in github.com/docker/docker/daemon.(*Daemon).containerRestart at ./daemon/restart.go:69 7 0x0000000003028485 in github.com/docker/docker/daemon.(*Daemon).ContainerRestart at ./daemon/restart.go:24 8 0x00000000028a9c95 in github.com/docker/docker/api/server/router/container.(*containerRouter).postContainersRestart at ./api/server/router/container/container_routes.go:267
$ nerdctl -n k8s.io ps -a | grep zgz d79ac72ea7a6 dockerhub.xxxxxxwan.cn/pub/nginx:latest "/docker-entrypoint.…" 57 seconds ago Up k8s://default/test-zgz/nginx a2f547c59e8f dockerhub.xxxxxxwan.cn/pub/pause:3.9 "/pause" About a minute ago Up k8s://default/test-zgz $ nerdctl -n k8s.io inspect a2f547c59e8f | grep Resolv "ResolvConfPath": "/data/lv/lib/io.containerd.grpc.v1.cri/sandboxes/a2f547c59e8fc1d0dd1b6e5a9e35232211e9d4ce811a9101c45ed4d6bc9cf343/resolv.conf", $ cat /data/lv/lib/io.containerd.grpc.v1.cri/sandboxes/a2f547c59e8fc1d0dd1b6e5a9e35232211e9d4ce811a9101c45ed4d6bc9cf343/resolv.conf search default.svc.cluster.local svc.cluster.local cluster.local nameserver 169.254.25.10 options ndots:5 $ nerdctl -n k8s.io restart a2f547c59e8f a2f547c59e8f $ nerdctl -n k8s.io ps -a | grep zgz 13be2f142023 dockerhub.xxxxxxwan.cn/pub/nginx:latest "/docker-entrypoint.…" Less than a second ago Up k8s://default/test-zgz/nginx cabe19fd2fd7 dockerhub.xxxxxxwan.cn/pub/pause:3.9 "/pause" 1 second ago Up k8s://default/test-zgz d79ac72ea7a6 dockerhub.xxxxxxwan.cn/pub/nginx:latest "/docker-entrypoint.…" 2 minutes ago Created k8s://default/test-zgz/nginx a2f547c59e8f dockerhub.xxxxxxwan.cn/pub/pause:3.9 "/pause" 2 minutes ago Up k8s://default/test-zgz $ nerdctl -n k8s.io ps -a | grep zgz 13be2f142023 dockerhub.xxxxxxwan.cn/pub/nginx:latest "/docker-entrypoint.…" 9 seconds ago Up k8s://default/test-zgz/nginx cabe19fd2fd7 dockerhub.xxxxxxwan.cn/pub/pause:3.9 "/pause" 10 seconds ago Up k8s://default/test-zgz d79ac72ea7a6 dockerhub.xxxxxxwan.cn/pub/nginx:latest "/docker-entrypoint.…" 2 minutes ago Created k8s://default/test-zgz/nginx a2f547c59e8f dockerhub.xxxxxxwan.cn/pub/pause:3.9 "/pause" 2 minutes ago Up k8s://default/test-zgz $ nerdctl -n k8s.io inspect cabe19fd2fd7 | grep Resolv "ResolvConfPath": "/data/lv/lib/io.containerd.grpc.v1.cri/sandboxes/cabe19fd2fd75677f4ce77883697a76f66c425977c7cb358a3beb7da36d9d847/resolv.conf", $ cat /data/lv/lib/io.containerd.grpc.v1.cri/sandboxes/cabe19fd2fd75677f4ce77883697a76f66c425977c7cb358a3beb7da36d9d847/resolv.conf search default.svc.cluster.local svc.cluster.local cluster.local nameserver 169.254.25.10 options ndots:5 $ nerdctl -n k8s.io inspect 13be2f142023 | grep Resolv "ResolvConfPath": "/data/lv/lib/io.containerd.grpc.v1.cri/sandboxes/cabe19fd2fd75677f4ce77883697a76f66c425977c7cb358a3beb7da36d9d847/resolv.conf", $ cat /data/lv/lib/io.containerd.grpc.v1.cri/sandboxes/cabe19fd2fd75677f4ce77883697a76f66c425977c7cb358a3beb7da36d9d847/resolv.conf search default.svc.cluster.local svc.cluster.local cluster.local nameserver 169.254.25.10 options ndots:5
containerd 没复现，算了，还是提 issue 吧。
重启偶尔重建Pod
偶尔发现重启 /pause 容器被 k8s 重建了
1 2 3 4 5 6 7 8 9 10 11 12 13
$ docker restart c213ae917535 c213ae917535 $ docker ps -a | grep testpod 427dbeb17e72 reg.xxx.lan:5000/xxx/nginx "/docker-entrypoint.…" 2 seconds ago Up 1 second k8s_vulnerable-container_testpod_default_ebb8ace6-84ab-4a28-814c-109c41827908_8 612bf2b23193 reg.xxx.lan:5000/xxx/pause:3.9 "/pause" 3 seconds ago Up 1 second k8s_POD_testpod_default_ebb8ace6-84ab-4a28-814c-109c41827908_8 cbd04bc66cac reg.xxx.lan:5000/xxx/nginx "/docker-entrypoint.…" 10 minutes ago Exited (0) 2 seconds ago k8s_vulnerable-container_testpod_default_ebb8ace6-84ab-4a28-814c-109c41827908_7 51a41d9884e4 reg.xxx.lan:5000/xxx/pause:3.9 "/pause" 10 minutes ago Exited (0) 2 seconds ago k8s_POD_testpod_default_ebb8ace6-84ab-4a28-814c-109c41827908_7 c213ae917535 reg.xxx.lan:5000/xxx/pause:3.9 "/pause" 2 hours ago Exited (0) 2 seconds ago k8s_POD_testpod_default_ebb8ace6-84ab-4a28-814c-109c41827908_6 $ kubectl get event | grep testpod 3s Normal Pulling pod/testpod Pulling image "reg.xxx.lan:5000/xxx/nginx" 3s Normal Created pod/testpod Created container vulnerable-container 3s Normal Started pod/testpod Started container vulnerable-container 4s Normal SandboxChanged pod/testpod Pod sandbox changed, it will be killed and re-created.
搜 Pod sandbox changed 大致看了下这块 kubelet 代码，就是刚好检测 Pod 的 sandbox 状态和重启行为重合就会启动新的容器避免了这个问题，但是这种情况发生的概率不是百分之百重合。
