zhangguanzhang's Blog

pod 的 /etc/resolv.conf 生成机制

字数统计: 1.7k阅读时长: 9 min
2023/05/22

从源码角度来看 pod 的 /etc/resolv.conf 生成机制

由来

客户护网发现会请求 8.8.8.88.8.4.4,客户最后查到是 coredns ,让我们关掉或者换国内的 DNS ,然后看了下发现 coredns 的 forward 插件源码里没有 8.8.8.8 字样,客户的所有机器上的 /etc/resolv.conf 都是空的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ docker logs 8f5
.:53
[INFO] plugin/reload: Running configuration SHA512 = effc0d92174bc24da10852943da4fbf83e3b8f4728dece4ea1526625309cb3c284a030c026949eaebcd26a317cf9dc0ff697c2e25c5abed5eb0daf8979c82a1c
CoreDNS-1.11.1
linux/amd64, go1.20.7, ae2bbc2
[ERROR] plugin/errors: 2 6437059289693292526.4644270481058418478. HINFO: read udp 10.187.0.6:44577->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 6437059289693292526.4644270481058418478. HINFO: read udp 10.187.0.6:59200->8.8.4.4:53: i/o timeout
[ERROR] plugin/errors: 2 6437059289693292526.4644270481058418478. HINFO: read udp 10.187.0.6:35901->8.8.4.4:53: i/o timeout

$ docker inspect 8f5 | grep -i reso
"ResolvConfPath": "/data/kube/docker/containers/85af19f670a4009fc33b170e4bfe606f967fbab89e8444e6d8f7f3c2be8a9a9a/resolv.conf",
$ cat /data/kube/docker/containers/85af19f670a4009fc33b170e4bfe606f967fbab89e8444e6d8f7f3c2be8a9a9a/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4

解决过程

k8s 的 dnsPolicy

runtime 是用的 docker,先看下 k8s pod 的 dnsPolicy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ kubectl explain pod.spec.dnsPolicy
KIND: Pod
VERSION: v1

FIELD: dnsPolicy <string>

DESCRIPTION:
Set DNS policy for the pod. Defaults to "ClusterFirst". Valid values are
'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'. DNS
parameters given in DNSConfig will be merged with the policy selected with
DNSPolicy. To have DNS options set along with hostNetwork, you have to
specify DNS policy explicitly to 'ClusterFirstWithHostNet'.

Possible enum values:
- `"ClusterFirst"` indicates that the pod should use cluster DNS first
unless hostNetwork is true, if it is available, then fall back on the
default (as determined by kubelet) DNS settings.
- `"ClusterFirstWithHostNet"` indicates that the pod should use cluster DNS
first, if it is available, then fall back on the default (as determined by
kubelet) DNS settings.
- `"Default"` indicates that the pod should use the default (as determined
by kubelet) DNS settings.
- `"None"` indicates that the pod should use empty DNS settings. DNS
parameters such as nameservers and search paths should be defined via
DNSConfig.
  • 默认 pod 就是 ClusterFirst 会使用 kubelet 配置的 clusterDNS 也就是 coredns 的 svc IP
  • 如果是 hostNetwork 没配置 dnsPolicy 则会使用 kubelet 配置的 resolvConf 也就是宿主机的 /etc/resolv.conf 内容,Default 也是
  • ClusterFirstWithHostNet 字面意思,就是 pod 配置 hostNetwork: true 下,设置成 coredns 的 svc IP,例如 ingress controller
  • None 就是不使用 DNS,可以配合 DNSConfig 定义 DNS 相关的参数

而 coredns 的 deploy 就设置了 dnsPolicy: Default,会使用宿主机的 /etc/resolv.conf 内容。

相关源码

coredns 源码搜不到,那就查看下 k8s dns 相关源码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
// kubernetes/pkg/kubelet/network/dns/dns.go
// GetPodDNS returns DNS settings for the pod.
func (c *Configurer) GetPodDNS(pod *v1.Pod) (*runtimeapi.DNSConfig, error) {
dnsConfig, err := c.getHostDNSConfig(c.ResolverConfig)
if err != nil {
return nil, err
}

dnsType, err := getPodDNSType(pod)
if err != nil {
klog.ErrorS(err, "Failed to get DNS type for pod. Falling back to DNSClusterFirst policy.", "pod", klog.KObj(pod))
dnsType = podDNSCluster
}
switch dnsType {
case podDNSNone:
// DNSNone should use empty DNS settings as the base.
dnsConfig = &runtimeapi.DNSConfig{}
case podDNSCluster:
if len(c.clusterDNS) != 0 {
// For a pod with DNSClusterFirst policy, the cluster DNS server is
// the only nameserver configured for the pod. The cluster DNS server
// itself will forward queries to other nameservers that is configured
// to use, in case the cluster DNS server cannot resolve the DNS query
// itself.
dnsConfig.Servers = []string{}
for _, ip := range c.clusterDNS {
dnsConfig.Servers = append(dnsConfig.Servers, ip.String())
}
dnsConfig.Searches = c.generateSearchesForDNSClusterFirst(dnsConfig.Searches, pod)
dnsConfig.Options = defaultDNSOptions
break
}
// clusterDNS is not known. Pod with ClusterDNSFirst Policy cannot be created.
nodeErrorMsg := fmt.Sprintf("kubelet does not have ClusterDNS IP configured and cannot create Pod using %q policy. Falling back to %q policy.", v1.DNSClusterFirst, v1.DNSDefault)
c.recorder.Eventf(c.nodeRef, v1.EventTypeWarning, "MissingClusterDNS", nodeErrorMsg)
c.recorder.Eventf(pod, v1.EventTypeWarning, "MissingClusterDNS", "pod: %q. %s", format.Pod(pod), nodeErrorMsg)
// Fallback to DNSDefault.
fallthrough
case podDNSHost:
// When the kubelet --resolv-conf flag is set to the empty string, use
// DNS settings that override the docker default (which is to use
// /etc/resolv.conf) and effectively disable DNS lookups. According to
// the bind documentation, the behavior of the DNS client library when
// "nameservers" are not specified is to "use the nameserver on the
// local machine". A nameserver setting of localhost is equivalent to
// this documented behavior.
if c.ResolverConfig == "" {
for _, nodeIP := range c.nodeIPs {
if utilnet.IsIPv6(nodeIP) {
dnsConfig.Servers = append(dnsConfig.Servers, "::1")
} else {
dnsConfig.Servers = append(dnsConfig.Servers, "127.0.0.1")
}
}
if len(dnsConfig.Servers) == 0 {
dnsConfig.Servers = append(dnsConfig.Servers, "127.0.0.1")
}
dnsConfig.Searches = []string{"."}
}
}

if pod.Spec.DNSConfig != nil {
dnsConfig = appendDNSConfig(dnsConfig, pod.Spec.DNSConfig)
}
return c.formDNSConfigFitsLimits(dnsConfig, pod), nil
}

看逻辑 case podDNSHost: 并没有为空的时候追加 DNS server 的逻辑,只是 kubelet 没配置 --resolv-conf 选项下使用 127.0.0.1 。然后去查看下 docker 源码找到了:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
// moby/libnetwork/resolvconf/resolvconf.go

var (
// Note: the default IPv4 & IPv6 resolvers are set to Google's Public DNS
defaultIPv4Dns = []string{"nameserver 8.8.8.8", "nameserver 8.8.4.4"}
defaultIPv6Dns = []string{"nameserver 2001:4860:4860::8888", "nameserver 2001:4860:4860::8844"}
...

// FilterResolvDNS cleans up the config in resolvConf. It has two main jobs:
// 1. It looks for localhost (127.*|::1) entries in the provided
// resolv.conf, removing local nameserver entries, and, if the resulting
// cleaned config has no defined nameservers left, adds default DNS entries
// 2. Given the caller provides the enable/disable state of IPv6, the filter
// code will remove all IPv6 nameservers if it is not enabled for containers
func FilterResolvDNS(resolvConf []byte, ipv6Enabled bool) (*File, error) {
cleanedResolvConf := localhostNSRegexp.ReplaceAll(resolvConf, []byte{})
// if IPv6 is not enabled, also clean out any IPv6 address nameserver
if !ipv6Enabled {
cleanedResolvConf = nsIPv6Regexp.ReplaceAll(cleanedResolvConf, []byte{})
}
// if the resulting resolvConf has no more nameservers defined, add appropriate
// default DNS servers for IPv4 and (optionally) IPv6
if len(GetNameservers(cleanedResolvConf, IP)) == 0 {
log.G(context.TODO()).Infof("No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: %v", defaultIPv4Dns)
dns := defaultIPv4Dns
if ipv6Enabled {
log.G(context.TODO()).Infof("IPv6 enabled; Adding default IPv6 external servers: %v", defaultIPv6Dns)
dns = append(dns, defaultIPv6Dns...)
}
cleanedResolvConf = append(cleanedResolvConf, []byte("\n"+strings.Join(dns, "\n"))...)
}
return &File{Content: cleanedResolvConf, Hash: hashData(cleanedResolvConf)}, nil
}

所以 /etc/resolv.conf 为空,docker 创建容器会追加谷歌 DNS,开头由来章节的日志就是本地复现的日志,避免手段就主要是利用 kubelet 在 case 后的逻辑判断:

1
2
3
if pod.Spec.DNSConfig != nil {
dnsConfig = appendDNSConfig(dnsConfig, pod.Spec.DNSConfig)
}

可以给 coredns 的 deploy 设置下 pod.Spec.DNSConfig 一个国内就行,是追加行为,例如下面设置的 223.5.5.5

1
2
3
4
5
6
7
8
9
$ docker ps -a | grep coredns
e3d1b5cc2405 cbb01a7bd410 "/coredns -conf /etc…" 4 seconds ago Up 3 seconds k8s_coredns_coredns-c4c868fc4-bxl7d_kube-system_22cf75d3-2f9c-4cd0-ba44-7a6509f7a3f2_0
15903cf3f557 reg.xxx.lan:5000/xxx/pause:3.9 "/pause" 4 seconds ago Up 3 seconds k8s_POD_coredns-c4c868fc4-bxl7d_kube-system_22cf75d3-2f9c-4cd0-ba44-7a6509f7a3f2_0
8f57eed59941 cbb01a7bd410 "/coredns -conf /etc…" 3 hours ago Up 3 hours k8s_coredns_coredns-5b69bc466f-ppdfx_kube-system_cd4df00e-a138-430a-8ceb-7e4092a3d986_0
85af19f670a4 reg.xxx.lan:5000/xxx/pause:3.9 "/pause" 3 hours ago Up 3 hours k8s_POD_coredns-5b69bc466f-ppdfx_kube-system_cd4df00e-a138-430a-8ceb-7e4092a3d986_0
$ docker inspect e3d | grep -i reso
"ResolvConfPath": "/data/kube/docker/containers/15903cf3f557b8bee70b44597258e634b3a951c51f27db4c969bdc91a7546428/resolv.conf",
$ cat /data/kube/docker/containers/15903cf3f557b8bee70b44597258e634b3a951c51f27db4c969bdc91a7546428/resolv.conf
nameserver 223.5.5.5

客户不配置宿主机 DNS 就是上面的,然后宿主机配置一个 dns,删除下 pod:

1
2
3
4
5
6
7
8
9
10
11
$ kubectl -n kube-system delete pod coredns-c4c868fc4-bxl7d 
pod "coredns-c4c868fc4-bxl7d" deleted
$ docker ps -a | grep coredns
73ff76b298d2 cbb01a7bd410 "/coredns -conf /etc…" 7 seconds ago Up 6 seconds k8s_coredns_coredns-c4c868fc4-98lqv_kube-system_5f67e9d4-71ca-48bc-857e-0edb79d32f37_0
ead6f87df3f1 reg.xxx.lan:5000/xxx/pause:3.9 "/pause" 7 seconds ago Up 7 seconds k8s_POD_coredns-c4c868fc4-98lqv_kube-system_5f67e9d4-71ca-48bc-857e-0edb79d32f37_0
15903cf3f557 reg.xxx.lan:5000/xxx/pause:3.9 "/pause" About a minute ago Exited (0) 2 seconds ago k8s_POD_coredns-c4c868fc4-bxl7d_kube-system_22cf75d3-2f9c-4cd0-ba44-7a6509f7a3f2_0
$ docker inspect 73ff76b298d2 | grep -i reso
"ResolvConfPath": "/data/kube/docker/containers/ead6f87df3f142386a18f945a8be351ea2c3c22ef7ee3434373372da26945dc9/resolv.conf",
$ cat /data/kube/docker/containers/ead6f87df3f142386a18f945a8be351ea2c3c22ef7ee3434373372da26945dc9/resolv.conf
nameserver 114.114.114.114
nameserver 223.5.5.5

这样就保证只有国内的了。

一些额外信息

CATALOG
  1. 1. 由来
  2. 2. 解决过程
    1. 2.1. k8s 的 dnsPolicy
    2. 2.2. 相关源码
  3. 3. 一些额外信息