zhangguanzhang's Blog

consul仅当服务发现在k8s无状态部署使用

字数统计: 2.7k阅读时长: 13 min
2019/10/24 Share

由来

      consul server在机器上的部署已经写完了,但是dba还是希望部署在k8s上,昨天搜了下相关文章扣出来部分步骤自己验证实现了。
      先说下需求,三台物理机单独跑mysql主从mha,利用consul client的服务注册+ttl向consul server注册成域名,然后配置k8s内部的coredns把域名*.service.consul转发到consul server。架构为下图

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Physical server
+---------------------+ K8S Cluster
| | +--------------------------------+
|mysql + consul client+-----------+ | +---------------------+ |
| | | | | +-------------+ | |
+---------------------+ | | | |consul server| | |
Physical server | | | D +------+------+ | |
+---------------------+ | | | e | | |
| | | | | p +------+------+ | |
|mysql + consul client+----------->------>+ | l |consul server| | |
| | | | | o +------+------+ | |
+---------------------+ | | | y | | |
Physical server | | | +------+------+ | |
+---------------------+ | | | |consul server| | |
| | | | | +-------------+ | |
|mysql + consul client+-----------+ | +---------------------+ |
| | +--------------------------------+
+---------------------+

consul有k8s client的相关代码,赋予RBAC后能够用selector能够自动join到其他成员

1
2
3
"retry_join": [
"provider=k8s label_selector=\"app=consul,component=server\""
],

      当然其实市面上还有其他的方案的,为啥我不用

镜像准备

      因为这里我是consul server跑k8s里,consul client在物理机上,通信我要配置成tls,tls我是secret导入的,cm写配置文件。挂载到pod里后因为官方docker镜像的entrypoint.sh里chown了配置文件目录,而挂载到pod内部是只读的,pod起来后执行到chown那就报错退出了。所以改造了下官方的entrypoint脚本

1
2
3
4
5
6
7
8
9
FROM consul:1.6.1
LABEL maintainer="zhangguanzhang <zhangguanzhang@qq.com>"

ARG TZ=Asia/Shanghai

RUN set -eux && \
ln -sf /usr/share/zoneinfo/${TZ} /etc/localtime && \
echo ${TZ} > /etc/timezone
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh

docker-entrypoint.sh见我github https://raw.githubusercontent.com/zhangguanzhang/Dockerfile/master/consul-k8s-dns/docker-entrypoint.sh
Dockerfile我是打算直接改官方的而不是FROM镜像,但是gpg key出问题所以我写成上面那样构建的镜像。见issue https://github.com/hashicorp/docker-consul/issues/137
自己构建镜像的话docker-entrypoint.sh记得加执行权限,我构建的镜像推送到dockerhub上可以直接拉取

tls

证书操作都是在容器里生成的

1
2
mkdir ssl
docker run --rm -ti --entrypoint sh --workdir /root -v $PWD/ssl:/root/ zhangguanzhang/consul-k8s-dns

我们先进入容器

step 1: 创建ca

为了简单起见,这里我使用Consul命令行的的内置TLS功能来创建基本的CA。您只需为数据中心创建一个CA。您应该在用于创建CA的同一服务器上生成所有证书。
ca默认五年,其他的证书默认1年,这里需要带参数-days=设置长点的日期

1
2
3
consul tls ca create -days=36500
==> Saved consul-agent-ca.pem
==> Saved consul-agent-ca-key.pem

step2: 创建server角色的证书

这里数据中心默认名字为dc1,其他的自行选项赋值。在创建CA的同一台服务器上重复此过程,直到每台服务器都有一个单独的证书。该命令可以反复调用,它将自动增加证书和密钥号。您将需要将证书分发到服务器。
严格来说每个server单独使用,也就是说假如三个server,下面命令应该执行三次。但是我们仅仅当无状态服务使用,如果每个server单独一套证书就不能放secret里,毕竟文件名不一样不能对照到pod上,这里我们仅仅生成一份

1
2
3
4
5
6
7
8
consul tls cert create -server -dc=dc1 -days=36500
==> WARNING: Server Certificates grants authority to become a
server and access all state in the cluster including root keys
and all ACL tokens. Do not distribute them to production hosts
that are not server nodes. Store them as securely as CA keys.
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-server-consul-0.pem
==> Saved dc1-server-consul-0-key.pem

step3: 创建client角色的证书

在Consul 1.5.2中,您可以使用替代过程来自动将证书分发给客户端。要启用此新功能,请设置auto_encrypt

您可以继续使用生成证书consul tls cert create -client并手动分发证书。对于需要高度保护的数据中心,仍然需要现有的工作流程。

如果您正在运行Consul 1.5.1或更早版本,则需要使用来为每个客户端创建单独的证书consul tls cert create -client。客户端证书也由您的CA签名,但是它们没有特殊性Subject Alternative Name,这意味着如果verify_server_hostname启用,则它们不能作为server角色启动。

这里我是高于1.5.2的,不需要为每个客户端创建证书,客户端只需要拥有consul-agent-ca.pem这个ca下,会自动从server获取证书存在内存中,并且不会持久保存。但是我测试了并没有成功,还是生成了证书

1
2
3
4
5
6
7
8
$ consul tls cert create -client -dc=dc1 -days=36500
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-0.pem
==> Saved dc1-client-consul-0-key.pem
$ consul tls cert create -client -dc=dc1 -days=36500
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-client-consul-1.pem
==> Saved dc1-client-consul-1-key.pem

客户端实体服务跑,可以单独去生成而不是共用一套

step4: 创建cli的证书

1
2
3
4
$ consul tls cert create -cli -dc=dc1 -days=36500
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-cli-consul-0.pem
==> Saved dc1-cli-consul-0-key.pem

创建完证书ctrl+d退出容器

k8s相关yaml

consul可以跨数据中心,wan相关配置(hostPort模拟)我没有测试成功,而且考虑到高可用,基本是每台机器一台。所以我用了硬性互斥+hostNetwork,关闭serfwan端口。

nodeSelector记得自己去打label固定住
apply之前我们先把证书导入成secret

1
2
3
4
5
6
7
cd ssl
kubectl create secret generic consul \
--from-file=consul-agent-ca.pem \
--from-file=dc1-server-consul-0.pem \
--from-file=dc1-server-consul-0-key.pem \
--from-file=dc1-cli-consul-0.pem \
--from-file=dc1-cli-consul-0-key.pem

yaml文件

另外要注意,因为每次pod重启基本相当于node-id变了,参数leave_on_terminate应该false,见 https://github.com/hashicorp/consul/issues/6672https://github.com/hashicorp/consul/issues/3938 所以preStop的部分没啥用,忽略即可
因为coredns的转发需要写ip或者文件而不能是域名,所以我们得把consul-server的svc的clusterIP给固定住(记住得在svc的cidr内选一个未使用的ip),或者干脆不创建svc直接使用hostIP,这里按照svc实例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
apiVersion: apps/v1
kind: Deployment
metadata:
name: consul-server
spec:
selector:
matchLabels:
app: consul
component: server
replicas: 3
template:
metadata:
labels:
app: consul
component: server
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- consul
topologyKey: kubernetes.io/hostname
serviceAccountName: consul-server
nodeSelector:
master: "true"
terminationGracePeriodSeconds: 7
hostNetwork: true
securityContext:
fsGroup: 1000
containers:
- name: consul
image: "zhangguanzhang/consul-k8s-dns"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: CONSUL_HTTP_ADDR
value: https://localhost:8501
- name: CONSUL_CACERT
value: /consul/config/ssl/consul-agent-ca.pem
- name: CONSUL_CLIENT_CERT
value: /consul/config/ssl/dc1-cli-consul-0.pem
- name: CONSUL_CLIENT_KEY
value: /consul/config/ssl/dc1-cli-consul-0-key.pem
# - name: RETRY_JOIN
# value: 172.19.0.5,172.19.0.6,172.19.0.7
args:
- agent
- -advertise=$(POD_IP)
- -node=$(NODE)
volumeMounts:
- name: data
mountPath: /consul/data
- name: config
mountPath: /consul/config/
- name: tls
mountPath: /consul/config/ssl/
lifecycle:
preStop:
exec:
command:
- consul leave
readinessProbe:
exec:
command:
- consul
- members
failureThreshold: 2
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 3
ports:
- containerPort: 8300
name: server
- containerPort: 8301
name: serflan
# - containerPort: 8302
# name: serfwan
- containerPort: 8400
name: alt-port
- containerPort: 8501
name: https
- containerPort: 8600
name: dns-udp
protocol: UDP
- containerPort: 8600
name: dns-tcp
protocol: TCP
volumes:
- name: config
configMap:
name: consul-server
- name: tls
secret:
secretName: consul
- name: data
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: consul-server
namespace: default
labels:
app: consul
spec:
clusterIP: 10.96.0.11
ports:
- name: https
port: 8501
targetPort: https
- name: serflan-tcp
protocol: "TCP"
port: 8301
targetPort: 8301
- name: serflan-udp
protocol: "UDP"
port: 8301
targetPort: 8301
# - name: serfwan-tcp
# protocol: "TCP"
# port: 8302
# targetPort: 8302
# - name: serfwan-udp
# protocol: "UDP"
# port: 8302
# targetPort: 8302
- name: server
port: 8300
targetPort: 8300
- name: dns-tcp
protocol: "TCP"
port: 8600
targetPort: dns-tcp
- name: dns-udp
protocol: "UDP"
port: 8600
targetPort: dns-udp
selector:
app: consul
component: server
---
apiVersion: v1
kind: ConfigMap
metadata:
name: consul-server
namespace: default
labels:
app: consul
role: server
data:
server.json: |-
{
"client_addr": "0.0.0.0",
"datacenter": "dc1",
"bootstrap_expect": 3,
"domain": "consul",
"skip_leave_on_interrupt": true,
"leave_on_terminate" : false,
"log_level": "INFO",
"retry_join": [
"provider=k8s label_selector=\"app=consul,component=server\""
],
"retry_interval": "2s",
"verify_incoming": true,
"verify_outgoing": true,
"verify_server_hostname": true,
"ca_file": "/consul/config/ssl/consul-agent-ca.pem",
"cert_file": "/consul/config/ssl/dc1-server-consul-0.pem",
"key_file": "/consul/config/ssl/dc1-server-consul-0-key.pem",
"ports": {
"http": -1,
"serf_wan": -1,
"https": 8501
},
"server": true,
"ui": false
}
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: consul-server
labels:
app: consul
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: consul
labels:
app: consul
rules:
- apiGroups: [""]
resources:
- pods
verbs:
- get
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: consul
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: consul
subjects:
- kind: ServiceAccount
name: consul-server
namespace: default

apply后

1
2
3
4
5
[root@k8s-m1 consul-tls-ansible]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
consul-server-7879498d78-8qjbs 1/1 Running 0 2m34s 172.19.0.6 172.19.0.6 <none> <none>
consul-server-7879498d78-drm7n 1/1 Running 0 2m34s 172.19.0.7 172.19.0.7 <none> <none>
consul-server-7879498d78-zjc6z 1/1 Running 0 2m34s 172.19.0.5 172.19.0.5 <none> <none>

client是实体服务,用我的ansible部署的,client配置服务注册

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
"services": [
{
"name": "r-3306-mysql",
"tags": [
"slave-3306"
],
"address": "172.16.0.2",
"port": 3306,
"checks": [
{
"args": ["echo"], //实际应该写健康检查脚本,并去掉此处注释
"interval": "5s"
}
]
}
]
}

client的consul reload后看到服务注册上去

1
2
3
4
2019/10/24 13:32:12 [INFO] agent: (LAN) joined: 3
2019/10/24 13:32:12 [INFO] agent: Join LAN completed. Synced with 3 initial agents
2019/10/24 13:32:12 [INFO] agent: Synced service "r-3306-mysql"
2019/10/24 13:32:21 [INFO] agent: Synced check "service:r-3306-mysql"

查看members

1
2
3
4
5
6
7
8
9
10
[root@k8s-m1 consul-tls-ansible]# kubectl exec consul-server-7879498d78-8qjbs consul members
Node Address Status Type Build Protocol DC Segment
172.19.0.5 172.19.0.5:8301 alive server 1.6.1 2 dc1 <all>
172.19.0.6 172.19.0.6:8301 alive server 1.6.1 2 dc1 <all>
172.19.0.7 172.19.0.7:8301 alive server 1.6.1 2 dc1 <all>
172.19.0.13 172.19.0.13:8301 alive client 1.6.1 2 dc1 <default>
[root@k8s-m1 consul-tls-ansible]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
consul-server ClusterIP 10.96.0.11 <none> 8501/TCP,8301/TCP,8301/UDP,8300/TCP,8600/TCP,8600/UDP 2m48s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2d22h

直接使用svc来测试域名解析

1
2
[root@k8s-m1 consul-tls-ansible]# dig -p 8600 @10.96.0.11 r-3306-mysql.service.consul +short
172.16.0.2

配置coredns转发,coredns的configmap添加下面内容,也可以添加上client的ip

1
2
3
4
5
6
...
service.consul:53 {
errors
cache 30
forward . 10.96.0.11:8600 172.19.0.13:8600
}

跑一个指定版本带解析的工具pod

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- name: busybox
image: busybox:1.28
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
restartPolicy: Always
EOF

测试解析,不影响集群的内部解析

1
2
3
4
5
6
7
8
9
10
11
12
$ kubectl exec -ti busybox -- nslookup r-3306-mysql.service.consul
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name: r-3306-mysql.service.consul
Address 1: 172.16.0.2
$ kubectl exec -ti busybox -- nslookup kubernetes
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name: kubernetes
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

参考

https://www.consul.io/docs/platform/k8s/run.html
https://github.com/kelseyhightower/consul-on-kubernetes

CATALOG
  1. 1. 由来
  2. 2. 镜像准备
  3. 3. tls
    1. 3.1. step 1: 创建ca
    2. 3.2. step2: 创建server角色的证书
    3. 3.3. step3: 创建client角色的证书
    4. 3.4. step4: 创建cli的证书
  4. 4. k8s相关yaml
    1. 4.1. yaml文件
  5. 5. 参考