zhangguanzhang's Blog

Killing container "docker://xxx with 30 second grace period

字数统计: 523阅读时长: 2 min
2020/01/15 Share

由来

开发反应他们环境上有个 pod 不正常。他们先操作然后卡住后来找我的。

1
2
3
4
5
$ kubectl -n xxx get pod -o wide | grep 0/1
xstorage-6d959cd8c8-rdscv 0/1 Init:0/1 0 15m xxxx
$ kubectl -n xxx delete po xstorage-6d959cd8c8-rdscv
pod "xstorage-6d959cd8c8-rdscv" deleted
^[[A^[[A^[[A^[[A

处理过程

查看日志:

1
2
3
4
$ journalctl -xe -u kubelet
...
kubelet_pods.go:1090] Killing unwanted pod "xstorage-6d959cd8c8-rdscv"
kuberuntime_container.go:581] Killing container "docker://7e33d0cdfe694050d3a7ef2a2553792eaff78286c92c52674320b2be23eb793e" with 30 second grace period

查看下状态显示运行,kubelet 显示走 api 删不掉,所以也不打算尝试 docker rm -f 之类的了,从 pid 层面去试试

1
2
3
4
5
6
7
8
$ docker ps -a | grep 7e33
7e33d0cdfe69 ed29c9ed519e "bash -ceux '\\cp -rf…" 23 minutes ago Up 23 minutes k8s_copy-configmap_xstorage-6d959cd8c8-rdscv_xxx_ee4423f2-de76-424c-b685-8d888446b2af_0
$ docker inspect 7e33 | grep -i pid
"Pid": 15370,
"PidMode": "",
"PidsLimit": 0,
$ kill -9 15370
-bash: kill: (15370) - 没有那个进程

绝了,尝试下 exec 看看:

1
2
3
4
$ docker exec 7e33  ps aux
cannot exec in a stopped state: unknown
$ docker ps -a | grep 7e33
7e33d0cdfe69 ed29c9ed519e "bash -ceux '\\cp -rf…" 23 minutes ago Up 23 minutes k8s_copy-configmap_xstorage-6d959cd8c8-rdscv_xxx_ee4423f2-de76-424c-b685-8d888446b2af_0

好吧,从上层的 containerd-shim 处理吧,按照容器 id 的前面,例如我这里取 7e33

1
2
3
4
5
$ ps aux | grep 7e33
root 15251 0.0 0.0 10724 3984 ? Sl 10:43 0:00 containerd-shim -namespace moby -workdir /data/kube/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/7e33d0cdfe694050d3a7ef2a2553792eaff78286c92c52674320b2be23eb793e -address /var/run/docker/containerd/containerd.sock -containerd-binary /data/kube/bin/containerd -runtime-root /var/run/docker/runtime-runc
root 28330 0.0 0.0 112728 984 pts/5 S+ 11:07 0:00 grep --color=auto 7e33
$ kill 15251
$ docker ps -a | grep 7e33

然后 pod 也没有了。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
$ docker info
Containers: 388
Running: 301
Paused: 0
Stopped: 87
Images: 378
Server Version: 18.09.3
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e6b3f5632f50dbc4e9cb6288d911bf4f5e95b18e
runc version: 6635b4f0c6af3810594d2770f662f34ddc15b40d
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-693.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.4 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.26GiB
Name: redhat7.4
ID: JJTI:G4BO:2VFT:PGNY:VYFD:OYHD:MNK3:5L6U:RY4Q:4WUC:NQDA:HP2D
Docker Root Dir: /data/kube/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
reg.xxx.lan:5000
treg.yun.xxx.cn
127.0.0.0/8
Registry Mirrors:
https://registry.docker-cn.com/
https://docker.mirrors.ustc.edu.cn/
Live Restore Enabled: false
Product License: Community Engine
CATALOG
  1. 1. 由来
  2. 2. 处理过程