zhangguanzhang's Blog

docker18.03 hang at 'restoring container'

字数统计: 3.4k阅读时长: 20 min
2020/12/04 Share

由来

起初是 k8s 有几个 node not ready,上去看了下 kubelet 日志刷 container runtime down,重启了下 docker 后还是没用,docker ps 命令都卡住。

环境信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
$ cat /etc/redhat-release 
Linux xxx-disk0 3.10.0-1127.13.1.el7.x86_64 #1 SMP Tue Jun 23 15:46:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.4.1708 (Core)

$ docker info
Containers: 91
Running: 63
Paused: 0
Stopped: 28
Images: 539
Server Version: 18.03.0-ce
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-1127.13.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.51GiB
Name: xxx-disk0
ID: UZRM:KRSL:TYWM:VAQY:KWCX:AVFD:NP53:TC35:YHOC:TLLO:YGXO:RMYS
Docker Root Dir: /app/kube/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
treg.yun.xxx.cn
reg.xxx.lan:5000
127.0.0.0/8
Registry Mirrors:
https://registry.docker-cn.com/
https://docker.mirrors.ustc.edu.cn/
Live Restore Enabled: false

排查过程

先停掉docker,然后前台启动加 debug 参数启动

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
$ pgrep dockerd
4659
$ kill 4659 && > /var/run/docker.pid
$ ps aux | grep dockerd
root 5628 0.0 0.0 112708 980 pts/0 S+ 22:33 0:00 grep --color=auto dockerd
$ ./dockerd -D
WARN[0000] The "graph" config file option is deprecated. Please use "data-root" instead.
WARN[2020-12-04T22:33:50.432804342+08:00] could not change group /var/run/docker.sock to docker: group docker not found
DEBU[2020-12-04T22:33:50.432936283+08:00] Listener created for HTTP on unix (/var/run/docker.sock)
INFO[2020-12-04T22:33:50.433612435+08:00] libcontainerd: started new docker-containerd process pid=5646
INFO[0000] starting containerd module=containerd revision=cfd04396dc68220d1cecbe686a6cc3aa5ce3667c version=v1.0.2
DEBU[0000] changing OOM score to -500 module=containerd
INFO[0000] loading plugin "io.containerd.content.v1.content"... module=containerd type=io.containerd.content.v1
INFO[0000] loading plugin "io.containerd.snapshotter.v1.btrfs"... module=containerd type=io.containerd.snapshotter.v1
WARN[0000] failed to load plugin io.containerd.snapshotter.v1.btrfs error="path /app/kube/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
INFO[0000] loading plugin "io.containerd.snapshotter.v1.overlayfs"... module=containerd type=io.containerd.snapshotter.v1
INFO[0000] loading plugin "io.containerd.metadata.v1.bolt"... module=containerd type=io.containerd.metadata.v1
WARN[0000] could not use snapshotter btrfs in metadata plugin error="path /app/kube/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt"
INFO[0000] loading plugin "io.containerd.differ.v1.walking"... module=containerd type=io.containerd.differ.v1
INFO[0000] loading plugin "io.containerd.gc.v1.scheduler"... module=containerd type=io.containerd.gc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.containers"... module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.content"... module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.diff"... module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.events"... module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.healthcheck"... module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.images"... module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.leases"... module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.namespaces"... module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.snapshots"... module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.monitor.v1.cgroups"... module=containerd type=io.containerd.monitor.v1
INFO[0000] loading plugin "io.containerd.runtime.v1.linux"... module=containerd type=io.containerd.runtime.v1
DEBU[0000] loading tasks in namespace module="containerd/io.containerd.runtime.v1.linux" namespace=moby
INFO[0000] loading plugin "io.containerd.grpc.v1.tasks"... module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.version"... module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.introspection"... module=containerd type=io.containerd.grpc.v1
INFO[0000] serving... address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
INFO[0000] serving... address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
INFO[0000] containerd successfully booted in 0.009604s module=containerd
DEBU[2020-12-04T22:33:50.456534148+08:00] Golang's threads limit set to 113940
DEBU[2020-12-04T22:33:50.457345643+08:00] Using default logging driver json-file
DEBU[2020-12-04T22:33:50.457466912+08:00] [graphdriver] priority list: [btrfs zfs overlay2 aufs overlay devicemapper vfs]
DEBU[2020-12-04T22:33:50.457623030+08:00] processing event stream module=libcontainerd namespace=plugins.moby
DEBU[2020-12-04T22:33:50.479691287+08:00] backingFs=xfs, projectQuotaSupported=false
INFO[2020-12-04T22:33:50.479712832+08:00] [graphdriver] using prior storage driver: overlay2
DEBU[2020-12-04T22:33:50.479724151+08:00] Initialized graph driver overlay2
DEBU[2020-12-04T22:33:50.510882767+08:00] Max Concurrent Downloads: 10
DEBU[2020-12-04T22:33:50.510930407+08:00] Max Concurrent Uploads: 5
DEBU[0000] garbage collected d=24.493383ms module="containerd/io.containerd.gc.v1.scheduler"
INFO[2020-12-04T22:33:50.608483121+08:00] Graph migration to content-addressability took 0.00 seconds
INFO[2020-12-04T22:33:50.610430840+08:00] Loading containers: start.
DEBU[2020-12-04T22:33:50.610704281+08:00] processing event stream module=libcontainerd namespace=moby
DEBU[2020-12-04T22:33:50.611446797+08:00] Loaded container 027a389c8c1e93629cc5f68af8d023b2ecfe350d7771ba6b87598ff705f6c19f, isRunning: false
DEBU[2020-12-04T22:33:50.611803503+08:00] Loaded container 24735e5aea2bd91b5fa5d729ca021a09532c2ea9b8b06f5171d0da23fc3bf4cc, isRunning: false
DEBU[2020-12-04T22:33:50.612174253+08:00] Loaded container 487a8c2f30986796c3948d1469d506e1d3ab394e17533040ef7a5444a32be0fc, isRunning: false
DEBU[2020-12-04T22:33:50.612494092+08:00] Loaded container 52d32b0e03c957b6cb9b4d793c47900e689a29d9ae0d63703ea29073a352fbe5, isRunning: false
DEBU[2020-12-04T22:33:50.612816495+08:00] Loaded container 5b7b0b52c71a14164f269853679211b3823e9eecc2d3829bf2db10c9b720217d, isRunning: false
DEBU[2020-12-04T22:33:50.613447082+08:00] Loaded container 62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be, isRunning: true
DEBU[2020-12-04T22:33:50.613769649+08:00] Loaded container 68ba211ec7328bebd3b241631a703639447c05056ffe07ed633b72d0bc210938, isRunning: false
DEBU[2020-12-04T22:33:50.614756585+08:00] Loaded container 73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212, isRunning: true
DEBU[2020-12-04T22:33:50.615381990+08:00] Loaded container 7a2a75c8a0c8dac8c5773ad92fa45ac7e1d33c9be85ecb65eb147929955dca50, isRunning: false
DEBU[2020-12-04T22:33:50.616222796+08:00] Loaded container 81095c01c4b99c7d2cc9e6bee8726c11f16d27204523727e7d067d980c26ac64, isRunning: false
DEBU[2020-12-04T22:33:50.616569394+08:00] Loaded container 94098167eb466dbf1a454f5491a162488d1fdb1eebe804c5c1f403f7fce62dc4, isRunning: false
DEBU[2020-12-04T22:33:50.616981038+08:00] Loaded container 9d4cbcce43b0262d972e73b2770f26ca762e2fa86f0de88a7909b8e59c0b805a, isRunning: false
DEBU[2020-12-04T22:33:50.617460452+08:00] Loaded container aecde8eb18924d8548d79d5e0383baa7ac3ab1cfc4c55e1f32c4089dfc153071, isRunning: false
DEBU[2020-12-04T22:33:50.617908975+08:00] Loaded container aed0618a325b4b84363357c1830515048d23af6afd79606cbb0ad64bf5f226a2, isRunning: false
DEBU[2020-12-04T22:33:50.618252961+08:00] Loaded container b43e4995720f235c40ffd60bde1fb54e87ece3598f8bd625996042f637896687, isRunning: false
DEBU[2020-12-04T22:33:50.618557604+08:00] Loaded container c1e6a1de9b9c2fd420e718c405c114e726ec5531561a4caf662b757a3724711e, isRunning: false
DEBU[2020-12-04T22:33:50.618942417+08:00] Loaded container c5eb3c941e562153e0cf0af738f1cb43f34591f0b48ad5458ab2002f5be9e0a8, isRunning: false
DEBU[2020-12-04T22:33:50.619380785+08:00] Loaded container e211ffccacb8f7982899097fbd0f9ce1d95f8f31f290fb10baf40d00f4980bc9, isRunning: false
DEBU[2020-12-04T22:33:50.619831551+08:00] Loaded container ef547d238cd01ff7ec048de3442fe9293aa1d5d932ea66c5aed34bfff014182b, isRunning: false
DEBU[2020-12-04T22:33:50.620192032+08:00] Loaded container f3bb916ec5d7847c3be4341975c47f4e2fe587fc726ca7d76e3dca15cb8dd21d, isRunning: false
DEBU[2020-12-04T22:33:50.620438678+08:00] Loaded container fa6de6f4aa8894c18a9737bac462f57c69893eca5e4b58bc3bd793a76b252951, isRunning: false
DEBU[2020-12-04T22:33:51.379861237+08:00] restoring container container=ef547d238cd01ff7ec048de3442fe9293aa1d5d932ea66c5aed34bfff014182b paused=false running=false
DEBU[2020-12-04T22:33:51.379910464+08:00] restoring container container=e211ffccacb8f7982899097fbd0f9ce1d95f8f31f290fb10baf40d00f4980bc9 paused=false running=false
DEBU[2020-12-04T22:33:51.379994141+08:00] restoring container container=7a2a75c8a0c8dac8c5773ad92fa45ac7e1d33c9be85ecb65eb147929955dca50 paused=false running=false
DEBU[2020-12-04T22:33:51.380029802+08:00] restoring container container=c1e6a1de9b9c2fd420e718c405c114e726ec5531561a4caf662b757a3724711e paused=false running=false
DEBU[2020-12-04T22:33:51.380084763+08:00] restoring container container=5b7b0b52c71a14164f269853679211b3823e9eecc2d3829bf2db10c9b720217d paused=false running=false
DEBU[2020-12-04T22:33:51.380127006+08:00] restoring container container=9d4cbcce43b0262d972e73b2770f26ca762e2fa86f0de88a7909b8e59c0b805a paused=false running=false
DEBU[2020-12-04T22:33:51.380121758+08:00] restoring container container=fa6de6f4aa8894c18a9737bac462f57c69893eca5e4b58bc3bd793a76b252951 paused=false running=false
DEBU[2020-12-04T22:33:51.380163318+08:00] restoring container container=52d32b0e03c957b6cb9b4d793c47900e689a29d9ae0d63703ea29073a352fbe5 paused=false running=false
DEBU[2020-12-04T22:33:51.380310029+08:00] restoring container container=027a389c8c1e93629cc5f68af8d023b2ecfe350d7771ba6b87598ff705f6c19f paused=false running=false
DEBU[2020-12-04T22:33:51.380382722+08:00] restoring container container=68ba211ec7328bebd3b241631a703639447c05056ffe07ed633b72d0bc210938 paused=false running=false
DEBU[2020-12-04T22:33:51.380419320+08:00] restoring container container=487a8c2f30986796c3948d1469d506e1d3ab394e17533040ef7a5444a32be0fc paused=false running=false
DEBU[2020-12-04T22:33:51.380433522+08:00] restoring container container=73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 paused=false running=true
DEBU[2020-12-04T22:33:51.380459224+08:00] restoring container container=c5eb3c941e562153e0cf0af738f1cb43f34591f0b48ad5458ab2002f5be9e0a8 paused=false running=false
DEBU[2020-12-04T22:33:51.380525276+08:00] restoring container container=b43e4995720f235c40ffd60bde1fb54e87ece3598f8bd625996042f637896687 paused=false running=false
DEBU[2020-12-04T22:33:51.380563957+08:00] restoring container container=81095c01c4b99c7d2cc9e6bee8726c11f16d27204523727e7d067d980c26ac64 paused=false running=false
DEBU[2020-12-04T22:33:51.380586567+08:00] restoring container container=62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be paused=false running=true
DEBU[2020-12-04T22:33:51.380599061+08:00] restoring container container=94098167eb466dbf1a454f5491a162488d1fdb1eebe804c5c1f403f7fce62dc4 paused=false running=false
DEBU[2020-12-04T22:33:51.380616220+08:00] restoring container container=aecde8eb18924d8548d79d5e0383baa7ac3ab1cfc4c55e1f32c4089dfc153071 paused=false running=false
DEBU[2020-12-04T22:33:51.380641090+08:00] restoring container container=f3bb916ec5d7847c3be4341975c47f4e2fe587fc726ca7d76e3dca15cb8dd21d paused=false running=false
DEBU[2020-12-04T22:33:51.380825356+08:00] restoring container container=24735e5aea2bd91b5fa5d729ca021a09532c2ea9b8b06f5171d0da23fc3bf4cc paused=false running=false
DEBU[2020-12-04T22:33:51.380953092+08:00] restoring container container=aed0618a325b4b84363357c1830515048d23af6afd79606cbb0ad64bf5f226a2 paused=false running=false

然后发现卡在这,正常是会像 gin 那样启动输出支持的 http api 路由信息的。开一个窗口,发送 SIGUSR1 信号打印 goroutine 堆栈信息看看卡在哪儿:

1
2
3
$ pgrep dockerd
3085
$ kill -USR1 3085

docker 的日志和系统日志都会有下面的类似输出:

1
Dec 04 22:33:52 xxxx dockerd[3085]: time="2020-12-33T58:15:52.906433650+08:00" level=info msg="goroutine stacks written to /var/run/docker/goroutine-stacks-2020-12-04T223358+0800.log"

查看了下下面这段比较可疑,daemon/daemon.go:364 附近

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
goroutine 1 [semacquire, 5 minutes]:
sync.runtime_Semacquire(0xc4204de73c)
/usr/local/go/src/runtime/sema.go:56 +0x3b
sync.(*WaitGroup).Wait(0xc4204de730)
/usr/local/go/src/sync/waitgroup.go:131 +0x74
github.com/docker/docker/daemon.(*Daemon).restore(0xc42009a480, 0x190c3a6, 0x4)
/go/src/github.com/docker/docker/daemon/daemon.go:364 +0xfeb
github.com/docker/docker/daemon.NewDaemon(0xc42018d200, 0x2ec75c0, 0xc4201be410, 0x2ea89e0, 0xc420087d40, 0xc4201323c0, 0x0, 0x0, 0x0)
/go/src/github.com/docker/docker/daemon/daemon.go:894 +0x258d
main.(*DaemonCli).start(0xc42051da40, 0xc4201c5d50, 0x0, 0x0)
/go/src/github.com/docker/docker/cmd/dockerd/daemon.go:223 +0x1320
main.runDaemon(0xc4201c5d50, 0xc42044b3b0, 0x0)
/go/src/github.com/docker/docker/cmd/dockerd/docker.go:78 +0x78
main.newDaemonCommand.func1(0xc420176000, 0xc4201359e0, 0x0, 0x1, 0x0, 0x0)
/go/src/github.com/docker/docker/cmd/dockerd/docker.go:29 +0x5d
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc420176000, 0xc42000c090, 0x1, 0x1, 0xc420176000, 0xc42000c090)
/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:646 +0x44f
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc420176000, 0x2194e40, 0x2419c01, 0xc420135980)
/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:742 +0x310
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(0xc420176000, 0xc420135980, 0x190fa00)
/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:695 +0x2d
main.main()
/go/src/github.com/docker/docker/cmd/dockerd/docker.go:105 +0xe3

按照docker info 的信息去找了下对应的 分支代码。364 行一个wg.Wait(),得看前面的 goroutine 是卡在哪儿,根据前面的堆栈信息,应该是卡在 github.com/docker/docker/daemon.(*Daemon).restore,也就是 238 行的 daemon.containerd.Restore 方法,卡在wg.Wait()说明有协程没释放锁,这里containerd.Restore方法的第一行就是锁,里面有个方法c.remote.LoadContainer,实际上是和docker-containerd通信的。

查看下 docker-containerd 进程:

1
2
3
4
5
$ ps aux | grep containerd
root 5646 0.2 0.0 606436 14048 ? Ssl 22:33 0:00 docker-containerd --config /var/run/docker/containerd/containerd.toml
appuser 6261 0.0 0.0 112708 984 pts/1 S+ 22:36 0:00 grep --color=auto containerd
root 8355 0.1 0.0 9052 4308 ? Sl Dec03 2:56 docker-containerd-shim -namespace moby -workdir /app/kube/dockercontainerd/daemon/io.containerd.runtime.v1.linux/moby/62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /app/kube/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc
root 11171 0.0 0.0 9052 4052 ? Sl Dec03 0:18 docker-containerd-shim -namespace moby -workdir /app/kube/dockercontainerd/daemon/io.containerd.runtime.v1.linux/moby/73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /app/kube/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc

有残留的,杀掉一个试试

1
$ kill 11171

然后原窗口有输出了一些日志,实际上是执行了 244行的 daemon.containerd.DeleteTask方法,说明思路是对的,进程通信有问题。

1
2
3
4
5
6
7
ERRO[0172] connecting to shim                            error=<nil> id=73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 module="containerd/io.containerd.runtime.v1.linux" namespace=moby
DEBU[2020-12-04T22:36:42.690975930+08:00] restored container alive=false container=73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 module=libcontainerd namespace=moby pid=0
DEBU[2020-12-04T22:36:42.701154551+08:00] Trying to unmount /app/kube/docker/containers/73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212/mounts
DEBU[2020-12-04T22:36:42.707909556+08:00] Unmounted /app/kube/docker/containers/73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212/mounts
DEBU[0172] event published module="containerd/io.containerd.runtime.v1.linux" ns=moby topic="/tasks/exit" type=containerd.events.TaskExit
DEBU[0172] event published module="containerd/io.containerd.runtime.v1.linux" ns=moby topic="/tasks/delete" type=containerd.events.TaskDelete
DEBU[2020-12-04T22:36:42.947670205+08:00] event module=libcontainerd namespace=moby topic=/tasks/exit

接着处理另一个:

1
2
3
4
5
$ ps aux | grep containerd
root 5646 0.2 0.0 606692 14048 ? Ssl 22:33 0:00 docker-containerd --config /var/run/docker/containerd/containerd.toml
root 6461 0.0 0.0 112708 984 pts/1 S+ 22:37 0:00 grep --color=auto containerd
root 8355 0.1 0.0 9052 4260 ? Sl Dec03 2:56 docker-containerd-shim -namespace moby -workdir /app/kube/dockercontainerd/daemon/io.containerd.runtime.v1.linux/moby/62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /app/kube/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc
$ kill 8355

然后前台 debug 的日志没有卡住,正常启动了:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
ERRO[0249] connecting to shim                            error=<nil> id=62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be module="containerd/io.containerd.runtime.v1.linux" namespace=moby
DEBU[2020-12-04T22:37:59.709825146+08:00] restored container alive=false container=62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be module=libcontainerd namespace=moby pid=0
DEBU[2020-12-04T22:37:59.710064357+08:00] event module=libcontainerd namespace=moby topic=/tasks/delete
INFO[2020-12-04T22:37:59.710093459+08:00] ignoring event module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
WARN[2020-12-04T22:37:59.710215638+08:00] Ignoring Exit Event, no such exec command found container=73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 exec-id=73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 exec-pid=11197
DEBU[2020-12-04T22:37:59.719102521+08:00] Trying to unmount /app/kube/docker/containers/62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be/mounts
DEBU[2020-12-04T22:37:59.722934436+08:00] Unmounted /app/kube/docker/containers/62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be/mounts
DEBU[0249] event published module="containerd/containers" ns=moby topic="/containers/delete" type=containerd.events.ContainerDelete
DEBU[2020-12-04T22:37:59.978450001+08:00] container mounted via layerStore: &{/app/kube/docker/overlay2/97a09a97cf8c3ae835fb0ca6526c0282b26379942dfb49081189a39ce0400596/merged 0x2f42600 0x2f42600}
DEBU[0249] event published module="containerd/io.containerd.runtime.v1.linux" ns=moby topic="/tasks/exit" type=containerd.events.TaskExit
DEBU[2020-12-04T22:38:00.169804015+08:00] event module=libcontainerd namespace=moby topic=/tasks/exit
DEBU[0249] event published module="containerd/io.containerd.runtime.v1.linux" ns=moby topic="/tasks/delete" type=containerd.events.TaskDelete
WARN[2020-12-04T22:38:00.169915440+08:00] Ignoring Exit Event, no such exec command found container=62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be exec-id=62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be exec-pid=8396
DEBU[2020-12-04T22:38:00.170037297+08:00] event module=libcontainerd namespace=moby topic=/tasks/delete
INFO[2020-12-04T22:38:00.170061445+08:00] ignoring event module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
DEBU[0249] event published module="containerd/containers" ns=moby topic="/containers/delete" type=containerd.events.ContainerDelete
DEBU[2020-12-04T22:38:00.199820461+08:00] container mounted via layerStore: &{/app/kube/docker/overlay2/2abb109b107ef7f0e5c31b1a100b446234118ae38afe43977c8c718f115cdfd6/merged 0x2f42600 0x2f42600}
DEBU[2020-12-04T22:38:00.208519823+08:00] Option Experimental: false
DEBU[2020-12-04T22:38:00.208542167+08:00] Option DefaultDriver: bridge
DEBU[2020-12-04T22:38:00.208549815+08:00] Option DefaultNetwork: bridge
DEBU[2020-12-04T22:38:00.208557480+08:00] Network Control Plane MTU: 1500
DEBU[2020-12-04T22:38:00.245647071+08:00] /sbin/iptables, [--wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[2020-12-04T22:38:00.247719844+08:00] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]
DEBU[2020-12-04T22:38:00.249828613+08:00] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[2020-12-04T22:38:00.251439314+08:00] /sbin/iptables, [--wait -t nat -D PREROUTING]
...

服务器上有安全狗,可能和安全狗有关系。

参考

CATALOG
  1. 1. 由来
    1. 1.1. 环境信息
    2. 1.2. 排查过程
  2. 2. 参考