zhangguanzhang's Blog

docker18.03 hang at 'restoring container'

hang
字数统计: 3.2k阅读时长: 18 min
2020/12/04

由来

起初是 k8s 有几个 node not ready，上去看了下 kubelet 日志刷 container runtime down，重启了下 docker 后还是没用，docker ps 命令都卡住。

环境信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
$ cat /etc/redhat-release 
Linux xxx-disk0 3.10.0-1127.13.1.el7.x86_64 #1 SMP Tue Jun 23 15:46:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.4.1708 (Core) 

$ docker info
Containers: 91
 Running: 63
 Paused: 0
 Stopped: 28
Images: 539
Server Version: 18.03.0-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-1127.13.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.51GiB
Name: xxx-disk0
ID: UZRM:KRSL:TYWM:VAQY:KWCX:AVFD:NP53:TC35:YHOC:TLLO:YGXO:RMYS
Docker Root Dir: /app/kube/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 treg.yun.xxx.cn
 reg.xxx.lan:5000
 127.0.0.0/8
Registry Mirrors:
 https://registry.docker-cn.com/
 https://docker.mirrors.ustc.edu.cn/
Live Restore Enabled: false

排查过程

先停掉docker，然后前台启动加 debug 参数启动

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
$ pgrep dockerd
4659
$ kill 4659 && > /var/run/docker.pid 
$ ps aux | grep dockerd
root      5628  0.0  0.0 112708   980 pts/0    S+   22:33   0:00 grep --color=auto dockerd
$ ./dockerd -D
WARN[0000] The "graph" config file option is deprecated. Please use "data-root" instead. 
WARN[2020-12-04T22:33:50.432804342+08:00] could not change group /var/run/docker.sock to docker: group docker not found 
DEBU[2020-12-04T22:33:50.432936283+08:00] Listener created for HTTP on unix (/var/run/docker.sock) 
INFO[2020-12-04T22:33:50.433612435+08:00] libcontainerd: started new docker-containerd process  pid=5646
INFO[0000] starting containerd                           module=containerd revision=cfd04396dc68220d1cecbe686a6cc3aa5ce3667c version=v1.0.2
DEBU[0000] changing OOM score to -500                    module=containerd
INFO[0000] loading plugin "io.containerd.content.v1.content"...  module=containerd type=io.containerd.content.v1
INFO[0000] loading plugin "io.containerd.snapshotter.v1.btrfs"...  module=containerd type=io.containerd.snapshotter.v1
WARN[0000] failed to load plugin io.containerd.snapshotter.v1.btrfs  error="path /app/kube/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
INFO[0000] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  module=containerd type=io.containerd.snapshotter.v1
INFO[0000] loading plugin "io.containerd.metadata.v1.bolt"...  module=containerd type=io.containerd.metadata.v1
WARN[0000] could not use snapshotter btrfs in metadata plugin  error="path /app/kube/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt"
INFO[0000] loading plugin "io.containerd.differ.v1.walking"...  module=containerd type=io.containerd.differ.v1
INFO[0000] loading plugin "io.containerd.gc.v1.scheduler"...  module=containerd type=io.containerd.gc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.containers"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.content"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.diff"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.events"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.healthcheck"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.images"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.leases"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.namespaces"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.snapshots"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.monitor.v1.cgroups"...  module=containerd type=io.containerd.monitor.v1
INFO[0000] loading plugin "io.containerd.runtime.v1.linux"...  module=containerd type=io.containerd.runtime.v1
DEBU[0000] loading tasks in namespace                    module="containerd/io.containerd.runtime.v1.linux" namespace=moby
INFO[0000] loading plugin "io.containerd.grpc.v1.tasks"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.version"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.introspection"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
INFO[0000] containerd successfully booted in 0.009604s   module=containerd
DEBU[2020-12-04T22:33:50.456534148+08:00] Golang's threads limit set to 113940         
DEBU[2020-12-04T22:33:50.457345643+08:00] Using default logging driver json-file       
DEBU[2020-12-04T22:33:50.457466912+08:00] [graphdriver] priority list: [btrfs zfs overlay2 aufs overlay devicemapper vfs] 
DEBU[2020-12-04T22:33:50.457623030+08:00] processing event stream                       module=libcontainerd namespace=plugins.moby
DEBU[2020-12-04T22:33:50.479691287+08:00] backingFs=xfs,  projectQuotaSupported=false  
INFO[2020-12-04T22:33:50.479712832+08:00] [graphdriver] using prior storage driver: overlay2 
DEBU[2020-12-04T22:33:50.479724151+08:00] Initialized graph driver overlay2            
DEBU[2020-12-04T22:33:50.510882767+08:00] Max Concurrent Downloads: 10                 
DEBU[2020-12-04T22:33:50.510930407+08:00] Max Concurrent Uploads: 5                    
DEBU[0000] garbage collected                             d=24.493383ms module="containerd/io.containerd.gc.v1.scheduler"
INFO[2020-12-04T22:33:50.608483121+08:00] Graph migration to content-addressability took 0.00 seconds 
INFO[2020-12-04T22:33:50.610430840+08:00] Loading containers: start.                   
DEBU[2020-12-04T22:33:50.610704281+08:00] processing event stream                       module=libcontainerd namespace=moby
DEBU[2020-12-04T22:33:50.611446797+08:00] Loaded container 027a389c8c1e93629cc5f68af8d023b2ecfe350d7771ba6b87598ff705f6c19f, isRunning: false 
DEBU[2020-12-04T22:33:50.611803503+08:00] Loaded container 24735e5aea2bd91b5fa5d729ca021a09532c2ea9b8b06f5171d0da23fc3bf4cc, isRunning: false 
DEBU[2020-12-04T22:33:50.612174253+08:00] Loaded container 487a8c2f30986796c3948d1469d506e1d3ab394e17533040ef7a5444a32be0fc, isRunning: false 
DEBU[2020-12-04T22:33:50.612494092+08:00] Loaded container 52d32b0e03c957b6cb9b4d793c47900e689a29d9ae0d63703ea29073a352fbe5, isRunning: false 
DEBU[2020-12-04T22:33:50.612816495+08:00] Loaded container 5b7b0b52c71a14164f269853679211b3823e9eecc2d3829bf2db10c9b720217d, isRunning: false 
DEBU[2020-12-04T22:33:50.613447082+08:00] Loaded container 62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be, isRunning: true 
DEBU[2020-12-04T22:33:50.613769649+08:00] Loaded container 68ba211ec7328bebd3b241631a703639447c05056ffe07ed633b72d0bc210938, isRunning: false 
DEBU[2020-12-04T22:33:50.614756585+08:00] Loaded container 73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212, isRunning: true 
DEBU[2020-12-04T22:33:50.615381990+08:00] Loaded container 7a2a75c8a0c8dac8c5773ad92fa45ac7e1d33c9be85ecb65eb147929955dca50, isRunning: false 
DEBU[2020-12-04T22:33:50.616222796+08:00] Loaded container 81095c01c4b99c7d2cc9e6bee8726c11f16d27204523727e7d067d980c26ac64, isRunning: false 
DEBU[2020-12-04T22:33:50.616569394+08:00] Loaded container 94098167eb466dbf1a454f5491a162488d1fdb1eebe804c5c1f403f7fce62dc4, isRunning: false 
DEBU[2020-12-04T22:33:50.616981038+08:00] Loaded container 9d4cbcce43b0262d972e73b2770f26ca762e2fa86f0de88a7909b8e59c0b805a, isRunning: false 
DEBU[2020-12-04T22:33:50.617460452+08:00] Loaded container aecde8eb18924d8548d79d5e0383baa7ac3ab1cfc4c55e1f32c4089dfc153071, isRunning: false 
DEBU[2020-12-04T22:33:50.617908975+08:00] Loaded container aed0618a325b4b84363357c1830515048d23af6afd79606cbb0ad64bf5f226a2, isRunning: false 
DEBU[2020-12-04T22:33:50.618252961+08:00] Loaded container b43e4995720f235c40ffd60bde1fb54e87ece3598f8bd625996042f637896687, isRunning: false 
DEBU[2020-12-04T22:33:50.618557604+08:00] Loaded container c1e6a1de9b9c2fd420e718c405c114e726ec5531561a4caf662b757a3724711e, isRunning: false 
DEBU[2020-12-04T22:33:50.618942417+08:00] Loaded container c5eb3c941e562153e0cf0af738f1cb43f34591f0b48ad5458ab2002f5be9e0a8, isRunning: false 
DEBU[2020-12-04T22:33:50.619380785+08:00] Loaded container e211ffccacb8f7982899097fbd0f9ce1d95f8f31f290fb10baf40d00f4980bc9, isRunning: false 
DEBU[2020-12-04T22:33:50.619831551+08:00] Loaded container ef547d238cd01ff7ec048de3442fe9293aa1d5d932ea66c5aed34bfff014182b, isRunning: false 
DEBU[2020-12-04T22:33:50.620192032+08:00] Loaded container f3bb916ec5d7847c3be4341975c47f4e2fe587fc726ca7d76e3dca15cb8dd21d, isRunning: false 
DEBU[2020-12-04T22:33:50.620438678+08:00] Loaded container fa6de6f4aa8894c18a9737bac462f57c69893eca5e4b58bc3bd793a76b252951, isRunning: false 
DEBU[2020-12-04T22:33:51.379861237+08:00] restoring container                           container=ef547d238cd01ff7ec048de3442fe9293aa1d5d932ea66c5aed34bfff014182b paused=false running=false
DEBU[2020-12-04T22:33:51.379910464+08:00] restoring container                           container=e211ffccacb8f7982899097fbd0f9ce1d95f8f31f290fb10baf40d00f4980bc9 paused=false running=false
DEBU[2020-12-04T22:33:51.379994141+08:00] restoring container                           container=7a2a75c8a0c8dac8c5773ad92fa45ac7e1d33c9be85ecb65eb147929955dca50 paused=false running=false
DEBU[2020-12-04T22:33:51.380029802+08:00] restoring container                           container=c1e6a1de9b9c2fd420e718c405c114e726ec5531561a4caf662b757a3724711e paused=false running=false
DEBU[2020-12-04T22:33:51.380084763+08:00] restoring container                           container=5b7b0b52c71a14164f269853679211b3823e9eecc2d3829bf2db10c9b720217d paused=false running=false
DEBU[2020-12-04T22:33:51.380127006+08:00] restoring container                           container=9d4cbcce43b0262d972e73b2770f26ca762e2fa86f0de88a7909b8e59c0b805a paused=false running=false
DEBU[2020-12-04T22:33:51.380121758+08:00] restoring container                           container=fa6de6f4aa8894c18a9737bac462f57c69893eca5e4b58bc3bd793a76b252951 paused=false running=false
DEBU[2020-12-04T22:33:51.380163318+08:00] restoring container                           container=52d32b0e03c957b6cb9b4d793c47900e689a29d9ae0d63703ea29073a352fbe5 paused=false running=false
DEBU[2020-12-04T22:33:51.380310029+08:00] restoring container                           container=027a389c8c1e93629cc5f68af8d023b2ecfe350d7771ba6b87598ff705f6c19f paused=false running=false
DEBU[2020-12-04T22:33:51.380382722+08:00] restoring container                           container=68ba211ec7328bebd3b241631a703639447c05056ffe07ed633b72d0bc210938 paused=false running=false
DEBU[2020-12-04T22:33:51.380419320+08:00] restoring container                           container=487a8c2f30986796c3948d1469d506e1d3ab394e17533040ef7a5444a32be0fc paused=false running=false
DEBU[2020-12-04T22:33:51.380433522+08:00] restoring container                           container=73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 paused=false running=true
DEBU[2020-12-04T22:33:51.380459224+08:00] restoring container                           container=c5eb3c941e562153e0cf0af738f1cb43f34591f0b48ad5458ab2002f5be9e0a8 paused=false running=false
DEBU[2020-12-04T22:33:51.380525276+08:00] restoring container                           container=b43e4995720f235c40ffd60bde1fb54e87ece3598f8bd625996042f637896687 paused=false running=false
DEBU[2020-12-04T22:33:51.380563957+08:00] restoring container                           container=81095c01c4b99c7d2cc9e6bee8726c11f16d27204523727e7d067d980c26ac64 paused=false running=false
DEBU[2020-12-04T22:33:51.380586567+08:00] restoring container                           container=62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be paused=false running=true
DEBU[2020-12-04T22:33:51.380599061+08:00] restoring container                           container=94098167eb466dbf1a454f5491a162488d1fdb1eebe804c5c1f403f7fce62dc4 paused=false running=false
DEBU[2020-12-04T22:33:51.380616220+08:00] restoring container                           container=aecde8eb18924d8548d79d5e0383baa7ac3ab1cfc4c55e1f32c4089dfc153071 paused=false running=false
DEBU[2020-12-04T22:33:51.380641090+08:00] restoring container                           container=f3bb916ec5d7847c3be4341975c47f4e2fe587fc726ca7d76e3dca15cb8dd21d paused=false running=false
DEBU[2020-12-04T22:33:51.380825356+08:00] restoring container                           container=24735e5aea2bd91b5fa5d729ca021a09532c2ea9b8b06f5171d0da23fc3bf4cc paused=false running=false
DEBU[2020-12-04T22:33:51.380953092+08:00] restoring container                           container=aed0618a325b4b84363357c1830515048d23af6afd79606cbb0ad64bf5f226a2 paused=false running=false

然后发现卡在这，正常是会像 gin 那样启动输出支持的 http api 路由信息的。开一个窗口，发送 SIGUSR1 信号打印 goroutine 堆栈信息看看卡在哪儿：

1
2
3
$ pgrep dockerd
3085
$ kill -USR1 3085

docker 的日志和系统日志都会有下面的类似输出：

1
Dec 04 22:33:52 xxxx dockerd[3085]: time="2020-12-33T58:15:52.906433650+08:00" level=info msg="goroutine stacks written to /var/run/docker/goroutine-stacks-2020-12-04T223358+0800.log"

查看了下下面这段比较可疑，daemon/daemon.go:364 附近

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
goroutine 1 [semacquire, 5 minutes]:
sync.runtime_Semacquire(0xc4204de73c)
	/usr/local/go/src/runtime/sema.go:56 +0x3b
sync.(*WaitGroup).Wait(0xc4204de730)
	/usr/local/go/src/sync/waitgroup.go:131 +0x74
github.com/docker/docker/daemon.(*Daemon).restore(0xc42009a480, 0x190c3a6, 0x4)
	/go/src/github.com/docker/docker/daemon/daemon.go:364 +0xfeb
github.com/docker/docker/daemon.NewDaemon(0xc42018d200, 0x2ec75c0, 0xc4201be410, 0x2ea89e0, 0xc420087d40, 0xc4201323c0, 0x0, 0x0, 0x0)
	/go/src/github.com/docker/docker/daemon/daemon.go:894 +0x258d
main.(*DaemonCli).start(0xc42051da40, 0xc4201c5d50, 0x0, 0x0)
	/go/src/github.com/docker/docker/cmd/dockerd/daemon.go:223 +0x1320
main.runDaemon(0xc4201c5d50, 0xc42044b3b0, 0x0)
	/go/src/github.com/docker/docker/cmd/dockerd/docker.go:78 +0x78
main.newDaemonCommand.func1(0xc420176000, 0xc4201359e0, 0x0, 0x1, 0x0, 0x0)
	/go/src/github.com/docker/docker/cmd/dockerd/docker.go:29 +0x5d
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc420176000, 0xc42000c090, 0x1, 0x1, 0xc420176000, 0xc42000c090)
	/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:646 +0x44f
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc420176000, 0x2194e40, 0x2419c01, 0xc420135980)
	/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:742 +0x310
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(0xc420176000, 0xc420135980, 0x190fa00)
	/go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:695 +0x2d
main.main()
	/go/src/github.com/docker/docker/cmd/dockerd/docker.go:105 +0xe3

按照docker info 的信息去找了下对应的 分支代码。364 行一个wg.Wait()，得看前面的 goroutine 是卡在哪儿，根据前面的堆栈信息，应该是卡在 github.com/docker/docker/daemon.(*Daemon).restore，也就是 238 行的 daemon.containerd.Restore 方法，卡在wg.Wait()说明有协程没释放锁，这里containerd.Restore方法的第一行就是锁，里面有个方法c.remote.LoadContainer，实际上是和docker-containerd通信的。

查看下 docker-containerd 进程：

1
2
3
4
5
$ ps aux | grep containerd
root      5646  0.2  0.0 606436 14048 ?        Ssl  22:33   0:00 docker-containerd --config /var/run/docker/containerd/containerd.toml
appuser   6261  0.0  0.0 112708   984 pts/1    S+   22:36   0:00 grep --color=auto containerd
root      8355  0.1  0.0   9052  4308 ?        Sl   Dec03   2:56 docker-containerd-shim -namespace moby -workdir /app/kube/dockercontainerd/daemon/io.containerd.runtime.v1.linux/moby/62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /app/kube/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc
root     11171  0.0  0.0   9052  4052 ?        Sl   Dec03   0:18 docker-containerd-shim -namespace moby -workdir /app/kube/dockercontainerd/daemon/io.containerd.runtime.v1.linux/moby/73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /app/kube/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc

有残留的，杀掉一个试试

1
$ kill 11171

然后原窗口有输出了一些日志，实际上是执行了 244行的 daemon.containerd.DeleteTask方法，说明思路是对的，进程通信有问题。

1
2
3
4
5
6
7
ERRO[0172] connecting to shim                            error=<nil> id=73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 module="containerd/io.containerd.runtime.v1.linux" namespace=moby
DEBU[2020-12-04T22:36:42.690975930+08:00] restored container                            alive=false container=73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 module=libcontainerd namespace=moby pid=0
DEBU[2020-12-04T22:36:42.701154551+08:00] Trying to unmount /app/kube/docker/containers/73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212/mounts 
DEBU[2020-12-04T22:36:42.707909556+08:00] Unmounted /app/kube/docker/containers/73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212/mounts 
DEBU[0172] event published                               module="containerd/io.containerd.runtime.v1.linux" ns=moby topic="/tasks/exit" type=containerd.events.TaskExit
DEBU[0172] event published                               module="containerd/io.containerd.runtime.v1.linux" ns=moby topic="/tasks/delete" type=containerd.events.TaskDelete
DEBU[2020-12-04T22:36:42.947670205+08:00] event                                         module=libcontainerd namespace=moby topic=/tasks/exit

接着处理另一个：

1
2
3
4
5
$ ps aux | grep containerd
root      5646  0.2  0.0 606692 14048 ?        Ssl  22:33   0:00 docker-containerd --config /var/run/docker/containerd/containerd.toml
root      6461  0.0  0.0 112708   984 pts/1    S+   22:37   0:00 grep --color=auto containerd
root      8355  0.1  0.0   9052  4260 ?        Sl   Dec03   2:56 docker-containerd-shim -namespace moby -workdir /app/kube/dockercontainerd/daemon/io.containerd.runtime.v1.linux/moby/62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be -address /var/run/docker/containerd/docker-containerd.sock -containerd-binary /app/kube/bin/docker-containerd -runtime-root /var/run/docker/runtime-runc
$ kill 8355

然后前台 debug 的日志没有卡住，正常启动了：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
ERRO[0249] connecting to shim                            error=<nil> id=62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be module="containerd/io.containerd.runtime.v1.linux" namespace=moby
DEBU[2020-12-04T22:37:59.709825146+08:00] restored container                            alive=false container=62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be module=libcontainerd namespace=moby pid=0
DEBU[2020-12-04T22:37:59.710064357+08:00] event                                         module=libcontainerd namespace=moby topic=/tasks/delete
INFO[2020-12-04T22:37:59.710093459+08:00] ignoring event                                module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
WARN[2020-12-04T22:37:59.710215638+08:00] Ignoring Exit Event, no such exec command found  container=73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 exec-id=73cfe941e7a948a77783c77f963efc66327323c2603e058e7ab61f85f8e98212 exec-pid=11197
DEBU[2020-12-04T22:37:59.719102521+08:00] Trying to unmount /app/kube/docker/containers/62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be/mounts 
DEBU[2020-12-04T22:37:59.722934436+08:00] Unmounted /app/kube/docker/containers/62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be/mounts 
DEBU[0249] event published                               module="containerd/containers" ns=moby topic="/containers/delete" type=containerd.events.ContainerDelete
DEBU[2020-12-04T22:37:59.978450001+08:00] container mounted via layerStore: &{/app/kube/docker/overlay2/97a09a97cf8c3ae835fb0ca6526c0282b26379942dfb49081189a39ce0400596/merged 0x2f42600 0x2f42600} 
DEBU[0249] event published                               module="containerd/io.containerd.runtime.v1.linux" ns=moby topic="/tasks/exit" type=containerd.events.TaskExit
DEBU[2020-12-04T22:38:00.169804015+08:00] event                                         module=libcontainerd namespace=moby topic=/tasks/exit
DEBU[0249] event published                               module="containerd/io.containerd.runtime.v1.linux" ns=moby topic="/tasks/delete" type=containerd.events.TaskDelete
WARN[2020-12-04T22:38:00.169915440+08:00] Ignoring Exit Event, no such exec command found  container=62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be exec-id=62b049d16d1fe03c193295329e7055b3e675a5e94b9566eee6accc35820530be exec-pid=8396
DEBU[2020-12-04T22:38:00.170037297+08:00] event                                         module=libcontainerd namespace=moby topic=/tasks/delete
INFO[2020-12-04T22:38:00.170061445+08:00] ignoring event                                module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
DEBU[0249] event published                               module="containerd/containers" ns=moby topic="/containers/delete" type=containerd.events.ContainerDelete
DEBU[2020-12-04T22:38:00.199820461+08:00] container mounted via layerStore: &{/app/kube/docker/overlay2/2abb109b107ef7f0e5c31b1a100b446234118ae38afe43977c8c718f115cdfd6/merged 0x2f42600 0x2f42600} 
DEBU[2020-12-04T22:38:00.208519823+08:00] Option Experimental: false                   
DEBU[2020-12-04T22:38:00.208542167+08:00] Option DefaultDriver: bridge                 
DEBU[2020-12-04T22:38:00.208549815+08:00] Option DefaultNetwork: bridge                
DEBU[2020-12-04T22:38:00.208557480+08:00] Network Control Plane MTU: 1500              
DEBU[2020-12-04T22:38:00.245647071+08:00] /sbin/iptables, [--wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[2020-12-04T22:38:00.247719844+08:00] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER] 
DEBU[2020-12-04T22:38:00.249828613+08:00] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[2020-12-04T22:38:00.251439314+08:00] /sbin/iptables, [--wait -t nat -D PREROUTING]
...

服务器上有安全狗，可能和安全狗有关系。

参考

原文作者：Zhangguanzhang

原文链接：http://zhangguanzhang.github.io/2020/12/04/docker1803-hang-container-restore/

发表日期：December 4th 2020, 7:42:08 pm

更新日期：December 4th 2020, 7:42:08 pm

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
  1. 1. 由来
    1. 1.1. 环境信息
    2. 1.2. 排查过程
  2. 2. 参考
Total : 186
2024
2023
2022
2021
2020
2019
2018
2017
2016
linux 微信 diy 8266 总结 ipmi CLP k8s Mismatch Altium Designer ansible consul docker lnmp 安卓 Linux kernel prometheus exporter boot grub cad tcp golang centos6 chromedp chrony kubernetes cni-plugins mips64le kubeconfig swap dlv adguardhome dns Dockerfile e2fsck fsck keepavlied ecs xmrig etcd fio flannel ftp go mod go gobot esp8266 hello world host-gw mqtt 原创 jira confluence kube-controller-manager java springboot vip keepalived ConfigMap PodAffinity kylin arm64 Kylin audit fs nginx microsoft graph onedrive nodeport usb-net overlay kernal preseed Prometheus openstack cloud-init python containerd keepalive shell bash skopeo ssh kubelet systemd dockerhub gcr.io api ubuntu18 initrd tftp uos x86 openwrt zabbix zombie 正则 awk EmuELEC ingress N1 cobra hang cifs http lvs ipvsadm kube-proxy 63s timeout lamp nfc pm3 coredns openshift ocp squashfs timezone rpm rpmbuild segfault suse wireguard raid kubeadm operator exsi redhat8.4 vxlan veth panic supervisor pyinstaller ttf woff hostname qemu binfmt_misc loongarch64 externalIP cgo nfs ipset proxmox non-root containers vmware hung apparmor serviceAccountToken gpu nvidia cri-dockerd pprof trivy redis /etc/resolv.conf hostPort hotplug android headscale derper iptables
日志 心得 闲搞 ipmi kubernetes ansible kubelet Linux 心得 prometheus boot Auto cad boot golang kubeconfig grub linux centos6 容器 docker docker 容器 openstack boot vsftp go version panic gobot 闲聊 iptables admission kube-controller-manager keepalived k8s Prometheus Job k8s lua microsoft openwrt preseed Prometheus proxmox 总结 CI Kylin ubuntu18 pxe uos graph 运维 zombie 技巧 travis docker onedrive kickstart consul fstab http vxlan coredns node-cache golang kmem suse wireguard raid kubeadm LVM kickstart prometheus raid