zhangguanzhang's Blog

docker-ce 18.09.3 启动panic: invalid freelist page: 56, page type is leaf的解决处理

k8s docker
字数统计: 2.3k阅读时长: 13 min
2021/05/26

这个问题和之前的docker-18.06.3-ce启动panic: invalid page type: 0: 0的解决处理差不多，不过 db 文件不同。客户停止 docker 后起不来了，查看日志：

1
journalctl -xe -u docker
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17+08:00" level=warning msg="The \"graph\" config file option is deprecated. Please use \"data-root\" instead."
5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17.841082978+08:00" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17.858839019+08:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /data/kube/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17.859857612+08:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found.\n": exit status 1"
5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17.860062002+08:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /data/kube/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17.860084186+08:00" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /data/kube/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17.860090453+08:00" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /data/kube/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17.860095653+08:00" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found.\n": exit status 1"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.866980   19061 flags.go:33] FLAG: --container-runtime="docker"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.866984   19061 flags.go:33] FLAG: --container-runtime-endpoint="unix:///var/run/dockershim.sock"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867014   19061 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867018   19061 flags.go:33] FLAG: --docker-endpoint="unix:///var/run/docker.sock"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867022   19061 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867025   19061 flags.go:33] FLAG: --docker-only="false"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867028   19061 flags.go:33] FLAG: --docker-root="/var/lib/docker"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867032   19061 flags.go:33] FLAG: --docker-tls="false"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867036   19061 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867039   19061 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867043   19061 flags.go:33] FLAG: --docker-tls-key="key.pem"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867138   19061 flags.go:33] FLAG: --experimental-dockershim="false"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867143   19061 flags.go:33] FLAG: --experimental-dockershim-root-directory="/var/lib/dockershim"
5月 26 18:42:18 xxxx dockerd[19053]: time="2021-05-26T18:42:18.208058217+08:00" level=error msg="Failed to load container 90e5da1293b04c3eab30e9f7a2d5714aba563b2ab20d15c67aee1d1d79d3154c: json: cannot unmarshal number into Go value of type container.Container"
5月 26 18:42:18 xxxx dockerd[19053]: time="2021-05-26T18:42:18.214213879+08:00" level=error msg="Failed to load container 99abceba7154b7dba08cc03bc1651fd18f38b0d485dcd2c195f479fc375a8216: invalid character 'e' looking for beginning of value"
5月 26 18:42:25 xxxx dockerd[19053]: panic: invalid freelist page: 56, page type is leaf
5月 26 18:42:25 xxxx dockerd[19053]: goroutine 1 [running]:
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*freelist).read(0xc424b121b0, 0x7fc46c7b1000)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/freelist.go:237 +0x2ff
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*DB).loadFreelist.func1()
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/db.go:292 +0x12d
5月 26 18:42:25 xxxx dockerd[19053]: sync.(*Once).Do(0xc420b70328, 0xc420fb1f58)
5月 26 18:42:25 xxxx dockerd[19053]: /usr/local/go/src/sync/once.go:44 +0xc0
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*DB).loadFreelist(0xc420b701e0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/db.go:285 +0x50
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/go.etcd.io/bbolt.Open(0xc422bddad0, 0x2b, 0x1a4, 0xc420fb2070, 0x20c754c, 0x23816c0, 0x25ceee8)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/db.go:262 +0x316
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libkv/store/boltdb.(*BoltDB).getDBhandle(0xc424b10960, 0x32ef098, 0xc424b109a4, 0x7fc47c368e38)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libkv/store/boltdb/boltdb.go:113 +0x93
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libkv/store/boltdb.(*BoltDB).List(0xc424b10960, 0xc422b99fe0, 0x1b, 0x0, 0x0, 0x0, 0x0, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libkv/store/boltdb/boltdb.go:274 +0xbb
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore.(*cache).kmap(0xc424a60fe0, 0x26b6dc0, 0xc422a0e4d0, 0x4467ea, 0xc422b99f9a, 0x1a668ce)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore/cache.go:43 +0x18a
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore.(*cache).list(0xc424a60fe0, 0x26b6dc0, 0xc422a0e4d0, 0x0, 0x0, 0x0, 0x0, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore/cache.go:164 +0x7b
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore.(*datastore).List(0xc42352b180, 0xc422b99f80, 0x1b, 0x26b6dc0, 0xc422a0e4d0, 0x0, 0x0, 0x0, 0x0, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore/datastore.go:517 +0x179
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge.(*driver).populateNetworks(0xc42203d5c0, 0x5, 0x1a6a443)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge/bridge_store.go:50 +0xe0
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge.(*driver).initStore(0xc42203d5c0, 0xc424b316b0, 0x0, 0xc423529bc0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge/bridge_store.go:35 +0x207
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge.(*driver).configure(0xc42203d5c0, 0xc424b316b0, 0x0, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge/bridge.go:378 +0x1b4
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge.Init(0x268abc0, 0xc424119740, 0xc424b316b0, 0x0, 0xffffffffffffffff)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge/bridge.go:161 +0xa2
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drvregistry.(*DrvRegistry).AddDriver(0xc424119740, 0x1a6a539, 0x6, 0x266fe78, 0xc424b316b0, 0xc42264df80, 0x8)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drvregistry/drvregistry.go:72 +0x48
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork.New(0xc42264df80, 0x9, 0x10, 0xc4209e03f0, 0xc420c7f620, 0xc42264df80, 0x9)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/controller.go:220 +0x4a5
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/daemon.(*Daemon).initNetworkController(0xc420a281e0, 0xc42087d900, 0xc420c7f620, 0xc420a281e0, 0xc42239c740, 0xc420c7f620, 0xc420c7f5f0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/daemon/daemon_unix.go:807 +0xa9
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/daemon.(*Daemon).restore(0xc420a281e0, 0xc4209b8300, 0xc4208e4790)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/daemon/daemon.go:419 +0xd6f
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/daemon.NewDaemon(0x26a6240, 0xc4209b8300, 0xc42087d900, 0xc4209e03f0, 0x0, 0x0, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/daemon/daemon.go:987 +0x2c4b
5月 26 18:42:25 xxxx dockerd[19053]: main.(*DaemonCli).start(0xc42024b890, 0xc42018f320, 0x0, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/cmd/dockerd/daemon.go:180 +0x74f
5月 26 18:42:25 xxxx dockerd[19053]: main.runDaemon(0xc42018f320, 0xc420876200, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:7 +0x47
5月 26 18:42:25 xxxx dockerd[19053]: main.newDaemonCommand.func1(0xc42087ac80, 0x32ef098, 0x0, 0x0, 0x0, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/cmd/dockerd/docker.go:29 +0x5d
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc42087ac80, 0xc42003a1e0, 0x0, 0x0, 0xc42087ac80, 0xc42003a1e0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:762 +0x46a
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc42087ac80, 0x267bb40, 0x2254580, 0x267bb50)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:852 +0x30c
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(0xc42087ac80, 0xc42000e020, 0x4d067f)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:800 +0x2d
5月 26 18:42:25 xxxx dockerd[19053]: main.main()
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/cmd/dockerd/docker.go:70 +0xa2
5月 26 18:42:25 xxxx systemd[1]: docker.service: main process exited, code=exited, status=2/INVALIDARGUMENT
5月 26 18:42:25 xxxx systemd[1]: Unit docker.service entered failed state.
5月 26 18:42:25 xxxx systemd[1]: docker.service failed.

首先得根据这个 panic 的堆栈，调用关系是 main.main -> cobra -> docker daemon -> daemon.(*Daemon).restore -> initNetworkController -> libnetwork/datastore/cache -> boltdb
docker 使用了 boltdb 存储了网络信息成 db 文件，但是这个 db 文件损坏了，导致读取字节序列化错误类型，去 docker 的目录 find 下:

1
2
3
4
5
6
7
8
$ find /data/kube/docker -type f -size -5M -name '*.db' | grep -v overlay2
/data/kube/docker/containerd/daemon/io.containerd.metadata.v1.bolt/meta.db
/data/kube/docker/volumes/metadata.db
/data/kube/docker/network/files/local-kv.db
/data/kube/docker/builder/fscache.db
/data/kube/docker/buildkit/snapshots.db
/data/kube/docker/buildkit/metadata.db
/data/kube/docker/buildkit/cache.db

改名 db 文件重启 docker 解决

1
2
mv /data/kube/docker/network/files/local-kv.db{,.bak}
systemctl restart docker

原文作者：Zhangguanzhang

原文链接：http://zhangguanzhang.github.io/2021/05/26/docker-panic-invalid-freelist-page/

发表日期：May 26th 2021, 7:52:37 pm

更新日期：May 26th 2021, 7:52:37 pm

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
Total : 178
2024
2023
2022
2021
2020
2019
2018
2017
2016
linux 微信 diy 8266 总结 ipmi CLP k8s Mismatch Altium Designer ansible consul docker lnmp 安卓 Linux kernel prometheus exporter boot grub cad tcp golang centos6 chromedp chrony kubernetes cni-plugins mips64le kubeconfig swap dlv adguardhome dns Dockerfile e2fsck fsck keepavlied ecs xmrig etcd fio flannel ftp go mod go gobot esp8266 hello world host-gw mqtt 原创 jira confluence kube-controller-manager java springboot vip keepalived ConfigMap PodAffinity kylin arm64 Kylin audit fs nginx microsoft graph onedrive nodeport usb-net overlay kernal preseed Prometheus openstack cloud-init python containerd keepalive shell bash skopeo ssh kubelet systemd dockerhub gcr.io api ubuntu18 initrd tftp uos x86 openwrt zabbix zombie 正则 awk EmuELEC ingress N1 cobra hang cifs http lvs ipvsadm kube-proxy 63s timeout lamp nfc pm3 coredns openshift ocp squashfs timezone rpm rpmbuild segfault suse wireguard raid kubeadm operator exsi redhat8.4 vxlan veth panic supervisor pyinstaller ttf woff hostname qemu binfmt_misc loongarch64 externalIP cgo nfs ipset proxmox non-root containers vmware hung apparmor serviceAccountToken gpu nvidia cri-dockerd pprof trivy
日志 心得 闲搞 ipmi kubernetes ansible kubelet Linux 心得 prometheus boot Auto cad boot golang kubeconfig grub linux centos6 容器 docker docker 容器 openstack boot vsftp go version panic gobot 闲聊 iptables admission kube-controller-manager keepalived k8s Prometheus Job k8s lua microsoft openwrt preseed Prometheus proxmox 总结 CI Kylin ubuntu18 pxe uos graph 运维 zombie 技巧 travis docker onedrive kickstart consul fstab http vxlan coredns node-cache golang kmem suse wireguard raid kubeadm LVM kickstart prometheus raid