docker-ce 18.09.3 启动panic: invalid freelist page: 56, page type is leaf的解决处理

k8s docker

字数统计: 2.3k阅读时长: 13 min

 2021/05/26 

这个问题和之前的docker-18.06.3-ce启动panic: invalid page type: 0: 0的解决处理差不多，不过 db 文件不同。客户停止 docker 后起不来了，查看日志：

1	journalctl -xe -u docker

5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17+08:00" level=warning msg="The \"graph\" config file option is deprecated. Please use \"data-root\" instead."
5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17.841082978+08:00" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17.858839019+08:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /data/kube/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17.859857612+08:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found.\n": exit status 1"
5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17.860062002+08:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /data/kube/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17.860084186+08:00" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /data/kube/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17.860090453+08:00" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /data/kube/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
5月 26 18:42:17 xxxx dockerd[19053]: time="2021-05-26T18:42:17.860095653+08:00" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found.\n": exit status 1"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.866980   19061 flags.go:33] FLAG: --container-runtime="docker"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.866984   19061 flags.go:33] FLAG: --container-runtime-endpoint="unix:///var/run/dockershim.sock"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867014   19061 flags.go:33] FLAG: --docker="unix:///var/run/docker.sock"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867018   19061 flags.go:33] FLAG: --docker-endpoint="unix:///var/run/docker.sock"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867022   19061 flags.go:33] FLAG: --docker-env-metadata-whitelist=""
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867025   19061 flags.go:33] FLAG: --docker-only="false"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867028   19061 flags.go:33] FLAG: --docker-root="/var/lib/docker"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867032   19061 flags.go:33] FLAG: --docker-tls="false"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867036   19061 flags.go:33] FLAG: --docker-tls-ca="ca.pem"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867039   19061 flags.go:33] FLAG: --docker-tls-cert="cert.pem"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867043   19061 flags.go:33] FLAG: --docker-tls-key="key.pem"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867138   19061 flags.go:33] FLAG: --experimental-dockershim="false"
5月 26 18:42:17 xxxx kubelet[19061]: I0526 18:42:17.867143   19061 flags.go:33] FLAG: --experimental-dockershim-root-directory="/var/lib/dockershim"
5月 26 18:42:18 xxxx dockerd[19053]: time="2021-05-26T18:42:18.208058217+08:00" level=error msg="Failed to load container 90e5da1293b04c3eab30e9f7a2d5714aba563b2ab20d15c67aee1d1d79d3154c: json: cannot unmarshal number into Go value of type container.Container"
5月 26 18:42:18 xxxx dockerd[19053]: time="2021-05-26T18:42:18.214213879+08:00" level=error msg="Failed to load container 99abceba7154b7dba08cc03bc1651fd18f38b0d485dcd2c195f479fc375a8216: invalid character 'e' looking for beginning of value"
5月 26 18:42:25 xxxx dockerd[19053]: panic: invalid freelist page: 56, page type is leaf
5月 26 18:42:25 xxxx dockerd[19053]: goroutine 1 [running]:
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*freelist).read(0xc424b121b0, 0x7fc46c7b1000)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/freelist.go:237 +0x2ff
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*DB).loadFreelist.func1()
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/db.go:292 +0x12d
5月 26 18:42:25 xxxx dockerd[19053]: sync.(*Once).Do(0xc420b70328, 0xc420fb1f58)
5月 26 18:42:25 xxxx dockerd[19053]: /usr/local/go/src/sync/once.go:44 +0xc0
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/go.etcd.io/bbolt.(*DB).loadFreelist(0xc420b701e0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/db.go:285 +0x50
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/go.etcd.io/bbolt.Open(0xc422bddad0, 0x2b, 0x1a4, 0xc420fb2070, 0x20c754c, 0x23816c0, 0x25ceee8)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/go.etcd.io/bbolt/db.go:262 +0x316
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libkv/store/boltdb.(*BoltDB).getDBhandle(0xc424b10960, 0x32ef098, 0xc424b109a4, 0x7fc47c368e38)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libkv/store/boltdb/boltdb.go:113 +0x93
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libkv/store/boltdb.(*BoltDB).List(0xc424b10960, 0xc422b99fe0, 0x1b, 0x0, 0x0, 0x0, 0x0, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libkv/store/boltdb/boltdb.go:274 +0xbb
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore.(*cache).kmap(0xc424a60fe0, 0x26b6dc0, 0xc422a0e4d0, 0x4467ea, 0xc422b99f9a, 0x1a668ce)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore/cache.go:43 +0x18a
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore.(*cache).list(0xc424a60fe0, 0x26b6dc0, 0xc422a0e4d0, 0x0, 0x0, 0x0, 0x0, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore/cache.go:164 +0x7b
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore.(*datastore).List(0xc42352b180, 0xc422b99f80, 0x1b, 0x26b6dc0, 0xc422a0e4d0, 0x0, 0x0, 0x0, 0x0, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/datastore/datastore.go:517 +0x179
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge.(*driver).populateNetworks(0xc42203d5c0, 0x5, 0x1a6a443)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge/bridge_store.go:50 +0xe0
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge.(*driver).initStore(0xc42203d5c0, 0xc424b316b0, 0x0, 0xc423529bc0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge/bridge_store.go:35 +0x207
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge.(*driver).configure(0xc42203d5c0, 0xc424b316b0, 0x0, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge/bridge.go:378 +0x1b4
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge.Init(0x268abc0, 0xc424119740, 0xc424b316b0, 0x0, 0xffffffffffffffff)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/bridge/bridge.go:161 +0xa2
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drvregistry.(*DrvRegistry).AddDriver(0xc424119740, 0x1a6a539, 0x6, 0x266fe78, 0xc424b316b0, 0xc42264df80, 0x8)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drvregistry/drvregistry.go:72 +0x48
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/docker/libnetwork.New(0xc42264df80, 0x9, 0x10, 0xc4209e03f0, 0xc420c7f620, 0xc42264df80, 0x9)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/controller.go:220 +0x4a5
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/daemon.(*Daemon).initNetworkController(0xc420a281e0, 0xc42087d900, 0xc420c7f620, 0xc420a281e0, 0xc42239c740, 0xc420c7f620, 0xc420c7f5f0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/daemon/daemon_unix.go:807 +0xa9
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/daemon.(*Daemon).restore(0xc420a281e0, 0xc4209b8300, 0xc4208e4790)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/daemon/daemon.go:419 +0xd6f
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/daemon.NewDaemon(0x26a6240, 0xc4209b8300, 0xc42087d900, 0xc4209e03f0, 0x0, 0x0, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/daemon/daemon.go:987 +0x2c4b
5月 26 18:42:25 xxxx dockerd[19053]: main.(*DaemonCli).start(0xc42024b890, 0xc42018f320, 0x0, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/cmd/dockerd/daemon.go:180 +0x74f
5月 26 18:42:25 xxxx dockerd[19053]: main.runDaemon(0xc42018f320, 0xc420876200, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:7 +0x47
5月 26 18:42:25 xxxx dockerd[19053]: main.newDaemonCommand.func1(0xc42087ac80, 0x32ef098, 0x0, 0x0, 0x0, 0x0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/cmd/dockerd/docker.go:29 +0x5d
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc42087ac80, 0xc42003a1e0, 0x0, 0x0, 0xc42087ac80, 0xc42003a1e0)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:762 +0x46a
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc42087ac80, 0x267bb40, 0x2254580, 0x267bb50)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:852 +0x30c
5月 26 18:42:25 xxxx dockerd[19053]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(0xc42087ac80, 0xc42000e020, 0x4d067f)
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:800 +0x2d
5月 26 18:42:25 xxxx dockerd[19053]: main.main()
5月 26 18:42:25 xxxx dockerd[19053]: /go/src/github.com/docker/docker/cmd/dockerd/docker.go:70 +0xa2
5月 26 18:42:25 xxxx systemd[1]: docker.service: main process exited, code=exited, status=2/INVALIDARGUMENT
5月 26 18:42:25 xxxx systemd[1]: Unit docker.service entered failed state.
5月 26 18:42:25 xxxx systemd[1]: docker.service failed.

首先得根据这个 panic 的堆栈，调用关系是 main.main -> cobra -> docker daemon -> daemon.(*Daemon).restore -> initNetworkController -> libnetwork/datastore/cache -> boltdb
docker 使用了 boltdb 存储了网络信息成 db 文件，但是这个 db 文件损坏了，导致读取字节序列化错误类型，去 docker 的目录 find 下:

$ find /data/kube/docker -type f -size -5M -name '*.db' | grep -v overlay2
/data/kube/docker/containerd/daemon/io.containerd.metadata.v1.bolt/meta.db
/data/kube/docker/volumes/metadata.db
/data/kube/docker/network/files/local-kv.db
/data/kube/docker/builder/fscache.db
/data/kube/docker/buildkit/snapshots.db
/data/kube/docker/buildkit/metadata.db
/data/kube/docker/buildkit/cache.db

改名 db 文件重启 docker 解决

1 2	mv /data/kube/docker/network/files/local-kv.db{,.bak} systemctl restart docker

原文作者：Zhangguanzhang

原文链接：http://zhangguanzhang.github.io/2021/05/26/docker-panic-invalid-freelist-page/

发表日期：May 26th 2021, 7:52:37 pm

更新日期：May 26th 2021, 7:52:37 pm

Next Post

openshift 4.5.9 etcd损坏+脑裂修复过程
Previous Post

一次单节点单个pod网络问题排查过程

CATALOG