golang 代码层面构建和推送容器镜像到远端仓库的 demo 参考
构建镜像
这里使用 kaniko 先举例：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
| $ cat Dockerfile
FROM alpine
COPY Dockerfile /opt
RUN chmod -R 644 /opt
$ docker run --rm -ti -v $PWD:/workspace gcr.io/kaniko-project/executor:v1.20.1 \
--no-push --context=/workspace --dockerfile /workspace/Dockerfile \
--destination test --tar-path image.tar
INFO[0000] Retrieving image manifest alpine
INFO[0000] Retrieving image alpine from registry index.docker.io
INFO[0002] Built cross stage deps: map[]
INFO[0002] Retrieving image manifest alpine
INFO[0002] Returning cached image manifest
INFO[0002] Executing 0 build triggers
INFO[0002] Building stage 'alpine' [idx: '0', base-idx: '-1']
INFO[0002] Unpacking rootfs as cmd COPY Dockerfile /opt requires it.
INFO[0005] COPY go.mod /opt
INFO[0005] Taking snapshot of files...
INFO[0005] RUN chmod -R 644 /opt
INFO[0005] Initializing snapshotter ...
INFO[0005] Taking snapshot of full filesystem...
INFO[0005] Cmd: /bin/sh
INFO[0005] Args: [-c chmod -R 644 /opt]
INFO[0005] Running: [/bin/sh -c chmod -R 644 /opt]
INFO[0005] Taking snapshot of full filesystem...
INFO[0007] Skipping push to container registry due to --no-push flag
|
然后目录就有 image.tar 镜像
代码
下面代码注意，需要自己设置两个环境变量
DOCKER_CONFIG=xxx.json 内容为登录后的 json 文件路径KANIKO_DIR=/opt/xxx/xxx 干净的空目录
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
| package main
import (
"fmt"
"os"
"github.com/GoogleContainerTools/kaniko/pkg/config"
"github.com/GoogleContainerTools/kaniko/pkg/executor"
)
func main() {
opts := &config.KanikoOptions{
RegistryOptions: config.RegistryOptions{
Insecure: true,
InsecurePull: true,
ImageDownloadRetry: 3,
InsecureRegistries: []string{""},
},
SrcContext: ".",
DockerfilePath: os.Args[1],
Destinations: []string{os.Args[2]},
Compression: "gzip",
SnapshotMode: "full",
}
err := executor.CheckPushPermissions(opts)
if err != nil {
fmt.Printf("检查推送权限失败: %v\n", err)
os.Exit(1)
}
image, err := executor.DoBuild(opts)
if err != nil {
fmt.Printf("构建镜像失败: %v\n", err)
os.Exit(1)
}
err = executor.DoPush(image, opts)
if err != nil {
fmt.Printf("推送镜像失败: %v\n", err)
os.Exit(1)
}
fmt.Println("镜像推送成功!")
}
|
编译为 build ，注意 KANIKO_DIR 的目录和 SrcContext 不能重合，还有建议去看看源码，KANIKO_DIR 修改很不方便，自行接入测试就知道了。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
| $ docker pull xxx.xxx.xxx:5000/build-test/test
Using default tag: latest
Error response from daemon: manifest for xxx.xxx.xxx:5000/build-test/test:latest not found: manifest unknown: manifest unknown
$ cat Dockerfile
FROM xxx.xxx.xxx:5000/xxx/font_image
COPY Dockerfile /opt
RUN chmod -R 644 /opt
$ docker run --net host --rm -ti -v $PWD:/v -w /v -v ~/.docker:/root/.docker centos:7
$ KANIKO_DIR=/opt ./build Dockerfile xxx.xxx.xxx:5000/build-test/test
INFO[0000] Retrieving image manifest xxx.xxx.xxx:5000/xxx/xxxx_image
INFO[0000] Retrieving image xxx.xxx.xxx:5000/xxx/xxxx_imagefrom registry xxx.xxx.xxx:5000
INFO[0000] Built cross stage deps: map[]
INFO[0000] Retrieving image manifest xxx.xxx.xxx:5000/xxx/xxxx_image
INFO[0000] Returning cached image manifest
INFO[0000] Executing 0 build triggers
INFO[0000] Building stage 'xxx.xxx.xxx:5000/xxx/font_image' [idx: '0', base-idx: '-1']
INFO[0000] Unpacking rootfs as cmd COPY Dockerfile /opt requires it.
INFO[0000] COPY Dockerfile /opt
INFO[0000] Taking snapshot of files...
INFO[0000] RUN chmod -R 644 /opt
INFO[0000] Initializing snapshotter ...
INFO[0000] Taking snapshot of full filesystem...
INFO[0004] Cmd: /bin/sh
INFO[0004] Args: [-c chmod -R 644 /opt]
INFO[0004] Running: [/bin/sh -c chmod -R 644 /opt]
INFO[0004] Taking snapshot of full filesystem...
INFO[0005] No files were changed, appending empty layer to config. No layer added to image.
INFO[0005] Pushing image to xxx.xxx.xxx:5000/build-test/test
INFO[0006] Pushed xxx.xxx.xxx:5000/build-test/test@sha256:06efb31a8d750f4e34099091c905562d25aa180ff6d0845179acc633e51f01f7
|
退出到宿主机上拉取 docker pull xxx.xxx.xxx:5000/build-test/test 能拉取
2024/04/17 发现，在容器内 golang 使用 kaniko 库构建镜像的时候，会把 FROM alpine 的 rootfs 解压到容器内覆盖，导致容器内 rootfs 乱了，轻则导致一些 so 问题，重则运行进程 core dump。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| $ KANIKO_DIR=/tmp/kaniko build Dockerfile 127.0.0.1:5000/build-test/test:test /tmp/kaniko/
Destinations: 127.0.0.1:5000/build-test/test:test
INFO[0010] Retrieving image manifest alpine
INFO[0010] Retrieving image alpine from registry index.docker.io
INFO[0012] Built cross stage deps: map[]
INFO[0012] Retrieving image manifest alpine
INFO[0012] Returning cached image manifest
INFO[0012] Executing 0 build triggers
INFO[0012] Building stage 'alpine' [idx: '0', base-idx: '-1']
INFO[0012] Unpacking rootfs as cmd COPY Dockerfile /opt requires it.
INFO[0015] COPY Dockerfile /opt
INFO[0015] Taking snapshot of files...
INFO[0015] RUN chmod -R 644 /opt
INFO[0015] Initializing snapshotter ...
INFO[0015] Taking snapshot of full filesystem...
ERRO[0019] Couldn't eval /usr/lib/libcrypto.so.3 with link /usr/lib/libcrypto.so.3
INFO[0019] Cmd: /bin/sh
INFO[0019] Args: [-c chmod -R 644 /opt]
INFO[0019] Running: [/bin/sh -c chmod -R 644 /opt]
INFO[0019] Taking snapshot of full filesystem...
ERRO[0022] Couldn't eval /usr/lib/libssl.so.3 with link /usr/lib/libssl.so.3
INFO[0025] Skipping push to container registry due to --no-push flag
|
例如上面是容器内是 ubuntu，FROM 的是 alpine，/usr/lib/libcrypto.so.3 是 alpine 的，并不是 ubuntu 的，构建镜像后有的，为啥之前没发现，是因为之前 FROM 的镜像是 busybox，基本只有 busybox sh，覆盖了容器内的也没多大问题。
官方的 kaniko-project/executor 实际是 scratch 空镜像加二进制和基础目录，并且是一次性运行，没有这种问题（但是看 issue 有人反馈过，在构建多个 FROM 的镜像会出现类似问题，哈哈哈）。
覆盖 issue：
并且发现 kaniko 并不能非 root 构建，调试过程中发现会例如前面的 FROM alpine 后 COPY ，COPY、ADD 和 RUN 都需要解开 rootfs 操作的，下面报错就是把 alpine 的层的 tar 内的目录读取后本地映射创建，获取目录的 uid，gid，在本地目录创建和 chown，相当于非 root 用户执行 mkdir bin后，chown 0:0 bin
1
2
3
4
5
6
7
8
9
10
11
| $ KANIKO_DIR=/tmp/kaniko build Dockerfile 127.0.0.1:5000/build-test/test:test
Destinations: 127.0.0.1:5000/build-test/test:test
INFO[0036] Retrieving image manifest alpine
INFO[0036] Retrieving image alpine from registry index.docker.io
INFO[0041] Built cross stage deps: map[]
INFO[0042] Retrieving image manifest alpine
INFO[0042] Returning cached image manifest
INFO[0042] Executing 0 build triggers
INFO[0044] Building stage 'alpine' [idx: '0', base-idx: '-1']
INFO[0066] Unpacking rootfs as cmd COPY Dockerfile /opt requires it.
构建镜像失败: error building stage: failed to get filesystem from image: chown /bin: operation not permitted
|
只推送镜像的代码
其实上面的 kaniko 底层也是使用 go-containerregistry 库推送 tar 的
参考 crane 的 push 命令代码：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
| package main
import (
"fmt"
"github.com/google/go-containerregistry/pkg/authn"
"github.com/google/go-containerregistry/pkg/name"
"github.com/google/go-containerregistry/pkg/v1/remote"
"github.com/google/go-containerregistry/pkg/v1/tarball"
"os"
)
func main() {
img, err := tarball.ImageFromPath(os.Args[1], nil)
if err != nil {
fmt.Printf("Error loading Docker image from tarball: %v\n", err)
return
}
h, _ := img.Digest()
fmt.Printf("local image Digest: %s\n", h.String())
repoName, err := name.ParseReference(os.Args[2], name.Insecure)
if err != nil {
fmt.Printf("Error parsing repository URL: %v\n", err)
return
}
err = remote.Write(
repoName,
img,
remote.WithAuthFromKeychain(authn.DefaultKeychain),
)
if err != nil {
fmt.Printf("Error push repository URL: %v\n", err)
return
}
remoteImg, err := remote.Image(repoName, remote.WithAuthFromKeychain(authn.DefaultKeychain))
if err != nil {
fmt.Printf("Error get: %v\n", err)
return
}
h, _ = remoteImg.Digest()
fmt.Printf("Image pushed successfully: %s\n", h.String())
}
|
编译后为 tarPush 镜像，测试：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| $ docker pull xxx.xxx.xxx:5000/test/test:test
Error response from daemon: manifest for xxx.xxx.xxx:5000/test/test:test not found: manifest unknown: manifest unknown
# 保存镜像
$ docker save centos:7 -o image.tar
# 容器里测试，没有挂载 docker.sock
$ docker run --rm -ti --net host -v ~/.docker:/root/.docker -v $PWD:/v -w /v centos:7
$ ./tarPush image.tar xxx.xxx.xxx:5000/test/test:test1
local image Digest: sha256:26ee6ac028a8c40861034deb292a6ce45cf713e9761250a32ef69b08c2abc67f
Image pushed successfully: sha256:26ee6ac028a8c40861034deb292a6ce45cf713e9761250a32ef69b08c2abc67f
# 退出去，宿主机上拉取
# docker pull xxx.xxx.xxx:5000/test/test:test
test: Pulling from test/test
Digest: sha256:26ee6ac028a8c40861034deb292a6ce45cf713e9761250a32ef69b08c2abc67f
Status: Downloaded newer image for xxx.xxx.xxx:5000/test/test:test
xxx.xxx.xxx:5000/test/test:test
|
我向 google/go-containerregistry 提交了 pr 支持 support gzip on tarball.ImageFromPath 离线镜像推送不支持 gzip 的，也就是：
1
| docker save alpine | gzip -> alpine.tar.gz
|
注意事项
KANIKO_DIR 为啥说要提前设置环境变量，是因为包下面的 import 级别设置了值，代码里 os.Setenv 是不行的，必须启动设置变量 KANIKO_DIR
1
2
3
4
5
6
7
| # https://github.com/GoogleContainerTools/kaniko/blob/v1.19.2/pkg/config/init.go#L29-L33
var KanikoDir = func() string {
if kd, ok := os.LookupEnv("KANIKO_DIR"); ok {
return kd
}
return constants.DefaultKanikoPath
}()
|
