qemu-user-static 下 cgo go build 卡住的一次解决

qemu cgo

字数统计: 1.1k阅读时长: 6 min

 2023/08/08 

记录 qemu-user-static 下 cgo go build 卡住的一次解决

由来

内部构建机器都是 amd64 架构，项目大多数都没用 cgo，用了的几个都是 sqlite 这种长期稳定的，编译 arm64 的是下面的步骤：

FROM xxx/golang:xxx 带 gcc-aarch64-linux-gnu 的 amd64 golang 镜像

COPY ...
RUN CC=aarch64-linux-gnu-gcc GOARCH=arm64 CGO_ENABLED=1 \
      go build xxxx

前不久有个 odbc 的 cgo 项目，这样报错：

1	/usr/lib/gcc-cross/aarch64-linux-gnu/7/../../../../aarch64-linux-gnu/bin/ld.gold: error: cannot find -lodbc

解决过程

上面报错是因为 amd64 的镜像里安装 unixodbc-dev 的是 amd64 架构的，sqlite 没问题应该是它单元测试覆盖率很高了，而且安卓是 arm64 ，头文件里很多差异性估计用 #define 做 if else 了所以没问题。

qemu go build hangs

FROM --platform=linux/arm64 xxx/golang:xxx
RUN apt install -y unixodbc-dev
COPY ...
RUN GOARCH=arm64 CGO_ENABLED=1 \
      go build xxxx

arm64 机器上编译是没问题的，但是我们构建机器都是 amd64 架构（centos 7，内核 419 版本）使用的利用 qemu user 模式和 binfmt_misc 构建其他架构的 docker 镜像，时不时卡在 go build 上

升级了 qemu 版本和 docker 版本，还是卡住，后面打算升级内核。

安装 arm64 lib

但是升级内核之前想了下，能不能在 amd64 里安装 arm64 的 odbc lib，然后传递 CGO_CFLAGS、CGO_LDFLAGS 指定 arm64 的 odbc path，最终搞出来了：

FROM --platform=linux/arm64 debian:buster-slim as deb
RUN set -eux; \
    if [  -e /etc/apt/sources.list ];then sed -ri 's/[a-zA-Z0-9.]+(debian.org|ubuntu.com)/mirrors.aliyun.com/g' /etc/apt/sources.list; fi; \
    apt update; \
    cd /tmp; \
    apt download \
      libodbc1 \
      libltdl7 \
      unixodbc-dev; \
    mkdir -p /test/usr/local/lib_arm64 /test/usr/local/include_arm64; \
    ls *.deb | while read deb;do dpkg-deb -x $deb /opt/; done; \
    find /opt; \
    find /opt/usr/lib/ -not -type d -exec cp -a {} /test/usr/local/lib_arm64 \; ; \
    find /opt/usr/include/ -not -type d -exec cp -a {} /test/usr/local/include_arm64 \; ; \
    ls -l /test/usr/local/lib_arm64 /test/usr/local/include_arm64

FROM scratch
COPY --from=deb /test /odbc

上面镜像构建成 xxxxx/file/odbc:v1

FROM xxxxx/file/odbc:v1 as odbc
FROM xxx/golang:xxx # apt install -y gcc-aarch64-linux-gnu 的 amd64 golang 镜像
COPY --from=odbc /odbc /

RUN CC=aarch64-linux-gnu-gcc GOARCH=arm64 CGO_ENABLED=1 \
        CGO_CFLAGS='-g -O2 -I/usr/local/include_arm64' \
      CGO_LDFLAGS='-g -O2 -L/usr/local/lib_arm64 -Wl,-rpath-link,/usr/local/lib_arm64 -lodbc' \
      go build xxx

-g -O2 是默认的值，主要是后面的参数

Dockerfile

2023/11/06 增加龙芯的

FROM --platform=linux/arm64 debian:buster-slim as deb
RUN set -eux; \
    if [  -e /etc/apt/sources.list ];then sed -ri 's/[a-zA-Z0-9.]+(debian.org|ubuntu.com)/mirrors.aliyun.com/g' /etc/apt/sources.list; fi; \
    apt update; \
    cd /tmp; \
    apt download \
      libodbc1 \
      libltdl7 \
      unixodbc-dev; \
    mkdir -p /test/usr/local/lib_arm64 /test/usr/local/include_arm64; \
    ls *.deb | while read deb;do dpkg-deb -x $deb /opt/; done; \
    find /opt; \
    find /opt/usr/lib/ -not -type d -exec cp -a {} /test/usr/local/lib_arm64 \; ; \
    find /opt/usr/include/ -not -type d -exec cp -a {} /test/usr/local/include_arm64 \; ; \
    ls -l /test/usr/local/lib_arm64 /test/usr/local/include_arm64

FROM cr.loongnix.cn/library/debian:buster-slim as loong64_deb
RUN set -eux; \
    if [  -e /etc/apt/sources.list ];then sed -ri 's/[a-zA-Z0-9.]+(debian.org|ubuntu.com)/mirrors.aliyun.com/g' /etc/apt/sources.list; fi; \
    apt update; \
    cd /tmp; \
    apt download \
        libodbc1 \
        unixodbc-dev \
        odbcinst1debian2 \
        #libltdl-dev \
        libltdl7; \
    mkdir -p /test/usr/local/lib_loong64 /test/usr/local/include_loong64; \
    ls *.deb | while read deb;do dpkg-deb -x $deb /opt/; done; \
    find /opt; \
    find /opt/usr/lib/ -not -type d -exec cp -a {} /test/usr/local/lib_loong64 \; ; \
    find /opt/usr/include/ -not -type d -exec cp -a {} /test/usr/local/include_loong64 \; ; \
    ls -l /test/usr/local/lib_loong64 /test/usr/local/include_loong64

FROM scratch
COPY --from=deb /test /odbc
COPY --from=loong64_deb /test /odbc

龙芯使用：

1
2
3

CC=loongarch64-linux-gnu-gcc CGO_CFLAGS='-g -O2 -I/usr/local/include_loong64' \
    CGO_LDFLAGS='-g -O2 -L/usr/local/lib_loong64 -Wl,-rpath-link,/usr/local/lib_loong64 -lodbc' \
    GOARCH=loong64 CGO_ENABLED=1 go build -a -o /xxx cmd/xxx

不加额外的 -Wl,-rpath-link,/usr/local/lib_loong64 就报错 warning: libltdl.so.7, needed by /usr/local/lib_loong64/libodbc.so, not found (try using -rpath or -rpath-link) ：

/usr/local/go/pkg/tool/linux_amd64/link: running loongarch64-linux-gnu-gcc failed: exit status 1
/loongson-gnu-toolchain/bin/../lib/gcc/loongarch64-linux-gnu/8.3.0/../../../../loongarch64-linux-gnu/bin/ld: warning: libltdl.so.7, needed by /usr/local/lib_loong64/libodbc.so, not found (try using -rpath or -rpath-link)
/loongson-gnu-toolchain/bin/../lib/gcc/loongarch64-linux-gnu/8.3.0/../../../../loongarch64-linux-gnu/bin/ld: /usr/local/lib_loong64/libodbc.so: undefined reference to `lt_dlclose'
/loongson-gnu-toolchain/bin/../lib/gcc/loongarch64-linux-gnu/8.3.0/../../../../loongarch64-linux-gnu/bin/ld: /usr/local/lib_loong64/libodbc.so: undefined reference to `lt_dlerror'
/loongson-gnu-toolchain/bin/../lib/gcc/loongarch64-linux-gnu/8.3.0/../../../../loongarch64-linux-gnu/bin/ld: /usr/local/lib_loong64/libodbc.so: undefined reference to `lt_dlopen'
/loongson-gnu-toolchain/bin/../lib/gcc/loongarch64-linux-gnu/8.3.0/../../../../loongarch64-linux-gnu/bin/ld: /usr/local/lib_loong64/libodbc.so: undefined reference to `lt_dlinit'
/loongson-gnu-toolchain/bin/../lib/gcc/loongarch64-linux-gnu/8.3.0/../../../../loongarch64-linux-gnu/bin/ld: /usr/local/lib_loong64/libodbc.so: undefined reference to `lt_dlsetsearchpath'
/loongson-gnu-toolchain/bin/../lib/gcc/loongarch64-linux-gnu/8.3.0/../../../../loongarch64-linux-gnu/bin/ld: /usr/local/lib_loong64/libodbc.so: undefined reference to `lt_dlsym'
collect2: error: ld returned 1 exit status

原文作者：Zhangguanzhang

原文链接：http://zhangguanzhang.github.io/2023/08/08/qemu-cgo-build-hang/

发表日期：August 8th 2023, 6:07:30 pm

更新日期：August 8th 2023, 6:07:30 pm