zhangguanzhang's Blog

利用 docker buildx 静态编译 nginx

字数统计: 2.7k阅读时长: 16 min
2022/01/26

由来

内部有需求需要静态编译 nginx,尝试了下,搞出来了。先是按照官方 nginx Dockerfile 的逻辑走不通,后面下载 nginx 官方源码编译才行。

buildx 使用

见文章 buildx 使用

nginx Dockerfile

先说下官方的失败尝试。先 clone 项目:

1
2
git clone https://github.com/nginxinc/docker-nginx.git
cd docker-nginx

分为 stablemainline。大概研究了下,发现 case "$apkArch" in x86_64|aarch64) 的情况是利用包管理直接安装的,其他的架构才是源码编译安装。改了下这个 case ,先用 arm64 的试下走源码编译。然后看下逻辑是下载源码进去 make all

1
2
3
4
5
curl -f -O https://hg.nginx.org/pkg-oss/archive/${NGINX_VERSION}-${PKG_RELEASE}.tar.gz
tar xzvf ${NGINX_VERSION}-${PKG_RELEASE}.tar.gz
&& cd pkg-oss-${NGINX_VERSION}-${PKG_RELEASE} \
&& cd alpine \
&& make all

而源码里 Makefile 的前面内容如下:

1
2
3
4
5
6
7
8
9
10
11
12
ifeq ($(MODULE_TARGET), plus)
APKBUILD_TEMPLATE= APKBUILD-plus-module.in
MODULE_SUFFIX= -plus
MODULE_SUMMARY_PREFIX=NGINX Plus
TARGET_VERSION=$(PLUS_VERSION)
MODULE_PACKAGE_PREFIX=nginx-plus-module
else
APKBUILD_TEMPLATE= APKBUILD-module.in
MODULE_SUMMARY_PREFIX=nginx
TARGET_VERSION=$(BASE_VERSION)
MODULE_PACKAGE_PREFIX=nginx-module
endif

因为不是构建 nginx-plus 所以我们看 cat APKBUILD-module.in 里找到了下面的:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
build() {
cd "$builddir"

_nproc=`getconf _NPROCESSORS_ONLN`
if [ $_nproc -gt 1 ]; then
_make_opts="-j$_nproc"
fi

%%MODULE_PREBUILD%%

cd "$builddir"

CFLAGS= %%MODULE_ENV%% ./configure %%BASE_CONFIGURE_ARGS%% %%MODULE_CONFIGURE_ARGS%% --with-cc-opt="$CFLAGS %%MODULE_CC_OPT_DEBUG%%" --with-ld-opt="$LDFLAGS %%MODULE_LD_OPT_DEBUG%%" --with-debug
make $_make_opts modules
for so in `find objs/ -maxdepth 1 -type f -name "*.so"`; do \
debugso=`echo ${so} | sed -e 's|\.so$|-debug.so|'` ; \
mv ${so} ${debugso} ; \
done
CFLAGS= %%MODULE_ENV%% ./configure %%BASE_CONFIGURE_ARGS%% %%MODULE_CONFIGURE_ARGS%% --with-cc-opt="$CFLAGS %%MODULE_CC_OPT%%" --with-ld-opt="$LDFLAGS %%MODULE_LD_OPT%%"
make $_make_opts modules
}

CFLAGSLDFLAGS 是支持环境变量传入的,所以改下 Dockerfile 加入下面的:

1
2
ARG CFLAGS
ARG LDFLAGS

然后开始构建:

1
2
3
docker buildx build -t zhangguanzhang/nginx:arm64-static . \
--platform linux/arm64 \
--build-arg=CFLAGS="-static -s" --build-arg=LDFLAGS=-static

然后失败,报错 ./configure: error: the invalid value in --with-ld-opt="-static"

1
2
3
4
5
6
7
8
9
#6 481.1 >>> nginx: Unpacking /tmp/tmp.OciAKf/pkg-oss-1.21.5-1/alpine/abuild-base/nginx-1.21.5.tar.gz...
#6 482.9 checking for OS
#6 482.9 + Linux 5.4.0-91-generic aarch64
#6 482.9 checking for C compiler ... found
#6 483.6 + using GNU C compiler
#6 483.7 + gcc version: 10.3.1 20211027 (Alpine 10.3.1_git20211027)
#6 483.7 checking for gcc -pipe switch ... found
#6 484.3 checking for --with-ld-opt="-static" ... not found
#6 484.3 ./configure: error: the invalid value in --with-ld-opt="-static"

然后在 Dockerfile 里 apk add 加了 glibc-static 还是一样报错。然后尝试下官方的

自己编译

手动步骤

1
docker run --rm -ti --name t1 test alpine

前置依赖

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
if [ -f /etc/apk/repositories ];then sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories; fi && \
if [ -f /etc/apt/sources.list ];then sed -ri 's/(deb|security).debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list; fi && \
if [ ! -e /etc/nsswitch.conf ];then echo 'hosts: files dns myhostname' > /etc/nsswitch.conf; fi

apk add --no-cache --virtual .build-deps \
gcc \
libc-dev \
make \
openssl-dev \
pcre2-dev \
zlib-dev \
openssl-libs-static zlib-static \
linux-headers \
libxslt-dev \
gd-dev \
geoip-dev \
perl-dev \
libedit-dev \
bash \
alpine-sdk \
findutils
bash

下载源码包

1
2
3
wget https://nginx.org/download/nginx-1.21.6.tar.gz
tar zxf nginx-1.21.6.tar.gz
cd nginx-1.21.6

找下官方的编译参数,下面是我 arm64 上:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
$ docker run --rm --entrypoint nginx nginx:alpine-perl -V
nginx version: nginx/1.21.6
built by gcc 10.3.1 20211027 (Alpine 10.3.1_git20211027)
built with OpenSSL 1.1.1l 24 Aug 2021
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --with-perl_modules_path=/usr/lib/perl5/vendor_perl --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-Os -fomit-frame-pointer -g' --with-ld-opt=-Wl,--as-needed,-O1,--sort-common
$ docker run --rm --entrypoint nginx nginx:alpine-perl -V |& grep -Po -- "--[a-z_-]+(=('[^']+'|\S+))?"
--prefix=/etc/nginx
--sbin-path=/usr/sbin/nginx
--modules-path=/usr/lib/nginx/modules
--conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log
--pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp
--with-perl_modules_path=/usr/lib/perl5/vendor_perl
--user=nginx
--group=nginx
--with-compat
--with-file-aio
--with-threads
--with-http_addition_module
--with-http_auth_request_module
--with-http_dav_module
--with-http_flv_module
--with-http_gunzip_module
--with-http_gzip_static_module
--with-http_mp
--with-http_random_index_module
--with-http_realip_module
--with-http_secure_link_module
--with-http_slice_module
--with-http_ssl_module
--with-http_stub_status_module
--with-http_sub_module
--with-http_v
--with-mail
--with-mail_ssl_module
--with-stream
--with-stream_realip_module
--with-stream_ssl_module
--with-stream_ssl_preread_module
--with-cc-opt='-Os -fomit-frame-pointer -g'
--with-ld-opt=-Wl,--as-needed,-O1,--sort-common

with-http_mpwith-http_v 似乎对应 --with-http_mp4_module--with-http_v2_module,更多编译参数参考官方文档 nginx-configure
改下 --with-cc-opt--with-ld-opt 后编译

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
./configure \
--prefix=/etc/nginx \
--sbin-path=/usr/sbin/nginx \
--modules-path=/usr/lib/nginx/modules \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--http-client-body-temp-path=/var/cache/nginx/client_temp \
--http-proxy-temp-path=/var/cache/nginx/proxy_temp \
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
--http-scgi-temp-path=/var/cache/nginx/scgi_temp \
--with-perl_modules_path=/usr/lib/perl5/vendor_perl \
--user=nginx \
--group=nginx \
--with-compat \
--with-file-aio \
--with-threads \
--with-http_addition_module \
--with-http_auth_request_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_mp4_module \
--with-http_random_index_module \
--with-http_realip_module \
--with-http_secure_link_module \
--with-http_slice_module \
--with-http_ssl_module \
--with-http_stub_status_module \
--with-http_sub_module \
--with-http_v2_module \
--with-mail \
--with-mail_ssl_module \
--with-stream \
--with-stream_realip_module \
--with-stream_ssl_module \
--with-stream_ssl_preread_module \
--with-cc-opt='-static -s' \
--with-ld-opt=-static

make

make install

Dockerfile 构建

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
FROM alpine AS build

ARG NGINX_VERSION=1.21.6
WORKDIR /opt
RUN if [ -f /etc/apk/repositories ];then sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories; fi && \
if [ -f /etc/apt/sources.list ];then sed -ri 's/(deb|security).debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list; fi && \
if [ ! -e /etc/nsswitch.conf ];then echo 'hosts: files dns myhostname' > /etc/nsswitch.conf; fi && \
apk add --no-cache --virtual .build-deps \
gcc \
libc-dev \
make \
openssl-dev \
pcre2-dev \
zlib-dev \
openssl-libs-static zlib-static \
linux-headers \
libxslt-dev \
gd-dev \
geoip-dev \
perl-dev \
libedit-dev \
bash \
alpine-sdk \
findutils && \
wget http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz && tar zxf nginx-${NGINX_VERSION}.tar.gz

RUN cd nginx-${NGINX_VERSION} && \
./configure \
--prefix=/etc/nginx \
--sbin-path=/usr/sbin/nginx \
--modules-path=/usr/lib/nginx/modules \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--http-client-body-temp-path=/var/cache/nginx/client_temp \
--http-proxy-temp-path=/var/cache/nginx/proxy_temp \
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
--http-scgi-temp-path=/var/cache/nginx/scgi_temp \
--with-perl_modules_path=/usr/lib/perl5/vendor_perl \
--user=nginx \
--group=nginx \
--with-compat \
--with-file-aio \
--with-threads \
--with-http_addition_module \
--with-http_auth_request_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_mp4_module \
--with-http_random_index_module \
--with-http_realip_module \
--with-http_secure_link_module \
--with-http_slice_module \
--with-http_ssl_module \
--with-http_stub_status_module \
--with-http_sub_module \
--with-http_v2_module \
--with-mail \
--with-mail_ssl_module \
--with-stream \
--with-stream_realip_module \
--with-stream_ssl_module \
--with-stream_ssl_preread_module \
--with-cc-opt='-static -s' \
--with-ld-opt=-static && \
make && \
make install && \
cp `which nginx` /nginx-$(cat /etc/apk/arch)

FROM scratch AS bin
COPY --from=build /nginx-* /

构建

1
2
3
docker buildx build  . --platform linux/amd64,linux/arm64 \
--target bin --output . \
--build-arg=NGINX_VERSION=1.21.6

结果:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# ll                          
total 4
drwxr-xr-x 4 root root 62 Jan 26 16:21 ./
drwxr-xr-x 6 root root 98 Jan 26 15:31 ../
-rw-r--r-- 1 root root 2718 Jan 26 16:05 Dockerfile
drwxr-xr-x 2 root root 26 Jan 26 16:21 linux_amd64/
drwxr-xr-x 2 root root 27 Jan 26 16:21 linux_arm64/
# ll linux_a*
linux_amd64:
total 20424
drwxr-xr-x 2 root root 26 Jan 26 16:21 ./
drwxr-xr-x 4 root root 62 Jan 26 16:21 ../
-rwxr-xr-x 1 root root 20910696 Jan 26 16:12 nginx-x86_64*

linux_arm64:
total 20444
drwxr-xr-x 2 root root 27 Jan 26 16:21 ./
drwxr-xr-x 4 root root 62 Jan 26 16:21 ../
-rwxr-xr-x 1 root root 20932656 Jan 26 16:21 nginx-aarch64*
# ./linux_amd64/nginx-x86_64 -V
nginx version: nginx/1.21.6
built by gcc 10.3.1 20211027 (Alpine 10.3.1_git20211027)
built with OpenSSL 1.1.1l 24 Aug 2021
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --with-perl_modules_path=/usr/lib/perl5/vendor_perl --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-static -s' --with-ld-opt=-static
# file ./linux_amd64/nginx-x86_64
./linux_amd64/nginx-x86_64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped
# file ./linux_arm64/nginx-aarch64
./linux_arm64/nginx-aarch64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, with debug_info, not stripped

符号链接可以 strip -s $(which nginx) 去掉减少大小。

对接到真实场景

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
FROM nginx:alpine AS conf
FROM alpine AS build

ARG NGINX_VERSION=1.21.6
WORKDIR /opt
RUN if [ -f /etc/apk/repositories ];then sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories; fi && \
if [ -f /etc/apt/sources.list ];then sed -ri 's/(deb|security).debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list; fi && \
if [ ! -e /etc/nsswitch.conf ];then echo 'hosts: files dns myhostname' > /etc/nsswitch.conf; fi && \
apk add --no-cache --virtual .build-deps \
gcc \
libc-dev \
make \
openssl-dev \
pcre2-dev \
zlib-dev \
openssl-libs-static zlib-static \
linux-headers \
libxslt-dev \
gd-dev \
geoip-dev \
perl-dev \
libedit-dev \
bash \
alpine-sdk \
findutils && \
wget http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz && tar zxf nginx-${NGINX_VERSION}.tar.gz

RUN cd nginx-${NGINX_VERSION} && \
./configure \
--prefix=/etc/nginx \
--sbin-path=/usr/sbin/nginx \
--modules-path=/usr/lib/nginx/modules \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--http-client-body-temp-path=/var/cache/nginx/client_temp \
--http-proxy-temp-path=/var/cache/nginx/proxy_temp \
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
--http-scgi-temp-path=/var/cache/nginx/scgi_temp \
--with-perl_modules_path=/usr/lib/perl5/vendor_perl \
--user=nginx \
--group=nginx \
--with-compat \
--with-file-aio \
--with-threads \
--with-http_addition_module \
--with-http_auth_request_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_mp4_module \
--with-http_random_index_module \
--with-http_realip_module \
--with-http_secure_link_module \
--with-http_slice_module \
--with-http_ssl_module \
--with-http_stub_status_module \
--with-http_sub_module \
--with-http_v2_module \
--with-mail \
--with-mail_ssl_module \
--with-stream \
--with-stream_realip_module \
--with-stream_ssl_module \
--with-stream_ssl_preread_module \
--with-cc-opt='-static -s' \
--with-ld-opt=-static

RUN cd nginx-${NGINX_VERSION} && \
mkdir /install_root && \
make && \
make DESTDIR=/install_root install && \
rm -f /install_root/etc/nginx/*.default && \
rmdir /install_root/var/run ; true && \
mkdir -p /install_root/var/cache/nginx/ \
/install_root/usr/lib/nginx/modules \
/install_root/etc/nginx/conf.d \
/install_root/usr/share/nginx && \
mv /install_root/etc/nginx/html /install_root/usr/share/nginx/ && \
ln -sf /dev/stdout /install_root/var/log/nginx/access.log && \
ln -sf /dev/stderr /install_root/var/log/nginx/error.log


COPY --from=conf /etc/nginx/nginx.conf /install_root/etc/nginx/nginx.conf
COPY --from=conf /etc/nginx/conf.d /install_root/etc/nginx/conf.d

RUN find /install_root

FROM alpine
COPY --from=build /install_root /
# alpine create nginx user/group
RUN addgroup -g 101 -S nginx && \
adduser -S -D -H -u 101 -h /var/cache/nginx -s /sbin/nologin -G nginx -g nginx nginx && \
if [ ! -e /etc/nsswitch.conf ];then echo 'hosts: files dns myhostname' >> /etc/nsswitch.conf; fi
# debian create nginx user/group
# RUN addgroup --system --gid 101 nginx && \
# adduser --system --disabled-login --ingroup nginx --no-create-home --home /nonexistent --gecos "nginx user" --shell /bin/false --uid 101 nginx && \

注意 worker_processes auto; 在高核心 cpu 上会非常吃配置和浪费配置,特别是 arm64 的国产服务器上,需要改成固定的数字。

后续一些额外模块测试

vts 模块验证不影响静态

1
2
3
4
5
6
7
    wget https://github.com/vozlt/nginx-module-vts/archive/refs/tags/v0.1.18.tar.gz && \
tar zxf v0.1.18.tar.gz -C /tmp/ && \
rm -rf v0.1.18.tar.gz

RUN cd nginx-${NGINX_VERSION} && \
...
--add-module=/tmp/nginx-module-vts-0.1.18 \

参考

CATALOG
  1. 1. 由来
    1. 1.1. buildx 使用
    2. 1.2. nginx Dockerfile
    3. 1.3. 自己编译
      1. 1.3.1. 手动步骤
      2. 1.3.2. Dockerfile 构建
      3. 1.3.3. 对接到真实场景
    4. 1.4. 后续一些额外模块测试
  2. 2. 参考