由来 之前那篇 ipvs svc 的文章，内部已经上生产了，客户的环境可能完全内网，包管理安装 keepalived 不现实，所以 keepalived 是部署容器里的。在容灾测试的时候，例如 3 台机器部署好业务，然后跑压测脚本模拟用户使用，发现关台机器的时候故障时间很短，但是这个机器开机的期间，还是很大概率故障时间很长，体现在接口的错误数量很多。大概看了下，是 keepalived 启动慢，先试启动 docker daemon，然后容器启动是顺序不固定，可能 keepalived 很后起来，于是就想着看看能不能 keepalived 拿出来，也就是静态编译。
buildx 使用 见文章 buildx 使用
折腾 在官方仓库提了 issue is there any way to static build 后，和开发者沟通尝试过不少姿势都不行，然后有个大佬 hack 下编译成功了。开发者参照改了下后我试了下最新源码试可以整出来了。
主要是在 alpine 容器构建:
1 2 3 git clone https://github.com/acassen/keepalived.git cd keepaliveddocker run -v $PWD :/opt --workdir /opt --rm -ti alpine
构建依赖参考仓库里的 Dockerfile.in , static 库之类的可以 alpinelinux 上去搜索
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 if [ -f /etc/apk/repositories ];then sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories; fi && \ if [ -f /etc/apt/sources.list ];then sed -ri 's/(deb|security).debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list; fi && \ if [ ! -e /etc/nsswitch.conf ];then echo 'hosts: files dns myhostname' > /etc/nsswitch.conf; fi apk --no-cache add \ binutils \ file \ file-dev \ gcc \ glib \ glib-dev \ ipset \ ipset-dev \ iptables \ iptables-dev \ libmnl-dev \ libnftnl-dev \ libnl3 \ libnl3-dev \ make \ musl-dev \ net-snmp-dev \ openssl \ openssl-dev \ openssl-libs-static \ pcre2 \ pcre2-dev \ autoconf \ automake zlib-static alpine-sdk linux-headers libmnl-static
当然，你会发现没有 configure 脚本，可以参照 INSTALL 里执行 ./autogen.sh 生成，INSTALL 里可以参考下，有些包名在 os 上可能换了名字。同时如果要折腾的话，建议看下 --help 的内容。
1 2 ./autogen.sh ./configure --help
静态编译的配置：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 CFLAGS='-static -s' LDFLAGS=-static ./configure --disable-dynamic-linking \ --prefix=/usr \ --exec-prefix=/usr \ --bindir=/usr/bin \ --sbindir=/usr/sbin \ --sysconfdir=/etc \ --datadir=/usr/share \ --localstatedir=/var \ --mandir=/usr/share/man \ --enable-bfd \ --enable-snmp \ --enable-snmp-rfc \ --enable-nftables \ --enable-regex \ --enable-json --with-init=systemd --enable-vrrp --enable-libnl-dynamic
配置信息：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 Keepalived version : 2.2.7 Compiler : gcc gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027 Preprocessor flags : -D_GNU_SOURCE -DNETSNMP_NO_INLINE Compiler flags : -g -static -s -Wall -Wextra -Wunused -Wstrict-prototypes -Wabi -Wabsolute-value -Waddress-of-packed-member -Walloca -Walloc-zero -Warith-conversion -Warray-bounds=2 -Wattribute-alias=2 -Wbad-function-cast -Wc11-c2x-compat -Wcast-align -Wcast-qual -Wdate-time -Wdisabled-optimization -Wdouble-promotion -Wduplicated-branches -Wduplicated-cond -Wfloat-conversion -Wfloat-equal -Wformat-overflow -Wformat-security -Wformat-signedness -Wformat-truncation -Wframe-larger-than=5120 -Wimplicit-fallthrough=3 -Winit-self -Winline -Winvalid-pch -Wjump-misses-init -Wlogical-op -Wmissing-declarations -Wmissing-field-initializers -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wnormalized -Wnull-dereference -Wold-style-definition -Woverlength-strings -Wpointer-arith -Wredundant-decls -Wshadow -Wshift-overflow=2 -Wstack-protector -Wstrict-overflow=4 -Wstringop-overflow=2 -Wstringop-truncation -Wsuggest-attribute=cold -Wsuggest-attribute=const -Wsuggest-attribute=format -Wsuggest-attribute=malloc -Wsuggest-attribute=noreturn -Wsuggest-attribute=pure -Wsync-nand -Wtrampolines -Wundef -Wuninitialized -Wunknown-pragmas -Wunsafe-loop-optimizations -Wunsuffixed-float-constants -Wunused-const-variable=2 -Wvariadic-macros -Wwrite-strings -fPIE -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -O2 Linker flags : -static -pie -Wl,-z,relro -Wl,-z,now -L/usr/lib Extra Lib : -lm -lssl -lcrypto -lnftnl -lmnl -lpcre2-8 -lnetsnmpmibs -lnetsnmpagent -lnetsnmp -lcrypto Use IPVS Framework : Yes IPVS use libnl : No IPVS syncd attributes : Yes IPVS 64 bit stats : Yes HTTP_GET regex support : Yes fwmark socket support : Yes Use VRRP Framework : Yes Use VRRP VMAC : Yes Use VRRP authentication : Yes With track_process : Yes With linkbeat : Yes Use BFD Framework : Yes SNMP vrrp support : Yes SNMP checker support : Yes SNMP RFCv2 support : Yes SNMP RFCv3 support : Yes SNMP send V3 for V2 : Yes DBUS support : No Use JSON output : Yes libnl version : None Use IPv4 devconf : Yes Use iptables : No Use nftables : Yes init type : systemd systemd notify : No Strict config checks : No Build documentation : No Default runtime options : -D *** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.
libnl/libnl-3 这个我试了下加不进去，apk add libnl-dev 可以加上去，但是我看了 alpine 里 keepalived -v 的 configure 里也没我上面开的多。强开我试了 CPPFLAGS='-I/usr/include/libnl3' LDLIBS='-lnl3 -lnl-genl-3' 和下载 libnl-3 编译安装后都不行，想折腾和传递参数啥的话，多看看 configure 文件里的内容。
编译和安装：
信息：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 /opt # keepalived -v Keepalived v2.2.7 (02/23,2022), git commit v2.2.7-22-geb533a93 Copyright(C) 2001-2022 Alexandre Cassen, <acassen@gmail.com> Built with kernel headers for Linux 5.10.41 Running on Linux 5.4.0-99-generic #112-Ubuntu SMP Thu Feb 3 13:50:55 UTC 2022 Distro: Alpine Linux v3.15 configure options: --disable-dynamic-linking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/share/man --enable-bfd --enable-snmp --enable-snmp-rfc --enable-nftables --enable-regex --enable-json --with-init=systemd --enable-vrrp --enable-libnl-dynamic CFLAGS=-static -s LDFLAGS=-static Config options: NFTABLES LVS REGEX VRRP VRRP_AUTH VRRP_VMAC JSON BFD OLD_CHKSUM_COMPAT SNMP_V3_FOR_V2 SNMP_VRRP SNMP_CHECKER SNMP_RFCV2 SNMP_RFCV3 INIT=systemd System options: VSYSLOG MEMFD_CREATE IPV6_MULTICAST_ALL IPV4_DEVCONF RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_PROTOCOL FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA NET_LINUX_IF_H_COLLISION NETINET_LINUX_IF_ETHER_H_COLLISION IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS IPVS_TUN_TYPE IPVS_TUN_CSUM IPVS_TUN_GRE VRRP_IPVLAN IFLA_LINK_NETNSID INET6_ADDR_GEN_MODE VRF SO_MARK /opt # file `which keepalived` /usr/sbin/keepalived: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped /opt # ldd `which keepalived` /lib/ld-musl-x86_64.so.1: /usr/sbin/keepalived: Not a valid dynamic program
buildx 一步到位 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 FROM alpine as buildRUN if [ -f /etc/apk/repositories ];then sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories; fi && \ if [ -f /etc/apt/sources.list ];then sed -ri 's/(deb|security).debian.org/mirrors.aliyun.com/g' /etc/apt/sources.list; fi && \ if [ ! -e /etc/nsswitch.conf ];then echo 'hosts: files dns myhostname' > /etc/nsswitch.conf; fi && \ apk --no-cache add \ binutils \ file \ file-dev \ gcc \ glib \ glib-dev \ ipset \ ipset-dev \ iptables \ iptables-dev \ libmnl-dev \ libnftnl-dev \ libnl3 \ libnl3-dev \ make \ musl-dev \ net-snmp-dev \ openssl \ openssl-dev \ openssl-libs-static \ pcre2 \ pcre2-dev \ autoconf \ automake zlib-static alpine-sdk linux-headers libmnl-static git WORKDIR /opt RUN git clone https://github.com/acassen/keepalived.git RUN set -ex && \ cd /opt/keepalived && \ ./autogen.sh && \ CFLAGS='-static -s' LDFLAGS=-static ./configure --disable-dynamic-linking \ --prefix=/usr \ --exec-prefix=/usr \ --bindir=/usr/bin \ --sbindir=/usr/sbin \ --sysconfdir=/etc \ --datadir=/usr/share \ --localstatedir=/var \ --mandir=/usr/share/man \ --enable-bfd \ --enable-snmp \ --enable-snmp-rfc \ --enable-nftables \ --enable-regex \ --enable-json --with-init=systemd --enable-vrrp --enable-libnl-dynamic RUN set -ex && \ cd /opt/keepalived && \ make && \ make DESTDIR=/install_root install && \ find /install_root && \ rm -rf /install_root/usr/share FROM scratch AS binCOPY --from=build /install_root /
构建:
1 2 docker buildx build . --platform linux/amd64,linux/arm64 \ --target bin --output .
信息：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 $ ./usr/sbin/keepalived -v Keepalived v2.2.7 (02/23,2022), git commit v2.2.7-22-geb533a93 Copyright(C) 2001-2022 Alexandre Cassen, <acassen@gmail.com> Built with kernel headers for Linux 5.10.41 Running on Linux 5.4.0-99-generic #112-Ubuntu SMP Thu Feb 3 13:50:55 UTC 2022 Distro: Ubuntu 20.04.3 LTS configure options: --disable-dynamic-linking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --localstatedir=/var --mandir=/usr/share/man --enable-bfd --enable-snmp --enable-snmp-rfc --enable-nftables --enable-regex --enable-json --with-init=systemd --enable-vrrp --enable-libnl-dynamic CFLAGS=-static -s LDFLAGS=-static Config options: NFTABLES LVS REGEX VRRP VRRP_AUTH VRRP_VMAC JSON BFD OLD_CHKSUM_COMPAT SNMP_V3_FOR_V2 SNMP_VRRP SNMP_CHECKER SNMP_RFCV2 SNMP_RFCV3 INIT=systemd System options: VSYSLOG MEMFD_CREATE IPV6_MULTICAST_ALL IPV4_DEVCONF RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_PROTOCOL FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA NET_LINUX_IF_H_COLLISION NETINET_LINUX_IF_ETHER_H_COLLISION IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS IPVS_TUN_TYPE IPVS_TUN_CSUM IPVS_TUN_GRE VRRP_IPVLAN IFLA_LINK_NETNSID INET6_ADDR_GEN_MODE VRF SO_MARK $ ldd ./usr/sbin/keepalived not a dynamic executable $ file ./usr/sbin/keepalived ./usr/sbin/keepalived: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped $ ls -lh ./usr/sbin/keepalived -rwxr-xr-x 1 root root 4.4M Feb 24 17:56 ./usr/sbin/keepalived
试了下 arm64的也可以构建 --platform linux/amd64,linux/arm64 即可。
