docker 和 apparmor

linux docker apparmor

字数统计: 2.3k阅读时长: 12 min

 2024/01/08 

从搜索和源码探索 docker 如何处理 apparmor

由来

suse12sp5 上古系统记得之前接手还是 17.05 rpm 安装的，后续找了个 19.03.15 的 rpm 安装上的，之所以使用 rpm 是因为 docker static bin 安装起不来（忘记啥报错了来着），然后现在尝试了 24.0.5 的能起来，但是容器无法启动，发现和 apparmor 有关。apparmor、seccomp和 SELinux 都是用于增强 Linux 系统安全性的安全性模块。它们提供了对进程、应用程序和系统资源的额外访问控制层。

过程

报错信息

系统信息：

suse12sp5:~ # cat /etc/os-release 
NAME="SLES"
VERSION="12-SP5"
VERSION_ID="12.5"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP5"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:12:sp5"
suse12sp5:~ # uname -a
Linux suse12sp5 4.12.14-120-default #1 SMP Thu Nov 7 16:39:09 UTC 2019 (fd9dc36) x86_64 x86_64 x86_64 GNU/Linux
suse12sp5:~ # apparmor_parser --version
AppArmor parser version 2.8.2
Copyright (C) 1999-2008 Novell Inc.
Copyright 2009-2012 Canonical Ltd.
suse12sp5:~ # rpm -qa | grep libsecc
libseccomp2-32bit-2.3.1-10.1.x86_64
libseccomp2-2.3.1-10.1.x86_64

24.0.5 的 docker static bin 安装好后导入镜像，无法起容器，报错：

Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded: running `/sbin/apparmor_parser apparmor_parser -Kr /data/kube/docker/tmp/docker-default1881723382` failed with output: AppArmor parser error for /data/kube/docker/tmp/docker-default1881723382 in /data/kube/docker/tmp/docker-default1881723382 at line 16: syntax error, unexpected TOK_OPENPAREN, expecting TOK_MODE

代码里搜报错 AppArmor enabled on system but the 搜到 ensureDefaultAppArmorProfile 方法里：

// https://github.com/moby/moby/blob/v24.0.5/daemon/apparmor_default.go#L27
func ensureDefaultAppArmorProfile() error {
	if apparmor.HostSupports() {
		loaded, err := aaprofile.IsLoaded(defaultAppArmorProfile)
		if err != nil {
			return fmt.Errorf("Could not check if %s AppArmor profile was loaded: %s", defaultAppArmorProfile, err)
		}

		// Nothing to do.
		if loaded {
			return nil
		}

		// Load the profile.
		if err := aaprofile.InstallDefault(defaultAppArmorProfile); err != nil {
			return fmt.Errorf("AppArmor enabled on system but the %s profile could not be loaded: %s", defaultAppArmorProfile, err)
		}
	}

	return nil
}

apparmor.HostSupports 是 os.Getenv("container") == "" 且 /sys/module/apparmor/parameters/enabled 里是 Y 且存在命令 /sbin/apparmor_parser 则满足
aaprofile.IsLoaded 是等同于 grep 'docker-default ' /sys/kernel/security/apparmor/profiles 判断 apparmor 策略 docker-default 加载没，没加载则加载

docker 的 apparmor 生成和加载

然后就是 profiles/apparmor/apparmor.go 里的 generateDefault 和 InstallDefault:

docker daemon 进程读取 /proc/self/attr/current 为空则策略名为 unconfined
创建 os.CreateTemp 临时文件，并用模板 profiles/apparmor/template.go 生成到临时文件里
使用 /sbin/apparmor_parser -Kr <tmp_file> 加载策略
无论加载成功还是失败，最后会把这个临时策略文件删掉

解决过程

上面报错就是策略文件内容有问题，但是 docker daemon 会把策略文件删掉，最开始 clone 源码修改不删掉临时文件后 make 替换，但是后面发现了个简单粗暴套路：

cp /sbin/apparmor_parser /sbin/apparmor_parser_REAL
cat > /sbin/apparmor_parser << 'EOF'
#!/bin/bash

echo "$*" >> /root/apparmor_parser_arg_log

if [ $# -eq 2 ]; then
    cat $2 > /root/apparmor_parser_profile_log
fi

/sbin/apparmor_parser_REAL $*
EOF

然后 docker start 后 /root/apparmor_parser_profile_log 就生成内容了：



#include <tunables/global>


profile docker-default flags=(attach_disconnected,mediate_deleted) {

  #include <abstractions/base>


  network,
  capability,
  file,
  umount,
  # Host (privileged) processes may send signals to container processes.
  signal (receive) peer=unconfined,
  # dockerd may send signals to container processes (for "docker kill").
  signal (receive) peer=unconfined
,
  # Container processes may send signals amongst themselves.
  signal (send,receive) peer=docker-default,

  deny @{PROC}/* w,   # deny write for all files directly in /proc (not in a subdir)
  # deny write to files not in /proc/<number>/** or /proc/sys/**
  deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9/]*}/** w,
  deny @{PROC}/sys/[^k]** w,  # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
  deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,  # deny everything except shm* in /proc/sys/kernel/
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/kcore rwklx,

  deny mount,

  deny /sys/[^f]*/** wklx,
  deny /sys/f[^s]*/** wklx,
  deny /sys/fs/[^c]*/** wklx,
  deny /sys/fs/c[^g]*/** wklx,
  deny /sys/fs/cg[^r]*/** wklx,
  deny /sys/firmware/** rwklx,
  deny /sys/kernel/security/** rwklx,

  # suppress ptrace denials when using 'docker ps' or using 'ps' inside a container
  ptrace (trace,read,tracedby,readby) peer=docker-default,
}

执行 /sbin/apparmor_parser -Kr /root/apparmor_parser_profile_log 和上面一样的报错，尝试后发现去掉 signal 和 ptrace 行才可以，然后在 rpm 19.03 环境上面这样 hack 后拿到的策略文件也是没有 signal 和 ptrace 行的，删掉这俩类型后加载就只报警告：

# -KR 卸载
$ apparmor_parser -Kr /root/apparmor_parser_profile_log
Warning from test (test line 38): profile docker-default mount rules not enforced
$ aa-status
...
   docker-default (xxxx) 
   docker-default (xxxx) 
   docker-default (xxxx)
   ...

容器也能启动了，固化就把这个文件存放到 /etc/apparmor.d/docker-default ，这样提前加载后，docker daemon 判断加载了就不加载它的内置模板了。后续加了下下面几行：

1	@{PROC}/sys/kernel/ r,

apparmor 和一些其他信息

规则文件解释

#include 后面不是就绝对路径则是 /etc/apparmor.d/ 内的，提供一些变量命名和目录权限
profile <name|bin-path> flags=(xx,xxx,xxxx) { 名字（供外部 docker/k8s 使用该策略）或者二进制路径限制，flags 后面可以逗号或者空格分隔
- attach_disconnected 允许连接端口的进程
- mediate_deleted 允许进程访问已被删除的文件
- complain 默认是 enforce 模式，操作会被拒绝，complain 则会记录日志，kill 则 deny 的时候 kill 掉，查看进程模式可以通过 cat /proc/<pid>/attr/current 可以查看进程的策略名和模式，例如 dodcker-default enforce
还有其他属性就不介绍了，例如文件，网络，mount 啥的，自行搜下文档

开启打印后，可以系统日志里看到信息

1 2	echo 0 > /proc/sys/kernel/printk_ratelimit echo -n "all" > /sys/module/apparmor/parameters/audit

docker 示例

搜相关 issue 的时候发现，之前的模板里有判断 apparmor_parser 的版本，后面又去掉了，是因为有人提交了

$ ls -l contrib/apparmor/
total 12
-rw-r--r-- 1 root root  916 Jan  8 21:25 main.go
-rw-r--r-- 1 root root 5357 Jan  8 21:24 template.go

看了下代码，可以生成参考的策略内容：

$ go run contrib/apparmor/*.go 111
$ cat 111
@{DOCKER_GRAPH_PATH}=/var/lib/docker

profile /usr/bin/docker (attach_disconnected, complain) {
  # Prevent following links to these files during container setup.
  deny /etc/** mkl,
  deny /dev/** kl,
  deny /sys/** mkl,
  deny /proc/** mkl,

  mount -> @{DOCKER_GRAPH_PATH}/**,
  mount -> /,
  mount -> /proc/**,
  mount -> /sys/**,
  mount -> /run/docker/netns/**,
  mount -> /.pivot_root[0-9]*/,

  / r,

  umount,
  pivot_root,
  signal (receive) peer=@{profile_name},
  signal (receive) peer=unconfined,
  signal (send),
  network,
  capability,
  owner /** rw,
  @{DOCKER_GRAPH_PATH}/** rwl,
  @{DOCKER_GRAPH_PATH}/network/files/boltdb.db k,
  @{DOCKER_GRAPH_PATH}/network/files/local-kv.db k,
  # For user namespaces:
  @{DOCKER_GRAPH_PATH}/[0-9]*.[0-9]*/network/files/boltdb.db k,
  @{DOCKER_GRAPH_PATH}/[0-9]*.[0-9]*/network/files/local-kv.db k,

  # For non-root client use:
  /dev/urandom r,
  /dev/null rw,
  /dev/pts/[0-9]* rw,
  /run/docker.sock rw,
  /proc/** r,
  /proc/[0-9]*/attr/exec w,
  /sys/kernel/mm/hugepages/ r,
  /etc/localtime r,
  /etc/ld.so.cache r,
  /etc/passwd r,

  ptrace peer=@{profile_name},
  ptrace (read) peer=docker-default,
  deny ptrace (trace) peer=docker-default,
  deny ptrace peer=/usr/bin/docker///bin/ps,

  /usr/lib/** rm,
  /lib/** rm,

  /usr/bin/docker pix,
  /sbin/xtables-multi rCx,
  /sbin/iptables rCx,
  /sbin/modprobe rCx,
  /sbin/auplink rCx,
  /sbin/mke2fs rCx,
  /sbin/tune2fs rCx,
  /sbin/blkid rCx,
  /bin/kmod rCx,
  /usr/bin/xz rCx,
  /bin/ps rCx,
  /bin/tar rCx,
  /bin/cat rCx,
  /sbin/zfs rCx,
  /sbin/apparmor_parser rCx,

  # Transitions
  change_profile -> docker-*,
  change_profile -> unconfined,

  profile /bin/cat (complain) {
    /etc/ld.so.cache r,
    /lib/** rm,
    /dev/null rw,
    /proc r,
    /bin/cat mr,

    # For reading in 'docker stats':
    /proc/[0-9]*/net/dev r,
  }
  profile /bin/ps (complain) {
    /etc/ld.so.cache r,
    /etc/localtime r,
    /etc/passwd r,
    /etc/nsswitch.conf r,
    /lib/** rm,
    /proc/[0-9]*/** r,
    /dev/null rw,
    /bin/ps mr,

    # We don't need ptrace so we'll deny and ignore the error.
    deny ptrace (read, trace),

    # Quiet dac_override denials
    deny capability dac_override,
    deny capability dac_read_search,
    deny capability sys_ptrace,

    /dev/tty r,
    /proc/stat r,
    /proc/cpuinfo r,
    /proc/meminfo r,
    /proc/uptime r,
    /sys/devices/system/cpu/online r,
    /proc/sys/kernel/pid_max r,
    /proc/ r,
    /proc/tty/drivers r,
  }
  profile /sbin/iptables (complain) {
    signal (receive) peer=/usr/bin/docker,
    capability net_admin,
  }
  profile /sbin/auplink flags=(attach_disconnected, complain) {
    signal (receive) peer=/usr/bin/docker,
    capability sys_admin,
    capability dac_override,

    @{DOCKER_GRAPH_PATH}/aufs/** rw,
    @{DOCKER_GRAPH_PATH}/tmp/** rw,
    # For user namespaces:
    @{DOCKER_GRAPH_PATH}/[0-9]*.[0-9]*/** rw,

    /sys/fs/aufs/** r,
    /lib/** rm,
    /apparmor/.null r,
    /dev/null rw,
    /etc/ld.so.cache r,
    /sbin/auplink rm,
    /proc/fs/aufs/** rw,
    /proc/[0-9]*/mounts rw,
  }
  profile /sbin/modprobe /bin/kmod (complain) {
    signal (receive) peer=/usr/bin/docker,
    capability sys_module,
    /etc/ld.so.cache r,
    /lib/** rm,
    /dev/null rw,
    /apparmor/.null rw,
    /sbin/modprobe rm,
    /bin/kmod rm,
    /proc/cmdline r,
    /sys/module/** r,
    /etc/modprobe.d{/,/**} r,
  }
  # xz works via pipes, so we do not need access to the filesystem.
  profile /usr/bin/xz (complain) {
    signal (receive) peer=/usr/bin/docker,
    /etc/ld.so.cache r,
    /lib/** rm,
    /usr/bin/xz rm,
    deny /proc/** rw,
    deny /sys/** rw,
  }
  profile /sbin/xtables-multi (attach_disconnected, complain) {
    /etc/ld.so.cache r,
    /lib/** rm,
    /sbin/xtables-multi rm,
    /apparmor/.null w,
    /dev/null rw,

    /proc r,

    capability net_raw,
    capability net_admin,
    network raw,
  }
  profile /sbin/zfs (attach_disconnected, complain) {
    file,
    capability,
  }
  profile /sbin/mke2fs (complain) {
    /sbin/mke2fs rm,

    /lib/** rm,

    /apparmor/.null w,

    /etc/ld.so.cache r,
    /etc/mke2fs.conf r,
    /etc/mtab r,

    /dev/dm-* rw,
    /dev/urandom r,
    /dev/null rw,

    /proc/swaps r,
    /proc/[0-9]*/mounts r,
  }
  profile /sbin/tune2fs (complain) {
    /sbin/tune2fs rm,

    /lib/** rm,

    /apparmor/.null w,

    /etc/blkid.conf r,
    /etc/mtab r,
    /etc/ld.so.cache r,

    /dev/null rw,
    /dev/.blkid.tab r,
    /dev/dm-* rw,

    /proc/swaps r,
    /proc/[0-9]*/mounts r,
  }
  profile /sbin/blkid (complain) {
    /sbin/blkid rm,

    /lib/** rm,
    /apparmor/.null w,

    /etc/ld.so.cache r,
    /etc/blkid.conf r,

    /dev/null rw,
    /dev/.blkid.tab rl,
    /dev/.blkid.tab* rwl,
    /dev/dm-* r,

    /sys/devices/virtual/block/** r,

    capability mknod,

    mount -> @{DOCKER_GRAPH_PATH}/**,
  }
  profile /sbin/apparmor_parser (complain) {
    /sbin/apparmor_parser rm,

    /lib/** rm,

    /etc/ld.so.cache r,
    /etc/apparmor/** r,
    /etc/apparmor.d/** r,
    /etc/apparmor.d/cache/** w,

    /dev/null rw,

    /sys/kernel/security/apparmor/** r,
    /sys/kernel/security/apparmor/.replace w,

    /proc/[0-9]*/mounts r,
    /proc/sys/kernel/osrelease r,
    /proc r,

    capability mac_admin,
  }
}