在开始之前，我对是否应该在 SuperUser 或 Stackoverflow 中提出这个问题有两种想法 - 如果它位于不正确的位置，请提前道歉。

我有一个执行音频/视频处理的 docker 容器(包含 C/C++ 可执行代码)。因此，我想测试使用 RT 调度约束运行容器的好处。在网上搜索时，我发现了各种各样的信息，但我很难将所有部分放在一起。

系统环境:

• 主机:Ubuntu(库存)Zesty 17.04(否 RT 内核补丁，内核:4.10.0-35-genric)
• Docker 版本:17.05.0-ce
• Docker 镜像操作系统:Ubuntu Zesty 17.04。

在嵌套在 docker 镜像/容器中的可执行文件中，执行以下代码以将调度程序从“SCHED_OTHER”更改为“SCHED_FIFO”(参见 docs):

    struct sched_param sched = {};

    const int nMin = sched_get_priority_min(SCHED_FIFO);
    const int nMax = sched_get_priority_max(SCHED_FIFO);

    const int nHlf = (nMax - nMin) / 2;
    const int nPriority = nMin + nHlf + 1;

    sched.sched_priority = boost::algorithm::clamp(nPriority, nMin, nMax);

    if (sched_setscheduler(0, SCHED_FIFO, &sched) < 0)
        std::cerr << "SETSCHEDULER failed - err = " << strerror(errno) << std::endl;
    else
        std::cout << "Priority set to \"" << sched.sched_priority << "\"" << std::endl;

我一直在阅读有关使用实时调度程序的各种 Docker 文档。一个有趣的page国家，

Verify that CONFIG_RT_GROUP_SCHED is enabled in the Linux kernel by running zcat /proc/config.gz | grep CONFIG_RT_GROUP_SCHED or by checking for the existence of the file /sys/fs/cgroup/cpu.rt_runtime_us. For guidance on configuring the kernel realtime scheduler, consult the documentation for your operating system.

根据上述建议，现有的 Ubuntu Zesty 17.04 操作系统似乎未能通过这些检查。

第一个问题:我不能使用 RT 调度程序吗？什么是“CONFIG_RT_GROUP_SCHED”？让我感到困惑的一件事是，从 2010 年到 2012 年，网络上有一些关于使用 RT 补丁修补内核的旧帖子。似乎从那时起，Linux 内核中就出现了一些与软 RT 相关的工作。

报价 here引发了我的问题:

From kernel version 2.6.18 onward, however, Linux is gradually becoming equipped with real-time capabilities, most of which are derived from the former realtime-preempt patches developed by Ingo Molnar, Thomas Gleixner, Steven Rostedt, and others. Until the patches have been completely merged into the mainline kernel (this is expected to be around kernel version 2.6.30), they must be installed to achieve the best real-time performance. These patches are named:

继续……

阅读了更多信息后，我注意到设置 ulimit 很重要。我已经修改了/etc/security/limits.conf:

#*               soft    core            0
#root            hard    core            100000
#*               hard    rss             10000

# NEW ADDITION
gavin            hard    rtprio          99

第二个问题:想必以上是需要让docker daemon运行RT的吧？看起来好像守护进程是通过 systemd 控制的。

我继续调查，在同一个 Docker 文档页面上看到了以下片段:

To run containers using the realtime scheduler, run the Docker daemon with the --cpu-rt-runtime flag set to the maximum number of microseconds reserved for realtime tasks per runtime period. For instance, with the default period of 10000 microseconds (1 second), setting --cpu-rt-runtime=95000 ensures that containers using the realtime scheduler can run for 95000 microseconds for every 10000-microsecond period, leaving at least 5000 microseconds available for non-realtime tasks. To make this configuration permanent on systems which use systemd, see Control and configure Docker with systemd.

关注 this page ，我发现守护进程有两个有趣的参数:

  --cpu-rt-period int                     Limit the CPU real-time period in microseconds
  --cpu-rt-runtime int                    Limit the CPU real-time runtime in microseconds

同一页面表示可以通过'/etc/docker/daemon.json'指定docker daemon参数，所以我试了一下:

{
    "cpu-rt-period": 92500,
    "cpu-rt-runtime": 100000
}

注意:文档没有将上述选项指定为“Linux 上允许的配置选项”。我想我还是会试一试。

重启后的 Docker 守护进程输出:

-- Logs begin at Wed 2017-10-04 09:58:38 BST, end at Wed 2017-10-04 10:01:32 BST. --
Oct 04 09:58:47 gavin systemd[1]: Starting Docker Application Container Engine...
Oct 04 09:58:47 gavin dockerd[1501]: time="2017-10-04T09:58:47.885882588+01:00" level=info msg="libcontainerd: new containerd process, pid: 1531"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.053986072+01:00" level=warning msg="failed to rename /var/lib/docker/tmp for background deletion: %!s(<nil>).
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.161303803+01:00" level=info msg="[graphdriver] using prior storage driver: aufs"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.303409053+01:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.304002725+01:00" level=warning msg="Your kernel does not support swap memory limit"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.304078792+01:00" level=warning msg="Your kernel does not support cgroup rt period"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.304201239+01:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.305534113+01:00" level=info msg="Loading containers: start."
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.730193030+01:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemo
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.784938130+01:00" level=info msg="Loading containers: done."
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.888035017+01:00" level=info msg="Daemon has completed initialization"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.888104120+01:00" level=info msg="Docker daemon" commit=89658be graphdriver=aufs version=17.05.0-ce
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.903280645+01:00" level=info msg="API listen on /var/run/docker.sock"
Oct 04 09:58:48 gavin systemd[1]: Started Docker Application Container Engine.

感兴趣的特定行:

Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.304078792+01:00" level=warning msg="Your kernel does not support cgroup rt period"
Oct 04 09:58:48 gavin dockerd[1501]: time="2017-10-04T09:58:48.304201239+01:00" level=warning msg="Your kernel does not support cgroup rt runtime"

鉴于我之前的发现，这并不奇怪。

最后一个问题:当这最终工作时，我将如何确定我的容器是否真正在 RT 调度下运行？ 'top' 之类的就足够了吗？

编辑:我运行了内核诊断 script我found通过 github 上的 moby。这是输出:

warning: /proc/config.gz does not exist, searching other paths for kernel config ...
info: reading kernel config from /boot/config-4.10.0-35-generic ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: missing
    (cgroup swap accounting is currently not enabled, you can enable it by setting boot option "swapaccount=1")
- CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled (as module)
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled (as module)
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module)
  - "ipvlan":
    - CONFIG_IPVLAN: enabled (as module)
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: enabled (as module)
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled (as module)
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

意义线:

- CONFIG_RT_GROUP_SCHED: missing

最佳答案

容器级别

在容器内进行 RT 调度有两种选择:

1. 添加 SYS_NICE 功能

docker run --cap-add SYS_NICE ...

1. 使用带有 --privileged 标志的特权模式

docker run --privileged ...

注意:--privileged 标志授予比必要更多的权限!

更有限的 --cap-add SYS_NICE 选项更安全。

操作系统系统配置

您可能还必须在 sysctl 中启用实时调度。 如果您以 root 用户身份运行(Docker 容器默认):

sysctl -w kernel.sched_rt_runtime_us=-1

要使其永久化(更新您的图像):

echo 'kernel.sched_rt_runtime_us=-1' >> /etc/sysctl.conf

https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities

关于c++ - 使用 Ubuntu (Linux) 主机的 Docker 守护进程/容器实时调度，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/46563332/