安装 NVSwitch GPU 服务器的 cuda 驱动版本、nvidia-docker 指南

TFATS 2023-08-30 原文

一，安装 Cuda 驱动

如果出现如下报错，则需安装 gcc、kernel-devel，请参考下面第二步安装 gcc、kernel-devel。

二，安装 gcc、kernel-devel

1，安装 gcc 和 kernel-devel

若直接执行如下命令安装，如果默认版本不一致，则会遇到如下图报错：

yum -y install gcc kernel-devel
./NVIDIA-Linux-x86_64-515.86.01.run

2，报错原因

使用如下命令查看内核版本是否一致

uname -r
rpm -q kernel-devel

正常结果应该是如下图所示，内核版一致，若不一致则通过下一步解决此问题。

3，解决内核版本不一致问题

卸载 kernel-devel，执行如下命令：

yum remove kernel-devel

4，安装版本匹配的 kernel-devel

1）查看内核版本号

uname -a

红框圈出来的则是内核版本号:

kernel-devel-3.10.0-1160.el7.x86_64.rpm

2）下载版本匹配的 kernel-devel

kernel-devel 下载地址： https://ftp.sjtu.edu.cn/sites/ftp.scientificlinux.org/linux/scientific/7.9/x86_64/os/Packages/
找到与内核版本号对应的 kernel-devel 版本： kernel-devel-3.10.0-1160.el7.x86_64.rpm

# 在终端 使用 wget 下载 kernel-devel 
wget https://ftp.sjtu.edu.cn/sites/ftp.scientificlinux.org/linux/scientific/7.9/x86_64/os/Packages/kernel-devel-3.10.0-1160.el7.x86_64.rpm

2）安装 kernel-devel

rpm -ivh kernel-devel-3.10.0-1160.el7.x86_64.rpm

3）查看是否成功安装内核开发环境包 kernel-devel

uname -a ; rpm -qa kernel\* | sort

若成功安装，如下图所示：

三，gcc和g++

yum install gcc
yum install gcc-c++

四，安装 Cuda 驱动

./NVIDIA-Linux-x86_64-515.86.01.run

五，安装 NVIDIA-Fabric Manager 软件包

NVIDIA NVSwitch A100 GPU卡，需额外安装与驱动版本对应的 nvidia-fabricmanager 服务使 GPU 卡间能够互联通过NVSwitch互联，如果仅安装NVIDIA GPU 驱动程序，会导致GPU不能正常使用。

1，安装 nvidia-fabricmanager 详解

安装 nvidia-fabricmanager 详解如下： https://www.volcengine.com/docs/6419/73634

GPU云服务器 -> 用户指南 -> 安装 NVIDIA 驱动 -> 安装NVIDIA-Fabric Manager软件包

2，整理 nvidia-fabricmanager 安装步骤

1) 查看系统驱动版本

nvidia-smi

2) 通过安装包安装

CentOS 8.x

wget https://developer.download.nvidia.cn/compute/cuda/repos/rhel8/x86_64/nvidia-fabric-manager-470.57.02-1.x86_64.rpm
rpm -ivh nvidia-fabric-manager-470.57.02-1.x86_64.rpm

CentOS 7.x

wget https://developer.download.nvidia.cn/compute/cuda/repos/rhel7/x86_64/nvidia-fabric-manager-470.57.02-1.x86_64.rpm
rpm -ivh nvidia-fabric-manager-470.57.02-1.x86_64.rpm

Ubuntu 20.04

wget https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2004/x86_64/nvidia-fabricmanager-470_470.57.02-1_amd64.deb
dpkg -i nvidia-fabricmanager-470_470.57.02-1_amd64.deb

Ubuntu 18.04

wget https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu1804/x86_64/nvidia-fabricmanager-470_470.57.02-1_amd64.deb
dpkg -i nvidia-fabricmanager-470_470.57.02-1_amd64.deb

Debain 10、veLinux 1.0

wget https://developer.download.nvidia.cn/compute/cuda/repos/debian10/x86_64/nvidia-fabricmanager-470_470.57.02-1_amd64.deb
dpkg -i nvidia-fabricmanager-470_470.57.02-1_amd64.deb

3) 通过源安装

CentOS 8.x

dnf config-manager --add-repo http://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
dnf module enable -y nvidia-driver:470
dnf install -y nvidia-fabric-manager-0:470.57.02-1

CentOS 7.x

yum -y install yum-utils 
yum-config-manager --add-repo https://developer.download.nvidia.cn/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo
yum install -y nvidia-fabric-manager-470.57.02-1

Ubuntu 20.04

wget https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
apt-key add 7fa2af80.pub
rm 7fa2af80.pub
echo "deb http://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2004/x86_64 /" | tee /etc/apt/sources.list.d/cuda.list
apt-get update
apt-get -y install nvidia-fabricmanager-470=470.57.02-1

Ubuntu 18.04

wget https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
apt-key add 7fa2af80.pub
rm 7fa2af80.pub
echo "deb http://developer.download.nvidia.cn/compute/cuda/repos/ubuntu1804/x86_64 /" | tee /etc/apt/sources.list.d/cuda.list
apt-get update
apt-get -y install nvidia-fabricmanager-470=470.57.02-1

4) 启动NVIDIA-Fabric Manager

# 1. 执行以下命令启动Fabric Manager服务。
sudo systemctl start nvidia-fabricmanager

# 2. 执行以下命令查看Fabric Manager服务是否正常启动，回显active（running）表示启动成功。
sudo systemctl status nvidia-fabricmanager

# 3. 执行以下命令配置Fabric Manager服务随实例开机自启动。
sudo systemctl enable nvidia-fabricmanager

六，安装 nvidia-docker

1, 安装 nvidia-docker 详解

安装 nvidia-docker 详解： https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker

2, 总结安装 nvidia-docker

1) 安装在 Ubuntu and Debian

以下步骤可用于在 Ubuntu LTS（18.04、20.04 和 22.04）和 Debian（Stretch、Buster）发行版上设置
NVIDIA Container Toolkit。

设置 Docker-CE

curl https://get.docker.com | sh && sudo systemctl --now enable docker

设置包存储库和 GPG 密钥：

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
      && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

要访问experimental功能和发布候选版本，您可能需要将experimental分支添加到存储库列表中：

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
      && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/experimental/$distribution/libnvidia-container.list | \
         sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
         sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

nvidia-docker2更新包列表后安装包（和依赖项）

sudo apt-get update
sudo apt-get install -y nvidia-docker2

设置默认运行时后重启Docker守护进程完成安装：

sudo systemctl restart docker

此时，可以通过运行基本 CUDA 容器来测试工作设置：

sudo docker run --rm --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi

这应该会产生如下所示的控制台输出：

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   34C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

2) 安装在 CentOS 7/8

以下步骤可用于在 CentOS 7/8 上设置 NVIDIA Container Toolkit。

设置官方 Docker CE 存储库：

CentOS 8.x

sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo

CentOS 7.x

sudo yum-config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo

现在您可以观察docker-ce存储库中可用的包：
- CentOS 8.x
```
sudo dnf repolist -v
```
- CentOS 7.x
```
sudo yum repolist -v
```

由于 CentOS 不支持containerd.io较新版本的 Docker-CE
所需的特定版本的软件包，因此一种选择是手动安装软件包containerd.io，然后继续安装docker-ce 软件包。

安装containerd.io包：

CentOS 8.x

sudo dnf install -y https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.4.3-3.1.el7.x86_64.rpm

CentOS 7.x

sudo yum install -y https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.4.3-3.1.el7.x86_64.rpm

现在安装最新的docker-ce软件包：

CentOS 8.x

sudo dnf install docker-ce -y

CentOS 7.x

sudo yum install docker-ce -y

使用以下命令确保 Docker 服务正在运行：
```
sudo systemctl --now enable docker
```
最后，通过运行hello-world容器来测试你的 Docker 安装：
```
sudo docker run --rm hello-world
```

这应该会产生如下所示的控制台输出：

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete
Digest: sha256:7f0a9f93b4aa3022c3a4c147a449bf11e0941a1fd0bf4a8e6c9408b2600777c5
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
   (amd64)
3. The Docker daemon created a new container from that image which runs the
   executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
   to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/

For more examples and ideas, visit:
https://docs.docker.com/get-started/

设置存储库和 GPG 密钥：

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
   && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

要访问experimental功能和发布候选版本，您可能需要将experimental分支添加到存储库列表中：
```
yum-config-manager --enable libnvidia-container-experimental
```

nvidia-docker2更新包列表后安装包（和依赖项）：

CentOS 8.x

sudo dnf clean expire-cache --refresh

CentOS 7.x

sudo yum clean expire-cache

CentOS 8.x

sudo dnf install -y nvidia-docker2

CentOS 7.x

sudo yum install -y nvidia-docker2

设置默认运行时后重启Docker守护进程完成安装：
```
sudo systemctl restart docker
```

此时，可以通过运行基本 CUDA 容器来测试工作设置：

sudo docker run --rm --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi