Skip to main content

Deploy the NVIDIA GPU Operator on CCE

The NVIDIA GPU Operator is a critical tool for effectively managing GPU resources in Kubernetes clusters. It serves as an abstraction layer over Kubernetes APIs, automating tasks such as dynamic provisioning, driver updates, resource allocation, and optimization for GPU-intensive workloads, thereby simplifying the deployment and management of GPU-accelerated applications. Its functionality extends to dynamic provisioning of GPUs on demand, managing driver updates, optimizing resource allocation for varied workloads, and integrating with monitoring tools for comprehensive insights into GPU usage and health. This guide outlines how to deploy the NVIDIA GPU Operator on CCE cluster. The process involves preparing GPU nodes, installing necessary components, configuring the cluster for GPU support, deploying an application leveraging GPUs, and verifying functionality.

Prerequisites​

This blueprint requires:

  • Access to the CCE cluster with kubectl.
  • Helm installed on your system.

Preparing & Configuring a GPU Node Pool​

Go to the Open Telekom Cloud console and choose the specific cluster you want to add the GPU node pool to. At the left sidebar select Nodes and click Create Node Pool.

Node Pool Configuration​

Use the following values to configure the newly created GPU Node Pool:

  • Name: Assign a meaningful name to your GPU node pool, such as gpu-workers.
  • Flavor Selection: Choose a flavor that includes GPU resources. Look for options like pi2.2xlarge or similar GPU-accelerated instances available.
  • Annotations: If required by your cluster's configuration, add any necessary annotations.
  • Taints or Tolerations: Set taints or tolerations to manage pod scheduling. For GPU nodes, you might set a taint like nvidia.com/gpu=true:NoExecute and ensure pods requiring GPUs have the appropriate toleration.

image

After creating the Node Pool scale it to the desired size.

Verification​

Wait for some minutes until the nodes get provisioned and check if they have successfully joined the cluster with the following command:

kubectl get nodes --show-labels | grep "nvidia"
info

New GPU nodes should contain a label with accelerator as key and nvidia* as value (e.g. accelerator=nvidia-t4).

Installing the NVIDIA GPU Plugin​

Installation​

From sidebar select Add-ons and install the CCE AI Suite (NVIDIA GPU).

image

Plugin Configuration​

For more information see CCE AI Suite (NVIDIA GPU). image

caution

The selected driver must be compatible with the GPU nodes and supported by NVIDIA GPU Operator, otherwise the cluster will not be able to allocate GPU resources. Check supported drivers at Platform Support.

Deploying the NVIDIA GPU Operator via Helm​

Create a values.yaml file to include the required Helm Chart configuration values:

values.yaml
  hostPaths:
    driverInstallDir: "/usr/local/nvidia/"

  driver:
    enabled: false

  toolkit:
    enabled: false
important
  • hostPaths.driverInstallDir: The driver installation directory on CCE is different. Do not change this value!
  • driver.enabled: Driver installation is disabled because it's already installed via CCE AI Suite.
  • toolkit.enabled: The container toolkit installation is disabled because it's already installed via CCE AI Suite.

Now deploy the operator via helm:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install --wait gpu-operator \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator \
  -f values.yaml \
  --version=v24.9.2

Deploying an application with GPU Support​

  1. Create a Pod Manifest: For example, deploying a CUDA job.
cuda-example.yaml
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  containers:
  - name: cuda-vectoradd
    image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubuntu20.04"
    resources:
      limits:
        nvidia.com/gpu: 1
  1. Apply the Manifest:
kubectl apply -f cuda-example.yaml

Validation​

  1. Check Pod Status: Ensure pods are running.
kubectl get pods -n default
  1. Verify Logs: Check logs for GPU activity.
kubectl logs -f cuda-example-<pod-name> -n default

The containers' logs should indicate that the operation was succesfull, e.g.:

[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
tip

If you are looking for more sample workloads visit NVIDIA GPU Operator Verification: Running Sample GPU Applications.

Troubleshooting Tips​

Verifying NVIDIA Drivers are Installed on Nodes​

Ensuring that the GPU nodes have the correct NVIDIA drivers is a critical first step. SSH into one of your GPU nodes and run:

# If the add-on version is earlier than 2.0.0, run the following command:
cd /opt/cloud/cce/nvidia/bin && ./nvidia-smi

# If the add-on version is 2.0.0 or later and the driver installation path is changed, run the following command:
cd /usr/local/nvidia/bin && ./nvidia-smi

or directly on the Container:

cd /usr/local/nvidia/bin && ./nvidia-smi

This command should display details such as the driver version, GPU utilization, and any active processes. If it fails or shows an outdated driver, this indicates that the node isn’t properly set up.

You can find more information here.

Verifying Driver Compatibility​

If drivers are missing or incompatible, verify that the CCE AI Suite is correctly installed and configured. Reinstalling or updating the suite might be necessary if the drivers aren’t correctly deployed. Follow the instructions.

Additionally, run the following command to check the CUDA version in the container:

cat /usr/local/cuda/version.txt

Check whether the CUDA version supported by the NVIDIA driver version of the node where the container is located contains the CUDA version of the container.

Reviewing Logs​

Check whether the NVIDIA driver is running properly. Log in to the node where the add-on is running and view the driver installation log in the following path:

/opt/cloud/cce/nvidia/nvidia_installer.log

View standard output logs of the NVIDIA container. Filter the container ID by running the following command:

docker ps -a | grep nvidia

View logs by running the following command:

docker logs Container ID

Validating Pod Resource Requests​

Make sure the nodes that have GPUs are properly decorated with the following, that instructs Kubernetes to schedule the pods only on nodes that have available GPUs.

resources:
  limits:
    nvidia.com/gpu: 1
tip

Ensure that the requested number of GPUs does not exceed what’s available on any node.

Addressing Scheduling Conflicts​

  • Resource Overcommitment:
    • If multiple pods are scheduled with GPU resource requests, ensure that the overall demand does not exceed the cluster’s capacity.
    • Overcommitting resources might lead to scheduling failures.
  • Taints and Tolerations:
    • GPU nodes may have specific taints (e.g., nvidia.com/gpu=true:NoExecute).
    • Verify that your GPU-enabled pods include the proper tolerations so that the scheduler can place the pods on the GPU nodes.

Checking Operator Status​

Any errors here might indicate issues that indirectly affect GPU resource allocation:

helm list -n gpu-operator
kubectl get pods -n gpu-operator

Additional Information​

see also