En

Whisper migration from CPU to GPU#

This guide describes how to move the speech recognition service Whisper from CPU to an NVIDIA GPU in the client Sherpa AI Server installation. No other services or the selected LLM launch mode need to be changed.

Requirements#

Before switching, make sure that:

  • an NVIDIA GPU with enough video memory is installed on the server;
  • nvidia-smi detects the GPU;
  • NVIDIA Container Toolkit and NVIDIA GPU support in Docker are installed;
  • the aiserver-whisper:latest image is already downloaded;
  • the Whisper model is unpacked in ./whisper/models.

Checking Docker access to the GPU#

nvidia-smi

If nvidia-smi does not detect the GPU, or Docker is not configured to work with NVIDIA Container Toolkit, resolve that issue first. Otherwise, the Whisper container will not start in GPU mode.

Configuring docker-compose.yml#

Open docker-compose.yml and find the aiserver-whisper service:

  aiserver-whisper:
    container_name: aiserver-whisper
    image: aiserver-whisper:latest

In the service block, uncomment the WHISPER_DEVICE and CUDA_DEVICE_INDEX variables, as well as the deploy section. The final configuration should look like this:

  aiserver-whisper:
    container_name: aiserver-whisper
    image: aiserver-whisper:latest
    restart: unless-stopped
    profiles: ["whisper", "full"]
    environment:
      - WHISPER_MODEL=${WHISPER_MODEL:-base}
      - WHISPER_MODEL_DIR=/models
      - WHISPER_API_KEY=${WHISPER_API_KEY:-}
      - WHISPER_DEVICE=cuda:0
      - CUDA_DEVICE_INDEX=0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ports:
      - 3005:8000
    volumes:
      - ./whisper/models:/models:ro

WHISPER_DEVICE=cuda:0 explicitly selects the first GPU available to the container. CUDA_DEVICE_INDEX is used as a fallback setting when WHISPER_DEVICE is not present; in the client Compose file, keep the indexes aligned. If Whisper should use another GPU, specify, for example, cuda:1 and CUDA_DEVICE_INDEX=1.

Important: when LLM, Whisper, and BGE Reranker are running at the same time, take total VRAM consumption into account. Insufficient video memory can cause a CUDA out of memory error.

Verifying the configuration#

Before restarting, check the Compose file syntax:

docker compose --profile whisper config --quiet

The command exits without output if the configuration is valid.

Restarting Whisper in GPU mode#

Recreate only the Whisper container so the new environment variables and GPU reservation are applied:

docker compose --profile whisper up -d --force-recreate aiserver-whisper

A regular docker compose restart is not enough: that command does not apply container configuration changes.

Checking status and logs#

docker compose --profile whisper ps aiserver-whisper
docker compose --profile whisper logs --tail=100 aiserver-whisper

The aiserver-whisper container should be in the Up state.

Checking the selected device#

Whisper publishes diagnostic information at the /health endpoint:

curl -sS http://127.0.0.1:3005/health

Expected response:

{
  "status": "healthy",
  "device": "cuda:0",
  "cuda_available": true,
  "cuda_device_count": 1
}

The number of available GPUs may differ. The key signs of a successful switch are that device starts with cuda and cuda_available is true.

While transcription is running, also check GPU usage:

watch -n 1 nvidia-smi

Possible issues#

could not select device driver "nvidia"#

Docker does not see the NVIDIA runtime. Check the NVIDIA Container Toolkit installation, then run:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

After restarting Docker, recreate the Whisper container again.

cuda_available is false#

Check nvidia-smi, the NVIDIA Container Toolkit settings, and the presence of the deploy.resources.reservations.devices section in the final configuration:

docker compose --profile whisper config

CUDA out of memory#

There is not enough free video memory on the selected GPU. Stop unnecessary GPU processes, choose another GPU, or reduce the Whisper model through the WHISPER_MODEL variable in .env.

Rolling back to CPU#

To return Whisper to CPU:

  1. Comment out or remove WHISPER_DEVICE, CUDA_DEVICE_INDEX, and the deploy section from the aiserver-whisper service.
  2. Verify the configuration and recreate the container:
docker compose --profile whisper config --quiet
docker compose --profile whisper up -d --force-recreate aiserver-whisper
curl -sS http://127.0.0.1:3005/health

After rollback, /health should return "device":"cpu" and "cuda_available":false.