En

Installing Sherpa AIServer (CPU-only, no GPU)#

This guide covers installation and startup when the language model runs on a CPU (without an NVIDIA GPU). Preparation follows the general installation. The LLM image and Docker Compose profile differ.

Suitable for#

  • A server without an NVIDIA GPU or where GPU is not required for LLMs.
  • Accepting significantly slower LLM response times compared to GPU.

Important: Services like Whisper and BGE Reranker in docker-compose.yml are designed for GPU. For CPU-only scenarios do not enable whisper, reranker, or full profiles — enable the embed and cpu profiles. Embedding and PostgreSQL do not require NVIDIA.

What to download (same as regular installation, replacing the LLM image)#

Download the client archive, models, and other images.

Differences from GPU installation#

Component GPU installation CPU installation
LLM image aiserver-llm-server aiserver-llm-server-cpu
Compose launch docker-compose.yml without cpu profile docker-compose.yml with cpu profile

Everything else follows the general preparation list if you don't use optional GPU services.

  • Client archive and images: use one selected version, with verified SHA-256 checksums, as described in the distribution download guide.
  • Images: aiserver, aiserver-pg, aiserver-embed, aiserver-code-interpreter, aiserver-mcp-orchestrator, mypy, aiserver-nginx, aiserver-websocket, and aiserver-llm-server-cpu; aiserver-whisper and aiserver-bge-reranker are needed only when their GPU profiles are used.
  • Models: model-paraphrase-multilingual-MiniLM-L12-v2.tar.gz for the embed service and one supported LLM model. The installer offers a suitable CPU option.

Download the aiserver-llm-server-cpu image instead of aiserver-llm-server (or additionally if you want both).

The aiserver-llm-server image is optional for CPU scenarios — you may skip it to save space and bandwidth.

Unpack and load images into Docker#

  1. Unpack the verified client archive of your selected version and place image archives of that same version alongside it. For offline installation, also prepare their .SHA256SUM files.
  2. Run sudo ./sh_scripts/install.sh from the installation directory and select CPU mode. The installer selects aiserver-llm-server-cpu, verifies the version and checksums, loads images, and configures startup.
  3. If you need to import the image manually for troubleshooting, specify its exact filename and verify its checksum first:
docker load --input aiserver-llm-server-cpu_2.5.1.tar.gz # example for version 2.5.1

For manual preparation, extract models with extract_models.sh and extract_vllm.sh, then configure .env and certificates as in the general installation guide. When using install.sh, follow its prompts.

Configure .env for CPU-LLM#

The main aiserver service connects to the LLM via LLM_MODEL_API_BASE_URL / LLM_API_HOST and LLM_API_PORT. For the aiserver-llm-server-cpu container the API listens on port 8000 inside the Docker network (host mapping in docker-compose.yml is 3007).

The installer sets LLM_API_HOST=aiserver-llm-server-cpu. Before startup, verify that these .env values agree:

LLM_API_HOST=aiserver-llm-server-cpu
LLM_API_PORT=8000
LLM_MODEL_API_BASE_URL=http://aiserver-llm-server-cpu:8000

Start with docker-compose.yml and cpu profile#

In the installation directory use the provided docker-compose.yml and explicitly enable the embed and cpu profiles so the aiserver-llm-server-cpu service starts and GPU profiles (gpu, gpu2) are not activated.

Basic start (CPU LLM only)#

docker compose -f docker-compose.yml --profile embed --profile cpu up -d

Stop#

docker compose -f docker-compose.yml --profile embed --profile cpu down

Verify containers#

docker compose -f docker-compose.yml --profile embed --profile cpu ps

The aiserver-llm-server-cpu container should be Up.

Optional host check for LLM#

The compose file maps port 8000 to 3007 on the host. Verify the model or health endpoint:

curl -sS "http://127.0.0.1:3007/v1/models" || curl -sS "http://127.0.0.1:3007/health"

It is necessary to check logs if needed: docker compose -f docker-compose.yml logs -f aiserver-llm-server-cpu.

Quick checklist#

  1. Download artifacts as for a normal installation, but use the aiserver-llm-server-cpu image instead of aiserver-llm-server.
  2. Unpack client files, load images, extract models.
  3. It is necessary to set .env: LLM_MODEL_API_BASE_URL=http://aiserver-llm-server-cpu:8000 (and matching LLM_API_HOST/LLM_API_PORT).
  4. It is necessary to start: docker compose -f docker-compose.yml --profile embed --profile cpu up -d.
  5. Do not enable whisper / reranker / full if no GPU is available.

For CPU-only installs the NVIDIA/Container Toolkit sections can be skipped unless you run GPU containers.