En

Running the LLM Server on a Separate Server#

This section describes the setup for running only the LLM server on a separate server, without the other components of Sherpa AI Server.

When This Is Needed#

Running the LLM server on a separate server is useful when:

  • You need to distribute the load between servers
  • The LLM server requires powerful GPUs and is better off on a separate server
  • Scaling is required - multiple LLM servers for load balancing
  • You need to isolate the LLM server from the main application

Requirements#

  • A server with NVIDIA GPU (CUDA 11.8+)
  • Docker and Docker Compose installed
  • NVIDIA Container Toolkit installed
  • LLM models loaded in the directory llm-server/models/

Setup#

Step 1: Prepare the Server#

Make sure all necessary components are installed on the server

# Check GPU
nvidia-smi

# Check Docker
docker --version
docker compose version

Step 2: Prepare the files and image#

It is necessary to create a working directory. Copy the model to llm-server/models/ and, if needed, templates to llm-server/templates/. It is necessary to create .env as shown in Step 4. The main server's client-files archive does not contain the LLM Docker image: on a new LLM host, obtain the image archive of the same version as the main Sherpa AI Server and its checksum separately. For example, for version 2.5.1:

set -e
RELEASE_VERSION=2.5.1
ARCHIVE=aiserver-llm-server_$RELEASE_VERSION.tar.gz
BASE_URL=https://downloads.sherparpa.ru/SherpaAIServer/aiserver-llm-server
curl -fL --retry 3 -o "$ARCHIVE" "$BASE_URL/$ARCHIVE"
curl -fL --retry 3 -o "$ARCHIVE.SHA256SUM" "$BASE_URL/$ARCHIVE.SHA256SUM"
sha256sum -c "$ARCHIVE.SHA256SUM"
docker load -i "$ARCHIVE"
docker image inspect "aiserver-llm-server:$RELEASE_VERSION" >/dev/null

Without access to the download server, transfer both files of the same version to the LLM host and run the last three commands. If the checksum fails or the image with the required tag is absent, stop and check the bundle. It is necessary to set APP_VERSION in .env to this same version. Do not use an independent :latest tag instead of the verified release.

Step 3: Configure Docker Compose#

It is necessary to save the following as docker-compose.llm-only.yml in the working directory. It runs one LLM container using the verified image tag. By default, HTTP is available only on 127.0.0.1:3003. For direct access from another server, change the port line to "3003:8000" and restrict port 3003 to the main server's address. For HTTPS, keep the local binding.

services:
  aiserver-llm-server:
    container_name: aiserver-llm-server
    image: aiserver-llm-server:${APP_VERSION}
    restart: always
    env_file:
      - .env
    ports:
      - "127.0.0.1:3003:8000"
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - "./llm-server/models:/model-store"
      - "./llm-server/templates:/model-templates"

Step 4: Configure environment variables#

It is necessary to create or edit the .env file with LLM server settings:

APP_VERSION=2.5.1
# LLM server settings
LLM_CUDA_VISIBLE_DEVICES=0
LLM_TENSOR_PARALLEL_SIZE=1
LLM_GPU_MEMORY_UTILIZATION=0.90
LLM_COMPLETION_MODEL_NAME=/model-store/meta-llama/Meta-Llama-3-8B-Instruct
LLM_DTYPE=auto
LLM_TRUST_REMOTE_CODE=false
LLM_QUANTIZATION=false
LLM_MAX_MODEL_LEN=8192
LLM_HOST=0.0.0.0
LLM_PORT=8000
LLM_MAX_NUM_BATCHED_TOKENS=16384
LLM_MAX_NUM_SEQS=16
LLM_ENABLE_TOOLS=true
LLM_TOOL_CALL_PARSER=llama3_json
LLM_EXCLUDE_TOOLS_WHEN_NONE=true

Important:

  • It is necessary to ensure that the model path is correct: LLM_COMPLETION_MODEL_NAME=/model-store/model-name

Step 5: Check the configuration#

Before starting, check the configuration:

docker compose -f docker-compose.llm-only.yml config
# Check that port 3003 is free before starting
ss -ltn '( sport = :3003 )'
ls -la llm-server/models/

Step 6: Start the LLM server#

docker compose -f docker-compose.llm-only.yml up -d aiserver-llm-server
docker compose -f docker-compose.llm-only.yml ps

Expected Result: Only the aiserver-llm-server container should start.

Step 7: Verify operation#

# Check the logs
docker logs aiserver-llm-server

# Check GPU usage
nvidia-smi

# Check API availability (should return model information)
curl http://localhost:3003/v1/models

Expected Result:

  • The container should start successfully
  • There should be no critical errors in the logs
  • The API should respond to requests
  • The GPU should be used for loading the model

Connecting from Another Server#

If the LLM server is running on a separate server, configure the connection from the main server.

On the Main Server (Direct HTTP)#

For direct HTTP access, first change the port line in docker-compose.llm-only.yml to "3003:8000" and apply the configuration with docker compose -f docker-compose.llm-only.yml up -d aiserver-llm-server. Restrict incoming connections to port 3003 to the main server's address. In the .env file of the main server, specify the address of the LLM server:

# Address of the LLM server (replace with the IP or domain of your LLM server)
LLM_MODEL_API_BASE_URL=http://192.168.1.100:3003
# or
LLM_MODEL_API_BASE_URL=http://llm-server.example.com:3003

After changing .env, recreate the backend container with docker compose up -d --force-recreate aiserver. For a separate model instance, also set its base URL in the instance settings.

Connecting over HTTPS#

On the LLM host, install nginx using your OS package manager, then put fullchain.pem and privkey.pem in a protected /etc/sherpa/llm-certs/ directory. The certificate must cover the LLM host's DNS name. The 2.5.1 client archive does not include a separate HTTPS Docker Compose file or nginx configuration, so create the configuration below on the LLM host. It proxies to the existing container and does not start a second LLM.

In docker-compose.llm-only.yml, ensure the port is bound to "127.0.0.1:3003:8000". If you previously exposed HTTP for direct access, restore the local binding and apply it:

docker compose -f docker-compose.llm-only.yml up -d aiserver-llm-server
docker compose -f docker-compose.llm-only.yml ps

It is necessary to create /etc/nginx/conf.d/llm-server.conf:

server {
    listen 443 ssl;
    server_name llm-server.example.com;
    ssl_certificate /etc/sherpa/llm-certs/fullchain.pem;
    ssl_certificate_key /etc/sherpa/llm-certs/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    client_max_body_size 500m;

    location / {
        proxy_pass http://127.0.0.1:3003;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_read_timeout 600s;
        proxy_buffering off;
    }
}

Validate the configuration, start nginx, and limit TCP/443 access to the main server's address:

sudo nginx -t
sudo systemctl enable --now nginx
sudo systemctl reload nginx

On the main server, set the URL matching the certificate name and enable TLS verification in .env:

LLM_MODEL_API_BASE_URL=https://llm-server.example.com
LLM_TLS_VERIFY=1

If you use a private certificate authority, add its root certificate to the backend container's trust store. It is necessary to recreate aiserver, then check the connection without curl -k:

docker compose up -d --force-recreate aiserver
curl -fsS https://llm-server.example.com/v1/models

If LLM_API_KEY is enabled, add an Authorization: Bearer <key> header to the check request and set that key on the model instance in Sherpa AI Server. Then send a test chat request. LLM_TLS_VERIFY is disabled by default for compatibility with older installations. It is necessary to use 1 for HTTPS with a trusted certificate.

Possible Issues#

Container Does Not Start#

Problem: The container crashes immediately after starting.

Solution:

  1. It is necessary to check the logs: docker logs aiserver-llm-server
  2. It is necessary to ensure the GPU is available: nvidia-smi
  3. It is necessary to check that the model exists: ls -la llm-server/models/
  4. It is necessary to check permissions for the model directory

Model Does Not Load#

Problem: Errors when loading the model.

Solution:

  1. It is necessary to check the model path in .env: LLM_COMPLETION_MODEL_NAME
  2. It is necessary to ensure the model is loaded: ls -la llm-server/models/
  3. It is necessary to check logs for loading errors: docker logs aiserver-llm-server | grep -i error

Insufficient GPU Memory#

Problem: The model does not fit in GPU memory.

Solution:

  • It is necessary to reduce LLM_GPU_MEMORY_UTILIZATION in .env
  • It is necessary to use the quantized version of the model (set LLM_QUANTIZATION=true)
  • It is necessary to use a smaller model

Port Not Accessible Externally#

Problem: Cannot connect to the LLM server from another server.

Solution:

  1. It is necessary to check the firewall: sudo ufw status
  2. It is necessary to check that the port is forwarded: docker port aiserver-llm-server
  3. It is necessary to check Docker network settings

Monitoring#

To monitor the operation of the LLM server:

# Container status
docker ps | grep llm-server

# Resource usage
docker stats aiserver-llm-server

# GPU usage
watch -n 1 nvidia-smi

# Real-time logs
docker logs -f aiserver-llm-server

# API check
curl http://localhost:3003/health
curl http://localhost:3003/v1/models

Performance Optimization#

To optimize the performance of the LLM server:

  1. Configure GPU Memory:

    LLM_GPU_MEMORY_UTILIZATION=0.90  # Use maximum available memory

  2. Batching Configuration:

    LLM_MAX_NUM_BATCHED_TOKENS=16384
LLM_MAX_NUM_SEQS=16

  3. Use Quantization:

    LLM_QUANTIZATION=true  # For models that support quantization

After these steps, one LLM container runs on the separate server. The API uses port 3003 for direct HTTP access. For HTTPS, connect to nginx on port 443.