En

Migration of MariaDB to the orchestrator-db:v5.9.2+ container#

⚠️ Important: For the orchestrator-db:v5.9.1 migration, it is safer to use a logical dump and a new volume. This avoids the risk of damaging the old data directory and allows rollback without data loss.

1. What to know in advance#

In the Sherpa Orchestrator Chainguard images for MariaDB, the following is used:

  • only MYSQL_ROOT_PASSWORD for initialization;
  • the existing mount ./backend/config/my.cnf:/etc/mysql/conf.d/my.cnf.

The old MYSQL_ALLOW_EMPTY_PASSWORD, MYSQL_USER, and MYSQL_PASSWORD parameters for the bundled container are no longer needed.

2. Check the current configuration#

cd /opt/SherpaOrchestrator

grep -E "^(DB_ENGINE|MYSQL_HOST|MYSQL_PORT|MYSQL_USER|MYSQL_PASSWORD|MYSQL_DB)=" .env

For the bundled MariaDB, at least the following must be correct:

  • DB_ENGINE=mysql
  • MYSQL_HOST=orchestrator-db
  • MYSQL_DB=orchestrator

Check the current container and volume:

docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}" | grep orchestrator-db

docker volume ls | grep orchestrator-mysql-data

3. Put the application into maintenance mode#

Stop the application services so no new writes occur during the dump:

docker compose stop orchestrator orchestrator-nginx orchestrator-websocket orchestrator-vnc-proxy orchestrator-autopilot

Make sure the orchestrator-db container is still running:

docker ps --format "table {{.Names}}\t{{.Status}}" | grep orchestrator-db

4. Create a logical backup of MariaDB#

Create a directory:

sudo mkdir -p ./backups/mariadb
sudo chown -R "$USER:$USER" ./backups
export DB_MIGRATION_TS="$(date +%Y%m%d_%H%M%S)"
export DB_BACKUP_FILE="./backups/mariadb/orchestrator_${DB_MIGRATION_TS}.sql"

Create a dump of the working databases:

docker exec orchestrator-db sh -lc 'mariadb-dump \
  -uroot \
  -p"$MYSQL_ROOT_PASSWORD" \
  --databases orchestrator orchestrator_archive \
  --single-transaction \
  --routines \
  --events \
  --triggers' > "$DB_BACKUP_FILE"

Verify the result:

ls -lh "${DB_BACKUP_FILE}"
tail -n 20 "${DB_BACKUP_FILE}"
💡 Why mariadb-dump is used

A logical dump transfers data at the SQL level and does not depend on the internal datadir format. The old volume remains untouched, and the new container starts on a clean directory.

5. Stop the old MariaDB container#

docker compose stop orchestrator-db
docker ps -a --format "table {{.Names}}\t{{.Status}}" | grep orchestrator-db

Do not remove the old volume yet.

6. Create a new volume for Chainguard MariaDB#

Check the old volume:

docker volume ls | grep orchestrator-mysql-data

Create a new one:

docker volume create orchestrator-mysql-data-chainguard

Inspect it:

docker volume inspect orchestrator-mysql-data-chainguard

7. Switch the client compose file to the new volume#

Open docker-compose.client.yml and in the orchestrator-db service leave the old volume commented out and enable the new one:

volumes:
  # The old volume is kept as a reference during migration:
  # - orchestrator-mysql-data:/var/lib/mysql
  - orchestrator-mysql-data-chainguard:/var/lib/mysql

In the volumes: section at the bottom of the file, the new volume should also be active, and the old one should remain commented out:

volumes:
  # orchestrator-mysql-data:
  #   driver: local

  orchestrator-mysql-data-chainguard:
    driver: local

Check that only the following remains in environment:

environment:
  MYSQL_ROOT_PASSWORD: '${MYSQL_PASSWORD}'
💡 Why this matters

For the new Chainguard MariaDB container, it is safer to use a single root password without the legacy MYSQL_ALLOW_EMPTY_PASSWORD combination and additional bootstrap users. This removes extra failure points on an empty volume.

8. Start the new MariaDB container#

docker compose --profile mariadb up -d orchestrator-db

Check the logs:

docker logs -f --tail=100 orchestrator-db

Check readiness for queries:

docker exec orchestrator-db sh -lc 'mariadb -uroot -p"$MYSQL_ROOT_PASSWORD" -Nse "SELECT 1;"'

Expected result:

1

9. Restore the backup into the new container#

docker exec -i orchestrator-db sh -lc 'mariadb \
  -uroot \
  -p"$MYSQL_ROOT_PASSWORD"' < "$DB_BACKUP_FILE"

10. Verify the databases and configuration#

Make sure both databases are present:

docker exec orchestrator-db sh -lc 'mariadb -uroot -p"$MYSQL_ROOT_PASSWORD" -Nse "SHOW DATABASES LIKE '\''orchestrator'\'';"'

docker exec orchestrator-db sh -lc 'mariadb -uroot -p"$MYSQL_ROOT_PASSWORD" -Nse "SHOW DATABASES LIKE '\''orchestrator_archive'\'';"'

Check test queries:

docker exec orchestrator-db sh -lc 'MYSQL_PWD="$MYSQL_ROOT_PASSWORD" mariadb \
  -uroot \
  orchestrator \
  -Nse "SELECT COUNT(*) FROM accounts;"'

docker exec orchestrator-db sh -lc 'MYSQL_PWD="$MYSQL_ROOT_PASSWORD" mariadb \
  -uroot \
  orchestrator \
  -Nse "SELECT COUNT(*) FROM processes;"'

Check that the custom my.cnf was applied:

docker exec orchestrator-db sh -lc 'MYSQL_PWD="$MYSQL_ROOT_PASSWORD" mariadb \
  -uroot \
  -Nse "SHOW VARIABLES LIKE '\''max_allowed_packet'\'';"'

If the configuration mount is set up correctly, the current value from backend/config/my.cnf will be shown.

11. Bring the application back up#

docker compose up -d orchestrator orchestrator-websocket orchestrator-nginx

If needed, start the additional services:

docker compose up -d orchestrator-vnc-proxy
docker compose up -d orchestrator-autopilot
docker compose up -d orchestrator-vault

Check the backend logs:

docker logs --tail=100 orchestrator

Check the UI and a few working screens.

12. Rollback if something goes wrong#

If the new MariaDB does not work correctly:

  1. Stop the services:
docker compose stop orchestrator orchestrator-nginx orchestrator-websocket orchestrator-db
  1. Restore the old volume in docker-compose.yml:
volumes:
  - orchestrator-mysql-data:/var/lib/mysql
  1. Start the old, verified MariaDB image that originally worked with the volume.
  2. Start the containers again:
docker compose --profile mariadb up -d orchestrator-db
docker compose up -d orchestrator orchestrator-websocket orchestrator-nginx

13. What to remove after a successful migration#

Do not remove the following immediately after switching:

  • the SQL backup;
  • the old volume;
  • the old image archives.

After a monitoring period, the old volume can be removed:

docker volume rm orchestrator-mysql-data

Do this only after a full verification.