silverlinux-infra/docs/orchestration.md

Service Orchestration: slcompose

Overview

slcompose is the central service orchestrator for SilverLinux. It manages all Docker services in /srv/docker/ with automated secret injection from Infisical.

Architecture

System Boot
    ↓
systemd: slcompose.service
    ↓
/usr/local/bin/slcompose boot
    ↓
For each service in /srv/docker/:
    ├─ Load Infisical Token from /etc/infisical/token
    ├─ Discover docker-compose.yml
    ├─ Extract service name from directory (e.g., gitea)
    ├─ Inject secrets: infisical run --path=/gitea
    └─ Execute: docker compose up -d

Installation

Copy slcompose.sh to /usr/local/bin/slcompose and make executable:

sudo cp docs/slcompose.sh /usr/local/bin/slcompose
sudo chmod +x /usr/local/bin/slcompose

Copy systemd service file:

sudo cp docs/slcompose.service /etc/systemd/system/slcompose.service
sudo systemctl daemon-reload
sudo systemctl enable slcompose.service

Usage

List Services

slcompose list

Output:

Available services:
====================
  - gitea
  - portainer
  - postgres
  - mssql
  - openproject
  - jitsi
  - nginx-proxy-manager
  - postgres
  - baget
  - dbgate

Boot All Services

slcompose boot

This is automatically called by systemd on server startup.

Start a Service

slcompose up gitea

• Loads Infisical token
• Injects secrets from Infisical path: /gitea
• Runs: docker compose up -d in /srv/docker/gitea

Stop a Service

slcompose down gitea

Restart a Service

slcompose restart gitea

View Logs

Stream live logs (Ctrl+C to exit):

slcompose logs gitea

View last N lines (default 200):

slcompose logs-tail gitea
slcompose logs-tail gitea 500

View Injected Environment Variables

View all environment variables injected into a specific service from Infisical:

slcompose env gitea

Output (color-coded):

========================================
Environment Variables for: gitea
========================================
DB_HOST                                    = postgres
DB_NAME                                    = gitea_db
DB_PASSWD                                  = secure_password_123
DB_TYPE                                    = postgres
DB_USER                                    = gitea
DOMAIN                                     = git.silveressence.net
...

(Blue: variable names | Green: values)

View injected environment variables for ALL services:

slcompose env-all

Output (color-coded):

========================================
All Services and Their Environment Variables
========================================

--- gitea ---
DB_HOST                                    = postgres
DB_NAME                                    = gitea_db
...

--- portainer ---
ADMIN_PASSWORD                             = secure_password
...

--- postgres ---
POSTGRES_DB                                = main_db
...

(Blue: variable names | Green: values)

Use Cases:

• Verify secrets are properly injected
• Debug missing or incorrect environment variables
• Audit which services have access to which secrets
• Troubleshoot authentication or configuration issues

How Secret Injection Works

Prerequisites

1. Infisical Token: Stored at /etc/infisical/token
2. Infisical Domain (optional): Stored at /etc/infisical/domain
3. Secrets Path: Each service has secrets at path /<SERVICE_NAME> in Infisical

Example: For gitea service, secrets must be at /gitea in Infisical.

Injection Process

For each service, slcompose executes:

infisical run \
  --token "$INFISICAL_TOKEN" \
  --env prod \
  --path="/$SERVICE_NAME" \
  --recursive \
  -- docker compose "$@"

This:

1. Authenticates with Infisical using the token
2. Fetches all secrets from the specified path
3. Recursively includes nested secrets
4. Sets them as environment variables
5. Passes them to the docker compose command

Environment Variables

All secrets from Infisical are available as environment variables inside the docker compose execution context and are used by docker-compose.yml files.

Example docker-compose.yml:

services:
  gitea:
    image: gitea/gitea:latest
    environment:
      DB_TYPE: postgres
      DB_HOST: postgres
      DB_NAME: ${DB_NAME}              # Injected from Infisical
      DB_USER: ${DB_USER}              # Injected from Infisical
      DB_PASSWD: ${DB_PASSWD}          # Injected from Infisical

Service Directory Structure

Each service is a subdirectory in /srv/docker/ containing:

/srv/docker/gitea/
├── docker-compose.yml
├── README.md (optional)
└── app.ini (optional, service-specific config)

Services are discovered by the presence of docker-compose.yml.

Systemd Integration

Service File: /etc/systemd/system/slcompose.service

[Unit]
Description=SilverLinux Docker Orchestrator (slCompose)
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/slcompose boot
RemainAfterExit=yes
User=root

[Install]
WantedBy=multi-user.target

Key Details:

• Type=oneshot — Runs once and exits (doesn't stay running)
• RemainAfterExit=yes — Systemd remembers the service as "active" after boot completes
• After=docker.service — Waits for Docker to start first
• Requires=docker.service — Fails if Docker is not available
• WantedBy=multi-user.target — Starts during normal multi-user boot

Enable Auto-Boot

sudo systemctl enable slcompose.service

Check Status

sudo systemctl status slcompose.service

View Boot Logs

sudo journalctl -u slcompose.service -n 100

Error Handling

Missing Infisical Token

If /etc/infisical/token doesn't exist, slcompose exits with:

ERROR: Missing Infisical token at /etc/infisical/token

Service Not Found

If you try to manage a non-existent service:

slcompose up nonexistent

Output:

ERROR: Service 'nonexistent' not found at /srv/docker/nonexistent

Docker Not Running

If Docker isn't available, slcompose waits and retries:

Waiting for Docker...
Waiting for Docker...

Troubleshooting

Services not starting on boot

1. Check if systemd service is enabled:

   sudo systemctl is-enabled slcompose.service
   

2. View boot logs:

   sudo journalctl -u slcompose.service -n 50
   

3. Test slcompose manually:

   slcompose boot
   

Secrets not injected

1. Verify Infisical token exists:

   cat /etc/infisical/token
   

2. Test Infisical access:

   infisical run --token "$(cat /etc/infisical/token)" --env prod --path=/gitea -- env | grep -v '^_'
   

3. Check service paths in Infisical match service directory names

Service fails to start

1. View detailed logs:

   slcompose logs gitea
   

2. Check docker-compose.yml for syntax errors:

   cd /srv/docker/gitea && docker-compose config
   

3. View full docker compose output:

   cd /srv/docker/gitea && infisical run --token "$(cat /etc/infisical/token)" --env prod --path=/gitea -- docker compose up
   

Adding New Services

1. Create service directory:

   mkdir -p /srv/docker/myservice
   

2. Add docker-compose.yml:

   cp template/docker-compose.yml /srv/docker/myservice/
   

3. Create secrets in Infisical at path /myservice

4. Test:

   slcompose up myservice
   slcompose logs myservice
   slcompose down myservice
   

5. Service will automatically boot on next system startup

Performance

• Boot Time: ~2 minutes for all 10+ services (includes Docker startup)
• Per-Service: ~5-10 seconds per service startup
• Memory Overhead: Minimal (orchestrator is bash script)

Security

• Secrets: Never logged or displayed (only injected into containers)
• Tokens: Read from files with appropriate permissions
• Docker Access: Requires root/docker group membership
• Service Isolation: Uses Docker networks to isolate services

Related Files

• slcompose.sh — Orchestrator script
• slcompose.service — Systemd service file
• AI_CONTEXT.md — Infrastructure context
• ../README.md — Repository overview