silverlinux-infra/docs/orchestration.md

# Service Orchestration: slcompose

## Overview

**slcompose** is the central service orchestrator for SilverLinux. It manages all Docker services in `/srv/docker/` with automated secret injection from Infisical.

## Architecture

```
System Boot
    ↓
systemd: slcompose.service
    ↓
/usr/local/bin/slcompose boot
    ↓
For each service in /srv/docker/:
    ├─ Load Infisical Token from /etc/infisical/token
    ├─ Discover docker-compose.yml
    ├─ Extract service name from directory (e.g., gitea)
    ├─ Inject secrets: infisical run --path=/gitea
    └─ Execute: docker compose up -d
```

## Installation

Copy `slcompose.sh` to `/usr/local/bin/slcompose` and make executable:

```bash
sudo cp docs/slcompose.sh /usr/local/bin/slcompose
sudo chmod +x /usr/local/bin/slcompose
```

Copy systemd service file:

```bash
sudo cp docs/slcompose.service /etc/systemd/system/slcompose.service
sudo systemctl daemon-reload
sudo systemctl enable slcompose.service
```

## Usage

### List Services

```bash
slcompose list
```

Output:
```
Available services:
====================
  - gitea
  - portainer
  - postgres
  - mssql
  - openproject
  - jitsi
  - nginx-proxy-manager
  - postgres
  - baget
  - dbgate
  - nextcloud
```

### Boot All Services

```bash
slcompose boot
```

This is automatically called by systemd on server startup.

### Start a Service

```bash
slcompose up gitea
```

- Loads Infisical token
- Injects secrets from Infisical path: `/gitea`
- Runs: `docker compose up -d` in `/srv/docker/gitea`

### Stop a Service

```bash
slcompose down gitea
```

### Restart a Service

```bash
slcompose restart gitea
```

### View Logs

Stream live logs (Ctrl+C to exit):
```bash
slcompose logs gitea
```

View last N lines (default 200):
```bash
slcompose logs-tail gitea
slcompose logs-tail gitea 500
```

### View Injected Environment Variables

View all environment variables injected into a specific service from Infisical:

```bash
slcompose env gitea
```

Output (color-coded):
```
========================================
Environment Variables for: gitea
========================================
DB_HOST                                    = postgres
DB_NAME                                    = gitea_db
DB_PASSWD                                  = secure_password_123
DB_TYPE                                    = postgres
DB_USER                                    = gitea
DOMAIN                                     = git.silveressence.net
...
```

*(Blue: variable names | Green: values)*

View injected environment variables for ALL services:

```bash
slcompose env-all
```

Output (color-coded):
```
========================================
All Services and Their Environment Variables
========================================

--- gitea ---
DB_HOST                                    = postgres
DB_NAME                                    = gitea_db
...

--- portainer ---
ADMIN_PASSWORD                             = secure_password
...

--- postgres ---
POSTGRES_DB                                = main_db
...
```

*(Blue: variable names | Green: values)*

**Use Cases:**
- Verify secrets are properly injected
- Debug missing or incorrect environment variables
- Audit which services have access to which secrets
- Troubleshoot authentication or configuration issues

## How Secret Injection Works

### Prerequisites

1. **Infisical Token**: Stored at `/etc/infisical/token`
2. **Infisical Domain** (optional): Stored at `/etc/infisical/domain`
3. **Secrets Path**: Each service has secrets at path `/<SERVICE_NAME>` in Infisical

Example: For `gitea` service, secrets must be at `/gitea` in Infisical.

### Injection Process

For each service, slcompose executes:

```bash
infisical run \
  --token "$INFISICAL_TOKEN" \
  --env prod \
  --path="/$SERVICE_NAME" \
  --recursive \
  -- docker compose "$@"
```

This:
1. Authenticates with Infisical using the token
2. Fetches all secrets from the specified path
3. Recursively includes nested secrets
4. Sets them as environment variables
5. Passes them to the docker compose command

### Environment Variables

All secrets from Infisical are available as environment variables inside the docker compose execution context and are used by `docker-compose.yml` files.

Example `docker-compose.yml`:
```yaml
services:
  gitea:
    image: gitea/gitea:latest
    environment:
      DB_TYPE: postgres
      DB_HOST: postgres
      DB_NAME: ${DB_NAME}              # Injected from Infisical
      DB_USER: ${DB_USER}              # Injected from Infisical
      DB_PASSWD: ${DB_PASSWD}          # Injected from Infisical
```

## Service Directory Structure

Each service is a subdirectory in `/srv/docker/` containing:

```
/srv/docker/gitea/
├── docker-compose.yml
├── README.md (optional)
└── app.ini (optional, service-specific config)
```

Services are discovered by the presence of `docker-compose.yml`.

## Systemd Integration

### Service File: `/etc/systemd/system/slcompose.service`

```ini
[Unit]
Description=SilverLinux Docker Orchestrator (slCompose)
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/slcompose boot
RemainAfterExit=yes
User=root

[Install]
WantedBy=multi-user.target
```

**Key Details:**
- `Type=oneshot` — Runs once and exits (doesn't stay running)
- `RemainAfterExit=yes` — Systemd remembers the service as "active" after boot completes
- `After=docker.service` — Waits for Docker to start first
- `Requires=docker.service` — Fails if Docker is not available
- `WantedBy=multi-user.target` — Starts during normal multi-user boot

### Enable Auto-Boot

```bash
sudo systemctl enable slcompose.service
```

### Check Status

```bash
sudo systemctl status slcompose.service
```

### View Boot Logs

```bash
sudo journalctl -u slcompose.service -n 100
```

## Error Handling

### Missing Infisical Token

If `/etc/infisical/token` doesn't exist, slcompose exits with:
```
ERROR: Missing Infisical token at /etc/infisical/token
```

### Service Not Found

If you try to manage a non-existent service:
```bash
slcompose up nonexistent
```

Output:
```
ERROR: Service 'nonexistent' not found at /srv/docker/nonexistent
```

### Docker Not Running

If Docker isn't available, slcompose waits and retries:
```
Waiting for Docker...
Waiting for Docker...
```

## Troubleshooting

### Services not starting on boot

1. Check if systemd service is enabled:
   ```bash
   sudo systemctl is-enabled slcompose.service
   ```

2. View boot logs:
   ```bash
   sudo journalctl -u slcompose.service -n 50
   ```

3. Test slcompose manually:
   ```bash
   slcompose boot
   ```

### Secrets not injected

1. Verify Infisical token exists:
   ```bash
   cat /etc/infisical/token
   ```

2. Test Infisical access:
   ```bash
   infisical run --token "$(cat /etc/infisical/token)" --env prod --path=/gitea -- env | grep -v '^_'
   ```

3. Check service paths in Infisical match service directory names

### Service fails to start

1. View detailed logs:
   ```bash
   slcompose logs gitea
   ```

2. Check docker-compose.yml for syntax errors:
   ```bash
   cd /srv/docker/gitea && docker-compose config
   ```

3. View full docker compose output:
   ```bash
   cd /srv/docker/gitea && infisical run --token "$(cat /etc/infisical/token)" --env prod --path=/gitea -- docker compose up
   ```

## Adding New Services

1. Create service directory:
   ```bash
   mkdir -p /srv/docker/myservice
   ```

2. Add `docker-compose.yml`:
   ```bash
   cp template/docker-compose.yml /srv/docker/myservice/
   ```

3. Create secrets in Infisical at path `/myservice`

4. Test:
   ```bash
   slcompose up myservice
   slcompose logs myservice
   slcompose down myservice
   ```

5. Service will automatically boot on next system startup

## Performance

- **Boot Time**: ~2 minutes for all 10+ services (includes Docker startup)
- **Per-Service**: ~5-10 seconds per service startup
- **Memory Overhead**: Minimal (orchestrator is bash script)

## Security

- **Secrets**: Never logged or displayed (only injected into containers)
- **Tokens**: Read from files with appropriate permissions
- **Docker Access**: Requires root/docker group membership
- **Service Isolation**: Uses Docker networks to isolate services

## Related Files

- [slcompose.sh](slcompose.sh) — Orchestrator script
- [slcompose.service](slcompose.service) — Systemd service file
- [AI_CONTEXT.md](AI_CONTEXT.md) — Infrastructure context
- [../README.md](../README.md) — Repository overview