Infrastructure Integration Specialist

🎯 Overview

This skill provides a systematic methodology for integrating and monitoring self-hosted infrastructure components, covering the full lifecycle from DNS setup → server configuration → deployment → monitoring dashboards → ongoing observability. It consolidates domain knowledge across Cloudflare DNS automation, Coolify server setup, dashboard development, and infrastructure hardening.

🧭 Methodology: RTCC + OODA Hybrid

RTCC Framework (Execution Focus)

• Rôle: DevOps Infrastructure Specialist
• Tâche: Integrate and monitor self-hosted services
• Context: Coolify environment with multiple services
• Contraintes: API limitations, permissions, automation requirements

OODA Framework (Analysis Focus)

• Observer: Audit current system state and dependencies
• Orienter: Choose optimal integration approach
• Décider: Select tools and configuration methods
• Agir: Implement with systematic verification

📊 Progressive Task Structure

🧭 Progression
⚙️ Diagnostic système ✅ (étape 1 / 6)
🧩 Audit des intégrations ✅ (étape 2 / 6)
🛠️ Finalisation configuration ✅ (étape 3 / 6)
🔧 Correction des API issues ✅ (étape 4 / 6)
📊 Activation dashboard 🟠 (étape 5 / 6)
🚀 Validation finale 

A. ☁️ Cloudflare DNS Automation

Purpose

Automate DNS record creation via Cloudflare API for professional subdomains with HTTPS proxy support.

Prerequisites

• Python 3.6+ with requests library
• Cloudflare API token with DNS Read, DNS Write, Zone Read permissions
• Domain registered with Cloudflare
• VPS IP for target record

Required Permissions

Avoid: Zone Settings Write, Zone Versioning Write, Firewall for AI, Custom Asset Write

Quick Start

# Create a subdomain with HTTPS proxy
python3 cloudflare-dns.py \
    --token "cfat_your_token" \
    --domain "example.com" \
    --subdomain "dashboard" \
    --ip "203.0.113.10"

Integration Workflow

1. ✅ DNS Creation: subdomain.example.com → VPS IP
2. ✅ Coolify App: Create and configure app
3. 🔄 User Action: Manual domain config in Coolify interface
4. 🔒 SSL Setup: Automatic via Let’s Encrypt
5. 🌐 Final Access: https://subdomain.example.com

Pitfalls

• 403 “Cannot use access token from location”: Token has IP restrictions. Remove IP restriction in Cloudflare dashboard.
• Proxy mode: Set proxied: True (orange cloud) for automatic HTTPS. TTL=1 for fast propagation.
• Always validate token first: curl -H "Authorization: Bearer $TOKEN" https://api.cloudflare.com/client/v4/tokens/verify

See Also

• references/cloudflare-dns-automation.md — full script and deployment integration

B. 🖥️ Coolify Server Setup — Fixing “No Available Server”

The Critical Problem

Coolify shows “no available server” when no server is configured to run applications.

Fix: Configure a Server in Coolify

1. Log into Coolify UI (https://cool.example.com)
2. Go to Servers → Add Server
3. Configure:
   • Name: dashboard-server
   • IP: VPS IP
   • Port: 22
   • User: root
   • SSH Key: your public key

Direct Deployment (when Coolify API v4 fails)

Coolify v4 Beta API often returns 404 on /api/v1/projects/{uuid}/applications POST.

Workaround: Deploy via docker-compose.yml on VPS directly, then use Coolify UI “Import” or “Docker Compose” feature.

Port Conflict Resolution

# Check port 3000
sudo lsof -i :3000
sudo kill -9 <PID>  # if needed

# Auto-redirect in deploy script
if netstat -tlnp | grep -q :3000; then
    pkill -f "node.*server.js" || true
    sleep 2
fi

Verification Pattern

# 1. Container running
docker ps | grep dashboard

# 2. Port open
netstat -tlnp | grep :3000

# 3. API health
curl -f http://localhost:3000/api/health

# 4. Interface
curl -I http://localhost:3000/

See Also

• references/coolify-server-setup.md — full automation scripts
• references/coolify-deployment-lessons.md — lessons learned
• scripts/coolify-deploy-direct.sh — direct deployment script
• scripts/coolify-verify-deployment.sh — verification script
• templates/coolify-docker-compose.prod.yml — production compose file

C. 📊 Monitoring Dashboard for Coolify

Purpose

Build a professional real-time monitoring dashboard for self-hosted infrastructure with Coolify integration, WebSocket updates, and optional HTTPS via Cloudflare.

Tech Stack

• Backend: Node.js/Express + Socket.IO + node-cron
• Frontend: Vanilla HTML/JS (responsive)
• Data Source: Coolify API (/api/v1/projects, /api/v1/applications)
• Real-time: WebSocket updates every 30 seconds

Project Structure

dashboard-monitoring/
├── server.js           # Express + WebSocket server
├── .env               # Configuration
├── start.sh           # Startup script
├── frontend/
│   └── index.html     # Responsive web interface
├── Dockerfile         # For production
├── docker-compose.yml # Local development
├── docker-compose.prod.yml # Coolify deployment
├── package.json       # Dependencies
└── .git/              # Repository

Environment Setup

HOST=0.0.0.0
PORT=3000
NODE_ENV=production
COOLIFY_URL=https://cool.iatuto.com
# Secret: définir COOLIFY_TOKEN dans ~/.hermes_env
COOLIFY_TOKEN=$COOLIFY_TOKEN
CORS_ORIGIN=https://dashboard.iatuto.com

Multi-Project Coolify API Pattern

The Coolify API splits applications across projects. Use this pattern to discover all apps:

// 1. Get all projects
const projects = await axios.get(`${url}/api/v1/projects`, ...);
// 2. For each project, get apps
for (const project of projects) {
  const apps = await axios.get(`${url}/api/v1/projects/${project.id}/applications`, ...);
}
// 3. Fallback: /api/v1/applications (main project)

Deployment Options

1. Via Coolify (recommended): Import repo → select docker-compose.prod.yml → set domain
2. Manual: docker build -t dashboard-monitoring . && docker run -d -p 3000:3000 dashboard-monitoring
3. Direct Node: NODE_ENV=production node server.js (with PM2 for persistence)

PM2 Process Monitoring

Paperclip AI Server Management

For managing a specific Node.js app (e.g., Paperclip AI) with PM2:

Pitfalls:

• Migration: Always run pnpm db:migrate after environment setup or infra changes.
• Dependency paths: If ERR_MODULE_NOT_FOUND from packages/db/src/index.ts, check relative import paths (./client.js vs ../client.js). Imports must match compiled dist/ structure.
• Persistence: Always pm2 save after adding processes to survive reboots.

Workflow:

cd /home/bf/paperclip-ai
pnpm run db:migrate
pm2 start dist/index.js --name paperclip
pm2 save

General PM2 Process Monitoring

// monitor.js — watch processes and notify n8n webhook
const pm2 = require('pm2');
const axios = require('axios');
pm2.connect(() => {
  pm2.launchBus((err, bus) => {
    bus.on('process:event', (data) => {
      axios.post('https://n8n.example.com/webhook/paperclip-monitor', {
        event: data.event, process: data.process.name, status: data.process.status
      }).catch(console.error);
    });
  });
});

VPS Remote Access

• Server must listen on 0.0.0.0 (not localhost)
• Open port 3000 in UFW: sudo ufw allow 3000/tcp
• Optional Cloudflare proxy (orange cloud) for HTTPS

Pitfalls

• Coolify api v4 fallback: If /api/v1/projects/{id}/applications fails, fall back to /api/v1/applications
• Pull access denied: Coolify tries Docker Hub image that doesn’t exist. Use explicit build: directive in compose.
• Cloudflare tunnel conflict: If domain is on Cloudflare proxy, ensure tunnel (cloudflared) has priority or disable A record.

See Also

• references/dashboard-monitoring-coolify.md — comprehensive deployment guide (French)
• references/dashboard-vps-config.md — VPS and network configuration details
• references/coolify-multi-projects.md — multi-project API technique
• scripts/deploy-monitoring-dashboard.sh — automated deploy script
• templates/dashboard-docker-compose.yml — Docker compose for Coolify deployment

D. 🏗️ Dashboard Development Patterns

Architecture

Frontend (React/Vanilla JS)
    ↕️ WebSocket
Backend (Node.js/Express)
    ↕️ REST API
Coolify / Vikunja / n8n / BookStack APIs

Key Technical Decisions

• Host binding: process.env.HOST || '0.0.0.0' for remote VPS access
• Real-time: Socket.IO with 30-second cron interval
• API debouncing: Cache frequent data, batch related calls
• Error handling: Graceful degradation on service failure
• CORS: Configure for production domain (not *)

UI Principles

• Overview section with key metrics at a glance
• Service cards with individual status
• Status colors: green/yellow/red
• Mobile-first responsive grid layout
• Loading states during data fetch

See Also

• references/dashboard-dev-patterns.md — full development patterns and code examples
• references/coolify-api-implementation.md — real-world API implementation notes
• scripts/dashboard-verify-setup.sh — setup verification script
• templates/dashboard-package.json.template — package template
• templates/dashboard-server.js.template — server code template
• templates/dashboard-start.sh.template — startup script template

E. 🔧 Infrastructure Hardening

SSH + UFW + fail2ban

Never modify UFW rules affecting SSH without an alternative access path tested and confirmed.

Sequence:

1. Add new SSH port (e.g., 2222) — do NOT remove port 22 yet
2. Restart SSH: sudo systemctl restart sshd
3. Test the new port from another terminal
4. Open new port in provider network firewall (separate from UFW!)
5. Only then configure UFW

sudo ufw default deny incoming
sudo ufw allow 22/tcp
sudo ufw allow 2222/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 3000/tcp
sudo ufw enable

fail2ban Configuration

sudo tee /etc/fail2ban/jail.local.d/sshd.conf > /dev/null << 'EOF'
[sshd]
enabled = true
port = 22,2222
maxretry = 3
findtime = 10m
bantime = 1h
EOF

Provider Firewall Awareness

Port Classification

See Also

• references/ssh-ufw-fail2ban-hardening.md — full hardening guide with duplicate rule cleanup
• references/port-service-audit.md — port/service discovery workflow

F. 🔒 Change Management: Backup → Modify → Log → Rollback

Mandatory policy for ALL system modifications. Every change to configs, services, deployments, or infrastructure must follow this sequence.

⚠️ Ce workflow est automatisé par le skill sysguard. Utilise la commande sysguard backup / sysguard log / sysguard rollback — c’est plus complet, plus fiable, et gère le token OAuth automatiquement.

Workflow

1️⃣ BACKUP  →  2️⃣ MODIFY  →  3️⃣ LOG  →  4️⃣ VERIFY  →  ✅ DONE
                                                          (or 🔄 ROLLBACK)

Step 1: Backup (always before modifying)

Using sysguard (recommended):

sysguard backup full        # Backup complet + log automatique
sysguard backup hermes-config   # Config Hermes uniquement

Manual fallback:

DATE=$(date +%Y%m%d_%H%M)
BACKUP_DIR=~/.hermes/backups/pre-$DATE
mkdir -p "$BACKUP_DIR"
cp ~/.hermes/config.yaml "$BACKUP_DIR/"
cp ~/.hermes/.env "$BACKUP_DIR/"
tar czf "$BACKUP_DIR/skills.tar.gz" -C ~/.hermes skills/
echo "✅ Backup → $BACKUP_DIR"

Step 3: Log to Google Sheet

Using sysguard:

sysguard log "Action" "Description détaillée" "Catégorie" "✅"

Legacy (OpenClaw — déprécié):

python3 ~/openclaw-workspace/scripts/log-change-sheets.py \
  "Action" "Description" "Catégorie" "✅"

Step 5 (if failed): Rollback

Using sysguard:

sysguard list          # Lister les backups disponibles
sysguard rollback 1    # Restaurer le backup #1

Manual:

RESTORE=$(ls -td ~/.hermes/backups/pre-* | head -1)
cp "$RESTORE/config.yaml" ~/.hermes/
cp "$RESTORE/.env" ~/.hermes/

Google Sheet

• Sheet URL: https://docs.google.com/spreadsheets/d/1U-GaOz6qmQaBP8NRwwRlVqY67wT_a1O7cA8dp60GGYs
• Columns: Date | Heure | Catégorie | Action | Détails | Statut
• Auth: OAuth Device Flow (token at ~/.config/gspread/token.json)
• Scripts: ~/scripts/sysguard/sysguard (recommandé) ou ~/openclaw-workspace/scripts/log-change-sheets.py (legacy)
• Détails OAuth: voir le skill sysguard pour le pattern d’injection client_id/secret

G. 🐳 Docker Bridge → Host Port Forwarding (iptables DNAT)

Problem

When a service runs natively on the host (not in Docker), Docker containers on bridge networks cannot reach it — even if the service listens on 0.0.0.0:PORT. The connection is silently dropped by Docker’s iptables rules.

Symptom: docker exec <container> curl http://HOST.GATEWAY:PORT/health → timeout or connection refused, while curl http://127.0.0.1:PORT/health on the host works fine.

Solution: iptables DNAT Rule

sudo iptables -A DOCKER -p tcp --dport PORT -j ACCEPT
sudo iptables -t nat -A DOCKER -p tcp --dport PORT -j DNAT --to-destination 127.0.0.1:PORT

This adds rules in the Docker iptables chain to allow and redirect traffic from the Docker bridge network to the host service.

Example (port 8787 for Hermes WebUI):

sudo iptables -A DOCKER -p tcp --dport 8787 -j ACCEPT
sudo iptables -t nat -A DOCKER -p tcp --dport 8787 -j DNAT --to-destination 127.0.0.1:8787

Persistence

iptables rules are not persistent across reboots or Docker service restarts. Options:

Traefik File Provider for Non-Docker Services

When a native service needs to be exposed via Coolify’s Traefik:

1. Ensure the service listens on 0.0.0.0:PORT (or at least the Docker bridge gateway IP)
2. Add the iptables DNAT rule above
3. Create a Traefik file provider config:

# /data/coolify/proxy/dynamic/my-service.yml
http:
  routers:
    my-service:
      rule: "Host(`service.example.com`)"
      entrypoints:
        - https
      tls:
        certResolver: letsencrypt
      service: my-service
  services:
    my-service:
      loadBalancer:
        servers:
          - url: "http://HOST_DOCKER_GATEWAY_IP:PORT"

sudo mkdir -p /data/coolify/proxy/dynamic
sudo tee /data/coolify/proxy/dynamic/my-service.yml > /dev/null << 'EOF'
# ... content ...
EOF

Traefik’s --providers.file.watch=true picks up changes automatically.

Note: The Coolify Docker proxy must mount /data/coolify/proxy to /traefik (default setup).

H. 📋 Vikunja v1.0.0-rc2 — Self-Hosted Task Management

Architecture (v1.0.0-rc2 Changes)

Vikunja v1.0.0-rc2 introduces a view-based bucket model:

• Views (project_views) replace direct project-to-bucket relationships
• Each project auto-creates 4 views: List(0), Gantt(1), Table(2), Kanban(3)
• Buckets belong to the Kanban view (project_view_id), not the project
• API endpoint for buckets: /projects/{id}/views/{view_id}/buckets (NOT /projects/{id}/buckets which returns 404)

Authentication

Login via API:

curl -sk https://vikunja.example.com/api/v1/login -X POST \
  -H "Content-Type: application/json" \
  -d '{"username":"USER","password":"PASS"}'
# Returns JWT in "token" field

Use JWT as Bearer token for all subsequent calls.

User Creation (Direct DB — When API Registration Fails)

Vikunja 403 error “code 1010” on /api/v1/register usually means no registration token is configured. Bypass via PostgreSQL:

-- Find columns
SELECT column_name FROM information_schema.columns WHERE table_name='users';

-- Create user (critical: issuer must be 'local' for password auth)
INSERT INTO users (username, email, password, name, status, issuer, created, updated)
VALUES ('myuser', 'my@email.com', 'BCRYPT_HASH', 'My Name', 0, 'local', NOW(), NOW());

Critical fields:

• status: integer. 0 = active
• issuer: must be 'local' (not NULL) for password-based login
• password: bcrypt hash with $2a$ prefix (not $2b$). Cost factor 11 matches Vikunja defaults
• email_hash column does NOT exist in v1.0.0-rc2

Bucket Management (via DB)

Since the API endpoint for bucket CRUD returns 404 in v1.0.0-rc2, manage buckets directly:

# Get kanban view ID
SELECT id FROM project_views WHERE project_id=PROJECT_ID AND view_kind=3;

# List existing buckets
SELECT id, title FROM buckets WHERE project_view_id=VIEW_ID;

# Rename default buckets (created as "To-Do", "Doing", "Done")
UPDATE buckets SET title='📋 Planifiée', updated=NOW() WHERE id=BUCKET_ID;
UPDATE buckets SET title='🔧 En cours', updated=NOW() WHERE id=BUCKET_ID;
UPDATE buckets SET title='✅ Terminée', updated=NOW() WHERE id=BUCKET_ID;

# Add extra bucket (must include updated=NOW() — NOT NULL constraint)
INSERT INTO buckets (title, project_view_id, created_by_id, created, updated)
VALUES ('🗑️ Abandonnée', VIEW_ID, USER_ID, NOW(), NOW());

Task Import

# Create task on project with bucket_id
curl -sk https://vikunja.example.com/api/v1/projects/PID/tasks -X PUT \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer JWT" \
  -d '{"title":"Task name","description":"...","bucket_id":BID,"priority":1}'

The bucket_id maps the task to its kanban column.

Key DB Schema Differences (v1.0.0-rc2 vs older)

Pitfalls

• Python SSL to Cloudflare-proxied Vikunja: urllib.request may timeout or return 403 behind Cloudflare. Use curl via subprocess instead.
• Bcrypt $2a vs $2b: Python’s bcrypt library generates $2b$ prefix. Vikunja (Go) expects $2a$. Replace: pwd_hash = "$2a" + pwd_hash[3:]
• Bucket API: /projects/{id}/buckets returns 404. Use /projects/{id}/views/{view_id}/buckets or DB directly.
• Dollar signs in SQL: bcrypt hashes contain $ which shell expands. Write SQL to a temp file and docker exec -i psql < file.sql.
• Login via curl works, Python urllib fails: Cloudflare SSL handling quirk. Wrap in subprocess calling curl.

See Also

• references/vikunja-v1-api.md — complete API workflow reference
• references/docker-db-direct-access.md — bypass expired JWT via Docker exec direct PostgreSQL queries (Coolify pattern)

I. 🔄 Vikunja ↔ Google Calendar Sync (3-Option Architecture)

Purpose

Synchroniser les tâches Vikunja avec due_date vers Google Calendar (et vice-versa), avec 3 solutions indépendantes servant de backup mutuel. L’idée : si une méthode tombe, les 2 autres tiennent.

Architecture 3 Options

┌──────────────────────────────────────────────────────┐
│                   VIKUNJA TASKS                       │
│  (due_date définie = visible dans calendrier)         │
└──────────┬──────────────────────┬────────────────────┘
           │                      │
           ▼                      ▼
┌──────────────────┐   ┌──────────────────────┐
│  Option A        │   │  Option C             │
│  (Prioritaire)   │   │  (CalDAV direct)      │
│                  │   │                       │
│  Python cron     │   │  Apple Calendar /     │
│  bidirectionnel  │   │  iPhone calendrier    │
│  toutes les 5min │   │  (lecture seule des   │
│                  │   │   tâches avec due_date)│
└────────┬─────────┘   └──────────────────────┘
         │
         ▼
┌──────────────────┐
│  Option B        │
│  (n8n fallback)  │
│                  │
│  Workflow n8n    │
│  visuel + logs   │
└──────────────────┘
         │
         ▼
┌──────────────────────────────────────────┐
│           GOOGLE CALENDAR                 │
│  (bfnous@gmail.com — événements créés)    │
└──────────────────────────────────────────┘

Option A — Python Cron Script (Prioritaire)

Script : /home/bf/scripts/vikunja-calendar-sync.py (créé si nécessaire) Cron : Toutes les 5 minutes State file : /home/bf/scripts/sync-state.json

Forward sync (Vikunja → Calendar) :

1. Lit toutes les tâches Vikunja avec due_date définie
2. Si tâche pas dans state → crée événement Google Calendar
3. Si tâche dans state mais date changée → update événement
4. Si tâche marquée done → supprime événement + retire du state
5. Marque les événements créés avec 🔁 Vikunja:{task_id} dans la description

Reverse sync (Calendar → Vikunja) :

1. Lit les événements Calendar des 7 prochains jours
2. Si événement sans task_id associé mais avec tag → crée tâche Vikunja

Logging 3 canaux :

• Fichier audit : /home/bf/central/audit/2026/05/ (section dédiée)
• Google Sheets : feuille “Log” avec [date, action, task_title, status]
• Vikunja : tâche de log dans projet cible avec label “Sync”

Flags : --dry-run pour mode simulation

Option B — n8n Workflow JSON

Fichier : /home/bf/scripts/vikunja-calendar-n8n-workflow.json Structure :

• Schedule Trigger (5 min) → HTTP GET Vikunja tasks → Filter → Google Calendar → Sheets log
• Importable directement dans l’interface n8n

Option C — CalDAV Apple Calendar

Endpoint CalDAV (vérifié) : https://vikunja.iatuto.com/caldav — HTTP 200 avec auth Auth : username + API token (comme mot de passe) Configuration :

• macOS : Calendrier → Fichier → Nouvel abonnement Calendrier → URL CalDAV
• iPhone : Réglages → Mots de passe & Comptes → Ajouter compte → Autre → Compte CalDAV

Pitfall important : Google Calendar web ne supporte PAS CalDAV natif. Utiliser Apple Calendar ou un client CalDAV tiers.

Endpoints API Vikunja (vérifiés)

Note : Les view IDs sont per-project (projet 7 a views 25-28, pas 1-4). Lire les vues du projet avant d’appeler l’endpoint tasks.

Pitfalls

• CalDAV 401 sur /dav/ : L’endpoint CalDAV fonctionne à /caldav, pas /dav/ ni /.well-known/caldav
• API token ≠ mot de passe CalDAV : Certaines instances Vikunja refusent le token comme password CalDAV. Solution : utiliser le mot de passe réel du compte Vikunja
• Concurrence cron : Le script 5min doit utiliser un fichier lock pour éviter les exécutions concurrentes (le script dure <1s normalement mais prévoir quand même)
• Due_date = 0001-01-01 : Vikunja renvoie cette date par défaut pour les tâches sans échéance. Filtrer avec due_date != '0001-01-01T00:00:00Z'
• PUT vs POST pour création tâche : Vikunja v1.0.0-rc2 utilise PUT (pas POST) pour créer des tâches via l’API

See Also

• references/vikunja-google-calendar-sync.md — full deployment reference, cron setup, state file format, real implementation details
• scripts/vikunja-calendar-sync.py — optional: copy to skill’s scripts/ dir

J. 🔑 Coolify API Token Injection (Direct Database)

When You Need It

Coolify’s web UI manages API tokens, but if you need programmatic API access and the UI doesn’t expose a token, you can create one directly in the database.

How It Works

Coolify uses Laravel Sanctum for API authentication. Tokens are stored as SHA-256 hashes in personal_access_tokens. The API auth header format is Bearer <id>|<plaintext>.

Procedure

import hashlib
plaintext = "my-secret-api-key"
hashed = hashlib.sha256(plaintext.encode()).hexdigest()

docker exec coolify-db psql -U coolify -c \
  "INSERT INTO personal_access_tokens (tokenable_type, tokenable_id, name, token, team_id, abilities, created_at, updated_at) VALUES ('App\\\\Models\\\\User', '0', 'my-token', 'HASHED_VALUE', '0', '[\"root\"]', NOW(), NOW());"

docker exec coolify-db psql -U coolify -t -A -c \
  "SELECT id FROM personal_access_tokens WHERE name='my-token';"

Database Layout

Pitfalls

• Missing team_id: Column has NOT NULL constraint. Always set team_id: '0'.
• SHA-256 hash: The token column stores the hash, but the client sends the plaintext. Bearer <id>|<plaintext>.
• Shell escaping: Single quotes inside abilities can break bash. Write SQL to a temp file.

Python Environment

• Problem: ModuleNotFoundError with system Python
• Solution: Use python3.10 for gspread/google-auth
• Check: python3.10 -c "import gspread; print('OK')"

Logging Permissions

• Problem: Permission denied to /var/log/
• Solution: Use local ./logs/ directory
• Code: os.makedirs(os.path.join(os.path.dirname(__file__), 'logs'), exist_ok=True)

Verification

• Problem: 404 on /api/status
• Solution: Test root / endpoint first
• Verification: curl -s -f "https://service.example.com/"

K. 🖥️ VPS System Diagnostic & Maintenance

Purpose

Systematic approach to VPS health diagnostics and routine maintenance: detect performance issues, clean wasted resources, and identify service conflicts (duplicate instances, port collisions).

Diagnostic Workflow

1️⃣ SNAPSHOT  →  2️⃣ REPORT  →  3️⃣ WAIT  →  4️⃣ ACT  →  5️⃣ VERIFY  →  6️⃣ REPORT

Step 1 — Data Collection (SNAPSHOT):

Collect these signals in parallel (stateless, read-only):

# System health
hostnamectl && free -h && df -h / && uptime && cat /proc/loadavg && nproc
# Log sizes (top consumers)
sudo du -sh /var/log/* | sort -rh | head -10
# Docker overview
docker ps -a && docker system df
# Docker volumes — check dangling count
docker volume ls -f dangling=true -q | wc -l
# Running processes by CPU/MEM
ps aux --sort=-%cpu | head -12 && ps aux --sort=-%mem | head -8
# Ports & network
ss -tlnp
# Systemd/Docker service overlap (detect duplicates)
# bracket trick: grep "[f]oo" matches "foo" but not "grep foo"
ps aux | grep "\[<service_name>" | sed 's/\\[//'
systemctl --user list-units --type=service --state=running | grep -i "<service_name>"
docker ps --format '{{.ID}} {{.Names}} {{.Status}}' | grep -i "<service_name>"
# Errors
systemctl list-units --type=service --state=failed
journalctl -p err -b -n 20

Step 2 — Diagnostic Report: Present findings with overall health (✅/🟠/🔴), data tables, flagged anomalies, and prioritised recommendations.

Step 3 — Wait for direction: Present report, do not act until user chooses a path.

Step 4 — Act: Execute agreed actions.

Step 5 — Verify: Re-run key metrics, confirm state changed.

Step 6 — Final Report: Before/after comparison + remaining items.

Docker Cleanup Nuances

# SEE what would be freed first
docker system df   # check "Reclaimable" column per type

# REMOVE everything truly unused
docker container prune -f          # 1. remove stopped containers FIRST
docker system prune --volumes -f   # 2. then remove dangling volumes

Critical: docker system prune --volumes -f only removes volumes NOT referenced by ANY container — including stopped ones. Volumes attached to stopped containers (even dangling ones) are NOT removed without step 1. Skipping step 1 can leave 50+ volumes (several GB) untouched.

Duplicate Service Discovery

When you suspect a duplicate or conflicting instance:

1. Find ALL process instances: ps aux | grep "[f]oo" (bracket trick replaces grep [f]oo with PID-less match)
2. Check NATIVE (systemd): systemctl --user list-units | grep -i "<service>" plus /etc/systemd/system/
3. Check DOCKER: docker ps --format '{{.ID}} {{.Names}} {{.Status}} {{.Ports}}' | grep -i "<service>"
4. Cross-reference by PID: Match PID to Docker (/proc/<PID>/cgroup) or systemd (systemctl --user status <service>)
5. Compare start times: ps -o pid,lstart,cmd -p <PID> to see launch order and mechanism
6. Assess management overlap: Coolify-managed Docker vs systemd user service = different deployment paths, not a bug

Pitfalls

• sudo du -sh /var/lib/docker timeout: Blocks 15+ seconds behind SSH. Use docker system df (instant).
• grep -v grep can miss PIDs: Use bracket trick — ps aux | grep "[h]ermes" matches “hermes” but not the grep command itself.
• Gateway duplicates found by port, not by name: Always cross-reference with ss -tlnp | grep <PORT>.
• monit summary returns empty if monit isn’t running: Confirm systemctl status monit first.

See Also

• references/vps-system-diagnostic.md — full session data with exact commands, output, cleanup results, and pitfalls

📋 Integration Checklist

Pre-Integration

• Verify service accessibility (curl -f)
• Check Python environment and modules
• Confirm authentication tokens
• Validate logging permissions

During Integration

• Configure correct endpoints
• Set up proper error handling
• Implement logging with timestamps
• Test cron job syntax

Post-Integration

• Run full synchronization
• Verify monitoring functionality
• Check dashboard updates
• Review automated workflows

📚 References

Infrastructure Hardening

• references/port-service-audit.md — Port/service discovery workflow for VPS with Coolify + Docker + Traefik
• references/ssh-ufw-fail2ban-hardening.md — SSH hardening with UFW + fail2ban

DNS Automation

• references/cloudflare-dns-automation.md — Full script and Cloudflare API integration

Coolify Server Setup

• references/coolify-server-setup.md — Server configuration and “no available server” fix
• references/coolify-deployment-lessons.md — Deployment lessons and verification patterns

Dashboard Monitoring & Development

• references/dashboard-monitoring-coolify.md — Comprehensive Coolify dashboard guide (French)
• references/dashboard-monitoring-coolify-flat.md — English version of dashboard setup
• references/dashboard-monitoring-complet.md — High-level monitoring methodology
• references/dashboard-dev-patterns.md — Development architecture and UI patterns
• references/dashboard-vps-config.md — VPS remote access configuration
• references/coolify-multi-projects.md — Multi-project Coolify API technique
• references/coolify-api-implementation.md — Real-world API implementation notes

Hermes Config Replication

• references/hermes-config-replication.md — Repliquer la config Hermès d’une machine (mx) à une autre (VPS) : diagnostic openai-codex brické, fetch SSH, adaptation terminal.cwd, contournement des guards, vérification. Voir la procedure quand le provider principal est configuré partout mais sans credentials.

Session Reference

• references/session-implementation-guide.md — Session-specific implementation notes
• references/docker-db-direct-access.md — Docker exec direct PostgreSQL queries (Coolify services, Vikunja bypass pattern)

Scripts

• scripts/coolify-deploy-direct.sh — Direct deployment to Coolify
• scripts/coolify-verify-deployment.sh — Deployment verification
• scripts/deploy-monitoring-dashboard.sh — Automated dashboard deployment
• scripts/dashboard-verify-setup.sh — Dashboard setup verification
• scripts/log-change-sheets.py — Google Sheet logger (OAuth Device Flow, no gspread needed)

Change Management

• references/change-management-sheet-auth.md — OAuth Device Flow auth pattern for Google Sheets logging

Templates

• templates/coolify-docker-compose.prod.yml — Production Docker compose
• templates/dashboard-docker-compose.yml — Dashboard Docker compose for Coolify
• templates/dashboard-package.json.template — Node.js package template
• templates/dashboard-server.js.template — Express server template
• templates/dashboard-start.sh.template — Startup script template

🔗 Related Skills

• coolify-gateway — POLITIQUE OPÉRATIONNELLE (complémentaire). Ce skill (infrastructure-integration) dit COMMENT configurer. coolify-gateway dit QUOI faire et QUOI interdire (règles, audits, notifications, backups, protocole d’intervention). Toujours charger les deux ensemble pour une intervention complète.
• integration-task-management-spreadsheets — Vikunja + Google Sheets integration
• paperclip-ai-server-management — Paperclip AI deployment with PM2
• github — Repository management and CI/CD
• mlops — Model serving and API optimization