Cron Job Maintenance — Diagnostic & Repair Workflow

When to Use

Use this skill when:

• One or more Hermes cron jobs show “error” status in cronjob(action='list')
• A cron job’s output is empty, truncated, or unexpected
• Jobs stopped running after an update (token rotation, script change, SSH key change)
• You need to audit all active cron jobs for health

Do NOT use for:

• First-time setup of new cron jobs (use cronjob action=create directly)
• General system monitoring (use dedicated monitoring/alerting)

Diagnostic Workflow

Step 1: List all jobs and identify failures

cronjob(action='list')

Look for jobs where status is anything other than ok. Note:

• status: unknown — job hasn’t run yet (first creation)
• status: running — currently executing, wait for it
• status: error — last run exited non-zero, needs investigation

Step 2: Capture the error output

Force-run the failing job to capture live output:

cronjob(action='run', job_id='<id>')

This runs the job immediately and returns its output. Key observations:

• Script output — any Python traceback, error message, or non-zero exit
• Silent failure — empty output with non-zero exit often means SSH timeout or script not found
• SSH errors — “Connection refused”, “Connection timeout”, “Host key verification failed”
• Auth errors — “Invalid Credentials”, “Token expired”, “401”, “403”

Step 3: Root cause categories

Work through these in order:

Z. Stale lock files blocking execution

When multiple cron jobs fail at the exact same timestamp, check for stale lock files.

# Check Hermes lock files
ls -la ~/.hermes/*.lock
ls -la ~/.hermes/native/*.lock 2>/dev/null
# Check /tmp lock files
ls -la /tmp/*.lock 2>/dev/null

If a lock file exists from a previous run that crashed (e.g., after an OOM or abrupt system shutdown), it will block ALL jobs that use the same locking mechanism. The symptom is identical timestamps on N failed jobs.

Fix: remove the stale lock files and re-run the jobs:

rm -f ~/.hermes/auth.lock ~/.hermes/gateway.lock

Also check for failover runner locks that may be blocking VPS task wrappers:

ls -la /tmp/failover-runner-*.lock 2>/dev/null

Before running rm, verify the lock is truly stale — check its modification time against the last successful job run. A lock from 2+ hours ago with no running python3 or ssh process is safe to remove. A fresh lock (seconds old) may indicate an active job.

A. Script/Command not found

The script path doesn’t exist or symlinks were rejected.

# Verify the script exists at the exact path
ls -la ~/.hermes/scripts/<relative-path>
# Note: Heremss rejects symlinks — use real files only

B. Exit code ≠ 0 but execution was successful

Many diagnostic scripts exit 1 (warnings) or 2 (non-fatal errors). Cron interprets any non-zero as failure.

Fix: Wrap the script to filter expected non-zero exits:

#!/bin/bash
python3 some-scanner.py 2>&1
rc=$?
if [ $rc -eq 1 ] || [ $rc -eq 2 ]; then
  exit 0
fi
exit $rc

C. SSH / network failure (transient)

If the script works when run locally but the cron shows error, it could be SSH congestion.

# Test SSH connectivity
ssh -q -o ConnectTimeout=10 -p <port> user@host exit && echo "SSH OK"

# Force-run on VPS directly
ssh -p <port> user@host "bash ~/path/to/script.sh"

Common causes:

• VPS under high load (swap thrashing, high I/O wait)
• SSH ControlMaster socket stale
• Network congestion / packet loss
• MaxStartups limit on VPS sshd

Fix: Add SSH retry in the wrapper, reduce frequency, or fix the underlying VPS load issue.

D. Token / credential expiry

Google OAuth tokens expire and need periodic re-authentication.

# Check token expiry
python3 -c "import json; d=json.load(open('path/to/token.json')); print(d.get('expiry','N/A'))"

# Test if gspread can authenticate
python3 -c "
import gspread
gc = gspread.oauth(credentials_filename='token.json')
sh = gc.open_by_key('<spreadsheet-id>')
print(f'OK: found {len(sh.worksheets())} worksheets')
"

Token expired → needs re-auth via OAuth flow (user interaction required).

E. Script bug / missing dependency

# Test the script standalone
bash ~/.hermes/scripts/path/to/wrapper.sh 2>&1

# Check Python syntax
python3 -c "import ast; ast.parse(open('script.py').read()); print('Syntax OK')"

# Check imports
python3 -c "import gspread, requests" 2>&1

Step 4: Fix and verify

After applying the fix:

1. Re-run the job manually:

   cronjob(action='run', job_id='<id>')

2. Verify the output shows success and exit code 0

3. Wait for the next scheduled run (or force one) to confirm cron accepts the status

4. Update the shared log:

   # Add to ~/shared_log.md
   [DATE] | <instance> | <agent> | cron-fix:<job-name> | COMPLETED | <git-hash-if-applicable>

Pitfalls

Google OAuth tokens expire silently

A cron job that calls Google APIs (Drive, Sheets) will fail with exit code 1 when the OAuth token in ~/.hermes/google_token.json expires. The error in shared_log.md is typically a 401 or credential error. Fix: generate a new token via PKCE OAuth flow — see references/google-oauth-reauth.md. On a headless VPS, use the copy-paste method (not a local server callback): the script prints the OAuth URL, the user opens it in their local browser, authorizes, then pastes the ?code=... from the resulting redirect URL back into the terminal. A reusable template is at templates/reauth-google-copypaste.py — copy to the VPS via base64 transfer (see references/vps-script-corruption-transfer.md).

Script transfer via SSH corrupts complex Python content

When a cron job needs a replacement script pushed to the VPS, scp and heredocs in SSH commands corrupt f-strings, dollar signs, quotes, and backslashes. The symptom is a SyntaxError on the remote side that does not exist locally. Solution: transfer via base64 encoding — see references/vps-script-corruption-transfer.md.

Verification Checklist

After any fix:

• cronjob(action='run', job_id='<id>') returns exit 0 and expected output
• VPS-requiring jobs: ssh user@host "bash script.sh" works standalone
• Token-auth jobs: gspread/Google API auth test passes
• Wrapper scripts: bash wrapper.sh; echo $? prints 0
• Cron schedule is appropriate (not too frequent for load-sensitive scripts)
• Shared log updated with fix summary

Concrete Session Reference

See references/session-20260608-cron-diagnostic.md for a worked example diagnosing 5 concurrent cron failures across local and VPS jobs — exit code filtering, SPREADSHEET_ID errors, OAuth token expiry, and SSH congestion false positives.

See references/multi-sheet-dashboard-diagnostic.md for diagnosing when a user says “the dashboard stopped updating” — the script writes to one sheet but the user watches another. Covers programmatic sheet enumeration and script-to-sheet mapping.