Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
285 lines (233 loc) · 9.59 KB

ARCHITECTURE.md

File metadata and controls

285 lines (233 loc) · 9.59 KB
Raw

Internal Architecture

Component Overview

┌─────────────────────────────────────────────────────┐
│                  Docker Container                   │
│                                                     │
│  ┌─────────┐                                       │
│  │  tini   │  (PID 1 - Init system)               │
│  └────┬────┘                                       │
│       │                                             │
│       ├──> entrypoint.sh (Setup logic)            │
│       │                                             │
│       └──> ┌──────────────────────────────────┐   │
│            │      Supervisor (PID 2)          │   │
│            │                                  │   │
│            │  ┌────────────────────────────┐ │   │
│            │  │ corosync-qnetd (Priority 10)│ │   │
│            │  │ - Port 5403                  │ │   │
│            │  │ - User: coroqnetd           │ │   │
│            │  │ - Autorestart: true         │ │   │
│            │  └────────────────────────────┘ │   │
│            │                                  │   │
│            │  ┌────────────────────────────┐ │   │
│            │  │ sshd (Priority 20) [*]     │ │   │
│            │  │ - Port 22                   │ │   │
│            │  │ - Setup mode only           │ │   │
│            │  │ - Autorestart: true         │ │   │
│            │  └────────────────────────────┘ │   │
│            │                                  │   │
│            │  ┌────────────────────────────┐ │   │
│            │  │ auto_lock.sh (Priority 30) [*]│  │
│            │  │ - Monitors cluster          │ │   │
│            │  │ - Setup mode only           │ │   │
│            │  │ - Autorestart: false        │ │   │
│            │  └────────────────────────────┘ │   │
│            └──────────────────────────────────┘   │
│                                                     │
│  [*] Only loaded when /data/SSH_DISABLED absent   │
└─────────────────────────────────────────────────────┘

Startup Flow

First Boot (Setup Mode)

  1. tini starts as PID 1
  2. entrypoint.sh detects no /data/SSH_DISABLED
  3. Generates/restores SSH host keys
  4. Sets root password from ROOT_PASSWORD
  5. Enables sshd.conf and autolock.conf
  6. Launches Supervisor with 3 processes
  7. Waits for Proxmox connection...

During pvecm qdevice setup

  1. Proxmox connects via SSH (port 22)
  2. Exchanges certificates with QNetd
  3. Cluster appears in corosync-qnetd-tool -l
  4. auto_lock.sh detects cluster (threshold check)
  5. Waits AUTO_LOCK_WAIT seconds
  6. Creates /data/SSH_DISABLED lock file
  7. Stops SSH via supervisorctl
  8. Removes sshd.conf

Subsequent Boots (Secure Mode)

  1. tini starts as PID 1
  2. entrypoint.sh detects /data/SSH_DISABLED
  3. Skips SSH setup entirely
  4. Only enables qnetd.conf
  5. Launches Supervisor with 1 process (qnetd only)

Security Layers

Layer Protection Details
Temporal Auto-disable SSH Active max 2-5 min
Persistent Lock file Survives reboots
Process Minimal attack surface Only QNetd runs after setup
Network Port isolation 5403 only after setup
Init Proper signal handling Tini prevents zombie processes

Data Persistence

persistent_data/
├── SSH_DISABLED          # Lock file (empty, presence-based)
├── ssh/                  # SSH host keys
│   ├── ssh_host_rsa_key
│   ├── ssh_host_rsa_key.pub
│   ├── ssh_host_ecdsa_key
│   ├── ssh_host_ecdsa_key.pub
│   ├── ssh_host_ed25519_key
│   └── ssh_host_ed25519_key.pub
└── corosync/             # QNetd database
    ├── nssdb/
    │   ├── cert9.db
    │   ├── key4.db
    │   └── pkcs11.txt
    └── qnetd-cluster-*.crt

Auto-lock Algorithm

DETECTION_COUNT = 0
THRESHOLD = 3  # Configurable via AUTO_LOCK_THRESHOLD

while True:
    if cluster_detected():
        DETECTION_COUNT += 1

        if DETECTION_COUNT >= THRESHOLD:
            wait(WAIT_SECONDS)

            if cluster_still_detected():
                create_lock()
                stop_ssh()
                exit(0)
            else:
                # Cluster lost during wait
                DETECTION_COUNT = 0
    else:
        # Reset counter on disconnection
        if DETECTION_COUNT > 0:
            DETECTION_COUNT = 0

    sleep(10)

Why threshold-based detection?

This prevents false positives during:

  • Initial certificate exchange (transient cluster states)
  • Network hiccups
  • Corosync service restarts on Proxmox nodes

The default threshold of 3 detections = 30 seconds of stable cluster presence (3 × 10s check interval).

Signal Handling

SIGTERM/SIGINT → tini → Supervisor → Graceful shutdown
                           ├─> corosync-qnetd (SIGTERM)
                           ├─> sshd (SIGTERM)
                           └─> auto_lock.sh (SIGTERM)

Tini ensures:

  • Proper signal forwarding to child processes
  • Zombie process reaping
  • Clean container stops

Network Ports

Port Protocol Service Availability
22 TCP SSH Setup mode only (auto-disabled)
5403 TCP QNetd Always (persistent)

File System Layout

/
├── data/                      # Persistent volume mount
│   ├── SSH_DISABLED           # Lock file
│   ├── ssh/                   # SSH keys backup
│   └── corosync/              # QNetd data (symlinked)
│
├── etc/
│   ├── corosync/
│   │   └── qnetd/             # Symlink → /data/corosync
│   │
│   ├── ssh/                   # SSH host keys (restored from /data)
│   │
│   └── supervisor/
│       ├── supervisord.conf
│       └── conf.d/
│           ├── qnetd.conf              # Always loaded
│           ├── sshd.conf.disabled      # Template
│           ├── sshd.conf               # Created in setup mode
│           ├── autolock.conf.disabled  # Template
│           └── autolock.conf           # Created in setup mode
│
├── auto_lock.sh               # Auto-lock monitor script
├── entrypoint.sh              # Container init script
├── healthcheck.sh             # Docker healthcheck
│
└── var/
    └── run/
        ├── corosync-qnetd/    # QNetd runtime
        └── sshd/              # SSH runtime (setup only)

Environment Variables

Variable Default Description
ROOT_PASSWORD - SSH root password (setup only)
AUTO_LOCK_WAIT 120 Seconds to wait before locking after detection
AUTO_LOCK_THRESHOLD 3 Number of successive cluster detections required
TZ Europe/Paris Timezone for logs

Healthcheck Logic

# Every 30 seconds:

1. Check if QNetd responds
   ├─> FAIL → Container unhealthy
   └─> OK → Continue

2. If locked mode: warn if no cluster
   └─> WARN (but don't fail)

3. Check if qnetd process exists
   ├─> FAIL → Container unhealthy
   └─> OK → Container healthy

The healthcheck allows Docker to automatically restart the container if QNetd crashes.

Recovery Scenarios

Scenario 1: Accidental lock file deletion while running

# Container remains secure - SSH already stopped
# Lock file only checked on container start

Scenario 2: Force re-setup

rm persistent_data/SSH_DISABLED
docker restart proxmox-qdevice
# Container enters setup mode again

Scenario 3: All Proxmox nodes reboot

# QNetd continues running (no dependency on cluster)
# Nodes reconnect automatically
# No intervention needed

Scenario 4: Container restart in locked mode

# entrypoint.sh detects lock file
# Skips SSH entirely
# Only starts QNetd
# Full security maintained

Logging

All services log to Docker stdout/stderr:

[INFO]    entrypoint.sh logs (container startup)
[AUTOLOCK] auto_lock.sh logs (cluster monitoring)
[qnetd]   corosync-qnetd logs (voting activity)
[sshd]    SSH connection logs (setup only)

View logs:

docker logs proxmox-qdevice -f
docker logs proxmox-qdevice | grep AUTOLOCK

Comparison: Before vs After Setup

Aspect Setup Mode Secure Mode
SSH Port 22 ✅ Open 🔒 Closed
QNetd Port 5403 ✅ Open ✅ Open
Running processes 3 1
Attack surface Temporary (2-5 min) Minimal
Auto-lock monitor ✅ Active ❌ Disabled
Lock file ❌ Absent ✅ Present