Skip to content

Standardize OTel Collector agent binary install for external nodes #23

Description

@castrojo

[WHAT] Harden just setup-otel-agent HOST=... as the canonical, well-documented way to instrument nodes that are outside the k3s cluster (contributor laptops, exo-dakota, any external machine).

[WHY] External nodes are not k3s cluster members — a Kubernetes DaemonSet cannot reach them. The binary systemd agent approach already works correctly and handles journald access, which a containerized agent cannot (journald volume mounts fail in distroless containers, and journal paths vary on Bluefin/ostree systems). This issue aligns versions and documents the two-tier model clearly.

[FIX]

  • Pin OTELCOL_VERSION as a single variable in Justfile — aggregator and agent must use the same version
  • Update agent endpoint config from localhost:4317 to the aggregator's address (ghost IP or k8s Service address once k3s is running)
  • Add just otel-agent-status HOST=... check recipe (exits nonzero on failure)
  • Document the two-tier observability model in README:
    • In-cluster nodes: future DaemonSet (when additional machines join the k3s cluster)
    • External nodes (today): just setup-otel-agent HOST=... binary systemd service
  • Verify exo-dakota agent is still sending data after aggregator moves to k3s

[NEXT] Do NOT convert the binary agent to a DaemonSet — containerized agents cannot reliably read journald files. Do NOT remove deploy-agent.sh — it is the correct mechanism for external nodes and is not replaced by any k3s feature.

Blocked by: #22 (OTel aggregator must be running in k3s before re-pointing agents)

Acceptance criteria:

  • OTELCOL_VERSION single variable controls both aggregator and agent versions
  • just setup-otel-agent HOST=jorge@192.168.1.247 deploys agent pointing at k3s aggregator
  • exo-dakota metrics and logs visible in Perses after migration
  • Two-tier model documented in README

Metadata

Metadata

Assignees

No one assigned

    Labels

    1-triageNew work awaiting human triage.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions