Skip to content

architecture.md

Latest commit

 

History

History
114 lines (86 loc) · 3.68 KB

architecture.md

File metadata and controls

114 lines (86 loc) · 3.68 KB
Raw

Project Architecture

General Overview

The project is built as a console orchestrator over a single application module, offline_llm_chat.

The run.py entry point does not contain model logic. It is responsible only for displaying the main menu and launching two scenarios:

  • Offline AI Chat;
  • System Analyzer.

When a scenario is selected, a dedicated Python entry point inside apps/offline_llm_chat/ is executed.

Module Composition

run.py

The main project entry point.

Functions:

  • display the menu;
  • start the selected module as a separate Python process;
  • verify that the entrypoint file exists on disk.

apps/offline_llm_chat/cli.py

A thin CLI layer for the Offline AI Chat user scenario.

Functions:

  • connect the library layer from lib/;
  • run the interactive model workflow.

apps/offline_llm_chat/system_analyzer_cli.py

CLI layer for the System Analyzer scenario.

Functions:

  • run host analysis;
  • display recommendations for .env parameters;
  • write the selected parameters to the configuration file.

apps/offline_llm_chat/lib/offline_llm_agent/

The main library layer of the project.

Key modules:

  • model_catalog.py — loading and validation of the model catalog;
  • download.py — downloading GGUF files and SHA256 verification;
  • docker_runtime.py — build context preparation, image build, and container launch;
  • chat.py — smoke test, message exchange, and interactive chat;
  • unix_http.py — minimal HTTP client over a Unix socket;
  • system_analyzer.py — host parameter collection and recommendation calculation;
  • settings.py — project path resolution and .env loading.

_work-models/

Catalog of models and local GGUF files.

Contents:

  • catalog.json — description of available models, files, URLs, SHA256 values, and runtime parameters;
  • models/<model_key>/ — local files for a specific model.

.runtime/offline-ai-chat/

A service runtime directory created automatically at startup.

Purpose:

  • Unix socket directory;
  • temporary build context for docker build.

This directory is not used as a user-facing results store.

Execution Flow

Offline AI Chat Scenario

flowchart TD
    A[run.py] --> B[apps/offline_llm_chat/cli.py]
    B --> C[Read catalog.json]
    C --> D[Model selection]
    D --> E[Check local GGUF files]
    E --> F[Download missing files]
    F --> G[Verify SHA256]
    G --> H[Create build context in .runtime]
    H --> I[Docker build]
    I --> J[Docker run]
    J --> K[Create Unix socket]
    K --> L[Check /v1/models]
    L --> M[Smoke test]
    M --> N[Interactive CLI chat]
Loading

System Analyzer Scenario

flowchart TD
    A[run.py] --> B[apps/offline_llm_chat/system_analyzer_cli.py]
    B --> C[Collect data about CPU RAM Swap disk GPU]
    C --> D[Generate startup profiles]
    D --> E[Select a profile]
    E --> F[Write parameters to .env]
Loading

Architectural Decisions

1. Baking the model into the image

Model files are not mounted into the runtime container from the host on every launch. During the build stage, the project creates a temporary build context and includes the model in the Docker image via COPY models/ /models/.

Practical effect:

  • the runtime becomes self-contained;
  • the number of host mounts is reduced;
  • the model is fixed within a specific image.

2. Communication over a Unix socket

llama.cpp does not publish a TCP port to the host. Communication is performed through the Unix socket /sock/llama.sock, which is mounted into the container via a bind mount.

Practical effect:

  • there is no external network API;
  • the network attack surface is reduced;
  • the local client communicates directly through a filesystem socket.