Netreka Akademi: Microservices Hub

Advanced Distributed System Architecture

A state-of-the-art microservices implementation demonstrating Broker Pattern, Asynchronous Messaging, gRPC Communication, and Distributed Logging.

---

📖 About The Project

Netreka Microservices Hub is a production-grade reference architecture designed to demonstrate how to build, deploy, and manage a robust distributed system using Go.

Key capabilities:

• Unified Gateway: Uses Caddy as a reverse proxy for secure, single-entry point access.
• Asynchronous Tasks: Non-blocking operations via RabbitMQ.
• Distributed Logging: Centralized log aggregation via gRPC and MongoDB.

---

Key Features

• Microservices Architecture: Fully decoupled services for Auth, Logging, Mail, and Listeners.
• Event-Driven Design: Uses RabbitMQ for asynchronous, non-blocking communication.
• High Performance: Critical paths utilize gRPC for low-latency inter-service calls.
• Docker Swarm Ready: Includes production-grade orchestration configurations (swarm.yml).
• Unified Gateway: Single entry point handling routing and protocol translation via Broker.
• Polyglot Persistence: Demonstrates optimal database usage with PostgreSQL (Relational) and MongoDB (NoSQL).

---

1. Project Structure

├── authentication-service  // User Identity & Auth (PostgreSQL)
├── broker-service          // API Gateway & Router (HTTP/REST)
├── front-end               // Web UI Dashboard (Go Templates)
├── listener-service        // Async Event Consumer (RabbitMQ)
├── logger-service          // Centralized Logging (MongoDB + gRPC)
├── mail-service            // Email Sandbox (MailHog)
├── project                 // Infrastructure (prop. `micro-caddy`, Docker, Make)
└── proto                   // gRPC Protocol Buffers Definitions

2. System Architecture

The system is designed as a decoupled cluster of independent services. The Broker Service acts as the API Gateway, handling ingress traffic and routing requests to specialized services via HTTP (REST), gRPC, or AMQP (Message Queue).

graph TD
    User((User)) -->|HTTPS/80| Caddy[Caddy Reverse Proxy]
    Caddy -->|HTTP/8081| FE[Frontend Service]
    FE -->|JSON/HTTP| Gateway[Broker Service / API Gateway]

    subgraph "Synchronous Layer"
        Gateway -->|HTTP| Auth[Authentication Service]
        Gateway -->|HTTP| Mail[Mail Service]
        Gateway -->|HTTP| LogHTTP[Logger Service]
        Gateway -->|gRPC/Protobuf| LogGRPC[Logger Service gRPC]
    end

    subgraph "Asynchronous Layer"
        Gateway -->|Publish Event| Rabbit{RabbitMQ}
        Rabbit -->|Consume Event| Listener[Listener Service]
        Listener -->|Async Log| LogHTTP
    end

    subgraph "Data Persistence"
        Auth -->|SQL| PG[(PostgreSQL)]
        LogHTTP -->|NoSQL| Mongo[(MongoDB)]
    end

Loading

---

3. Unified Request Flow (Step-by-Step Visualization)

This diagram visualizes exactly what happens when a user clicks "Authenticate". It traces the main request, the database lookup, the error handling, and the two parallel logging strategies (HTTP vs gRPC).

sequenceDiagram
    autonumber
    participant U as User (FrontEnd)
    participant B as Broker (Gateway)
    participant A as Auth Service
    participant P as PostgreSQL
    participant R as RabbitMQ
    participant L as Listener
    participant G as Logger (gRPC)

    Note over U, G: SCENARIO: User attempts to Login

    %% Step 1: Authentication Request
    U->>B: 1. POST /handle ("action": "auth")
    activate B
    
    B->>A: 2. POST /authenticate
    activate A
    A->>P: 3. Query User
    P-->>A: Match Found
    
    alt Login Successful
        A-->>B: 202 Accepted
    else Login Failed
        A-->>B: 401 Unauthorized
    end
    deactivate A

    %% Step 2: Asynchronous Logging (RabbitMQ)
    Note right of B: Log the attempt asynchronously
    B->>R: 4. Publish Event ("log.INFO")
    
    %% Step 3: High-Speed Logging (gRPC) - Parallel
    Note right of B: Critical System Log via gRPC
    B->>G: 5. RPC Call: WriteLog()
    activate G
    G->>G: Write to Mongo (Direct)
    G-->>B: Ack
    deactivate G

    B-->>U: 6. Final JSON Response
    deactivate B

    %% Step 4: Background Processing
    R->>L: 7. Push Message to Listener
    activate L
    L->>L: Processing Logic...
    deactivate L

Loading

---

4. Request Timeline & Latency Analysis

The following diagram illustrates the Authentication Flow on a timeline, highlighting the synchronous blocking nature of the request and the estimated latency at each hop.

sequenceDiagram
    autonumber
    participant Client as Web Client
    participant Broker as Broker Gateway
    participant Auth as Auth Service
    participant DB as PostgreSQL
    participant Logger as Logger Service

    Note over Client, Broker: Total Latency Target: < 200ms

    Client->>Broker: POST /handle (action: auth)
    activate Broker
    
    Note right of Broker: T+5ms: JSON Decoding
    
    Broker->>Auth: POST /authenticate
    activate Auth
    
    Note right of Auth: T+15ms: Validation
    
    Auth->>DB: Query User By Email
    activate DB
    DB-->>Auth: Result (User Found)
    deactivate DB
    Note right of Auth: T+45ms: Password Hash Compare
    
    alt Invalid Credentials
        Auth-->>Broker: 401 Unauthorized
        Broker->>Logger: Async Log (Auth Failed)
    else Valid Credentials
        Auth-->>Broker: 202 Accepted
    end
    deactivate Auth
    
    Broker-->>Client: JSON Response
    deactivate Broker
    Note over Client: T+80ms: UI Update

Loading

---

5. Protocol & API Reference (Technical Deep Dive)

3.1 REST API (Broker Service)

Endpoint: POST /v1/handle
Description: The primary ingress point. Routes requests based on the action field.

Payload Schema:

{
  "action": "auth", // required: [auth, log, mail, broker]
  "auth": {
    "email": "admin@example.com",
    "password": "password"
  },
  "log": {
    "name": "event_name",
    "data": "description"
  },
  "mail": {
    "to": "user@example.com",
    "subject": "Hello",
    "message": "Body"
  }
}

Implementation Details (Go/Chi): The router uses a switch statement to delegate business logic. This avoids "Controller Bloat" by keeping the handler focused solely on routing.

// broker-service/cmd/api/handlers.go
func (app *Config) HandleSubmission(w http.ResponseWriter, r *http.Request) {
    // ... decode json ...
    switch requestPayload.Action {
    case "auth":
        app.authenticate(w, r, requestPayload.Auth)
    case "log":
        app.logItem(w, requestPayload.Log) // HTTP Handler
    // ...
    }
}

---

3.2 gRPC Protocol (Logger Service)

Purpose: High-throughput, low-latency logging where HTTP overhead is unacceptable. Definition: proto/logs/v1/logs.proto

syntax = "proto3";
package logs;

message LogRequest {
  Log log_entry = 1;
}

message LogResponse {
  string result = 1;
}

service LogService {
  rpc WriteLog(LogRequest) returns (LogResponse);
}

Server Implementation (Logger): The logger service implements the generated UnimplementedLogServiceServer interface.

// logger-service/cmd/api/grpc.go
func (l *LogServer) WriteLog(ctx context.Context, req *logsv1.LogRequest) (*logsv1.LogResponse, error) {
    input := req.GetLogEntry()
    // Direct MongoDB Insertion (Bypassing HTTP middleware)
    logEntry := data.LogEntry{
        Name: input.Name,
        Data: input.Data,
    }
    err := logEntry.Insert() 
    return &logsv1.LogResponse{Result: "logged"}, nil
}

3.3 Asynchronous Event Bus (RabbitMQ)

Exchange Type: topic (Exchange Name: logs_topic) Binding Keys: log.INFO, log.ERROR

Consumer Implementation (Listener): The listener uses a Push-based model with Manual Acknowledgement. This ensures that if the service crashes while processing a message, the message is returned to the queue and not lost.

Correction: We set autoAck to false. This requires us to explicitly call d.Ack(false) after successful processing.

// listener-service/event/consumer.go
func (consumer *Consumer) Listen(topics []string) error {
    // ... setup exchange & queue declarations ...

    // Start Consuming (AutoAck = false for reliability)
    messages, err := ch.Consume(q.Name, "", false, false, false, false, nil)
    if err != nil {
        return err
    }

    // Non-blocking processing loop
    go func() {
        for d := range messages {
            var payload Payload
            _ = json.Unmarshal(d.Body, &payload)
            
            // Execute Business Logic
            err := handlePayload(payload)
            
            if err != nil {
                // Negative Acknowledge if failed (don't requeue loop)
                _ = d.Nack(false, true)
            } else {
                // Acknowledge logic completion
                _ = d.Ack(false)
            }
        }
    }()
    
    // Block forever using a channel
    <-forever
    return nil
}

3.4 RabbitMQ Deep Dive: Why do we use it?

The following diagram and points explain exactly what value RabbitMQ adds to this code architecture, distinguishing it from a simple HTTP call.

graph LR
    subgraph "Producer (Broker-Service)"
        Metrics[Response Time: <50ms] 
        Handler[HandleSubmission] -->|1. Fire Event| Exchange{Exchange: logs_topic}
        style Metrics fill:#e1f5fe,stroke:#01579b
    end

    subgraph "RabbitMQ Server"
        Exchange -->|2. Routing Key: 'log.INFO'| Queue[(Queue: Durable)]
        style Queue fill:#fff9c4,stroke:#fbc02d
    end

    subgraph "Consumer (Listener-Service)"
        Queue -->|3. Push Msg| Worker[Listener Worker]
        Worker -.->|4. Manual Ack| Queue
        Worker -->|5. Slow Operation| DB[Logger / IO]
    end

Loading

What this provides to the code:

1. Asynchronous Decoupling (Fire & Forget):

   • Without Rabbit: The Broker would have to wait for the Logger service to write to MongoDB before sending a response to the user. If MongoDB is slow, the user waits.
   • With Rabbit: The Broker "fires" the event to the Exchange (taking ~2ms) and immediately returns 202 Accepted to the user. The heavy lifting happens in the background.

2. Reliability & Data Safety (Durability):

   • The Manual Acknowledgement (d.Ack(false)) mechanism we verified ensures that if the listener-service crashes while processing a log, the message is not lost. It stays in the queue and is delivered to the next available listener instance.

3. Traffic Spike Smoothing (Buffering):

   • If 10,000 users hit "Authenticate" at once, the broker-service accepts them all instantly. The listener-service processes them one by one at a safe pace, preventing the MongoDB database from being overwhelmed.

---

6. Technical Specifications

System Requirements

Component	Metric	Requirement
CPU	Core Count	Min: 2 Cores (Shared)
Memory	RAM	Min: 4GB (Recommended: 8GB)
Storage	Volume Type	SSD (Required for DB Performance)
Network	Host Ports	8081 (Web), 8082 (Broker), 1025 (Mail), 27117 (Mongo), 5432 (Postgres)

Service Configuration

Service	Language Version	Base Image	Connection Pool	Timeout Policy
Broker	Go 1.21	Alpine 3.19	N/A	5s HTTP / 2s gRPC
Logger	Go 1.21	Alpine 3.19	Min: 5, Max: 50	10s Connect / 5s Write
Auth	Go 1.21	Alpine 3.19	Max: 25	5s Query
Database	PostgreSQL 15	Postgres:15-alpine	Default	N/A
NoSQL	MongoDB 8.0	Mongo:8.0	WiredTiger Engine	N/A

---

7. Deployment & Usage

Option A: Local Development (Docker Compose)

Best for testing changes rapidly on your machine.

1. Start System:

   cd project
   make up_build

2. Access Dashboard: Open http://netreka.local
3. View Emails: Open http://netreka.local:8025
4. Stop System:

   make down

Prerequisites: Hosts File Setup

To access the application via netreka.local, you must map your local IP to this domain.

1. Open Notepad as Administrator.
2. Open the file: C:\Windows\System32\drivers\etc\hosts
3. Add the following line to the bottom:

   127.0.0.1    netreka.local
   

4. Save the file.

Option B: Production Lifecycle (Docker Swarm)

This section outlines the complete lifecycle for deploying, managing, and updating the cluster in a production environment.

1. Cluster Initialization

Initialize the Swarm on your manager node. If you already have a swarm, skip this.

docker swarm init

If you have multiple nodes, run the join command provided by the output on your worker nodes.

2. Build & Publish Images

Before deploying, all microservice images must be built and pushed to a registry (Docker Hub) so nodes can pull them.

cd project
.\publish_images.ps1

This script builds all services (Broker, Auth, Logger, Mail, Listener, Front, Caddy) and pushes them to yasinenginexpert/micro-*.

3. Deployment (Start the Stack)

Deploy the stack using the swarm.yml configuration.

docker stack deploy -c swarm.yml netreka-stack

Verify Deployment:

docker stack services netreka-stack

4. Observability (Logs)

Swarm aggregates logs from all replicas. You don't need to know which node a container is on.

# View logs for a specific service
docker service logs -f netreka-stack_logger-service

# View logs for the gateway/proxy
docker service logs -f netreka-stack_caddy

5. Scaling

Scale services up or down instantly to handle traffic spikes.

# Scale listener service to 3 replicas for faster queue processing
docker service scale netreka-stack_listener-service=3

# Scale frontend to 2 replicas for high availability
docker service scale netreka-stack_front-end=2

6. Zero-Downtime Updates

To update a service (e.g., you changed the code):

1. Run .\publish_images.ps1 to push the new image.
2. Update the service to pull the latest image:

   docker service update --image yasinenginexpert/micro-broker:latest netreka-stack_broker-service

   Swarm will perform a rolling update, replacing containers one by one to ensure no downtime.

7. Shutdown (Teardown)

To remove the stack and stop all services:

docker stack rm netreka-stack
# Optional: Leave the swarm
# docker swarm leave --force

---

Developed by Yasin Engin via Netreka Akademi