Architected and implemented by Leonid Lachmann
A highly optimized, secure, and lightweight log management and observability pipeline deployed via Docker Compose.
The primary goal of this project is to deploy a robust observability stack capable of running in resource-constrained environments (e.g., edge servers or local development machines).
Initially architected as a traditional ELK stack, the system encountered Out-Of-Memory (OOM) constraints due to the heavy JVM footprint of Logstash. To ensure high availability and stability, the architecture was intentionally refactored to an EFK stack. By removing Logstash and configuring Filebeat to route logs directly to Elasticsearch, overall RAM consumption was reduced significantly while maintaining full log parsing capabilities.
- Nginx (Alpine): Configured to write physical access and error logs for ingestion.
- Filebeat: Lightweight shipper configured to parse Nginx logs and forward them directly to Elasticsearch via the
filestreaminput. - Elasticsearch (Single-Node): Core search engine configured with
xpack.securityfor native authentication. - Kibana: UI for data discovery, monitoring, and real-time visualization.
The infrastructure is containerized and orchestrated via Docker Compose. The refactored architecture allows the entire observability network to spin up quickly and run smoothly without overwhelming the host Docker engine.
Security is implemented at the core. System passwords and authentication tokens were generated and injected directly via the Elasticsearch REST API using terminal commands, bypassing default insecure setups.
This ensures that Kibana and the underlying data indices are fully protected by user authentication.
Direct validation of the Elasticsearch backend using the _cat/indices API confirms that Filebeat successfully created the datastreams (.ds-filebeat-*) and is actively indexing documents.
Nginx access logs are successfully ingested, parsed, and mapped to the @timestamp field. The Discover module natively visualizes traffic spikes using auto-interval tracking alongside the parsed document metrics.
Utilized Kibana Lens to construct custom vertical stacked bar charts, defining axes and distributions based on specific log fields for deeper traffic analysis.
Created dedicated operational dashboards aggregating multiple visual panels, allowing for at-a-glance monitoring of HTTP request distribution and server activity.
1. Clone the repository and navigate to the directory:
git clone <your-repository-url>
cd secure-efk-nginx-observability2. Optimize Host Memory Limits (Linux/WSL): Elasticsearch requires a higher virtual memory map count to function correctly:
sudo sysctl -w vm.max_map_count=2621443. Launch the Infrastructure:
docker-compose up -d4. Generate Traffic:
Visit http://localhost:8080 and refresh multiple times to generate fresh Nginx access logs.
5. Access the Dashboard:
Navigate to http://localhost:5601. When prompted to create a Data View, use the index pattern filebeat-* and set the primary time field to @timestamp.