Simplifying Elastic Stack Management with Docker Compose

5 min readDec 8, 2023

Managing and deploying the Elastic Stack, which includes Elasticsearch and Kibana, can be complex. Docker Compose simplifies this process, making it easier to set up and integrate additional components like Fleet Server. This blog post will guide you through a detailed process, covering setup, troubleshooting, and management tips I learned when setting it up for my SIEM.

Setting Up Elasticsearch and Kibana

Docker Compose Configuration

Begin with a well-structured Docker Compose file. Define services for Elasticsearch and Kibana, ensuring proper environment variable settings for security measures like ELASTIC_PASSWORD and KIBANA_PASSWORD.

# .env file
KIBANA_PASSWORD=CHANGEME
STACK_VERSION=8.10.2
CLUSTER_NAME=docker-cluster
LICENSE=basic
ES_PORT=9200
KIBANA_PORT=5601
MEM_LIMIT=1073741824
XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=<randomly generated 32 bit character string>
XPACK_SECURITY_ENCRYPTIONKEY=<32 bit character string>
XPACK_REPORTING_ENCRYPTIONKEY=<randomly generated 32 bit character string>Example of 32 bit randomly generated character string
mckkdwg7fsdkgwx3ytv867pb5revvupx

SSL Certificates for Security

Enhance security by implementing SSL certificates. Generate a Certificate Authority (CA) and create certificates for Elasticsearch and Kibana, enabling secure communication between services. This should be in the docker compose YAML file as a script in the commands section

if [ ! -f config/certs/ca.zip ]; then
  echo "Creating CA";
  bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
  unzip config/certs/ca.zip -d config/certs;
fi;

Health Checks for Initialization

Ensure service reliability by incorporating health checks in the Docker Compose file. This guarantees that Elasticsearch and Kibana are initialized before other services depend on them.

healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

Integrating Fleet Server

Expanding Docker Compose Configuration

Extend the Docker Compose setup to include Fleet Server. Adjust dependencies and configurations, paying attention to specific settings required for smooth integration. I recommend creating a new folder for the YAML and .env files in the same directory as your ELK stack

Fleet Server Configuration

Configure Fleet Server settings, such as the Elasticsearch host, service tokens, and policies. Address potential issues like version compatibility checks during the setup.

#YAML FILE
version: "3.6"
services:
  fleet-server:
    image: docker.elastic.co/beats/elastic-agent:8.10.2
    container_name: fleet-server
    restart: always
    volumes:
      - ../certs:/certs
    ports: [8220:8220]
    user: root
    environment:
      - FLEET_SERVER_ENABLE=true
      - FLEET_SERVER_POLICY_NAME=fleet-server-policy
      - FLEET_SERVER_ELASTICSEARCH_HOST=https://es01:9200
      - FLEET_SERVER_SERVICE_TOKEN=${FLEET_SERVER_SERVICE_TOKEN}
      - FLEET_SERVER_ELASTICSEARCH_CA=/certs/ca/ca.crt
      - FLEET_INSECURE=true
networks:
  default:
    name: elastic-stack-network

Troubleshooting Fleet Server

Anticipate and troubleshoot common errors that might occur during Fleet Server integration. Reference logs and follow Elastic’s troubleshooting guides to resolve issues effectively.

Troubleshooting and Error Handling

1. Common Setup Errors

During the setup of Elasticsearch, Kibana, and Fleet Server, users may encounter various errors that can impede the smooth deployment of the ELK stack. Here are some common setup errors and how to resolve them:

Elasticsearch Connection Issues:
Error Message: “Could not connect to Elasticsearch.”
Resolution: Verify the Elasticsearch container is running, check network configurations, and ensure the correct Elasticsearch URL is used in configurations.
Kibana Initialization Failures:

Error Message: “Kibana server is not ready yet.”

Resolution: Check Kibana logs for initialization errors, ensure Elasticsearch is accessible, and validate Kibana configuration settings.

Fleet Server Startup Problems:

Error Message: “Fleet Server failed: context canceled.”

Resolution: Inspect Fleet Server logs for detailed error messages, confirm Elasticsearch connectivity, and check Fleet Server configuration.

2. Practical Troubleshooting Tips

Docker Logs:

Tip: Utilize docker logs [container_name] to view container-specific logs. Look for error messages and warnings.

Network Configurations:

Tip: Verify network configurations, including hostnames and ports. Ensure necessary ports are open, and firewalls aren’t blocking communication.

Environment Variable Validation:

Tip: Check environment variables in the Docker Compose file for typos or missing values. Validate that sensitive information like passwords is correctly set.

Managing Docker Volumes

1. Understanding Docker Volumes

Data Persistence:Explanation: Docker volumes are crucial for maintaining configuration durability. They ensure that data, such as Elasticsearch indices and Kibana configurations, persists even if containers are stopped or removed.

2. Inspecting and Managing Volumes

Volume Inspection:

Guide: Use docker volume inspect [volume_name] to inspect volume details, including mount points and configuration. Understand the structure of the stored data.

Managing Volumes:

Guide: Utilize docker volume ls, docker volume rm, and other commands to manage volumes. This knowledge aids in cleaning up unused or problematic volumes.

3. Handling Volume-Related Issues

Permission Problems:

Issue: Permission errors preventing data writes.

Solution: Adjust volume permissions using chown or chmod to ensure the correct user has write access.

Missing Directories:

Issue: Directories not present in the volume.

Solution: Manually create missing directories within the volume to match expected paths.

Correcting File Paths:

Issue: Incorrect file paths leading to data retrieval problems.

Solution: Update file paths in configurations to match the actual paths within the Docker volumes.

Reboot and Cleanup

1. Reboot Impact on Docker Volumes

Data Persistence:

Explanation: Docker volumes ensure data persistence, so rebooting should not impact stored data. However, services might need manual restarts after a system reboot.

2. Using docker-compose down -v for Cleanup

Configuration Reset:

Importance: This command resets configurations, helping to start with a clean slate. It removes volumes associated with the Docker Compose project.

3. Token and Configuration Implications

Token Regeneration:

Implication: Kibana may generate new tokens after volume cleanup.

Advice: Retrieve and securely store essential tokens before running cleanup commands to ensure uninterrupted service.

Conclusion

Docker Compose simplifies the management of the Elastic Stack, providing a robust solution for log and data analysis. By following this comprehensive guide, users can navigate setup complexities, troubleshoot effectively, and manage Docker volumes with confidence. Documenting configurations, securing tokens, and staying informed about Elastic’s updates will ensure a seamless and reliable Elastic Stack deployment. I hope you loved this! Let me know in the comment if you have any questions or reviews.