Container Log Aggregation with Fluentd

Fluentd is an open-source log collector that unifies logging across containers, applications, and infrastructure. For Docker environments, Fluentd collects container logs, parses and enriches them, and forwards them to storage backends like Elasticsearch, S3, or Loki. This guide covers deploying Fluentd for Docker container log aggregation.

Architecture

# Docker containers → Fluentd (collect/parse/route) → Storage backends
#                                                    → Elasticsearch
#                                                    → S3/MinIO
#                                                    → Loki
#                                                    → Kafka

Deploying Fluentd with Docker

# docker-compose.yml
services:
  fluentd:
    image: fluent/fluentd:v1.17-1
    volumes:
      - ./fluentd/conf:/fluentd/etc
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    restart: always

Basic Configuration

# fluentd/conf/fluent.conf

# Accept logs from Docker logging driver

  @type forward
  port 24224
  bind 0.0.0.0


# Parse JSON logs

  @type parser
  key_name log
  reserve_data true
  
    @type json
    time_key time
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  


# Add metadata

  @type record_transformer
  
    hostname "#{Socket.gethostname}"
    environment production
  


# Route to Elasticsearch

  @type elasticsearch
  host elasticsearch
  port 9200
  logstash_format true
  logstash_prefix docker
  include_tag_key true
  flush_interval 5s
  retry_limit 3

Configure Docker to Use Fluentd

# Per-container
docker run -d --log-driver=fluentd --log-opt fluentd-address=localhost:24224 --log-opt tag="docker.{{.Name}}" nginx

# Docker daemon default (daemon.json)
{
    "log-driver": "fluentd",
    "log-opts": {
        "fluentd-address": "localhost:24224",
        "tag": "docker.{{.Name}}",
        "fluentd-async": "true"
    }
}

# Docker Compose
services:
  web:
    image: nginx
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: "docker.web"

Multi-Output Configuration

# Send to multiple destinations

  @type copy

  # Elasticsearch for search
  
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
  

  # S3 for archival
  
    @type s3
    aws_key_id YOUR_KEY
    aws_sec_key YOUR_SECRET
    s3_bucket docker-logs
    s3_region us-east-1
    path logs/%Y/%m/%d/
    
      timekey 3600
      timekey_wait 10m
    
  

  # Stdout for debugging
  
    @type stdout
  

Log Parsing Patterns

# Nginx access logs

  @type parser
  key_name log
  
    @type nginx
  


# Apache access logs

  @type parser
  key_name log
  
    @type apache2
  


# Custom regex

  @type parser
  key_name log
  
    @type regexp
    expression /^(?[^ ]*) (?[^ ]*) (?.*)$/
  

Monitoring Fluentd

# Enable monitoring

  @type monitor_agent
  bind 0.0.0.0
  port 24220


# Prometheus metrics

  @type prometheus
  bind 0.0.0.0
  port 24231


# Check status
curl http://localhost:24220/api/plugins.json | jq

Best Practices

• Use fluentd-async: true in Docker log options to prevent container blocking if Fluentd is unavailable
• Set buffer limits to prevent Fluentd from consuming too much memory
• Use file-based buffers for reliability (survives Fluentd restarts)
• Parse logs at collection time — structured logs are much easier to query
• Add container metadata (name, image, hostname) for filtering and correlation
• Monitor Fluentd's own metrics to catch backpressure and delivery failures