AWS EC2 Log Monitoring with Loki, Promtail, and Grafana

Monitoring logs in real time is essential for any production system. Instead of the heavy ELK stack, I opted for a lightweight alternative: Loki + Promtail + Grafana. In this post, I’ll share how I built a working log pipeline on an AWS EC2 server from scratch.

✨ Why Loki, not ELK?

• Lightweight
• Easy to set up
• Integrates well with Grafana (which we already use)

Loki by Grafana Labs, paired with Promtail, turned out to be a great fit.

🏗️ Architecture

[EC2 Log Files] → Promtail → Loki → Grafana

• Promtail: Tails log files and sends them to Loki.
• Loki: Stores and indexes log data.
• Grafana: Provides visualization and querying interface.

📁 Installing & Configuring Promtail

1. Install Promtail

wget https://github.com/grafana/loki/releases/download/v2.9.4/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip
chmod +x promtail-linux-amd64
mv promtail-linux-amd64 /usr/local/bin/promtail

2. Sample promtail.yaml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /opt/promtail/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: varlogs
          __path__: /var/log/*log

  - job_name: example
    static_configs:
      - targets: [localhost]
        labels:
          job: example
          env: live
          __path__: /opt/example/logs/server-*.json
    pipeline_stages:
      - regex:
          expression: '.*/server-(?P<component>[^/]+)\\.json'
          source: filename
      - json:
          expressions:
            ts: time
            lvl: level
            msg: message
            thread: thread
      - timestamp:
          source: ts
          format: RFC3339
      - labels:
          lvl:
          thread:
          component:
      - output:
          source: msg

3. Create Promtail User

sudo useradd --no-create-home --shell /usr/sbin/nologin promtail
sudo usermod -aG adm promtail

⚠️ Note: Promtail needs adm group access to read /var/log/*

4. Register systemd Service

[Unit]
Description=Promtail
After=network.target

[Service]
User=promtail
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/config.yaml
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable promtail
sudo systemctl start promtail

📁 Installing & Running Loki

1. Install Loki

wget https://github.com/grafana/loki/releases/download/v2.9.4/loki-linux-amd64.zip
unzip loki-linux-amd64.zip
chmod +x loki-linux-amd64
mv loki-linux-amd64 /usr/local/bin/loki

2. loki-config.yaml

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

3. Run Loki

/usr/local/bin/loki -config.file=/etc/loki/loki-config.yaml

📊 Visualizing Logs in Grafana

1. Open http://localhost:3000
2. Default credentials: admin / admin
3. Add Loki as a data source (http://localhost:3100)
4. Go to Explore and run query:
   {job="example", lvl="ERROR"}

🔍 Troubleshooting Tips

• Logs not appearing? Disable Live Mode and click Run Query.
• Check time range settings.
• Check Promtail’s parsing logic and regex.
• Test Loki connection:
  curl -G http://localhost:3100/loki/api/v1/labels

✅ Takeaways

• Promtail must be granted permission to read system logs.
• Loki stack is a great lightweight alternative to ELK.
• Grafana makes log querying and visualization intuitive.

🚀 What’s Next?

• Dockerize the entire logging stack
• Deploy backend with GCP Cloud Run
• Use Firebase Hosting for the frontend