CloudWatch: Logs, Insights & Container Monitoring

CloudWatch Logs

# Log groups and streams
aws logs create-log-group --log-group-name /myapp/prod
aws logs create-log-stream   --log-group-name /myapp/prod   --log-stream-name web-server-1

# Set retention
aws logs put-retention-policy   --log-group-name /myapp/prod   --retention-in-days 30   # 1,3,5,7,14,30,60,90,120,150,180,365,400,545,731,1827,3653

# List log groups
aws logs describe-log-groups
aws logs describe-log-groups --log-group-name-prefix /myapp

# Tail logs (like tail -f)
aws logs tail /myapp/prod --follow
aws logs tail /myapp/prod --since 1h
aws logs tail /myapp/prod --filter-pattern "ERROR"

# Get log events
aws logs get-log-events   --log-group-name /myapp/prod   --log-stream-name web-server-1   --start-time $(date -d "1 hour ago" +%s000)

# Filter log events (across streams in a log group)
aws logs filter-log-events   --log-group-name /myapp/prod   --filter-pattern "{ $.level = ERROR }"   --start-time $(date -d "1 hour ago" +%s000)

CloudWatch Logs Insights

Logs Insights is an interactive query language for analyzing log data. Queries run across log groups and return results in seconds.

-- Count errors in last hour
fields @timestamp, @message
| filter @message like /ERROR/
| stats count() as error_count by bin(5m)
| sort @timestamp desc

-- Lambda cold starts
filter @type = "REPORT"
| stats count() as invocations, avg(@duration) as avg_duration,
        sum(@initDuration > 0) as cold_starts
| display invocations, avg_duration, cold_starts

-- API latency percentiles
filter @message like /END/
| parse @message "Duration: * ms" as duration
| stats avg(duration), percentile(duration, 50) as p50,
        percentile(duration, 95) as p95, percentile(duration, 99) as p99

-- Top IPs from ALB access logs
fields @timestamp, @message
| parse @message "* * * * * * * * * * * "* *" * *" as time, elb, client, target,
    request_processing, target_processing, response_processing, elb_status,
    target_status, received_bytes, sent_bytes, request, user_agent, ssl_cipher, ssl_protocol
| stats count() as requests by client
| sort requests desc
| limit 20

-- Find specific user's requests
fields @timestamp, @message
| filter @message like /user-123/
| sort @timestamp desc
| limit 100

CloudWatch Agent (EC2 & On-Premises)

// /opt/aws/amazon-cloudwatch-agent/bin/config.json
{
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "cwagent"
  },
  "metrics": {
    "namespace": "MyApp/EC2",
    "metrics_collected": {
      "cpu": {
        "measurement": ["cpu_usage_idle", "cpu_usage_user", "cpu_usage_system"],
        "metrics_collection_interval": 60
      },
      "mem": {
        "measurement": ["mem_used_percent"],
        "metrics_collection_interval": 60
      },
      "disk": {
        "measurement": ["used_percent"],
        "resources": ["/", "/data"],
        "metrics_collection_interval": 300
      }
    }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/myapp/app.log",
            "log_group_name": "/myapp/prod",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
    }
  }
}

# Install and start agent
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl   -a fetch-config   -m ec2   -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json   -s

Container Insights

• Container Insights: enhanced monitoring for ECS and EKS — pod/task-level CPU, memory, network metrics.

• Enable on ECS cluster: aws ecs update-cluster-settings --cluster my-cluster --settings name=containerInsights,value=enabled

• Enable on EKS: install CloudWatch agent via Helm or EKS add-on; use FluentBit for log forwarding.

• EMF (Embedded Metric Format): log structured JSON with _aws.CloudWatchMetrics to publish metrics via logs — no SDK required.

• Synthetics Canaries: run Node.js/Python scripts on a schedule to monitor endpoints — like website uptime checks.

• RUM (Real User Monitoring): collect browser-side metrics (Core Web Vitals, errors) from actual users.

• Application Signals: automatic application performance monitoring with traces + SLI/SLO tracking (requires AWS X-Ray agent).