FixTheVuln

Log Management Guide

By FixTheVuln Team Peer-reviewed security content Sources: CISA, NVD, OWASP

Key Takeaways

Test Your Knowledge

CySA+ Practice Quiz Security+ Practice Quiz

Latest from the Blog

This Week in Threats: Jan 31–Feb 14, 2026 This Week in Threats: Feb 03–Feb 17, 2026 This Week in Threats: Feb 06–Feb 20, 2026
← Back to Home

Security Logging Essentials

Quick reference for security logging - what events to capture, how to structure logs, retention requirements, and SIEM integration basics.

What to Log

Authentication Events (Critical)

Event Fields to Capture Priority
Successful login username, IP, timestamp, user-agent, MFA method High
Failed login username, IP, timestamp, failure reason Critical
Logout username, session duration, IP Medium
Password change username, IP, timestamp, changed_by High
Password reset request username, IP, email sent to High
MFA enabled/disabled username, IP, MFA type, admin_override Critical
Account lockout username, IP, failed_attempts, duration Critical
Session invalidation username, reason, admin_action High

Authorization Events (Critical)

Event Fields to Capture Priority
Access denied username, resource, action, reason Critical
Privilege escalation username, from_role, to_role, approved_by Critical
Role change username, old_role, new_role, changed_by High
Permission grant/revoke username, permission, resource, admin High
Admin action admin_user, action, target, details Critical

Data Access Events

Event Fields to Capture Priority
Sensitive data access username, data_type, record_count, purpose Critical
Data export/download username, data_type, format, record_count Critical
Bulk data operations username, operation, record_count, filters High
Data modification username, table, record_id, before/after High
Data deletion username, table, record_id, soft/hard Critical

System Events

Event Fields to Capture Priority
Application start/stop app_name, version, timestamp, triggered_by Medium
Configuration change setting, old_value, new_value, changed_by Critical
Scheduled job execution job_name, status, duration, records_processed Medium
API rate limit hit client_id, endpoint, limit, current_count High
Error/Exception error_type, message, stack_trace, user_context High
Security scan results scan_type, findings_count, severity_breakdown High

Log Structure & Format

Structured Log Format (JSON)

{
  "timestamp": "2026-01-15T10:30:45.123Z",
  "level": "INFO",
  "service": "auth-service",
  "environment": "production",
  "trace_id": "abc123def456",
  "span_id": "789xyz",
  "event": {
    "type": "authentication",
    "action": "login_success",
    "outcome": "success"
  },
  "actor": {
    "user_id": "user_12345",
    "username": "[email protected]",
    "ip_address": "192.168.1.100",
    "user_agent": "Mozilla/5.0...",
    "session_id": "sess_abc123"
  },
  "resource": {
    "type": "user_account",
    "id": "user_12345"
  },
  "metadata": {
    "mfa_method": "totp",
    "login_method": "password",
    "geo_location": "US-CA"
  }
}

Common Log Fields

Field Description Example
timestamp ISO 8601 format, UTC timezone 2026-01-15T10:30:45.123Z
level Log severity level DEBUG, INFO, WARN, ERROR, CRITICAL
service Source application/service name auth-service, api-gateway
trace_id Distributed tracing correlation ID abc123def456
event.type Category of event authentication, authorization, data_access
event.action Specific action taken login_success, access_denied
actor.user_id User who performed action user_12345
actor.ip_address Client IP address 192.168.1.100
resource.type Type of resource accessed user_account, document, api_endpoint
resource.id Identifier of resource doc_789, user_12345

Python Logging Example

import logging
import json
from datetime import datetime

class SecurityLogger:
    def __init__(self, service_name):
        self.service = service_name
        self.logger = logging.getLogger(service_name)

    def log_auth_event(self, action, outcome, user_id, ip_address, **kwargs):
        log_entry = {
            "timestamp": datetime.utcnow().isoformat() + "Z",
            "level": "INFO" if outcome == "success" else "WARN",
            "service": self.service,
            "event": {
                "type": "authentication",
                "action": action,
                "outcome": outcome
            },
            "actor": {
                "user_id": user_id,
                "ip_address": ip_address
            },
            "metadata": kwargs
        }
        self.logger.info(json.dumps(log_entry))

# Usage
security_log = SecurityLogger("auth-service")
security_log.log_auth_event(
    action="login_attempt",
    outcome="success",
    user_id="user_123",
    ip_address="192.168.1.100",
    mfa_method="totp"
)

Log Retention Requirements

Retention by Compliance Framework

Framework Minimum Retention Log Types
PCI DSS 1 year (3 months immediately accessible) All security-relevant logs
HIPAA 6 years PHI access logs, security logs
SOX 7 years Financial system audit logs
GDPR As long as necessary (minimize) Personal data processing logs
SOC 2 1 year minimum Security event logs
NIST 800-53 Organization-defined AU-11 specifies requirements

Recommended Retention by Log Type

Log Type Hot Storage Cold Storage Total
Security/Audit logs 90 days 2+ years 3-7 years
Authentication logs 90 days 1+ years 2-3 years
Application logs 30 days 90 days 6-12 months
Debug logs 7 days N/A 7-30 days
Access logs (web) 30 days 90 days 6-12 months
Firewall/Network logs 30 days 1 year 1-2 years

SIEM Integration

Popular SIEM Solutions

Solution Type Best For
Splunk Commercial Enterprise, large scale
Microsoft Sentinel Cloud (Azure) Microsoft ecosystem
Elastic Security Open Source/Commercial Flexible, scalable
Sumo Logic Cloud Cloud-native apps
Wazuh Open Source Budget-conscious, compliance
Graylog Open Source/Commercial Log management focus

Key SIEM Detection Rules

# Example: Brute Force Detection (Splunk SPL)
index=auth sourcetype=auth_logs action=login_failure
| stats count by src_ip, user
| where count > 5
| table src_ip, user, count

# Example: Impossible Travel (pseudo-code)
SELECT user_id,
       prev_location, current_location,
       time_diff_minutes,
       distance_km
FROM login_events
WHERE time_diff_minutes < (distance_km / 900) * 60
  AND distance_km > 500

# Example: Data Exfiltration (high volume download)
index=app_logs event_type=data_export
| stats sum(record_count) as total_records by user, _time span=1h
| where total_records > 10000

# Example: Privilege Escalation
index=auth_logs event_type=role_change
| where new_role IN ("admin", "superuser", "root")
| table _time, user, old_role, new_role, changed_by

Alert Priority Matrix

Alert Priority Response Time
Multiple failed logins (brute force) High < 15 minutes
Admin account login from new IP Critical < 5 minutes
Bulk data export High < 15 minutes
Configuration change Medium < 1 hour
MFA disabled Critical < 5 minutes
Impossible travel High < 15 minutes
After-hours access Medium < 1 hour
New admin account created Critical < 5 minutes

Centralized Logging Architecture

Log Shipping Methods

# Filebeat configuration (ship to Elasticsearch)
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/app/*.log
  json.keys_under_root: true
  json.add_error_key: true

output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]
  username: "filebeat_internal"
  password: "${ES_PASSWORD}"
  ssl.certificate_authorities: ["/etc/pki/ca.crt"]

# Fluentd configuration

  @type tail
  path /var/log/app/*.log
  pos_file /var/log/fluentd/app.log.pos
  tag app.logs
  
    @type json
  



  @type elasticsearch
  host elasticsearch
  port 9200
  logstash_format true
  logstash_prefix app-logs

Log Integrity Protection

  • Immutable storage - Use WORM (Write Once Read Many) storage
  • Log signing - Sign log entries with HMAC or digital signatures
  • Separate log servers - Logs should be sent to dedicated infrastructure
  • Access control - Restrict who can delete/modify logs
  • Hash chains - Link log entries cryptographically
  • Timestamps - Use trusted time sources (NTP)

What NOT to Log

Sensitive Data to Exclude

  • Passwords - Never log passwords, even hashed
  • API keys/Secrets - Mask or exclude entirely
  • Credit card numbers - PCI DSS violation
  • SSN/Government IDs - Personally identifiable information
  • Full session tokens - Log only last 4 characters
  • Health information - HIPAA protected
  • Biometric data - Cannot be changed if leaked
  • Encryption keys - Defeats purpose of encryption

Data Masking Example

import re

def mask_sensitive_data(log_entry):
    # Mask credit card numbers
    log_entry = re.sub(
        r'\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b',
        '****-****-****-XXXX',
        log_entry
    )

    # Mask email addresses (partial)
    log_entry = re.sub(
        r'(\w{2})\w*@(\w{2})\w*\.(\w+)',
        r'\1***@\2***.\3',
        log_entry
    )

    # Mask API keys
    log_entry = re.sub(
        r'(api[_-]?key["\s:=]+)["\']?[\w-]{20,}["\']?',
        r'\1[REDACTED]',
        log_entry,
        flags=re.IGNORECASE
    )

    return log_entry

Related Resources

Incident Response Linux Hardening CIS Controls

Need Detailed Logging Implementation Guides?

For comprehensive tutorials and SIEM setup guides:

Visit FixTheVuln.com →

FixTheVuln Store

Get the CompTIA CySA+ Study Planner

Fillable PDF study planners with domain trackers, weekly schedules, and progress tracking. Available in Standard, ADHD-Friendly, Dark Mode, and 4-Format Bundle.

CompTIA CySA+ Planner

60+ certifications available — from $5.99

FixTheVuln Store

Patch Tuesday Sprint Kit

5 fillable templates for vulnerability triage, sprint planning, testing, SLA tracking, and executive reporting. Free blank templates or $4.99/mo for pre-filled intelligence with real CISA KEV data.

Try Free Templates Subscribe — $4.99/mo

FixTheVuln Store

Patch Tuesday Sprint Kit

5 fillable templates for vulnerability triage, sprint planning, testing, SLA tracking, and executive reporting. Free blank templates or $4.99/mo for pre-filled intelligence with real CISA KEV data.

Try Free Templates Subscribe — $4.99/mo

FixTheVuln Store

Patch Tuesday Sprint Kit

5 fillable templates for vulnerability triage, sprint planning, testing, SLA tracking, and executive reporting. Free blank templates or $4.99/mo for pre-filled intelligence with real CISA KEV data.

Try Free Templates Subscribe — $4.99/mo

CyberFolio

Choosing your next cert? Track them all in one place.

Build a shareable cybersecurity portfolio that highlights your certifications, projects, and skills — free.

Build Your Portfolio →

FixTheVuln Store

Patch Tuesday Sprint Kit

5 fillable templates for vulnerability triage, sprint planning, testing, SLA tracking, and executive reporting. Free blank templates or $4.99/mo for pre-filled intelligence with real CISA KEV data.

Try Free Templates Subscribe — $4.99/mo

FixTheVuln Store

Patch Tuesday Sprint Kit

5 fillable templates for vulnerability triage, sprint planning, testing, SLA tracking, and executive reporting. Free blank templates or $4.99/mo for pre-filled intelligence with real CISA KEV data.

Try Free Templates Subscribe — $4.99/mo

FixTheVuln Store

Patch Tuesday Sprint Kit

5 fillable templates for vulnerability triage, sprint planning, testing, SLA tracking, and executive reporting. Free blank templates or $4.99/mo for pre-filled intelligence with real CISA KEV data.

Try Free Templates Subscribe — $4.99/mo