Key Takeaways
- Log authentication events, authorization failures, and data access at minimum
- Never log sensitive data like passwords, tokens, or credit card numbers
- Centralize logs using a SIEM (Splunk, ELK Stack, or cloud-native solutions)
- Set log retention policies based on compliance requirements (PCI DSS: 1 year, HIPAA: 6 years)
- Set up real-time alerts for critical security events like multiple failed logins
Test Your Knowledge
Latest from the Blog
Security Logging Essentials
Quick reference for security logging - what events to capture, how to structure logs, retention requirements, and SIEM integration basics.
What to Log
Authentication Events (Critical)
| Event | Fields to Capture | Priority |
|---|---|---|
| Successful login | username, IP, timestamp, user-agent, MFA method | High |
| Failed login | username, IP, timestamp, failure reason | Critical |
| Logout | username, session duration, IP | Medium |
| Password change | username, IP, timestamp, changed_by | High |
| Password reset request | username, IP, email sent to | High |
| MFA enabled/disabled | username, IP, MFA type, admin_override | Critical |
| Account lockout | username, IP, failed_attempts, duration | Critical |
| Session invalidation | username, reason, admin_action | High |
Authorization Events (Critical)
| Event | Fields to Capture | Priority |
|---|---|---|
| Access denied | username, resource, action, reason | Critical |
| Privilege escalation | username, from_role, to_role, approved_by | Critical |
| Role change | username, old_role, new_role, changed_by | High |
| Permission grant/revoke | username, permission, resource, admin | High |
| Admin action | admin_user, action, target, details | Critical |
Data Access Events
| Event | Fields to Capture | Priority |
|---|---|---|
| Sensitive data access | username, data_type, record_count, purpose | Critical |
| Data export/download | username, data_type, format, record_count | Critical |
| Bulk data operations | username, operation, record_count, filters | High |
| Data modification | username, table, record_id, before/after | High |
| Data deletion | username, table, record_id, soft/hard | Critical |
System Events
| Event | Fields to Capture | Priority |
|---|---|---|
| Application start/stop | app_name, version, timestamp, triggered_by | Medium |
| Configuration change | setting, old_value, new_value, changed_by | Critical |
| Scheduled job execution | job_name, status, duration, records_processed | Medium |
| API rate limit hit | client_id, endpoint, limit, current_count | High |
| Error/Exception | error_type, message, stack_trace, user_context | High |
| Security scan results | scan_type, findings_count, severity_breakdown | High |
Log Structure & Format
Structured Log Format (JSON)
{
"timestamp": "2026-01-15T10:30:45.123Z",
"level": "INFO",
"service": "auth-service",
"environment": "production",
"trace_id": "abc123def456",
"span_id": "789xyz",
"event": {
"type": "authentication",
"action": "login_success",
"outcome": "success"
},
"actor": {
"user_id": "user_12345",
"username": "[email protected]",
"ip_address": "192.168.1.100",
"user_agent": "Mozilla/5.0...",
"session_id": "sess_abc123"
},
"resource": {
"type": "user_account",
"id": "user_12345"
},
"metadata": {
"mfa_method": "totp",
"login_method": "password",
"geo_location": "US-CA"
}
}
Common Log Fields
| Field | Description | Example |
|---|---|---|
| timestamp | ISO 8601 format, UTC timezone | 2026-01-15T10:30:45.123Z |
| level | Log severity level | DEBUG, INFO, WARN, ERROR, CRITICAL |
| service | Source application/service name | auth-service, api-gateway |
| trace_id | Distributed tracing correlation ID | abc123def456 |
| event.type | Category of event | authentication, authorization, data_access |
| event.action | Specific action taken | login_success, access_denied |
| actor.user_id | User who performed action | user_12345 |
| actor.ip_address | Client IP address | 192.168.1.100 |
| resource.type | Type of resource accessed | user_account, document, api_endpoint |
| resource.id | Identifier of resource | doc_789, user_12345 |
Python Logging Example
import logging
import json
from datetime import datetime
class SecurityLogger:
def __init__(self, service_name):
self.service = service_name
self.logger = logging.getLogger(service_name)
def log_auth_event(self, action, outcome, user_id, ip_address, **kwargs):
log_entry = {
"timestamp": datetime.utcnow().isoformat() + "Z",
"level": "INFO" if outcome == "success" else "WARN",
"service": self.service,
"event": {
"type": "authentication",
"action": action,
"outcome": outcome
},
"actor": {
"user_id": user_id,
"ip_address": ip_address
},
"metadata": kwargs
}
self.logger.info(json.dumps(log_entry))
# Usage
security_log = SecurityLogger("auth-service")
security_log.log_auth_event(
action="login_attempt",
outcome="success",
user_id="user_123",
ip_address="192.168.1.100",
mfa_method="totp"
)
Log Retention Requirements
Retention by Compliance Framework
| Framework | Minimum Retention | Log Types |
|---|---|---|
| PCI DSS | 1 year (3 months immediately accessible) | All security-relevant logs |
| HIPAA | 6 years | PHI access logs, security logs |
| SOX | 7 years | Financial system audit logs |
| GDPR | As long as necessary (minimize) | Personal data processing logs |
| SOC 2 | 1 year minimum | Security event logs |
| NIST 800-53 | Organization-defined | AU-11 specifies requirements |
Recommended Retention by Log Type
| Log Type | Hot Storage | Cold Storage | Total |
|---|---|---|---|
| Security/Audit logs | 90 days | 2+ years | 3-7 years |
| Authentication logs | 90 days | 1+ years | 2-3 years |
| Application logs | 30 days | 90 days | 6-12 months |
| Debug logs | 7 days | N/A | 7-30 days |
| Access logs (web) | 30 days | 90 days | 6-12 months |
| Firewall/Network logs | 30 days | 1 year | 1-2 years |
SIEM Integration
Popular SIEM Solutions
| Solution | Type | Best For |
|---|---|---|
| Splunk | Commercial | Enterprise, large scale |
| Microsoft Sentinel | Cloud (Azure) | Microsoft ecosystem |
| Elastic Security | Open Source/Commercial | Flexible, scalable |
| Sumo Logic | Cloud | Cloud-native apps |
| Wazuh | Open Source | Budget-conscious, compliance |
| Graylog | Open Source/Commercial | Log management focus |
Key SIEM Detection Rules
# Example: Brute Force Detection (Splunk SPL)
index=auth sourcetype=auth_logs action=login_failure
| stats count by src_ip, user
| where count > 5
| table src_ip, user, count
# Example: Impossible Travel (pseudo-code)
SELECT user_id,
prev_location, current_location,
time_diff_minutes,
distance_km
FROM login_events
WHERE time_diff_minutes < (distance_km / 900) * 60
AND distance_km > 500
# Example: Data Exfiltration (high volume download)
index=app_logs event_type=data_export
| stats sum(record_count) as total_records by user, _time span=1h
| where total_records > 10000
# Example: Privilege Escalation
index=auth_logs event_type=role_change
| where new_role IN ("admin", "superuser", "root")
| table _time, user, old_role, new_role, changed_by
Alert Priority Matrix
| Alert | Priority | Response Time |
|---|---|---|
| Multiple failed logins (brute force) | High | < 15 minutes |
| Admin account login from new IP | Critical | < 5 minutes |
| Bulk data export | High | < 15 minutes |
| Configuration change | Medium | < 1 hour |
| MFA disabled | Critical | < 5 minutes |
| Impossible travel | High | < 15 minutes |
| After-hours access | Medium | < 1 hour |
| New admin account created | Critical | < 5 minutes |
Centralized Logging Architecture
Log Shipping Methods
# Filebeat configuration (ship to Elasticsearch)
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/app/*.log
json.keys_under_root: true
json.add_error_key: true
output.elasticsearch:
hosts: ["https://elasticsearch:9200"]
username: "filebeat_internal"
password: "${ES_PASSWORD}"
ssl.certificate_authorities: ["/etc/pki/ca.crt"]
# Fluentd configuration
@type tail
path /var/log/app/*.log
pos_file /var/log/fluentd/app.log.pos
tag app.logs
@type json
@type elasticsearch
host elasticsearch
port 9200
logstash_format true
logstash_prefix app-logs
Log Integrity Protection
- Immutable storage - Use WORM (Write Once Read Many) storage
- Log signing - Sign log entries with HMAC or digital signatures
- Separate log servers - Logs should be sent to dedicated infrastructure
- Access control - Restrict who can delete/modify logs
- Hash chains - Link log entries cryptographically
- Timestamps - Use trusted time sources (NTP)
What NOT to Log
Sensitive Data to Exclude
- Passwords - Never log passwords, even hashed
- API keys/Secrets - Mask or exclude entirely
- Credit card numbers - PCI DSS violation
- SSN/Government IDs - Personally identifiable information
- Full session tokens - Log only last 4 characters
- Health information - HIPAA protected
- Biometric data - Cannot be changed if leaked
- Encryption keys - Defeats purpose of encryption
Data Masking Example
import re
def mask_sensitive_data(log_entry):
# Mask credit card numbers
log_entry = re.sub(
r'\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b',
'****-****-****-XXXX',
log_entry
)
# Mask email addresses (partial)
log_entry = re.sub(
r'(\w{2})\w*@(\w{2})\w*\.(\w+)',
r'\1***@\2***.\3',
log_entry
)
# Mask API keys
log_entry = re.sub(
r'(api[_-]?key["\s:=]+)["\']?[\w-]{20,}["\']?',
r'\1[REDACTED]',
log_entry,
flags=re.IGNORECASE
)
return log_entry
Related Resources
Need Detailed Logging Implementation Guides?
For comprehensive tutorials and SIEM setup guides:
Visit FixTheVuln.com →FixTheVuln Store
Get the CompTIA CySA+ Study Planner
Fillable PDF study planners with domain trackers, weekly schedules, and progress tracking. Available in Standard, ADHD-Friendly, Dark Mode, and 4-Format Bundle.
CompTIA CySA+ Planner60+ certifications available — from $5.99
FixTheVuln Store
Patch Tuesday Sprint Kit
5 fillable templates for vulnerability triage, sprint planning, testing, SLA tracking, and executive reporting. Free blank templates or $4.99/mo for pre-filled intelligence with real CISA KEV data.
Try Free Templates Subscribe — $4.99/moFixTheVuln Store
Patch Tuesday Sprint Kit
5 fillable templates for vulnerability triage, sprint planning, testing, SLA tracking, and executive reporting. Free blank templates or $4.99/mo for pre-filled intelligence with real CISA KEV data.
Try Free Templates Subscribe — $4.99/moFixTheVuln Store
Patch Tuesday Sprint Kit
5 fillable templates for vulnerability triage, sprint planning, testing, SLA tracking, and executive reporting. Free blank templates or $4.99/mo for pre-filled intelligence with real CISA KEV data.
Try Free Templates Subscribe — $4.99/moCyberFolio
Choosing your next cert? Track them all in one place.
Build a shareable cybersecurity portfolio that highlights your certifications, projects, and skills — free.
Build Your Portfolio →FixTheVuln Store
Patch Tuesday Sprint Kit
5 fillable templates for vulnerability triage, sprint planning, testing, SLA tracking, and executive reporting. Free blank templates or $4.99/mo for pre-filled intelligence with real CISA KEV data.
Try Free Templates Subscribe — $4.99/moFixTheVuln Store
Patch Tuesday Sprint Kit
5 fillable templates for vulnerability triage, sprint planning, testing, SLA tracking, and executive reporting. Free blank templates or $4.99/mo for pre-filled intelligence with real CISA KEV data.
Try Free Templates Subscribe — $4.99/moFixTheVuln Store
Patch Tuesday Sprint Kit
5 fillable templates for vulnerability triage, sprint planning, testing, SLA tracking, and executive reporting. Free blank templates or $4.99/mo for pre-filled intelligence with real CISA KEV data.
Try Free Templates Subscribe — $4.99/mo