Last updated: April 29, 2026
Why this module exists. Logs are the SOC’s primary data. Bad log architecture means missed detections, slow investigations, and impossible audit response. Good architecture means hunts complete in seconds and forensic timelines reconstruct in hours. The difference is mostly upfront planning.
The log-management problem in 2026 numbers
A medium Indian enterprise (5,000 endpoints, 200 servers, multi-cloud) generates roughly:
- 50-200 GB/day of logs at the source
- 15-30 TB/year of compressed retained logs
- Tens of thousands of distinct log sources / event types
Costs at scale (Splunk pricing): ~₹1.5-3 crore/year. Open-source paths are cheaper but require significant engineering investment.
The four-tier architecture pattern
- Hot tier (last 30-90 days) — fast search, expensive storage. Daily SOC operations.
- Warm tier (90 days – 1 year) — slower, cheaper. Investigations, audits.
- Cold tier (1-7 years) — archived in object storage. Compliance retention.
- Frozen / glacier (7+ years) — regulatory retention only. Restore on demand.
Cost-saver: aggressive parser-side filtering (don’t ingest noise) + tiered storage with auto-aging.
The platform decision matrix
| Platform | Strengths | Weaknesses | Indian enterprise fit |
|---|---|---|---|
| Splunk | Mature, vast app library, SPL is powerful | Expensive at scale; pricing per GB ingested | Banks, large enterprises with budget |
| Elastic Stack | Open source, flexible, KQL+ESQL strong | Operational complexity; sizing is real engineering | Engineering-heavy SaaS, fintech |
| Microsoft Sentinel | Cloud-native, KQL, Microsoft 365 integration | Vendor lock-in; cost not always lower | Microsoft-shop enterprises |
| Google Chronicle | 1-year hot retention by default; UDM is powerful | Newer; smaller app ecosystem | Multi-cloud, security-first orgs |
| Wazuh | Open source, batteries-included SIEM | Limited scale; some operations gaps | SMBs, Indian fintech under regulatory pressure |
| Datadog SIEM | Same platform as observability | SIEM features less mature than Splunk | SaaS already on Datadog |
What to log — the security minimum
- Endpoint — Sysmon (Windows), auditd (Linux), EDR alerts
- Authentication — Windows 4624/4625/4769, Linux PAM, SSO IdP
- Network — flow data (NetFlow/IPFIX), DNS queries, proxy logs
- Cloud — CloudTrail, Azure Activity Log, GCP Audit Logs
- Application — auth events, privileged actions, errors
- SaaS — Microsoft 365, Google Workspace, Slack, Salesforce admin activity
- Email — DMARC reports, attachment analytics
- Security tools — firewall, WAF, IDS, EDR
What NOT to ingest (the cost-savers)
- Application debug logs at INFO level — keep on-host, ingest only WARN+
- Health-check requests — filter at the edge
- Static-asset access logs — sample or skip
- Verbose system events (Windows 4634 logoff, normal service starts)
- Cloud audit metadata for every read API call (sampling acceptable)
Aggressive pre-filtering is the difference between ₹1cr and ₹3cr Splunk bills.
Parsing and normalisation
Different sources, different formats. Normalise to a unified schema:
- Elastic Common Schema (ECS) — broad adoption
- OCSF — newer, multi-vendor backed
- UDM (Chronicle) — Google’s schema
- CIM (Splunk Common Information Model) — Splunk-native
Normalised data lets you write detections that work across sources. Without it, every detection is per-source.
Retention strategy aligned to compliance
| Regulator/Standard | Minimum retention |
|---|---|
| CERT-In Direction | 180 days for specific log types |
| RBI | 10 years for transaction-related logs |
| SEBI CSCRF | 3-5 years per log category |
| PCI-DSS | 1 year (3 months online, 9 months archived) |
| ISO 27001 | Risk-based; typically 1-3 years |
| SOC 2 | Audit-period + 1 year typically |
Take the maximum of applicable requirements. Tier accordingly.
Performance budget — the queries that matter
Three classes of query, three different optimisations:
- Real-time alerts — must run every minute or stream. Index-time fields, summary indices.
- Daily reports — must run in 5-10 minutes. Pre-computed metrics, scheduled summary.
- Ad-hoc investigation — must complete in 1-30 minutes. Hot tier coverage of last 30 days.
Slow queries kill SOC productivity. Budget time for query optimisation as part of detection engineering.
Defender’s checklist
- Documented log sources — every system that logs, what it logs, retention.
- Daily ingest reconciliation — alert if a critical source stops sending.
- Cost monitoring — daily ingest GB; alert on +20% spike (DDoS-of-logs is a real attack).
- Schema normalisation — ECS / OCSF for detection portability.
- Tier auto-aging — hot → warm → cold automatic.
- Quarterly query-performance review — slow detections rebuilt.
- Audit-readiness drills — “give me all auth events for user X in 2025” should complete in <5 min.
Module Quiz · 6 questions
Pass with 80%+ to mark this module complete. Unlimited retries. Each question shows an explanation.
Custom team training + practitioner advisory
Beyond the free academy — we run private workshops, vCISO advisory, and red-team exercises tailored to your stack. For Indian SMBs scaling past their first hire.