Chronicle Forwarder data recovery

[ev13]

In the event a forwarder crashed, let's say 24-48 hours of downtime.

1. How can we recover the events that were meant to be ingested by the forwarder?
2. How much data, in GB, will be ingested into CSIEM after the downtime? Is there a specific interval for this data transfer? If so, what is the volume of data sent to Chronicle?
3. Once the events are ingested into CSIEM, how can we determine if any alerts were missed?
4. Is there a method, either manual or automated, to validate whether any alerts were missed from the late ingested logs?

[samsulrijal]

Recovering Events from a Crashed Forwarder Downtime

Recovering Lost Events (24-48 hour downtime):

• Limited options: Unfortunately, recovering all lost events during a complete downtime might be impossible. There are a few approaches to consider, depending on your setup:
• Source logs: If your sources (e.g., firewalls, applications) retain logs locally, you can potentially retrieve them for the downtime period. This requires manual intervention and depends on your source log retention policies.
• Secondary forwarder: If you have a redundant forwarder configured, it might have captured some of the events during the crash. However, this depends on the configuration and how long the primary forwarder was down before failover.
• Third-party tools: Some SIEM vendors offer log replay capabilities. These tools can ingest archived logs and attempt to reconstruct the event stream, potentially recovering some lost data.

Data Ingestion into CSIEM after Downtime:

• Uncertain quantity: The amount of data ingested after the forwarder recovers depends on the volume of events generated during the downtime. It's impossible to predict the exact size in GB.
• Ingestion interval: The data transfer interval depends on your forwarder configuration. It could be set to forward events in real-time (continuous), in batches (e.g., every 5 minutes), or at specific times.
• Chronicle data volume: The volume of data sent to Chronicle depends on your filtering and data forwarding configuration. Only relevant events forwarded to Chronicle will be ingested.

Identifying Missed Alerts:

• Limited methods: There's no foolproof way to definitively determine if any alerts were missed due to the downtime. Here are some approaches that might offer insights:
• Alert history: Review the alert history in your SIEM for the period before and after the downtime. Look for any recurring alerts that might have been triggered during the downtime based on historical patterns.
• Security team awareness: Inform your security team about the downtime. They might be aware of potential security incidents that could correlate with the timeframe.
• Log analysis tools: You can use advanced log analysis tools to compare historical trends with the timeframe after the forwarder recovers. This might reveal anomalies that could indicate missed alerts.

Validation Methods (Manual vs. Automated):

• Manual validation: This is a time-consuming process. You would need to analyze the recovered logs from your sources and compare them with the events ingested into the SIEM. This might involve filtering logs based on timestamps and searching for potential security events.
• Automated validation: Some SIEM solutions offer limited automated validation capabilities. These might involve predefined rules or anomaly detection algorithms that could potentially flag suspicious activity based on historical data. However, these methods are not perfect and require careful configuration.