Solved: Re: metrics - data is being bucketed on a daily ba...

mountaincode2

I am reaching out in relation to:

https://www.googlecloudcommunity.com/gc/Community-Blog/New-to-Google-SecOps-Using-Metrics-in-YARA-L-...

I have a query regarding the time period. The post says:

> In this metric, period:1d, window:30d means that the data is being bucketed on a daily basis across a window of 30 days.

$max_bytes_window = max(metrics.network_bytes_outbound(
    period:1d, window:30d,
    metric:value_sum,
    agg:max,
    principal.asset.ip:$ip
))

Now what does "data is being bucketed on a daily basis" mean. In the output screenshot in the post, i see the detection for each day. Is that what the phrase means, that is, the data will be grouped for 1 day?

jstoner

To put together the example in the blog, I used the test rule function. However that only allows me to run 2 weeks worth of data to test my rule. In the example shown, I am showing a detection each day with a metric of max byte count for the previous 30 days. So on the detection at the top of the list, the metric would be for the period of Feb 4 - March 6, the next one for March 8, the window is Feb 5 - March 7, and so forth. It happens that during the time shown in those examples that the max byte count did not roll out of that window so we don't see a change.

In the example below, here we have the max event count for the period changing over that rolling 30 day window. The outcome variable is showing the max event count for the past 30 days as of April 29 in the last example.

View solution in original post

jstoner

The metric period of 1 day and window of 30 days means that a metric, in the example in the blog, of the maximum sum of the outbound network bytes is calculated for that day n - 30 days. Then the next day, the calculation is performed on that day back 30 days with that last day of the previous window rolling off.

Because the periods in the examples are 1 day periods, all of the rules demonstrated are also aggregating data over the course of a day so that we can compare UDM data from the current day with the day periods across the 30 day window.

mountaincode2

@jstoneri am missing something and certainly unable to connect the dots.

Why do we need a "window" property. And how far back in time then would the data be calculated for?

For example, in my head, it can then keep going back for perpetuity:

May 05 - 30; April 04 - 30; March 03 - 30; Feb 02 - 30....

jstoner

The metrics syntax currently supports two different windows, 30 days and today. Perhaps down the road other windows may become available. The 30 day window is calculated based on the whichever day the metric ran and went back 30 days. That set of values is written to the entity graph and are available for use with retrohunts and the like on historical data if desired. The service was only turned up in the Feb/Mar timeframe so those metrics would only be created from roughly that point forward.

mountaincode2

Thanks @jstoner .

So would i be right in understanding the window is calculated as follows, if the query ran on May 06, 2024:

May 06 - 30
April 05- 30
March 03 - 30
Feb 02

However, i have a sense that my understanding might be wrong as the detections output screenshot in the following post https://www.googlecloudcommunity.com/gc/Community-Blog/New-to-Google-SecOps-Using-Metrics-in-YARA-L-...

shows the results from March 07 onward as opposed to running back 30 days from March 07:

What am i missing here please?

jstoner

To put together the example in the blog, I used the test rule function. However that only allows me to run 2 weeks worth of data to test my rule. In the example shown, I am showing a detection each day with a metric of max byte count for the previous 30 days. So on the detection at the top of the list, the metric would be for the period of Feb 4 - March 6, the next one for March 8, the window is Feb 5 - March 7, and so forth. It happens that during the time shown in those examples that the max byte count did not roll out of that window so we don't see a change.

In the example below, here we have the max event count for the period changing over that rolling 30 day window. The outcome variable is showing the max event count for the past 30 days as of April 29 in the last example.

mountaincode2

ah okay, @jstoner. i think i am closer to an understanding now.

So here are the takeaways:

data is being bucketed on a daily basis across a window of 30 days.
The metric period of 1 day and window of 30 days means that a metric, in the example in the blog, of the maximum sum of the outbound network bytes is calculated for that day n - 30 days.
...showing a detection each day with a metric of max byte count for the previous 30 days.

A quick followp. In one of your earlier responses, you say:

> Because the periods in the examples are 1 day periods, all of the rules demonstrated are also aggregating data over the course of a day so that we can compare UDM data from the current day with the day periods across the 30 day window.

In the following screenshot, detections are shown for each day with a metric of max byte count for the previous 30 days. The max byte count for the previous 30 days is highlighted in a red box. So where is aggregated data for the course of a day displayed.

Thank you!

jstoner

In the early examples I showed, I was really focused on the metric construction and what you could see based on adding metrics to rules. I believe I had a few rule examples where the rubber hit the road but not for every metric.

Here's an example rule working with that metric for byte count over time. The events section is just giving me all of my network connection events from my IP block that have a value in the sent bytes field.

My match statement aggregates all of these network connection events on the principal ip address for the day and then in my outcome section, I have two variables, one for my metric, and another which is the aggregation function of sum for the sent_bytes in the event section. That is my daily bytes for the past day.

In the condition section, i start with $net since that is the event variable for my events from today and I need to have events that meet my event criteria. The other portion of the condition (separated with AND) is a comparison between the daily outcome variable (which is our sum for the past day, the current data) and our metric value (which is our past 30 days) and in this case I want the daily to be greater than the max.

rule metric_examples_network {

  meta:
    author = "Google Cloud Security"

  events:
    $net.metadata.event_type = "NETWORK_CONNECTION"
    $net.network.sent_bytes > 0
    net.ip_in_range_cidr($net.principal.ip, "10.128.0.0/24")
    $net.principal.ip = $ip

  match:
    $ip over 1d

  outcome:
    $max_byte_count_30d = metrics.network_bytes_outbound(
        period:1d, window:30d,
        metric:value_sum,
        agg:max,
        principal.asset.ip:$ip
    )
    $daily_byte_sum = sum($net.network.sent_bytes)
  condition:
    $net and $daily_byte_sum > $max_byte_count_30d
}

The result of this when I run my test over a 2 week time frame is that only one system on one day exceeded that metric for that specific system and that is the detection you see here.

metrics - data is being bucketed on a daily basis across a window of 30 days.