Referencing extracted label from log in alert documentation

[the4thamigo-uk]

Hi, we are trying to create an alert policy based on data in the logs for a cloudfunction. We are using https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.alertPolicies, and specifying a `LogMatch` condition (https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.alertPolicies#LogMatch).

In `LogMatch` I can configure `labelExtractors` to extract data from the logs. This data shows up in the default alert message as a JSON block of labels, which is great, but I want to be able to reference them in the markdown `documentation`, and every way I have tried this results in `null`.

I notice the comment about null values here https://cloud.google.com/monitoring/alerts/doc-variables#null-values, but I cant figure out quite what this means in practice.

Is there a working example anywhere, or some guidance about if/how this should work?

Thanks for any help you can give on this

[ghayas_muhammad]

Here is the explanation of The section [1] which cover "null values" in the message aspect, and perhaps your log metrics are apart of this section, such as the resource.label.KEY and metric.label.KEY variables can have null values if your alerting policy uses cross-series aggregation (reduction), for example, calculating the SUM across each of the time-series that match a filter). When using cross-series aggregation, any labels not used in grouping are dropped and as a result they render as null when the variable is replaced with its value. All labels are retained when there is no cross-series aggregation. Please let me know if this is the case.

https://cloud.google.com/monitoring/alerts/doc-variables#null-values

[Fox]

Hello there,

I allow myself to post to this topic as I have the same kind of issue.
I would like to create a log-based alert policy (not creating a log-based metric though), and display the value of an extracted label into the documentation part as well. But in my case I am doing this through Terraform.

I tried to pass different variables from this doc but all I got from this is "null" values. 
Like the author, I can see the extracted labels as a block behind the title in the notification email.
So I guess there might be a way to export it in the documentation section ?

Note : As a workaround I can make it work by creating a log-based metric (with labels) + an alert policy, but not by creating a log-based alert policy directly, but this solution implies an overhead with creating twice as many resources.

[agoldstein]

This is now possible through the API, you can reference the log data dynamically in the notification payload: https://cloud.google.com/logging/docs/alerting/log-based-alerts#lba-by-api 

[satyendrasrivas]

Hello Google Team,

@agoldstein  @ghayas_muhammad 

I am able to configure labelExtractor in the log match alerting policy however these labels are now part of Alert Message and Incident Description. The downsize of adding extracted label is that policy "Notification rate limit(=15 min)" is not into effect. Earlier I was getting 1 alert in 15 minutes but now I am getting 3 or 4 alerts within 15 minutes window after adding label extractor.

I believe it is due to that fact that these labels are part of alert Title/Message so it is thinking it is a different message. The notification rate limit should be at Policy Name level.

Please let me know if anything incorrect in the usage of Alert Policy.