Alerting in GCP Cloud Monitoring

[shegde]

I want to create an alert in Cloud Monitoring using alerting policies when the number of 5xx errors over the last 5 mins across all proxies exceeds a certain threshold. Currently, am getting an alert for each error code and for each proxy separately. This is resulting in too many alerts. Can you please let me know how to send an aggregated alert ? Also is it possible to list the proxies, their error codes and a count of it in the Incident details ?

[ravenhedden]

Hi there,

Let's say we want to create an alert any time your Apigee org experiences 300 or more 5XX errors over 5 minutes. Follow the steps below:

1. Open the Create alerting policy dashboard in the Google Cloud console of the GCP project where your Apigee organization lives
2. Click Select a Metric
3. Deselect Show only active resources and metrics
4. Copy and paste the following in the Select metric field: apigee.googleapis.com/proxyV2/request_count
5. In the pane to the right, select Apigee proxy (v2) > Proxyv2 > Apigee proxy request cumulative count
6. Click Apply
7. Under Add filters, add a filter where response_code starts_with 5. Optionally, add another filter where env = [desired_environment_name]
8. Under Transform data, leave Rolling Window at 5 min and Rolling window function at rate, but update Time series aggregation to sum

Next configure the alert trigger

1. Under Condition type, select Threshold
2. In the Alert trigger field, select Any time series violates.
3. In the Threshold position field, select Above threshold.
4. In the Threshold value field, enter 1 (300 seconds in 5 min, so 1/second).

Finally, define your notification channel and alert name as you see fit before saving the policy. This should cover your request of an aggregated alert. For more examples of alerts, see here.

As for your other request, "is it possible to list the proxies, their error codes and a count of it in the Incident details", no I don't see a way to add this to the aggregated alert I described above. There are restrictions on the types of variables that you can add to an alert notification body. Manual workarounds surely exist, but they'd take a fair bit of effort to engineer and would warrant a separate post.