LinkedIn
Twitter
In the continuous journey of digital transformation and macroeconomic conditions, cost savings are top of mind for businesses. Cloud FinOps teams are tasked with exploring potential cloud cost savings and optimization opportunities for their businesses.
In this article, you'll learn how to use Google Cloud's recommender service, which provides recommendations and insights based on machine learning and current resource usage. The recommender service is part of Google Cloud's Active Assist portfolio, which is a set of tools to generate recommendations for six value pillars, as shown in the image below.
The scope of this article is limited to the cost value pillar of recommendations to help you manage, optimize, and reduce your cloud costs.
What are Recommenders, Recommendations, and Insights?
To be very precise with the terminology, a recommender is a Google Cloud service that generates recommendations and insights. A recommendation is a machine-generated suggestion for optimizing usage of Cloud resources. Insights are findings that can be used to proactively focus on important patterns in resource usage.
Cost Recommenders
The recommender service has eight primary cost recommenders in place under the cost value pillar, which provide recommendations and insights for potential cost savings with different Google Cloud products and resource types.
These recommendations are generated based on pre-defined parameters and algorithms of cost recommenders. Interested readers can dive into the details of these parameters and algorithms available on Google cloud public documentation, listed further below.
For your understanding, here's an example of how the “Idle VM instance recommender” logic works.
- Idle VM recommender (google.compute.instance.IdleResourceRecommender)
Idle VM recommendations help to find and stop idle VM instances to reduce cloud waste and optimize your compute expenses.
Predefined parameters and algorithm to qualify for idle VM:
|
Usage metrics of last 14 days or for new VM, 5 days since VM creation and following conditions to be met: |
|
1- CPU utilization < 0.03 vCPUs for 97% of VM runtime |
|
2- Received N/W traffic < 2600 Bytes/second for 95% of VM runtime |
|
3- Sent N/W traffic < 1000 Bytes/ second for 95% of VM runtime |
- Idle custom image recommender (google.compute.image.IdleResourceRecommender)
These recommendations help to find and remove unused images. - Idle IP address recommender (google.compute.address.IdleResourceRecommender)
These recommendations help to find and remove unused IPs. - Idle persistent disk recommender (google.compute.disk.IdleResourceRecommender)
These recommendations help find, backup, and remove unused disks. - BigQuery slot recommender (google.bigquery.capacityCommitments.Recommender)
These recommendations help optimize BigQuery spend with slot reservations based on the bigquery slot usage and capacity needs. - Cloud SQL idle instance recommender (google.cloudsql.instance.IdleRecommender)
These recommendations help identify idle/unused SQL instances and remove them to reduce the waste. - Cloud SQL overprovisioned instance recommender (google.cloudsql.instance.OverprovisionedRecommender)
These recommendations help identity unnecessarily large instances for a given workload and resize those overprovisioned SQL instances to optimize cost. - Committed use discount recommender google.compute.commitment.UsageCommitmentRecommender
google.cloudbilling.commitment.SpendBasedCommitmentRecommender
These recommendations are generated based on historical usage metrics and help to reduce costs thru commitments which are available for both spend based and resource based.
How to get started with recommendations
You can navigate to the Recommendation Hub in the Google Cloud console to check for generated recommendations. Recommendations can also be explored and actioned using gCloud CLI or REST API. Recommendations are generated by Organization, Billing Account, or Project.
Recommendation Hub
Recommendations can be analyzed manually in the Google Cloud console Recommendation Hub for each and every project or at the Organization level.
In an organization, if there are hundreds or thousands of projects, then it's not feasible to navigate to each and every project console to check the recommendations and take actions. In these types of scenarios, the best option is to configure Recommendations Export to BigQuery and then analyze these recommendations for potential cost saving actions.
Configure recommendations export to BigQuery at the organization level
Recommendations export to BigQuery can be configured from the Recommendations tab in the Google cloud console. This is a one time configurable activity at an organizational level.
Recommendation and insights tables in BigQuery after export
After completing a successful export, both the recommendations and insights tables will be available in the target BigQuery data set.
Deep dive into the recommendations export
Recommendations export tables in BigQuery can be explored further to understand the granular details. A summary of a few important fields are documented below, which can be used for building cost insights dashboards and can be shared across integrated external systems.
When the value of column primary _impact.category = COST then potential savings can be calculated.
Recommendations export to BigQuery table schema
|
Field name |
Description |
|---|---|
|
Cloud_entity_type |
project number or billing account |
|
Cloud_entity_id |
Value of the project number or billing account id |
|
Name |
Name of project recommendation projects/[PROJECT_NUMBER]/locations/[LOCATION]/recommenders/[RECOMMENDER_ID]/recommendations/[RECOMMENDATION_ID] |
|
Location |
Location for which this recommendation is generated |
|
Recommender |
Recommender ID of the recommender |
|
Recommender_subtype |
Contains an identifier for a subtype of recommendations |
|
Target_resources |
fully qualified resource name |
|
Description |
Free-form human readable summary |
|
Last_refresh_time |
Last time this recommendation was refreshed by the system |
|
State |
state of the recommendation |
|
primary _impact.category |
COST / SECURITY / PERFORMANCE |
|
primary_impact.cost_projection.cost.currency_code |
Currency |
|
primary_impact.cost_projection.cost.units |
value |
|
ancestors.organization_id |
Organization of the recommendation project |
|
ancestors.folder_ids |
Up to 5 levels of parent folders for the recommendation project |
|
Associated_insights |
Reference to an associated insight |
|
Recommendation_details |
Additional details of recommendation in JSON format |
|
Priority |
P4; P3; P2; P1: Highest priority |
When the value of column primary _impact.category = COST then potential savings can be calculated:
If primary_impact.category = COST
Then currency = primary_impact.cost_projection.cost.currency_code
And value = primary_impact.cost_projection.cost.units
For example, if Currency = USD and Value = -100
In this case, negative value means saving, so potential saving is USD 100
Deep dive into the insights export
Insights export tables in BigQuery can be explored further to understand the granular details. A summary of a few important fields are documented below.
Insights export to BigQuery table schema
|
Field name |
Description <example> |
|---|---|
|
Cloud_entity_type |
project number or billing account |
|
Cloud_entity_id |
Value of the project number or billing account id |
|
Name |
Name of project projects/[PROJECT_NUMBER]/locations/[LOCATION]/insightTypes/[INSIGHT_TYPE_ID]/insights/[insight_id] |
|
Location |
Location for which this insight is generated |
|
insight_type |
<google.billing.cost.insight> |
|
insight_subtype |
<cost_below_forecasted> |
|
Target_resources |
fully qualified resource name |
|
Description |
Free-form human readable summary |
|
Last_refresh_time |
Last time this recommendation was refreshed by the system |
|
State |
state of the recommendation |
|
primary _impact.category |
COST / SECURITY / PERFORMANCE |
|
ancestors.organization_id |
Organization of the insight related project |
|
ancestors.folder_ids |
Up to 5 levels of parent folders for the insight related project |
|
Associated_recommendations |
Reference to an associated recommendation |
|
insight_details |
Additional details of insight in JSON format |
|
Severity |
low; medium; high; critical |
High level architecture of business use cases
Once recommendations export is configured and data is available in BigQuery tables, then recommendations data can be leveraged to implement various use cases to achieve the maximum benefits.
You can refer to the high level architecture diagrams of a few business use cases documented below. These can be used as a reference to design and implement cost savings use cases as per your business requirements.
1. Looker, Google’s business intelligence software and big data analytics platform
Recommendations export and Billing export, along with required external data sources can be integrated with Looker for building cost optimization recommendations and insights dashboards. These dashboards help to proactively make cost optimization decisions based on recommendations and cost insights.
2. Looker Studio turns data into fully customizable, informative reports and dashboards
It's easy to develop, read, and share reports with Looker Studio (formerly Google Data Studio). Recommendations export and Billing export, along with required external data sources can be integrated with Looker for building cost optimization recommendations and insights reports.
3. JIRA or other external systems
Integrate with JIRA to create and assign issues (tickets) to respective business owners for analyzing and acting on potential cost saving opportunities. In this use case, script (python code) can be hosted on Cloud Functions or Cloud Run to manipulate and extract the recommendations as per required business logic. Then this information can be used to create the JIRA issues and assign them to their owners.
By referencing these architectural patterns, recommendations information can be integrated with email notifications and other customer systems to effectively communicate with your target audience.
As part of your organization's overarching Cloud FinOps strategy, using Google Cloud recommendations is key to identifying and acting on potential cloud cost savings that will maximize your investment in the cloud.
To engage with your peers and Googlers on FinOps discussions, join the Cloud FinOps and Cost Optimization Community!