This article ultimately addresses how a customer who either plans to enable VPC-Service Controls on their Apigee X deployment or currently has VPC-Service Controls enabled on their Apigee X deployment can establish connectivity to invoke Google APIs using private access from their Apigee X runtime.
For context, I recently was working with one of my customers in regards to their Apigee X deployment. The customer had followed a few of our great Apigee articles on our Google Cloud Community page that highlighted the ability to Control your Internet Egress traffic from Apigee & How to securely access Google APIs using restricted access from Apigee X. If you’d like to take a look at both of those great articles created by Daniel & Joel please take a look at them below as a precursor to this article.
With that being said the customer implemented the steps in both articles to ultimately Remove the unrestricted internet egress access that the Apigee X runtime has in place by enabling VPC-SC on their Apigee X tenant project. The customer had a hard requirement to lock down their Apigee deployment.
The customer was also well aware that with VPC-SC enabled on Apigee that the service perimeter would only allow connectivity to the Google APIs in the restricted IP range 199.36.153.4/30. This range also includes the Apigee APIs (apigee.googleapis.com). With this setup in place the customer reached out to our team because they wanted to be able to access the Google APIs configured under the private.googleapis.com VIP as well.
It is important to note that all public Google APIs are not accessible once VPC-SC’s have been enabled on an Apigee X tenant project. Public APIs or IPs are not accessible because the Apigee tenant no longer has internet egress enabled. VPC Service Controls supports the products listed in the Google Cloud documentation.
The documentation related to Apigee X/Hybrid and VPC-SC can be found here.
The APIs supported under the private.googleapis.com VIP can be found here.
At the end of day this was a big blocker for the customer in Production and I stepped in to come up with a workable solution to address the connectivity to that particular set of APIs. From a networking standpoint I reached out to some amazing colleagues to have joint working sessions to address this task from the customer.
There were numerous options we looked at from either utilizing a Next Generation Firewall appliance to handle the next hop to establish connectivity or by utilizing an Envoy Proxy with a Forwarding rule (with some Cloud DNS configurations) to do the trick.
We ended up deciding that a Regional load balancer with an External backend (internet NEG) should be the most feasible option to establish connectivity to the Google APIs configured under the private.googleapis.com VIP. In particular the customer wanted to access the Maps API. The Maps API will be the API service that we want to establish connectivity to for this article.
The end architecture that this article addresses is below.
If you would also like to do this in your Apigee X deployment then follow the high level steps below. It’s also important to note here that the assumption is that you’ve already done steps 1 & 2 in the list below. In Step 3 you will follow those steps in our public documentation to set up the regional load balancer with an internet NEG. I will be showing some of the configurations in the commands to connect to the Maps API in particular.
To follow this guide you will need to create an Application Load Balancer (LB) and create an internet Network Endpoint Group (NEG) in a project. For this article the LB and NEG will be created in the Apigee customer project. To complete these steps you should either be a project Owner or Editor, or you should have both of the following Compute Engine IAM roles listed in the documentation here.
To begin, ensure that you have already created the VPC network and subnet you wanted to use for the configuration. Since this is being done in my Apigee project I will just be using the network I already configured for use in my Apigee deployment, and for the subnet I will also be using the same VPC subnet I used with my Apigee deployment. In this setup you will also be creating another separate proxy-only subnet to be utilized by the regional LB.
Also to make this implementation easier for you I would recommend setting these environment variables in Cloud Shell or in the terminal (You will need the gcloud SDK if you plan to use the terminal option) ahead of following these steps.
export LB_NETWORK=[replace with network you used with Apigee]
export SUBNET=[replace with vpc subnet you used with Apigee]
export PROXY_ONLY_SUBNET_NAME=[replace with the proxy only subnet name of your choosing]
export PROXY_ONLY_SUBNET_RANGE=[replace with the proxy subnet range of your choosing]
export REGION=[replace with the same region as your Apigee deployment]
export ROUTER_NAME=[replace with the name of your choosing]
export LB_NAT_NAME=[replace with the name of your NAT]
export LB_IP_ADDRESS=[replace with the name of your choosing]
export LB_SUBNET_NAME=[replace with vpc subnet you used with Apigee]
export INTERNET_NEG_NAME=[replace with the name of your choosing]
export HTTP_HEALTH_CHECK_NAME=[replace with the name of your choosing]
export BACKEND_SERVICE=[replace with the name of your choosing]
export URL_MAP_NAME=[replace with the name of your choosing]
Here are the steps to setup our external backend environment outside of Google Cloud:
Create the VPC network and subnet (Assumption is that you’ve already created this / are using your networking from the Apigee project).
gcloud compute networks subnets create $PROXY_ONLY_SUBNET_NAME --purpose=REGIONAL_MANAGED_PROXY --role=ACTIVE --region=$REGION --network=$LB_NETWORK --range=$PROXY_ONLY_SUBNET_RANGE
Set up a Cloud NAT gateway (I followed the Dynamically allocated option but you can determine if Manual allocation is applicable to your configuration).
gcloud compute routers create $ROUTER_NAME --network=$LB_NETWORK --region=$REGION
gcloud compute routers nats create $LB_NAT_NAME --router=$ROUTER_NAME --endpoint-types=ENDPOINT_TYPE_MANAGED_PROXY_LB --nat-custom-subnet-ip-ranges=$PROXY_ONLY_SUBNET_NAME --auto-allocate-nat-external-ips --region=$REGION
Reserve the load balancer’s IP address
gcloud compute addresses create $LB_IP_ADDRESS --region=$REGION --subnet=$LB_SUBNET_NAME
gcloud compute addresses describe $LB_IP_ADDRESS --region=$REGION
Set up the internet NEG:
gcloud compute network-endpoint-groups create $INTERNET_NEG_NAME --network-endpoint-type=INTERNET_FQDN_PORT --default-port=443 --network=$LB_NETWORK --region=$REGION
gcloud compute network-endpoint-groups update $INTERNET_NEG_NAME --add-endpoint="fqdn=maps.googleapis.com,port=443" --region=$REGION
Create the load balancer:
gcloud compute health-checks create https $HTTP_HEALTH_CHECK_NAME --region=$REGION --use-serving-port
gcloud compute backend-services create $BACKEND_SERVICE --load-balancing-scheme=INTERNAL_MANAGED --protocol=HTTPS --health-checks=$TCP_HEALTH_CHECK_NAME --health-checks-region=$REGION --region=$REGION
gcloud compute backend-services add-backend $BACKEND_SERVICE --network-endpoint-group=$INTERNET_NEG_NAME --network-endpoint-group-region=$REGION --region=$REGION
gcloud compute url-maps create $URL_MAP_NAME --default-service=$BACKEND_SERVICE --region=$REGION
For my implementation I did not perform steps 5, 6, 7 to make the Load Balancer into an HTTPS LB. My LB is an HTTP LB but the backend service is configured for HTTPS. The customer did proceed with ensuring they did steps 5, 6 and 7 which is definitely ideal for a Production environment and a real implementation.
Now we're done with the setup! Let’s test.
First, before we test the actual connectivity through an Apigee proxy, let's re-confirm that we cannot hit the Maps API by running a few tests.
To test connectivity I decided to utilize the Timezone API configured under the Maps API portfolio. Before you can actually utilize this API you will need to generate an API Key.
Test #1. Test connectivity to the Timezone API via a curl call directly to the maps.googleapis.com endpoint. This curl is not utilizing a configured Apigee proxy and should work due to the fact that we set up a service perimeter via VPC-SC on the project (which by default the connectivity to the Google APIs in the restricted IP range 199.36.153.4/30 is established). At this point we just want to confirm that the service perimeter is working as intended.
curl -L -X GET 'https://maps.googleapis.com/maps/api/timezone/json?location=39.6034810%2C-119.6822510×tamp=1331161200&key=[replacewithyourapikey]'
So with this curl call we can see that it’s successfully connecting to the Timezone API while the service perimeter is enabled.
Test #2. Now let’s create an Apigee Proxy and deploy it. For the proxy configuration we are going to have the target endpoint configured in the same aspect as the curl command we used above but we will be utilizing the Apigee hostname to call the proxy in the environment the proxy is mapped to. The expectation is that the curl call through the proxy should fail.
Now try the curl call with the Apigee hostname and ensure that you pass the API key you created above.
curl -X GET 'https://[replacewithyourApigee Hostname]/maps/api/timezone/json?location=39.6034810%2C-119.6822510×tamp=1331161200&key=[replacewithyourapikey]'
With this we can see we're getting a 403 Forbidden. Optionally, you can also run a trace session in tandem to look at the end to end details of the API calls.
Now that we’ve run the two tests, let’s configure the Apigee proxy in the third test to use the regional LB we created to establish connectivity via the end architecture that was shared above.
Test #3. For this test just simply change the XML in the target endpoint postflow to point to the regional LB IP that was created above. Also, feel free to remove the SSLInfo block if applicable.
Now let’s test this proxy by running the curl again in Cloud Shell or a Terminal connected to your GCP project:
curl -X GET 'https://[replacewithyourApigee Hostname]/maps/api/timezone/json?location=39.6034810%2C-119.6822510×tamp=1331161200&key=[replacewithyourapikey]'
Success! We are now able to connect to the Timezone API, a part of the Maps API that resides under the private.googleapis.com VIP while we have VPC-SC enabled on our Apigee X deployment.
There are further testing methods as mentioned in the articles above that may be beneficial to you after implementing this recommendation.
Thanks a lot to @ssvaidyanathan & @gonzalezruben for the amazing support with the architecture and feedback during the drafting of this article!