Using network appliances to overcome the transitiv...

strebel · ‎06-03-2021

This article proposes one possible approach to overcome the missing transitive peering functionality when having multiple VPC networks peered in sequence. It introduces VM based network appliances that can help you achieve the intended TCP routing over two peering hops.

Options for connectivity across transitively peered VPC networks

The difficulty of connectivity across transitively peered VPC networks was already introduced in a previous article when we proposed a solution to achieving the connectivity that is leveraging shared VPC networks. In a nutshell the difficulty comes from the fact that VPC peering does not propagate peering routes and therefore in a scenario where a VPC A is peered with a VPC B and VPC B is in turn peered with VPC C then VPC A cannot send packets to a target in VPC C because the routes to reach VPC C are not propagated.

For Apigee X the transitive peering is especially challenging because the Apigee X tenant project is peered with the customer project meaning that any additional peering to the customer project e.g for a backend service will face the transitive peering problem.

As described in the previous article there is currently no fully managed option to turn on routing between transitively peered networks. In order to still be able to communicate between multiple peered networks, there are three common ways to work around this challenge:

Use shared VPCs such that all the backend services are located in service projects of the same host project.
Use Private Service Connect (PSC) to make a backend service that is running in one VPC available and consume it from another VPC.
Use a network appliance in a central VPC network that is peered to the Apigee X tenant network, the customer consumer networks and the customer backend networks. Then use custom routing rules to propagate the network appliance routes to the peered networks
Use VPN as a replacement for the second peering connection to the backend or from the consumer to the hub VPC.

In this article we will focus on the third option. This option is especially useful if there is an already established network design that leverages VPC peering and cannot be re-architected in a shared VPC model or using PSC.

Peering Hub Network Appliance

One option to enable routing across a transitively-peered networking path is to deploy a network appliance in the central VPC network (the one that has routes to both peered networks). For the purpose of this article we will consider a hub and spoke deployment topology where the hub VPC is the central VPC connected to the Apigee X tenant project network and all the required backends and consumer networks are peered at. It also assumes that the VPC peerings are configured to export/import custom routes from the hub to the spokes.

To create the appliance we first need to set a few configuration parameters. Here the NETWORK, SUBNET and REGION describe the hub subnet where the network appliance should be located.

NETWORK=hub
SUBNET=hub-euw6
REGION=europe-west6

Additionally we need to specify the routing CIDR blocks for the peered networks. For simplicity we will only consider two blocks here one for the peering range that is provided for the Apigee tenant project and the other for the range that spans our consumer and backend VPCs.

It is important to note that the ranges will need to be larger than the individually peered networks. As an example if your peering range for Apigee X is 10.111.0.0/23 the Apigee CIDR will have to be 10.111.0.0/22.

APIGEE_CIDR='10.111.0.0/22'# needs to be bigger than the Apigee Peering CIDR<br />SERVICES_CIDR='10.2.0.0/23'# spans both backend and consuming services

The actual appliances are created as a managed instance group (MIG) to improve availability and prepare for a horizontal scale-out if needed.

gcloud compute instance-templates create hub-gw-template \
--network $NETWORK \
--subnet $SUBNET \
--region $REGION \
--no-address \
--machine-type e2-small \
--can-ip-forward \
--tags hub-gw \
--scopes default \
--metadata startup-script='#! /bin/bash
echo "OK" > index.html
python3 -m http.server 80 &
iptables -F
iptables -t nat -A POSTROUTING -j MASQUERADE
echo 1 > /proc/sys/net/ipv4/ip_forward'

gcloud compute health-checks create tcp \
hub-gw-mig-health-check \
--check-interval 10 --unhealthy-threshold 3 --port 80

gcloud compute firewall-rules create hub-gw-health-checks \
--network $NETWORK --allow tcp:80 --target-tags hub-gw \
--source-ranges 130.211.0.0/22,35.191.0.0/16

gcloud compute instance-groups managed create \
hub-gw-mig \
--region $REGION --size=2 \
--template hub-gw-template \
--health-check hub-gw-mig-health-check \
--initial-delay 15

To have a single target for the network appliance MIG we also create an internal TCP load balancer (ILB) in the hub network.

gcloud compute backend-services create hub-gw-backend \
--load-balancing-scheme=internal \
--protocol=tcp \
--health-checks=hub-gw-mig-health-check

gcloud compute backend-services add-backend \
hub-gw-backend \
--instance-group=hub-gw-mig \
--instance-group-region=$REGION

gcloud compute forwarding-rules create \
hub-gw-ilb \
--load-balancing-scheme=internal \
--network=$NETWORK \
--subnet=$SUBNET \
--ip-protocol=TCP \
--ports=443 \
--backend-service=hub-gw-backend \
--backend-service-region=$REGION

Once the ILB is set up, we create custom routes that point traffic that is intended for the spoke networks to network appliance’s ILB.

gcloud compute routes create hub-gw-apigee-x \
--network=$NETWORK \
--destination-range=$APIGEE_CIDR \
--next-hop-ilb=hub-gw-ilb \
--next-hop-ilb-region=$REGION \
--priority=800

gcloud compute routes create hub-gw-backend-1 \
--network=$NETWORK \
--destination-range=$SERVICES_CIDR \
--next-hop-ilb=hub-gw-ilb \
--next-hop-ilb-region=$REGION \
--priority=800

After applying the routes, the hub network will have two routes that concern each peered range. One is the automatically generated peering-route-xyz and has a priority of 0 (highest) and the second one is the custom route that we just created with a priority of 800. Because only the custom routes are exported to the peered networks, the peered network will send all traffic to any other peered network to the network appliance. The network appliance in turn also sees the peering routes with the higher priority and is then able to send the traffic to the right peered VPC.

The last remaining step is to configure the firewall to allow traffic to the hub’s network appliance from the peered networks. In this example we allow the Apigee X tenant project to talk to the network appliance on both HTTP and HTTPS ports whilst the consumers are only allowed to use the HTTPS port on the appliance. Note that the firewall on the backends themselves might also need additional firewall permissions to allow traffic coming from the network appliance.

gcloud compute firewall-rules create apigee-x-hub-gw \
--network $NETWORK --allow tcp:443,tcp:80 --target-tags hub-gw \
--source-ranges $APIGEE_CIDR

gcloud compute firewall-rules create services-hub-gw \
--network $NETWORK --allow tcp:443 --target-tags hub-gw \
--source-ranges $SERVICES_CIDR

With this setup we can now communicate with backends that are using a transitive peering with the hub network by routing traffic through the hub’s network appliance. This setup also allows us to have consumers in another peered network and have them consume service across three peered networks.

Limitations

Note that there are limits and quotas to be considered when planning this network topology. Especially the number of peerings as well as the number of routes that can be defined within a network need to be taken into consideration for evaluating this approach.