Apigee X Southbound Connectivity with Private Service Connect (PSC) from Outside Apigee VPC

[greatdevaks]

Background

During February 2022, Apigee X software received a new feature release which enabled backend target routing with Private Service Connect (PSC). With PSC for Apigee X Southbound (“southbound” is used to refer to the connectivity/flow between Apigee and backend targets), Apigee can connect with backend target services running in VPC networks other than the one that is peered with the Apigee X Organization.

Problem Statement

For provisioning Apigee X, a VPC Peering is required between the customer VPC network and Apigee VPC network. This VPC Peering allows traffic exchange between the said two VPC networks and the Apigee API Proxies are able to call backend targets deployed in the peered customer VPC network. In order to call a backend target in some other VPC network, an Endpoint Attachment is needed in the Apigee VPC (which should route the traffic to a Service Attachment in the destination VPC network). Refer to the diagram below for understanding the flow.

Though the pattern described above is really great and could be one of the recommended patterns in the future for such connectivity requirements, there are primarily two reasons, as stated below, for why one would not want to create an Endpoint Attachment in the Apigee VPC and instead utilize Private Service Connect (PSC) Endpoint (Forwarding Rule) configured in the peered customer VPC network to reach the backend target (published through a Service Attachment) in the destination VPC network. Note: PSC Endpoint (Forwarding Rule) makes a consumer utilize an internal IP address through which the request can be routed (network address translation is performed) to the target service producer; this is different from the Apigee Endpoint Attachment in a way that the Apigee Endpoint Attachment is created in the Apigee VPC (in the Google-managed Apigee Tenant Project).

1. PSC for Apigee X Southbound Networking is in pre-GA as of time of writing (September 2022).
2. There may be a requirement for having a Private Service Connect (PSC) Endpoint (Forwarding Rule) in the customer VPC network which can be utilized by some other consumers (workloads deployed in the customer VPC network) along with Apigee to reach a backend target in the destination VPC network.

This article aims at introducing a Southbound connectivity pattern for Apigee X which uses Private Service Connect (PSC) Endpoint (Forwarding Rule) configured outside the Apigee VPC.

Solution

This section describes how to have a Private Service Connect (PSC) Endpoint (Forwarding Rule) configured outside the Apigee VPC, in the peered customer VPC network, so that it can be used by Apigee for Southbound connectivity.

A pre-requisite is to have a Service Attachment created for the backend target in the destination VPC network.

The diagram below shows how PSC Endpoint (Forwarding Rule) can be configured in the peered customer VPC network in order to access the backend target in the destination VPC network. An important thing to note is that since Apigee is VPC Peered with the customer VPC network, it cannot reach the PSC Endpoint (Forwarding Rule) directly. To overcome this connectivity issue, an Internal Load Balancer (ILB) and a Managed Instance Group (MIG) have been added which forwards the request to the PSC Endpoint (Forwarding Rule). The Managed Instance Group (MIG) uses iptables to forward the request to PSC Endpoint (Forwarding Rule). This use of MIG is very similar to the well-known Apigee Northbound MIG-based connectivity pattern.

Note: The architecture shown above demonstrates how a PSC Endpoint can uniquely connect to a PSC Service Attachment. The next section provides some reference scripts for implementing the Egress MIG functionality and configuring PSC Endpoint.

Reference Scripts

Egress MIG Startup Script

The following startup script requires PSC_ENDPOINT (IP address for the PSC Endpoint) as an input through Metadata.

Note: The script can be modified to support traffic routing to multiple PSC Endpoints; that would involve setting some Routing rules in the PSC consumer (customer VPC network). For simplicity purposes, the example below demonstrates forwarding traffic to a single PSC endpoint.

#!/bin/bash

# Note: If the PSC_ENDPOINT metadata variable is not set, the Startup Script will fail, making the associated MIG unhealthy.
psc_endpoint=$(curl -s http://metadata.google.internal/computeMetadata/v1/instance/attributes/PSC_ENDPOINT -H "Metadata-Flavor: Google")

sudo sysctl -w net.ipv4.ip_forward=1

sudo iptables -t nat -A POSTROUTING -j MASQUERADE

sudo iptables -t nat -A PREROUTING -p tcp --match multiport --dports 80,443 -j DNAT --to-destination "$psc_endpoint"

sudo iptables -A FORWARD -p tcp --match multiport --dports 80,443 -d "$psc_endpoint" -j ACCEPT

sudo iptables -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT

sudo sysctl -ew net.netfilter.nf_conntrack_buckets=1048576

sudo sysctl -ew net.netfilter.nf_conntrack_max=8388608​

Terraform Script for Private Service Connect (PSC) Endpoint (Forwarding Rule)

Note: Please supply values for the variables before executing the script. This script should be executed before creating the Egress resources like ILB + MIG because the MIG Startup Script requires PSC_ENDPOINT (IP address for the PSC Endpoint) as an input.

# Creation of the Internal Compute IP for PSC Endpoint.
resource "google_compute_address" "psc_ilb_consumer_address" {
 name = "psc-ilb-consumer-address"
 project = var.project_id # ID of the GCP Project to create compute IP in.
 region = var.region # Region to host PSC Endpoint in.
 subnetwork = var.psc_subnet # Name of the Subnet to host PSC Endpoint.
 address_type = "INTERNAL" # Address type for the PSC Endpoint.
}

# Creation of the Forwarding Rule for PSC Endpoint.
resource "google_compute_forwarding_rule" "psc_ilb_consumer" {
 name = "psc-ilb-consumer-fr"
 project = var.project_id # ID of the GCP Project to create PSC Forwarding Rule in.
 region = var.region # Region to host PSC Endpoint in.
 target = var.psc_ilb_service_attachment_id # ID of the Service Attachment to forward the PSC requests to. The Service Attachment URI has this format: "projects/SERVICE_PROJECT/regions/REGION/serviceAttachments/SERVICE_NAME".
 load_balancing_scheme = "" # Need to override EXTERNAL default when target is a Service Attachment.
 network = var.network # Name of the VPC Network to create the PSC Endpoint Forwarding Rule in.
 ip_address = google_compute_address.psc_ilb_consumer_address.id # IP Address of the PSC Endpoint.
}

 Acknowledgements: Thanks @omidt and @ncardace for valuable feedback on the draft of this article.

[williamspaul-1]

This was a very creative solution offered in a very challenging environment.  It works well, and shows the power of point-to-point connectivity in managing multiple workload endpoint networks.  The pattern helps to promote segregation of resources, while still allowing the necessary connectivity between the Apigee runtime environment in a way that controls cost and complexity.

Thanks Anmol for helping deliver this solution.

[greatdevaks]

Thanks a lot Paul.