Error 504: Upstream Request Timeout in Serverless Backend of Load Balancer

[harrus]

Introduction: In a setup where CloudRun services are accessed via private IP through a load balancer, I'm encountering a timeout restriction of 15 seconds when invoking the private IP from a virtual machine (VM). This timeout is significantly short, my CloudRun function executes in about 30 seconds to 1 minute.

VM (private network) => invoke private ip => via loadbalancer (url mapping...etc) => cloudrun

Issue Description:

• Utilizing a load balancer to access CloudRun services via private IP.
• The timeout for CloudRun services is set to 5 minutes (300 seconds) for handling incoming requests.
• However, when invoking the private IP from a VM, a maximum timeout of 15 seconds is enforced.
• Latencies exceeding this 15-second limit result in a 504 error response.
• Default resource timeout for serverless backend services is 1 hour (https://cloud.google.com/load-balancing/docs/negs/serverless-neg-concepts#backend-service-limitation...).

data "google_cloud_run_service" "cloudrun_api" {
  provider = google-beta
  location = "us-central1"
  name     = var.cloudrun_service
}
resource "google_compute_region_network_endpoint_group" "cloudrun_api" {
  provider = google-beta
  name     = "serverless-cloudrun-api"
  region   = "us-central1"

  cloud_run {
    service = data.google_cloud_run_service.cloudrun_api.name
    
  }
}

# Begin: Configure internal load balancing for cloudrun API

resource "google_compute_backend_service" "cloudrun_api" {
  provider                        = google-beta
  name                            = "lb-be-cloudrun-api"
  connection_draining_timeout_sec = 300
  load_balancing_scheme           = "INTERNAL_MANAGED"
  port_name                       = "http"
  protocol                        = "HTTP"
  session_affinity                = "NONE"
  locality_lb_policy              = "ROUND_ROBIN"

  backend {
    group = google_compute_region_network_endpoint_group.cloudrun_api.self_link
  }
}

resource "google_compute_url_map" "cloudrun_api" {
  provider        = google-beta
  name            = "lb-cloudrun-api"
  default_service = google_compute_backend_service.cloudrun_api.id
  path_matcher {
    name = "lb-cloudrun-path"
    default_service = google_compute_backend_service.cloudrun_api.id
    path_rule {
      paths   = ["/*"]
      service = google_compute_backend_service.cloudrun_api.id
      route_action {
        timeout {
          seconds = 80
        }
      }
    }

  }
}

resource "google_compute_target_http_proxy" "cloudrun_api" {
  provider = google-beta
  name     = "http-lb-proxy-cloudrun-api"
  url_map  = google_compute_url_map.cloudrun_api.id
}


resource "google_compute_address" "cloudrun_api" {
  provider     = google-beta
  address      = "10.128.0.10"
  name         = "lb-ipv4-cloudrun-api"
  ip_version   = "IPV4"
  address_type = "INTERNAL"
  purpose      = "GCE_ENDPOINT"
  subnetwork   = data.google_compute_subnetwork.subnetwork.name
}
resource "google_compute_global_forwarding_rule" "cloudrun_api" {
  provider              = google-beta
  name                  = "http-content-rule-cloudrun-api"
  ip_protocol           = "TCP"
  load_balancing_scheme = "INTERNAL_MANAGED"
  port_range            = "80-80"
  target                = google_compute_target_http_proxy.cloudrun_api.id
  ip_address            = google_compute_address.cloudrun_api.address
}

When I try with the public URL of the Cloud Run function, it works well. Can I ask which network component restricts the timeout to 15 seconds, and how can I increase it?